Circuit Breakers and Bulkheads in Rust Service Clients
Stop one failing dependency from taking the whole service down. A circuit breaker fails fast after repeated errors and probes for recovery; a bulkhead caps concurrency per dependency. Implement both with Tokio.
When a dependency slows down, the symptoms spread. Calls wait their full timeout, tasks pile up, connection pools drain, and soon unrelated endpoints are failing because they share the resources. Two patterns, borrowed from electrical and ship design, contain this: a circuit breaker that stops calling something that is failing, and bulkheads that keep a failure inside its compartment. This post implements both in plain Tokio and points at ready-made crates. Versions: Rust 1.90 with edition 2024, Tokio 1.47.
The circuit breaker states
| State | Behavior | Transition |
|---|---|---|
| Closed | Calls pass through; failures are counted | Failures reach the threshold: open |
| Open | Calls fail immediately; the dependency is not contacted | After the cooldown: half-open |
| Half-open | A limited number of trial calls pass | Success: closed. Failure: open again |
A compact implementation
use std::{
sync::Mutex,
time::{Duration, Instant},
};
#[derive(Debug)]
enum State {
Closed { consecutive_failures: u32 },
Open { until: Instant },
HalfOpen { trial_in_flight: bool },
}
pub struct CircuitBreaker {
state: Mutex<State>,
failure_threshold: u32,
cooldown: Duration,
}
#[derive(Debug)]
pub enum CallError<E> {
CircuitOpen,
Inner(E),
}
impl CircuitBreaker {
pub fn new(failure_threshold: u32, cooldown: Duration) -> Self {
CircuitBreaker { state: Mutex::new(State::Closed { consecutive_failures: 0 }), failure_threshold, cooldown }
}
/// Returns false when the call must not be attempted.
fn allow(&self) -> bool {
let mut s = self.state.lock().unwrap();
match &mut *s {
State::Closed { .. } => true,
State::Open { until } if Instant::now() >= *until => {
*s = State::HalfOpen { trial_in_flight: true }; // this caller is the trial
true
}
State::Open { .. } => false,
State::HalfOpen { trial_in_flight } => {
if *trial_in_flight { false } else { *trial_in_flight = true; true }
}
}
}
fn record(&self, ok: bool) {
let mut s = self.state.lock().unwrap();
*s = match (&*s, ok) {
(State::Closed { .. }, true) => State::Closed { consecutive_failures: 0 },
(State::Closed { consecutive_failures }, false) => {
let n = consecutive_failures + 1;
if n >= self.failure_threshold {
State::Open { until: Instant::now() + self.cooldown }
} else {
State::Closed { consecutive_failures: n }
}
}
(State::HalfOpen { .. }, true) => State::Closed { consecutive_failures: 0 },
(State::HalfOpen { .. }, false) => State::Open { until: Instant::now() + self.cooldown },
(State::Open { until }, _) => State::Open { until: *until },
};
}
pub async fn call<T, E, F, Fut>(&self, f: F) -> Result<T, CallError<E>>
where
F: FnOnce() -> Fut,
Fut: std::future::Future<Output = Result<T, E>>,
{
if !self.allow() {
return Err(CallError::CircuitOpen);
}
match f().await {
Ok(v) => { self.record(true); Ok(v) }
Err(e) => { self.record(false); Err(CallError::Inner(e)) }
}
}
}The mutex is held only for state changes, never across the awaited call, so it does not serialize requests. After failure_threshold consecutive failures the breaker opens; calls then return CircuitOpen in nanoseconds. After the cooldown exactly one trial call is admitted; its outcome decides whether to close or reopen. A production version counts failures in a sliding window or by error rate (failures over total in the last N seconds), which handles intermittent errors better than consecutive counts, and admits several trial calls in half-open. The failsafe crate and the Tower tower-resilience-style layers provide tested versions; the value of reading a small one is understanding the knobs.
Using it with a timeout
use std::sync::Arc;
pub struct InventoryClient {
http: reqwest::Client,
breaker: Arc<CircuitBreaker>,
}
impl InventoryClient {
pub async fn stock(&self, sku: &str) -> Option<u32> {
let url = format!("https://inventory.internal/stock/{sku}");
let result = self
.breaker
.call(|| async {
let res = tokio::time::timeout(Duration::from_millis(300), self.http.get(&url).send())
.await
.map_err(|_| "timeout")?
.map_err(|_| "connect")?;
if res.status().is_server_error() {
return Err("5xx");
}
res.json::<u32>().await.map_err(|_| "decode")
})
.await;
match result {
Ok(n) => Some(n),
Err(CallError::CircuitOpen) => None, // fallback: unknown stock; caller shows "check availability"
Err(CallError::Inner(_)) => None,
}
}
}The timeout is inside the breaker so that timeouts count as failures. 4xx responses are not treated as failures here (a missing sku is not a sick dependency). The fallback, None, is a product decision: a cached value, a default, a degraded page or an error. Without a fallback plan the breaker only changes the error message.
The bulkhead
use std::sync::Arc;
use tokio::sync::Semaphore;
#[derive(Clone)]
pub struct Bulkhead {
permits: Arc<Semaphore>,
}
impl Bulkhead {
pub fn new(max_concurrent: usize) -> Self {
Bulkhead { permits: Arc::new(Semaphore::new(max_concurrent)) }
}
pub async fn run<T, Fut: std::future::Future<Output = T>>(&self, f: Fut) -> Result<T, &'static str> {
let permit = self.permits.try_acquire().map_err(|_| "bulkhead full")?; // fail fast, do not queue
let out = f.await;
drop(permit);
Ok(out)
}
}Give each dependency its own semaphore sized to what it can handle: 20 concurrent calls to recommendations, 100 to the payment service. When the recommendation service slows down and its 20 permits are in use, further recommendation calls fail immediately while checkout, with its own permits, is unaffected. try_acquire rejects instead of waiting, which keeps latency bounded; use acquire with a timeout if brief queueing is acceptable (semaphores). For connection pools, separate pools per dependency or per priority provide the same isolation (pool sizing).
Tuning
- Threshold and window: trip on a clear signal, not a blip; a 50 percent error rate over 20 or more calls in 10 seconds is a common start.
- Cooldown: long enough for recovery (5 to 30 seconds), with jitter across instances to avoid synchronized probing.
- Per-instance state is simple but each instance learns separately; that is usually fine, and cheaper than shared state.
- Expose breaker state as a metric and log transitions; an open circuit is an incident signal (incident runbook).
Test it
Drive the breaker with a fake dependency that fails on command and a paused clock, asserting the open, half-open and closed transitions and that no calls reach the dependency while open (testing retries and timeouts). For the wider set of client policies, see async resilience patterns.
