Async Retries Timeouts and Circuit Breakers in Rust Clients
Retries fix transient failures and amplify real outages. The right combination is a timeout per attempt, capped jittered backoff, a total deadline and a breaker that stops hammering.
A dependency starts failing 30% of the time. Your service retries each failed call three times. Now every user request produces up to four calls, the dependency sees 2.2 times its normal load at the moment it can least afford it, and a brownout becomes an outage. Retries are the first resilience tool people add and the one that does the most damage when added alone. This page builds the full set in Rust: a timeout per attempt, capped and jittered backoff, a deadline over the whole operation, and a circuit breaker. It compiles on Rust 1.99 with edition 2024 and tokio 1.54, using no resilience crate, so you can see every moving part.
Why a bare retry loop makes things worse
Start with the arithmetic, because it is what justifies everything after. Suppose a service handles 1,000 requests per second and each makes one call to a dependency that is now failing 30% of the time. A policy of "retry up to 3 times, immediately" sends, per request, 1 call plus 0.3 retries plus 0.09 plus 0.027, which is 1.417 calls on average. That is 1,417 calls per second against a system that was already struggling. If the failures are caused by overload, they rise, which raises the retries, which raises the load. That feedback loop is called a retry storm, and it is why a retry needs a budget and a pause.
A timeout on every attempt
No retry policy works without a timeout per attempt. A call that hangs for 30 seconds is not a failure you can retry, it is a held resource. tokio::time::timeout wraps any future and returns Err(Elapsed) if it does not finish in time, dropping the future so the work stops at its next await point.
use std::time::Duration;
use tokio::time::{sleep, timeout};
async fn slow_dependency() -> &'static str {
sleep(Duration::from_millis(300)).await;
"response"
}
#[tokio::main]
async fn main() {
match timeout(Duration::from_millis(100), slow_dependency()).await {
Ok(v) => println!("got {v}"),
Err(elapsed) => println!("gave up: {elapsed}"),
}
}gave up: deadline has elapsed
Choose the per-attempt limit from the dependency's measured tail latency, such as its p99 plus some margin, not from a round number. If p99 is 80 ms, a 100 ms timeout fails about one call in a hundred and bounds the damage of the rest (timeouts in Tokio).
Backoff with full jitter
Waiting between attempts gives the dependency room to recover. Doubling the wait each time (exponential backoff) spreads attempts out, a cap stops it growing to minutes, and randomizing the wait stops a thousand clients from retrying in lockstep. The "full jitter" form picks a random delay between zero and the current cap, and it performs well in practice.
use std::time::Duration;
struct Backoff {
base: Duration,
cap: Duration,
state: u64, // xorshift state; use the rand crate in real code
}
impl Backoff {
fn new(base: Duration, cap: Duration, seed: u64) -> Self {
Self { base, cap, state: seed.max(1) }
}
fn next_random(&mut self) -> u64 {
self.state ^= self.state << 13;
self.state ^= self.state >> 7;
self.state ^= self.state << 17;
self.state
}
fn delay(&mut self, attempt: u32) -> Duration {
let exp = self.base.saturating_mul(2u32.saturating_pow(attempt));
let ceiling = exp.min(self.cap);
let jitter = self.next_random() % (ceiling.as_millis() as u64 + 1);
Duration::from_millis(jitter)
}
}
fn main() {
let mut b = Backoff::new(Duration::from_millis(100), Duration::from_secs(2), 42);
for attempt in 0..8 {
let ceiling = Duration::from_millis(100 * 2u64.pow(attempt)).min(Duration::from_secs(2));
println!("attempt {attempt}: ceiling {ceiling:>6?}, wait {:>6?}", b.delay(attempt));
}
}attempt 0: ceiling 100ms, wait 94ms
attempt 1: ceiling 200ms, wait 133ms
attempt 2: ceiling 400ms, wait 72ms
attempt 3: ceiling 800ms, wait 166ms
attempt 4: ceiling 1.6s, wait 973ms
attempt 5: ceiling 2s, wait 1.293s
attempt 6: ceiling 2s, wait 1.498s
attempt 7: ceiling 2s, wait 291ms
The ceiling doubles from 100 ms and stops at 2 s, and each actual wait is a random point below it, which is why attempt 2 happened to wait only 72 ms. That is the design, not a bug: the average wait follows the ceiling while any single wait can be short. Without the cap the eighth wait could be 12.8 seconds. Without jitter, a thousand clients that failed together would all retry together at 100 ms, 200 ms, 400 ms, and hit the dependency in synchronized waves.
A retry helper with a deadline
Now combine the pieces. The helper retries only errors you mark as transient, applies the per-attempt timeout, and stops when an overall deadline is spent. The deadline is what keeps a request's total latency bounded no matter how many attempts occur.
use std::future::Future;
use std::time::Duration;
use tokio::time::{Instant, sleep, timeout};
#[allow(dead_code)] // the messages are read through Debug when printed
#[derive(Debug)]
enum CallError {
Transient(&'static str),
Permanent(&'static str),
TimedOut,
}
async fn retry<F, Fut, T>(deadline: Duration, per_attempt: Duration, mut op: F) -> Result<T, CallError>
where
F: FnMut(u32) -> Fut,
Fut: Future<Output = Result<T, CallError>>,
{
let started = Instant::now();
let mut attempt = 0;
loop {
let result = match timeout(per_attempt, op(attempt)).await {
Ok(r) => r,
Err(_) => Err(CallError::TimedOut),
};
match result {
Ok(v) => return Ok(v),
Err(CallError::Permanent(m)) => return Err(CallError::Permanent(m)),
Err(e) => {
let pause = Duration::from_millis(20 * 2u64.pow(attempt));
if started.elapsed() + pause >= deadline {
return Err(e); // out of time: report the last real error
}
println!("attempt {attempt} failed ({e:?}), retrying in {pause:?}");
sleep(pause).await;
attempt += 1;
}
}
}
}
#[tokio::main]
async fn main() {
let r = retry(Duration::from_millis(500), Duration::from_millis(50), |attempt| async move {
match attempt {
0 => Err(CallError::Transient("connection reset")),
1 => {
sleep(Duration::from_millis(200)).await; // hangs past the 50 ms limit
Ok("late")
}
_ => Ok("fresh data"),
}
})
.await;
println!("result: {r:?}");
let r: Result<&str, _> = retry(Duration::from_millis(500), Duration::from_millis(50), |_| async {
Err(CallError::Permanent("401 unauthorized"))
})
.await;
println!("result: {r:?}");
}attempt 0 failed (Transient("connection reset")), retrying in 20ms
attempt 1 failed (TimedOut), retrying in 40ms
result: Ok("fresh data")
result: Err(Permanent("401 unauthorized"))
The first call fails fast and retries after 20 ms, the second hangs and is cut off at 50 ms, the third succeeds. The second scenario shows the rule that matters most: an authentication failure is permanent, so it returns immediately and is never retried. Retrying a 401 or a 400 only multiplies the traffic. Retry connection resets, timeouts and 502, 503 or 504 responses, and be careful with writes: a retried POST may run twice unless the operation is idempotent (idempotency keys). For a real HTTP client, the error classification lives in reqwest error handling.
A circuit breaker that stops calling
Backoff slows each caller. A circuit breaker makes all of them stop. After enough consecutive failures it opens and fails calls immediately, without touching the dependency, and after a cool-down it lets a single trial call through. If that succeeds, the breaker closes.
use std::sync::Mutex;
use std::time::{Duration, Instant};
#[derive(Debug, PartialEq)]
enum State {
Closed { failures: u32 },
Open { since: Instant },
HalfOpen,
}
struct Breaker {
state: Mutex<State>,
threshold: u32,
cooldown: Duration,
}
impl Breaker {
fn new(threshold: u32, cooldown: Duration) -> Self {
Self { state: Mutex::new(State::Closed { failures: 0 }), threshold, cooldown }
}
/// Returns false when the call should not even be attempted.
fn allow(&self) -> bool {
let mut s = self.state.lock().unwrap();
match *s {
State::Open { since } if since.elapsed() >= self.cooldown => {
*s = State::HalfOpen; // let one trial call through
true
}
State::Open { .. } => false,
_ => true,
}
}
fn record(&self, ok: bool) {
let mut s = self.state.lock().unwrap();
*s = match (&*s, ok) {
(_, true) => State::Closed { failures: 0 },
(State::Closed { failures }, false) if failures + 1 < self.threshold => {
State::Closed { failures: failures + 1 }
}
_ => State::Open { since: Instant::now() },
};
}
}
fn main() {
let breaker = Breaker::new(3, Duration::from_millis(100));
for call in 1..=8 {
if !breaker.allow() {
println!("call {call}: rejected by open breaker");
std::thread::sleep(Duration::from_millis(30));
continue;
}
let dependency_up = call >= 7;
breaker.record(dependency_up);
println!("call {call}: dependency {}", if dependency_up { "ok" } else { "failed" });
}
}call 1: dependency failed
call 2: dependency failed
call 3: dependency failed
call 4: rejected by open breaker
call 5: rejected by open breaker
call 6: rejected by open breaker
call 7: dependency ok
call 8: dependency ok
Three failures opened the breaker. Calls 4 to 6 were rejected in microseconds without touching the dependency, which is the load that the failing system did not have to absorb. Each rejection slept 30 ms in this demo, so by call 7 the 100 ms cool-down had passed, the breaker went half-open, the trial call succeeded and the breaker closed again. In production you would count failures within a time window, treat slow calls as failures and expose the state as a metric, or reach for a tested crate (circuit breakers and bulkheads).
A retry budget, not a retry count
A per-request limit of three retries still sends 4x the traffic when everything is failing. A budget limits retries relative to total traffic instead: for example, retries may be at most 10% of the requests in the recent window. When the dependency is healthy the budget is never used up, and when it is not, retries are cut off before they become a storm.
use std::sync::atomic::{AtomicU32, Ordering};
struct RetryBudget {
requests: AtomicU32,
retries: AtomicU32,
percent: u32,
}
impl RetryBudget {
fn new(percent: u32) -> Self {
Self { requests: AtomicU32::new(0), retries: AtomicU32::new(0), percent }
}
fn record_request(&self) {
self.requests.fetch_add(1, Ordering::Relaxed);
}
fn try_retry(&self) -> bool {
let allowed = self.requests.load(Ordering::Relaxed) * self.percent / 100;
let used = self.retries.load(Ordering::Relaxed);
if used < allowed {
self.retries.fetch_add(1, Ordering::Relaxed);
true
} else {
false
}
}
}
fn main() {
let budget = RetryBudget::new(10);
let (mut sent, mut refused) = (0, 0);
for _request in 0..100 {
budget.record_request();
sent += 1; // the first attempt, always allowed
// every call fails; each request wants up to 3 retries
for _ in 0..3 {
if budget.try_retry() {
sent += 1;
} else {
refused += 1;
}
}
}
println!("100 requests produced {sent} calls, {refused} retries refused");
}100 requests produced 110 calls, 290 retries refused
Without the budget the same 100 failing requests would have produced 400 calls. With it, they produced 110: the 100 first attempts plus 10 retries. Production budgets decay over a sliding window so old traffic stops counting, but the idea is the same, and it is the mechanism that prevents the 4x amplification at the top of this page.
Where this advice stops
- Layers multiply. If a client retries three times and the service behind it also retries three times, one failure becomes nine calls. Retry at one layer, usually the outermost that knows the operation is safe to repeat.
- A retry budget caps the damage: allow retries to add at most 10% extra traffic over a window, and drop the rest. A fixed per-request count does not do this.
- Rate limits are a different signal. A 429 with
Retry-Aftermeans wait that long, not use your own backoff (token bucket rate limiting). - None of this replaces fixing the cause. Resilience patterns buy time while the dependency recovers; they do not make a permanently broken one work.
