Testing Retry and Timeout Logic in Rust Without Sleeping
Retry and timeout code is full of delays, and sleeping in tests makes them slow and flaky. Use Tokio paused time, a scripted flaky dependency and an injected backoff to test every path in milliseconds.
A retry loop with exponential backoff and a timeout has more branches than it looks like: success on the first try, success after failures, a permanent error that must not retry, exhaustion, a timeout in the middle. Testing those with real sleeps is slow and still racy. The solution is to control time. This post builds a small retry helper and tests every path in a few milliseconds. It compiles on Rust 1.90 with edition 2024 and Tokio 1.47 with the test-util feature.
The code under test
use std::{future::Future, time::Duration};
use tokio::time::{sleep, timeout};
#[derive(Debug, PartialEq)]
pub enum CallError {
Transient,
Permanent,
}
#[derive(Debug, PartialEq)]
pub enum RetryError {
Exhausted(u32),
Permanent,
TimedOut,
}
pub async fn retry<T, F, Fut>(
max_attempts: u32,
base_delay: Duration,
per_call_timeout: Duration,
mut op: F,
) -> Result<T, RetryError>
where
F: FnMut() -> Fut,
Fut: Future<Output = Result<T, CallError>>,
{
let mut attempt = 0;
loop {
attempt += 1;
let failure = match timeout(per_call_timeout, op()).await {
Ok(Ok(v)) => return Ok(v),
Ok(Err(CallError::Permanent)) => return Err(RetryError::Permanent),
Ok(Err(CallError::Transient)) => RetryError::Exhausted(attempt),
Err(_) => RetryError::TimedOut,
};
if attempt >= max_attempts {
return Err(failure);
}
sleep(base_delay * 2u32.pow(attempt - 1)).await;
}
}Each attempt is bounded by a timeout, a permanent error stops at once, and the last failure is reported when attempts run out. The delays are 1x, 2x, 4x the base. Nothing here reads real time except through tokio::time, which is what makes it controllable.
A scripted dependency
use std::{collections::VecDeque, sync::{Arc, Mutex}};
#[derive(Clone)]
struct Script<T> {
results: Arc<Mutex<VecDeque<Result<T, CallError>>>>,
calls: Arc<Mutex<u32>>,
}
impl<T: Clone> Script<T> {
fn new(results: Vec<Result<T, CallError>>) -> Self {
Script { results: Arc::new(Mutex::new(results.into())), calls: Arc::new(Mutex::new(0)) }
}
async fn call(&self) -> Result<T, CallError> {
*self.calls.lock().unwrap() += 1;
self.results.lock().unwrap().pop_front().expect("script exhausted")
}
fn calls(&self) -> u32 {
*self.calls.lock().unwrap()
}
}"Fail twice, then succeed" is Script::new(vec![Err(Transient), Err(Transient), Ok(7)]): no mock framework, and the expectation reads as data.
The tests, with paused time
#[cfg(test)]
mod tests {
use super::*;
use tokio::time::Instant;
const BASE: Duration = Duration::from_secs(1);
const CALL_TIMEOUT: Duration = Duration::from_secs(5);
#[tokio::test(start_paused = true)]
async fn succeeds_after_two_transient_failures_with_exponential_backoff() {
let script = Script::new(vec![Err(CallError::Transient), Err(CallError::Transient), Ok(7)]);
let start = Instant::now();
let out = retry(5, BASE, CALL_TIMEOUT, || script.call()).await;
assert_eq!(out, Ok(7));
assert_eq!(script.calls(), 3);
assert_eq!(start.elapsed(), Duration::from_secs(1 + 2)); // waited 1s then 2s, virtually
}
#[tokio::test(start_paused = true)]
async fn permanent_error_is_not_retried() {
let script = Script::<u32>::new(vec![Err(CallError::Permanent)]);
let out = retry(5, BASE, CALL_TIMEOUT, || script.call()).await;
assert_eq!(out, Err(RetryError::Permanent));
assert_eq!(script.calls(), 1);
}
#[tokio::test(start_paused = true)]
async fn gives_up_after_max_attempts() {
let script = Script::<u32>::new(vec![Err(CallError::Transient); 3]);
let out = retry(3, BASE, CALL_TIMEOUT, || script.call()).await;
assert_eq!(out, Err(RetryError::Exhausted(3)));
assert_eq!(script.calls(), 3);
}
#[tokio::test(start_paused = true)]
async fn a_hung_call_is_cut_off_by_the_timeout() {
let start = Instant::now();
let out: Result<u32, _> = retry(1, BASE, CALL_TIMEOUT, || async {
tokio::time::sleep(Duration::from_secs(3600)).await; // never returns in time
Ok::<u32, CallError>(1)
})
.await;
assert_eq!(out, Err(RetryError::TimedOut));
assert_eq!(start.elapsed(), CALL_TIMEOUT);
}
}With start_paused = true the clock only moves when every task is blocked on a timer, at which point Tokio jumps straight to the next deadline. The whole module runs in milliseconds but observes exact virtual durations, so the assertion "waited 3 seconds in total" is precise rather than approximate. This only works because the code uses tokio::time; std::thread::sleep or std::time::Instant inside the code under test would still take real time or read the real clock (testing async with paused time).
Other behaviors worth covering
- Cancellation: if the caller drops the retry future mid-backoff, no further calls happen. Wrap it in an outer
timeoutshorter than the schedule and assertcalls()stops. - A total deadline: if your policy caps total elapsed time, test it with a script that always fails and assert the virtual elapsed time equals the cap.
- Jitter: take the random factor as a parameter (a closure returning 0.0 to 1.0) so tests use zero and a property test checks bounds.
- Idempotency: a retry that repeats a write needs the receiver to be safe against duplicates (idempotency keys).
The production side of these policies is in async retries, timeouts and circuit breakers, and the same technique tests periodic work (missed ticks).
