Request Coalescing to Stop Cache Stampedes in Rust Services
When a hot cache entry expires, a hundred requests recompute it at once. Coalesce concurrent misses so one loads and the rest wait, with moka get_with, a hand-rolled version and a note on stale-while-revalidate.
Caches fail in an unusual way: they work until the moment they matter most. A homepage query is cached for a minute. At the minute boundary, with a thousand requests per second, the entry expires and for the next few hundred milliseconds every request misses and queries the database at once. The database slows, the load takes longer, more requests pile up: a stampede. The fix is small. This post shows it with moka and by hand. Versions: Rust 1.90 with edition 2024, moka 0.12, Tokio 1.47, futures 0.3.
The problem in code
use moka::future::Cache;
use std::{sync::atomic::{AtomicU32, Ordering}, sync::Arc, time::Duration};
async fn load_homepage(counter: Arc<AtomicU32>) -> String {
counter.fetch_add(1, Ordering::SeqCst);
tokio::time::sleep(Duration::from_millis(200)).await; // an expensive query
"homepage".to_string()
}
async fn naive(cache: &Cache<&'static str, String>, loads: Arc<AtomicU32>) -> String {
if let Some(v) = cache.get("home").await {
return v;
}
let v = load_homepage(loads).await; // every concurrent miss runs this
cache.insert("home", v.clone()).await;
v
}The check and the load are separate steps. If fifty tasks call naive while the entry is absent, all fifty see a miss and all fifty run the load for 200 ms before the first insert lands.
The fix with moka
use moka::future::Cache;
use std::{sync::{atomic::{AtomicU32, Ordering}, Arc}, time::Duration};
#[tokio::main]
async fn main() {
let cache: Cache<&'static str, String> = Cache::builder()
.max_capacity(10_000)
.time_to_live(Duration::from_secs(60))
.build();
let loads = Arc::new(AtomicU32::new(0));
let mut tasks = Vec::new();
for _ in 0..50 {
let (cache, loads) = (cache.clone(), loads.clone());
tasks.push(tokio::spawn(async move {
cache
.get_with("home", async {
loads.fetch_add(1, Ordering::SeqCst);
tokio::time::sleep(Duration::from_millis(200)).await;
"homepage".to_string()
})
.await
}));
}
for t in tasks {
t.await.unwrap();
}
println!("loads = {}", loads.load(Ordering::SeqCst));
}loads = 1get_with checks the cache and, on a miss, runs the supplied future once; concurrent callers for the same key wait for that one result. Fifty tasks, one load. Different keys load concurrently, so unrelated entries never wait for each other. A Cache is cheap to clone (it is a handle), so share it through application state (in-process caches).
Fallible loads
use std::sync::Arc;
async fn fetch_user(cache: &moka::future::Cache<u64, String>, id: u64) -> Result<String, Arc<sqlx::Error>> {
cache
.try_get_with(id, async move { load_user_from_db(id).await })
.await
}
async fn load_user_from_db(_id: u64) -> Result<String, sqlx::Error> {
Ok("ada".into())
}try_get_with coalesces in the same way. If the load fails, every waiter receives the error (wrapped in an Arc), and nothing is cached, so the next request tries again. Decide whether that is acceptable: a failing dependency will be called by the next request immediately, so combine it with a retry policy that has backoff, or cache failures briefly yourself (a "negative cache") to protect the dependency.
Without moka: a shared future per key
use futures::future::{BoxFuture, FutureExt, Shared};
use std::{collections::HashMap, future::Future, sync::{Arc, Mutex}};
type InFlight = Shared<BoxFuture<'static, Arc<String>>>;
#[derive(Clone, Default)]
pub struct Coalescer {
inflight: Arc<Mutex<HashMap<u64, InFlight>>>,
}
impl Coalescer {
pub async fn get<F, Fut>(&self, key: u64, load: F) -> Arc<String>
where
F: FnOnce() -> Fut,
Fut: Future<Output = String> + Send + 'static,
{
let shared = {
let mut map = self.inflight.lock().unwrap();
if let Some(f) = map.get(&key) {
f.clone()
} else {
let fut: BoxFuture<'static, Arc<String>> = async move { Arc::new(load().await) }.boxed();
let shared = fut.shared();
map.insert(key, shared.clone());
shared
}
};
let result = shared.await;
self.inflight.lock().unwrap().remove(&key); // the first finisher removes the entry
result
}
}The map holds one Shared future per key; later callers clone it and await the same computation. The guard is dropped before the await so the mutex is never held across it (mutex choices). This minimal version also needs your own handling of errors, cancellation (if the first caller is dropped, the shared future is dropped with it only when no clones remain) and removal races. For production use, prefer moka or the async-singleflight style crates over rolling your own.
Defense in depth
- Jitter the TTL. Many entries created together expire together. Use a randomized lifetime, for example 60 seconds plus or minus 10 percent, so expirations spread out.
- Refresh before expiry. A background task reloads popular keys shortly before they expire, so requests never see a miss.
- Serve stale while revalidating. Keep an expired value for a short grace period and return it immediately while one task refreshes (stale-while-revalidate).
- Limit the loader's concurrency. A semaphore around the expensive call caps the damage across different keys (semaphores).
- Timeouts on loads so waiters are not held hostage by a slow dependency (timeouts and retries).
Cache design more broadly, including invalidation, is in cache invalidation; a two-tier setup with an in-process cache and Redis is in caching with moka and Redis.
