Stale While Revalidate Caching in Rust Services Explained
Serve the cached value immediately, even if slightly old, and refresh it in the background. Learn the idea, the HTTP Cache-Control directives and an in-process implementation with a freshness window and a grace window.
A cache with a time to live has a built-in unfairness: one request per expiry pays the full cost of the reload while everyone else gets a fast answer. Stale-while-revalidate removes the unfairness by relaxing one thing, the guarantee that data is never older than the TTL. Instead you decide how stale is acceptable and refresh quietly. This post covers the HTTP form and an in-process implementation. It compiles on Rust 1.90 with edition 2024 and Tokio 1.47.
The HTTP version
Cache-Control: public, max-age=60, stale-while-revalidate=300, stale-if-error=3600For 60 seconds a cache (browser, CDN or reverse proxy) serves the response without asking. From 60 to 360 seconds it serves the stale copy and revalidates in the background. If the origin errors, it may serve stale for up to an hour. With Axum, set it with a response header or a layer:
use axum::{http::{header, HeaderValue}, response::IntoResponse, routing::get, Json, Router};
use serde_json::json;
async fn news() -> impl IntoResponse {
(
[(header::CACHE_CONTROL, HeaderValue::from_static("public, max-age=60, stale-while-revalidate=300"))],
Json(json!({ "items": [] })),
)
}
fn app() -> Router {
Router::new().route("/news", get(news))
}Support for the directive varies across caches (check your CDN's documentation), but it is the cheapest way to get the effect for public, cacheable responses (ETag and cache headers).
The in-process version
use std::{collections::HashMap, future::Future, sync::{Arc, Mutex}, time::{Duration, Instant}};
#[derive(Clone)]
struct Entry<V> {
value: V,
loaded_at: Instant,
}
enum State<V> {
Fresh(V),
Stale(V),
Missing,
}
pub struct SwrCache<V> {
fresh_for: Duration,
stale_for: Duration,
map: Mutex<HashMap<String, Entry<V>>>,
refreshing: Mutex<std::collections::HashSet<String>>,
}
impl<V: Clone + Send + 'static> SwrCache<V> {
pub fn new(fresh_for: Duration, stale_for: Duration) -> Arc<Self> {
Arc::new(SwrCache { fresh_for, stale_for, map: Default::default(), refreshing: Default::default() })
}
fn lookup(&self, key: &str) -> State<V> {
match self.map.lock().unwrap().get(key) {
Some(e) if e.loaded_at.elapsed() < self.fresh_for => State::Fresh(e.value.clone()),
Some(e) if e.loaded_at.elapsed() < self.fresh_for + self.stale_for => State::Stale(e.value.clone()),
_ => State::Missing,
}
}
fn store(&self, key: &str, value: V) {
self.map.lock().unwrap().insert(key.to_string(), Entry { value, loaded_at: Instant::now() });
}
pub async fn get<F, Fut>(self: &Arc<Self>, key: &str, load: F) -> V
where
F: FnOnce() -> Fut + Send + 'static,
Fut: Future<Output = V> + Send + 'static,
{
match self.lookup(key) {
State::Fresh(v) => v,
State::Stale(v) => {
// start at most one background refresh for this key
if self.refreshing.lock().unwrap().insert(key.to_string()) {
let (this, key) = (self.clone(), key.to_string());
tokio::spawn(async move {
let fresh = load().await;
this.store(&key, fresh);
this.refreshing.lock().unwrap().remove(&key);
});
}
v
}
State::Missing => {
let v = load().await; // too old or never loaded: the caller waits
self.store(key, v.clone());
v
}
}
}
}The three states map to the behaviors: fresh values are returned at once; stale ones are returned at once and one task is spawned to refresh (the refreshing set ensures only one); missing or expired-beyond-grace entries make the caller wait for a real load. Guards are dropped before any await, so no lock is held across one. The cold-start case, a missing key, still has a stampede, so combine with request coalescing (cache stampede coalescing). For real services, moka offers expiry policies and the pieces to build this on top.
Failures: stale if error
Improve the refresh task: if load returns an error, keep the old value and retry later with backoff instead of replacing it. During an outage of the dependency the service continues to answer with data that is old but valid. Surface the situation in a metric (stale serves, refresh failures) so someone notices that the data has stopped updating.
When it is the wrong tool
- Data that must be exact at read time: account balances, inventory at checkout, permissions and revocations. A revoked token served from a stale cache is a security bug.
- Personalized responses cached under a shared key, which would leak data between users.
- Very low-traffic keys: a stale value is used by almost nobody, and a refresh triggered by a rare request does not help it; a plain TTL is simpler.
Choosing the windows
Set fresh_for to how stale the data may be without anyone minding, and the grace window to how long you are willing to show old data if the origin is slow or down. A product catalog might be fresh for a minute and usable for an hour. Pick numbers from product requirements and verify with metrics on hit ratio and refresh latency (cache invalidation).
