Graceful Degradation in Rust Services Under Heavy Load

Services do not fail at the load they were designed for. They fail at 1.5 times that, when queues grow, latencies stretch, clients time out and retry, and the retries make it worse. A service that degrades gracefully keeps serving a useful share of traffic during the spike and recovers on its own afterward. The techniques are standard, and the work is deciding the policy beforehand. Versions: Rust 1.90, Axum 0.8, Tower 0.5, tower-http 0.6.

How overload kills a service

Suppose a handler takes 50 ms and the service has capacity for 200 requests per second. At 250 requests per second, the extra 50 queue up each second. The queue grows without bound, so the wait grows, so by the time a request is processed the client has given up and sent a retry, work is done for nobody, and goodput (useful work) falls to near zero while the CPU is at 100 percent. The fix is to stop accepting what you cannot finish.

Bound concurrency and shed the rest

use axum::{error_handling::HandleErrorLayer, http::StatusCode, routing::get, BoxError, Router};
use std::time::Duration;
use tower::{limit::ConcurrencyLimitLayer, load_shed::LoadShedLayer, timeout::TimeoutLayer, ServiceBuilder};

async fn handle_overload(err: BoxError) -> (StatusCode, [(&'static str, &'static str); 1], &'static str) {
    if err.is::<tower::load_shed::error::Overloaded>() {
        (StatusCode::SERVICE_UNAVAILABLE, [("retry-after", "2")], "busy, retry shortly")
    } else if err.is::<tower::timeout::error::Elapsed>() {
        (StatusCode::GATEWAY_TIMEOUT, [("retry-after", "5")], "request timed out")
    } else {
        (StatusCode::INTERNAL_SERVER_ERROR, [("retry-after", "5")], "internal error")
    }
}

pub fn app() -> Router {
    let protect = ServiceBuilder::new()
        .layer(HandleErrorLayer::new(handle_overload))
        .layer(LoadShedLayer::new())
        .layer(ConcurrencyLimitLayer::new(256))
        .layer(TimeoutLayer::new(Duration::from_secs(5)));

    Router::new().route("/search", get(search)).layer(protect)
}

async fn search() -> &'static str {
    tokio::time::sleep(Duration::from_millis(50)).await;
    "results"
}

ConcurrencyLimitLayer allows 256 requests in flight. Without LoadShedLayer, the 257th request waits for a permit, which is a queue; with it, the layer returns Overloaded immediately when the inner service is not ready, which the error handler turns into a 503 with Retry-After. Layers are applied from the outside in order listed: error handling outermost, then shedding, then concurrency, then timeout around the handler. The timeout bounds the time a single slow request holds a permit. Note that the stock concurrency limit protects the whole router. Per-route limits (expensive search versus cheap health check) need separate layers on separate routes.

Timeouts everywhere

Every call crossing a boundary needs a deadline: incoming requests, database queries, outgoing HTTP calls. Budget them so inner timeouts are shorter than outer ones, or the outer gives up while the inner keeps working (Axum timeouts and limits, Tokio timeouts). Propagate the remaining deadline if you can, so downstream services stop working on abandoned requests.

Fallbacks and stale data

When a dependency is slow, serve something rather than fail:

use std::time::Duration;
use tokio::time::timeout;

async fn recommendations(user_id: u64, cache: &Cache) -> Vec<String> {
    match timeout(Duration::from_millis(200), fetch_recommendations(user_id)).await {
        Ok(Ok(items)) => {
            cache.store(user_id, items.clone());
            items
        }
        _ => cache.get_stale(user_id).unwrap_or_else(popular_items),
    }
}

Ladder of fallbacks: fresh result, stale cached result, generic default, omit the feature. Stale-while-revalidate does this systematically (stale while revalidate). A circuit breaker avoids hammering a failing dependency while it recovers (circuit breakers and bulkheads).

Priorities

Not all requests are equal. Health checks must always answer. Writes that complete a purchase matter more than a recommendations widget. Options:

• Separate concurrency limits per route group, so low-priority traffic cannot consume all permits.
• Separate listeners or pools for critical paths and background work.
• Bulkheads: a dedicated connection pool for the critical database operations (database timeouts and retries).
• Shed by class: reject anonymous traffic first, then free tier, then paid.

let critical = Router::new().route("/checkout", post(checkout)).layer(ConcurrencyLimitLayer::new(128));
let optional = Router::new().route("/recommendations", get(recs)).layer(ConcurrencyLimitLayer::new(32));
let app = Router::new().merge(critical).merge(optional);

Each group has its own permit pool, so a flood of recommendation requests cannot starve checkout.

Degrade features on purpose

Under pressure, switch off the expensive extras: fancy sorting, personalization, large page sizes, synchronous analytics. A flag toggled automatically (when queue depth or latency crosses a threshold) or by an operator makes it a lever rather than a code change (feature flags). Return smaller pages, drop optional fields, skip enrichment calls. Document these modes and test them.

Measure overload

• Requests in flight, queue depth, p50/p95/p99 latency and rejection rate per route.
• Goodput: successful responses per second.
• Saturation signals: worker busy ratio, connection pool wait time (Tokio runtime metrics, tail latency).

Alert on rising shed counts and on latency, and not on CPU alone.

Clients matter too

A 503 with Retry-After works only if clients respect it. Retries need backoff, jitter and budgets, or they amplify the overload (async resilience patterns). Document the behavior for your API consumers.

Test it

Run a load test above capacity and check the shape of the curve: latency for accepted requests should stay bounded, rejection rate should rise, and recovery after the spike should be quick (load testing). Also kill a dependency during the test, and verify fallbacks engage. Related design: backpressure in pipelines, Axum rate limiting and service performance checklist.