Tail Latency in Rust Services and Why Averages Mislead You

A dashboard that says "average response time 25 ms" is reassuring and almost useless. Users do not experience averages; they experience individual requests, and the slow ones are the ones they remember and complain about. Operating a service means looking at the tail of the latency distribution. This post covers how to record it in a Rust service and why it behaves the way it does. Versions: Rust 1.90 with edition 2024, Axum 0.8, metrics 0.24 and metrics-exporter-prometheus 0.17.

Averages versus percentiles

Suppose 99 requests take 20 ms and one takes 2,000 ms. The mean is 39.8 ms and looks fine; the 99th percentile (p99) is the value 99 percent of requests stay under, here 20 ms at p99 and 2,000 ms at p100. Now suppose a page makes 40 such calls. The chance that at least one hits the slow case is 1 - 0.99^40, about 33 percent, so a third of page loads are slow even though each backend call is "99 percent fast". That is why the tail matters more as systems get more distributed.

Recording a histogram

use axum::{extract::{MatchedPath, Request}, middleware::Next, response::Response};
use std::time::Instant;

pub async fn track_latency(req: Request, next: Next) -> Response {
    let route = req
        .extensions()
        .get::<MatchedPath>()
        .map(|p| p.as_str().to_owned())
        .unwrap_or_else(|| "unmatched".to_owned());
    let method = req.method().to_string();
    let start = Instant::now();

    let response = next.run(req).await;

    let status = response.status().as_u16().to_string();
    metrics::histogram!(
        "http_request_duration_seconds",
        "route" => route,
        "method" => method,
        "status" => status
    )
    .record(start.elapsed().as_secs_f64());
    response
}

use metrics_exporter_prometheus::{Matcher, PrometheusBuilder};

fn install_metrics() -> metrics_exporter_prometheus::PrometheusHandle {
    PrometheusBuilder::new()
        .set_buckets_for_metric(
            Matcher::Full("http_request_duration_seconds".into()),
            &[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0],
        )
        .unwrap()
        .install_recorder()
        .unwrap()
}

Label by the matched route template (/users/{id}), not the raw path, or every id creates a new time series and the metric store explodes. Bucket boundaries should bracket your objective: if the target is 300 ms, include 0.25 and 0.5 so percentiles near it are accurate. Prometheus then computes percentiles from the buckets: histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))). Because the quantile comes from merged buckets, it stays valid across instances; averaging per-instance p99 values does not.

Why tails are long in practice

Measure honestly: coordinated omission

A closed-loop load generator that sends the next request only after the previous one returns slows down exactly when the server does, so during a stall it sends nothing and records one slow sample instead of the hundreds of requests that would have been delayed. The result understates the tail. Use a tool with an open-loop mode and latency correction, such as oha --latency-correction, wrk2 or k6 with arrival-rate executors, and look at the full distribution, not only the mean (load testing a Rust service).

Finding what a slow request did

1. Add a trace span per request and per downstream call, with a request id, so a slow request shows where time went (request ids).
2. Log requests over a threshold (slow-request log) with their route and key parameters.
3. Compare slow and fast requests for the same route: different parameters? a cache miss? a particular tenant?
4. Profile under load for the busy-time causes (flamegraphs) and use runtime metrics for scheduling delays (Tokio runtime metrics).

Reducing the tail

• Put a timeout on every dependency call, and a deadline on the request, so one slow call cannot dominate.
• Hedge idempotent reads: after a short delay, send a second request to another replica and take the first answer.
• Shed load before queues grow: reject early when saturated rather than letting every request wait.
• Avoid synchronized behavior: jitter timers, TTLs and retry delays.
• Keep tasks short and yield in long loops; move CPU-heavy work off the async threads.

Tie it to an objective ("99 percent of requests under 300 ms over 30 days") so the numbers drive decisions instead of arguments. A broader launch list is in the performance checklist.