Skip to content
rusthacks.dev
Security

Request Limits and Slow Client Attacks in Rust Services

Slow clients and oversized requests exhaust connections and memory. Set header and body timeouts, body size limits, concurrency limits and keep-alive bounds at the proxy and in the Axum service with Tower layers.

5 min read
Summarize in AI
Flat illustration for the post “Request Limits and Slow Client Attacks in Rust Services”, composed around web.

Denial of service does not need a flood. A client that connects and then sends one byte every ten seconds can hold a connection indefinitely, and a few thousand such clients can exhaust file descriptors, memory or a worker pool. Rust and Tokio make idle connections cheap, which raises the threshold but does not remove it. The cure is explicit limits at every layer. This post lists them with working Axum code. Versions: Rust 1.90 with edition 2024, Axum 0.8, Tower 0.5, tower-http 0.6, Hyper 1, Tokio 1.47.

What can be abused

ResourceAbuseLimit
Connection slots, file descriptorsMany idle or half-open connectionsHeader read timeout, keep-alive timeout, connection cap
Memory per requestHuge bodies, huge headersBody and header size limits
Task timeA handler waiting forever on a slow upload or dependencyRequest timeout
ConcurrencyMany simultaneous expensive requestsConcurrency limit, load shedding, rate limits
Downstream resourcesDatabase connections held by slow requestsAcquire after the body is read; statement timeouts

Header read timeout in Hyper

rust
use std::time::Duration;
use hyper_util::{rt::{TokioExecutor, TokioIo, TokioTimer}, server::conn::auto::Builder};
use tokio::net::TcpListener;

#[tokio::main]
async fn main() -> std::io::Result<()> {
    let listener = TcpListener::bind("0.0.0.0:3000").await?;
    let app = axum::Router::new().route("/", axum::routing::get(|| async { "ok" }));
    let svc = hyper_util::service::TowerToHyperService::new(app);

    loop {
        let (stream, _) = listener.accept().await?;
        let svc = svc.clone();
        tokio::spawn(async move {
            let mut builder = Builder::new(TokioExecutor::new());
            builder
                .http1()
                .timer(TokioTimer::new())
                .header_read_timeout(Duration::from_secs(10))   // slow headers: connection closed
                .max_buf_size(32 * 1024);                       // cap header buffer
            let _ = builder.serve_connection(TokioIo::new(stream), svc).await;
        });
    }
}

axum::serve hides the connection builder, so setting hyper options needs the loop above (or a crate that exposes them). Hyper 1 applies a default header read timeout for HTTP/1 only when a timer is configured; configuring it explicitly documents the intent. The buffer cap bounds the size of the header section. When this is too much ceremony, terminate connections at a proxy that has these knobs by default and keep the service on a private network.

Timeouts, body limits, concurrency in Tower

rust
use std::time::Duration;
use axum::{error_handling::HandleErrorLayer, http::StatusCode, routing::post, BoxError, Router};
use tower::{limit::ConcurrencyLimitLayer, load_shed::LoadShedLayer, ServiceBuilder};
use tower_http::{limit::RequestBodyLimitLayer, timeout::TimeoutLayer};

fn app() -> Router {
    Router::new()
        .route("/items", post(|body: String| async move { body.len().to_string() }))
        .layer(
            ServiceBuilder::new()
                .layer(HandleErrorLayer::new(|e: BoxError| async move {
                    if e.is::<tower::load_shed::error::Overloaded>() {
                        (StatusCode::SERVICE_UNAVAILABLE, "overloaded")
                    } else {
                        (StatusCode::INTERNAL_SERVER_ERROR, "internal error")
                    }
                }))
                .layer(LoadShedLayer::new())                         // reject instead of queueing
                .layer(ConcurrencyLimitLayer::new(512))              // at most 512 in flight
                .layer(TimeoutLayer::new(Duration::from_secs(15)))   // whole request, including the body
                .layer(RequestBodyLimitLayer::new(1024 * 1024)),     // 1 MB
        )
}

Order matters. The outermost layers see the request first: load shedding and the concurrency limit decide whether to admit it, the timeout bounds its total time, and the body limit caps its size. LoadShedLayer turns "no capacity" into an immediate error that the handler maps to 503, instead of letting requests queue and time out slowly (queued work still holds memory). TimeoutLayer from tower-http returns 408 or 504 when the inner service exceeds the duration. A body limit applies while the body is read, so a stream that exceeds it fails mid-way, and a request with a large Content-Length can be rejected before reading. Choose numbers from your traffic: p99 latency times a small factor for the timeout, and the largest legitimate request for the body.

The slow body

A timeout on the whole request covers a client that trickles a valid body slowly, but a long total timeout still allows ten seconds of a held connection per request. For upload routes that legitimately need longer, set a longer timeout only there, with a smaller concurrency cap. A per-chunk read timeout is stricter: wrap the body stream with a timeout on each frame (tokio_stream::StreamExt::timeout) and abort when no data arrives for a few seconds.

At the proxy

Nginx
client_header_timeout   10s;
client_body_timeout     10s;
send_timeout            10s;
keepalive_timeout       30s;
client_max_body_size    1m;
limit_conn_zone $binary_remote_addr zone=perip:10m;
limit_conn perip 50;
limit_req_zone $binary_remote_addr zone=reqs:10m rate=20r/s;
limit_req zone=reqs burst=40 nodelay;

The reverse proxy (Nginx, HAProxy, Envoy, a cloud load balancer) is built to hold many slow connections cheaply and to cut them off. It also gives per-IP connection and request-rate limits (rate limiting and abuse prevention, Axum rate limiting). Remember that behind a proxy all connections come from the proxy's address, so per-client logic in the service needs the real client IP from a trusted header.

Do not hold scarce resources while waiting

  • Read and validate the request body before acquiring a database connection, so a slow client cannot pin one.
  • Put statement timeouts on queries (timeouts and retries).
  • Bound channels and queues so slow consumers cannot make memory grow (backpressure).

Test it

rust
#[tokio::test]
async fn slow_headers_are_cut_off() {
    use tokio::{io::{AsyncReadExt, AsyncWriteExt}, net::TcpStream, time::{timeout, Duration}};
    let addr = spawn_test_server().await;
    let mut s = TcpStream::connect(addr).await.unwrap();
    s.write_all(b"GET / HTTP/1.1\r\nHost: x\r\n").await.unwrap();   // never finish the headers
    let mut buf = [0u8; 64];
    let r = timeout(Duration::from_secs(15), s.read(&mut buf)).await;
    assert!(r.is_ok(), "server should close or respond within the header timeout");
}

async fn spawn_test_server() -> std::net::SocketAddr { "127.0.0.1:0".parse().unwrap() }

Send incomplete headers and assert the server closes the connection within the configured time, and send a body over the limit and assert a 413. Limits that are not tested tend to be disabled by a refactor.

Also inWeb
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.