Axum Graceful Shutdown and Draining Connections Step by Step

A deploy that kills the process kills every request in flight. Clients see resets, jobs stop half done, and the error graph has a spike at every release. Axum already knows how to drain: you give it a future that resolves when it is time. The part people get wrong is the signal, because ctrl_c is not what containers send. This post is the shutdown piece of the Axum production service guide.

Axum 0.8, tokio 1, Rust 1.90.

The signal future

use tokio::signal;

async fn shutdown_signal() {
    let ctrl_c = async {
        signal::ctrl_c().await.expect("install Ctrl+C handler");
    };

    #[cfg(unix)]
    let terminate = async {
        signal::unix::signal(signal::unix::SignalKind::terminate())
            .expect("install SIGTERM handler")
            .recv()
            .await;
    };

    #[cfg(not(unix))]
    let terminate = std::future::pending::<()>();

    tokio::select! {
        _ = ctrl_c => {},
        _ = terminate => {},
    }
}

On Unix it resolves on either SIGINT or SIGTERM. On other platforms the pending future never resolves, leaving only Ctrl+C.

Wire it in

#[tokio::main]
async fn main() {
    let app = axum::Router::new().route("/", axum::routing::get(|| async { "ok" }));
    let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await.unwrap();
    axum::serve(listener, app)
        .with_graceful_shutdown(shutdown_signal())
        .await
        .unwrap();
    println!("server stopped");
}

Send SIGTERM with kill -TERM <pid> while a slow request is running: the request completes, then server stopped prints. New connections during the drain are refused.

Bound the drain

A single client holding a connection open can keep the server alive forever. Start a deadline the moment the signal arrives:

async fn shutdown_with_deadline(deadline: std::time::Duration) {
    shutdown_signal().await;
    tokio::spawn(async move {
        tokio::time::sleep(deadline).await;
        eprintln!("drain timed out, exiting");
        std::process::exit(1);
    });
}

Pass shutdown_with_deadline(Duration::from_secs(25)) to with_graceful_shutdown. If every request finishes sooner, serve returns, main ends and the spawned task dies with the runtime. Wrapping serve itself in tokio::time::timeout would be wrong, because that clock starts at boot and not at the signal. Keep the deadline below the platform grace period: Kubernetes waits terminationGracePeriodSeconds (30 by default), and systemd waits TimeoutStopSec.

Order with the load balancer

On Kubernetes, readiness probes and endpoint removal race with SIGTERM. A common fix is to set a flag that makes /ready return 503 when the signal arrives, sleep a few seconds so the balancer notices, then begin the drain. The probes are described in health, readiness and liveness.

Long lived connections

WebSockets and server sent events never finish by themselves. Give them a cancellation token cloned from the same signal, and close them with a message when it fires. Without it, shutdown always hits the drain deadline.

Background work

Tasks you spawned continue until main returns, which then drops the runtime. Keep their JoinHandles, signal them, and await them after serve returns. The cancellation concerns are the same as in timeouts in Tokio.

When this is not enough

Handlers that talk to external systems still need timeouts, and a crash skips all of this. The axum::serve docs describe the API at docs.rs, and the deployment side is in deploying a Rust service with systemd.