Graceful Shutdown on Kubernetes for Rust Services Explained

A rolling update is a controlled outage of every pod, one at a time. If the old pod exits carelessly, a few requests fail every deploy. The sequence Kubernetes runs on shutdown is well defined, and a Rust service can follow it in about forty lines. Versions: Rust 1.90, Axum 0.8, Tokio 1.47, Kubernetes 1.31.

What Kubernetes does

When a pod is deleted:

1. The pod is marked Terminating.
2. In parallel, the endpoints controller removes the pod from Service endpoints, and the preStop hook (if any) runs.
3. After the hook, the container receives SIGTERM.
4. After terminationGracePeriodSeconds, it receives SIGKILL.

Step 2's removal takes time to reach kube-proxy, ingress controllers and external load balancers. Requests can arrive for several seconds after SIGTERM. An app that closes its listener immediately resets those connections.

The shutdown sequence in the app

1. On SIGTERM, set a flag so /ready returns 503.
2. Wait a short delay for endpoint removal to propagate (3 to 10 seconds).
3. Stop accepting new connections and wait for in-flight requests to finish, with a timeout.
4. Exit 0.

The code

use axum::{extract::State, http::StatusCode, routing::get, Router};
use std::sync::{
    atomic::{AtomicBool, Ordering},
    Arc,
};
use std::time::Duration;
use tokio::signal::unix::{signal, SignalKind};

#[derive(Clone)]
struct AppState {
    draining: Arc<AtomicBool>,
}

async fn ready(State(s): State<AppState>) -> StatusCode {
    if s.draining.load(Ordering::SeqCst) {
        StatusCode::SERVICE_UNAVAILABLE
    } else {
        StatusCode::OK
    }
}

async fn work() -> &'static str {
    tokio::time::sleep(Duration::from_millis(200)).await;
    "done"
}

#[tokio::main]
async fn main() {
    let state = AppState { draining: Arc::new(AtomicBool::new(false)) };
    let app = Router::new()
        .route("/work", get(work))
        .route("/ready", get(ready))
        .route("/live", get(|| async { "ok" }))
        .with_state(state.clone());

    let listener = tokio::net::TcpListener::bind("0.0.0.0:8080").await.unwrap();

    let shutdown = async move {
        let mut term = signal(SignalKind::terminate()).unwrap();
        let mut int = signal(SignalKind::interrupt()).unwrap();
        tokio::select! {
            _ = term.recv() => {}
            _ = int.recv() => {}
        }
        tracing::info!("shutdown signal received, failing readiness");
        state.draining.store(true, Ordering::SeqCst);
        tokio::time::sleep(Duration::from_secs(5)).await;
        tracing::info!("draining connections");
    };

    axum::serve(listener, app)
        .with_graceful_shutdown(shutdown)
        .await
        .unwrap();
    tracing::info!("clean exit");
}

After the shutdown future completes, Axum stops accepting new connections and waits for existing ones to finish. Keep-alive connections that are idle close, and connections with a request in flight complete first. If a handler might run for minutes, wrap the whole serve in a timeout so a stuck request cannot hold shutdown forever (Axum graceful shutdown, Tokio graceful shutdown).

The manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: api
          image: registry.example.test/api:1.4.0
          ports: [{ containerPort: 8080 }]
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }
            periodSeconds: 2
            failureThreshold: 1
          livenessProbe:
            httpGet: { path: /live, port: 8080 }
            periodSeconds: 10

Choose the grace period larger than: the readiness delay (5 s) plus your longest request plus a margin. periodSeconds: 2 and failureThreshold: 1 make the readiness failure take effect fast. The liveness probe must keep passing during drain, or Kubernetes will kill the pod early, which is why /live is separate (health checks).

Alternative: preStop

If you do not want sleep logic in the app:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "8"]

This needs a sleep binary in the image, absent in distroless and scratch images. Newer Kubernetes versions offer a sleep action natively: preStop: { sleep: { seconds: 8 } }. The grace period includes the hook's time.

Testing the behavior

kubectl rollout restart deployment/api &
oha -z 60s -c 20 https://api.example.test/work

Count non 2xx responses during the rollout. Zero is the target. Failures that are connection resets point to a too-short delay, and 503s from the app point to readiness flipping before the load balancer stops sending. Add long-lived connections and WebSockets to the test, since they need a policy to close them with a close frame (zero downtime deploys).

Pitfalls

• PID 1 handling: run the binary directly, not through a shell script that swallows the signal. Use exec or tini.
• Background tasks must stop too: cancel them with a token (cancelling tasks).
• Jobs with long work should checkpoint, since the grace period is finite.
• Log the phases with timestamps so you can see where time went.