Health Checks Readiness and Liveness for Rust Services

Liveness says the process is stuck, readiness says it cannot take traffic yet. Implement both in Axum, decide what each checks and avoid the restart loops a bad probe causes.

3 min read

Summarize in AI

Quick Summary

Expose a liveness endpoint that returns 200 if the process can answer at all, and a readiness endpoint that returns 200 only when dependencies the service needs, such as the database pool, are usable and the service is not draining. Liveness failures restart the process, and readiness failures remove it from the load balancer, so keep liveness trivial and shallow.

Key takeaways

Liveness answers "should this process be killed", and readiness answers "should this instance receive traffic".
A liveness probe that checks the database restarts every instance when the database is down, which turns an outage into a bigger one.
Readiness should fail during startup before dependencies are ready and during shutdown while draining.
Keep probes cheap, with timeouts shorter than the probe interval, and unauthenticated on an internal port or path.
A startup probe or a long initial delay avoids killing a slow-starting service.

Two probes, two questions, and mixing them up is a classic way to take down a healthy system. Liveness asks whether the process is wedged and needs a restart. Readiness asks whether this instance should be given traffic right now. This post implements both in Axum and shows the shutdown interplay. It is the health step of shipping Rust to production.

Axum 0.8, SQLx 0.8, Rust 1.90.

The endpoints

rust

use std::sync::{atomic::{AtomicBool, Ordering}, Arc};
use axum::{extract::State, http::StatusCode, routing::get, Router};
use sqlx::PgPool;

#[derive(Clone)]
pub struct AppState {
    pub pool: PgPool,
    pub draining: Arc<AtomicBool>,
}

async fn live() -> StatusCode {
    StatusCode::OK
}

async fn ready(State(s): State<AppState>) -> StatusCode {
    if s.draining.load(Ordering::Relaxed) {
        return StatusCode::SERVICE_UNAVAILABLE;
    }
    let ping = tokio::time::timeout(
        std::time::Duration::from_millis(500),
        sqlx::query("SELECT 1").execute(&s.pool),
    )
    .await;
    match ping {
        Ok(Ok(_)) => StatusCode::OK,
        _ => StatusCode::SERVICE_UNAVAILABLE,
    }
}

pub fn health_routes() -> Router<AppState> {
    Router::new().route("/livez", get(live)).route("/readyz", get(ready))
}

/livez returns 200 if the runtime can poll the handler, which proves the process is not deadlocked. /readyz checks the draining flag and does a 500 ms-bounded SELECT 1. The timeout matters: a hung database call must not hang the probe, or the probe itself times out and the orchestrator concludes the same thing with more delay.

What each should check

	Liveness	Readiness
Purpose	restart a stuck process	route traffic only to capable instances
Dependencies	none	the ones needed to serve requests
Failure action	kill and restart	remove from load balancer
Cost	trivial	a cheap query with a timeout
During shutdown	stays 200	goes 503

The shutdown sequence

On SIGTERM, set draining to true first. Readiness turns 503, the load balancer stops sending new requests (after its next probe, a few seconds), and in-flight requests finish. Then begin the graceful shutdown from Axum graceful shutdown. Sleeping a few seconds between setting the flag and closing the listener covers the probe interval.

Kubernetes probes

YAML

livenessProbe:
  httpGet: { path: /livez, port: 3000 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: 3000 }
  periodSeconds: 5
startupProbe:
  httpGet: { path: /readyz, port: 3000 }
  failureThreshold: 30
  periodSeconds: 2

The startup probe gives the service up to a minute to start before liveness applies. Without one, a service that runs migrations at boot can be killed mid-migration.

Mistakes that cause outages

A liveness probe that checks the database: a database blip restarts every instance at once.
A readiness probe without a timeout: a slow dependency makes every instance appear down, and with no healthy backend, traffic goes nowhere.
Checking every dependency in readiness: an optional service that is down should degrade features, not remove the instance.
Heavy probes: a readiness check that runs an expensive query every 5 seconds from every pod is its own load.

Docker and Compose

A container HEALTHCHECK or a Compose healthcheck can call the same endpoint with curl or a tiny built-in subcommand of your binary, since distroless images have no curl. See Compose healthchecks. The Kubernetes documentation explains the three probe types under configure liveness, readiness and startup probes.

Also inWeb

Taggedhealth