Rust on Kubernetes Probes Resources and Graceful Shutdown

A Rust service is a good Kubernetes citizen because it is small, starts fast and uses predictable memory. But Kubernetes does not run your program so much as supervise it: it starts it, decides whether it is ready, restarts it when a probe fails, kills it when it exceeds its memory limit, and sends it a signal when it wants it gone. A service written without that supervision in mind works on a laptop and misbehaves in a cluster, usually during a deploy, which is the worst time to find out.

This guide takes one small axum service through the Kubernetes behaviors that matter: configuration from a ConfigMap, readiness and liveness probes, resource requests and limits, graceful shutdown during a pod deletion, and rolling updates. Everything was run on a local single-node cluster (the Docker Desktop one, Kubernetes 1.34.3) with a throwaway namespace, so the outputs are real. What was not tested is a multi-node cluster, ingress, autoscaling, or any cloud provider's load balancer; where the guide mentions them, it says so. The service is the same one used in the Fly.io deployment guide, built into an 842 KB static image.

The service, again

The program reads PORT and GREETING from the environment, exits with status 78 on a configuration error, serves /healthz, / and a deliberately slow /slow, and drains on SIGTERM:

use std::net::SocketAddr;
use std::process::ExitCode;
use std::time::Duration;

use axum::{Router, routing::get};
use tokio::signal::unix::{SignalKind, signal};

struct Config {
    addr: SocketAddr,
    greeting: String,
}

impl Config {
    /// Read and validate everything at startup. A service that starts with bad configuration
    /// and fails on the first request is harder to diagnose than one that refuses to start.
    fn from_env() -> Result<Config, String> {
        let port = std::env::var("PORT").unwrap_or_else(|_| "8080".into());
        let port: u16 = port.parse().map_err(|_| format!("PORT must be a number between 1 and 65535, got {port:?}"))?;
        let greeting = std::env::var("GREETING").map_err(|_| "GREETING is required".to_string())?;
        if greeting.trim().is_empty() {
            return Err("GREETING must not be empty".into());
        }
        Ok(Config { addr: SocketAddr::from(([0, 0, 0, 0], port)), greeting })
    }
}

async fn shutdown_signal() {
    let mut term = signal(SignalKind::terminate()).expect("install SIGTERM handler");
    let mut int = signal(SignalKind::interrupt()).expect("install SIGINT handler");
    tokio::select! {
        _ = term.recv() => println!("got SIGTERM, draining"),
        _ = int.recv() => println!("got SIGINT, draining"),
    }
}

#[tokio::main]
async fn main() -> ExitCode {
    let config = match Config::from_env() {
        Ok(c) => c,
        Err(e) => {
            eprintln!("config error: {e}");
            return ExitCode::from(78); // EX_CONFIG, so supervisors can tell it from a crash
        }
    };
    let greeting = config.greeting.clone();
    let app = Router::new()
        .route("/healthz", get(|| async { "ok" }))
        .route("/", get(move || async move { greeting.clone() }))
        .route("/slow", get(|| async {
            tokio::time::sleep(Duration::from_secs(2)).await;
            "finished the slow request"
        }));
    let listener = match tokio::net::TcpListener::bind(config.addr).await {
        Ok(l) => l,
        Err(e) => {
            eprintln!("cannot bind {}: {e}", config.addr);
            return ExitCode::from(1);
        }
    };
    println!("listening on {}", config.addr);
    if let Err(e) = axum::serve(listener, app).with_graceful_shutdown(shutdown_signal()).await {
        eprintln!("server error: {e}");
        return ExitCode::from(1);
    }
    println!("clean exit");
    ExitCode::SUCCESS
}

Each choice has a Kubernetes consequence. Configuration comes from the environment because that is what a ConfigMap or Secret injects. The 0.0.0.0 bind is required because a Service reaches the pod's IP, not its loopback. The /healthz route is what the probes will call. And the SIGTERM handler is what makes pod deletion graceful: Kubernetes sends SIGTERM to the container's main process, waits up to terminationGracePeriodSeconds, and then sends SIGKILL.

The manifest

One file holds a Deployment, a ConfigMap and a Service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: svc
spec:
  replicas: 2
  selector:
    matchLabels: {app: svc}
  template:
    metadata:
      labels: {app: svc}
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: svc
          image: rh-svc:v3
          imagePullPolicy: Never
          ports:
            - containerPort: 8080
          env:
            - name: GREETING
              valueFrom:
                configMapKeyRef: {name: svc-config, key: greeting}
          readinessProbe:
            httpGet: {path: /healthz, port: 8080}
            periodSeconds: 2
          livenessProbe:
            httpGet: {path: /healthz, port: 8080}
            periodSeconds: 10
            failureThreshold: 3
          resources:
            requests: {cpu: 50m, memory: 16Mi}
            limits: {memory: 64Mi}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: svc-config
data:
  greeting: "hello from kubernetes"
---
apiVersion: v1
kind: Service
metadata:
  name: svc
spec:
  selector: {app: svc}
  ports:
    - port: 80
      targetPort: 8080

Read it top to bottom. The Deployment asks for two replicas of a pod. The container reads GREETING from the ConfigMap with configMapKeyRef, so a missing key is a deployment error, shown below. Both probes call /healthz. The container requests 50 millicores of CPU and 16 MiB of memory and is limited to 64 MiB. imagePullPolicy: Never is specific to this local experiment, where the image was loaded into the node directly; in a real cluster the image is pushed to a registry and the default policy applies. The Service routes port 80 to the pods' port 8080 by label.

Apply it and check:

$ kubectl --context docker-desktop -n rh-lab apply -f app.yaml
deployment.apps/svc created
configmap/svc-config created
service/svc created
$ kubectl get deploy svc
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
svc    2/2     2            2           52s
$ kubectl port-forward svc/svc 18090:80 &
$ curl -s localhost:18090/
hello from kubernetes

Note the --context flag. A machine that talks to several clusters should name the one it means every time; a bare kubectl apply goes to whatever the current context is, which on a developer machine is sometimes production. In writing this guide every command named the local context explicitly.

The first apply failed in an instructive way. The pods sat in ErrImageNeverPull, because the node had no copy of the image:

NAME                   READY   STATUS              RESTARTS   AGE
svc-659b886675-j249w   0/1     ErrImageNeverPull   0          20s
Warning  ErrImageNeverPull  kubelet  Container image "rh-svc:v3" is not present with pull policy of Never

Kubernetes reports image problems on the pod, not on the Deployment, so kubectl describe pod and its Events section are the first place to look when pods will not start. Loading the image into the node fixed this local case. In a real cluster the equivalent errors are ErrImagePull and ImagePullBackOff, caused by a wrong tag or missing registry credentials.

Probes: three questions, three different answers

Kubernetes asks three questions about a container, and they should not be confused.

A readiness probe asks whether the pod should receive traffic. A failing readiness probe removes the pod from the Service's endpoints but does not restart it. A liveness probe asks whether the process is stuck and should be restarted; a failing one makes the kubelet kill the container. A startup probe (not used above) gives a slow-starting container time before the other two begin.

The service here answers both with the same cheap route, which is fine for a program with no dependencies. Once a service depends on a database, the two should differ, and the rule is: readiness may depend on dependencies, liveness must not. If liveness fails whenever the database is down, a database outage makes Kubernetes restart every pod in a loop, which adds load to a database that is already struggling and turns a dependency problem into a full outage. Readiness, by contrast, should report that the pod cannot serve right now, so the pod stops getting traffic until it can. The detailed pattern is in health, readiness and liveness checks.

Choose timings from what they cost. periodSeconds: 2 on readiness means a pod becomes eligible for traffic within about two seconds of starting. failureThreshold: 3 on liveness with a ten-second period means a stuck process is restarted after about thirty seconds, which is slow enough not to kill a pod during a brief GC-like pause or a burst of load. A probe that is too aggressive is a source of outages of its own.

Configuration errors during a rollout

The configuration experiment is the most useful one in this guide. Change the Deployment to read a key that does not exist in the ConfigMap, and watch what happens to a running service:

$ kubectl patch deploy svc --type=json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/env/0/valueFrom/configMapKeyRef/key","value":"nope"}]'
$ kubectl get pods
NAME                   READY   STATUS                       RESTARTS   AGE
svc-6bb9f8cdd5-rqzcq   0/1     CreateContainerConfigError   0          12s
svc-789cd9b96-2v487    1/1     Running                      0          33s
svc-789cd9b96-jvqbt    1/1     Running                      0          34s
$ kubectl describe pod -l app=svc | grep couldn
  Warning  Failed  kubelet  Error: couldn't find key nope in ConfigMap rh-lab/svc-config

The new pod never became ready, and the rolling update stopped there. The two old pods kept running and kept serving, because a Deployment only removes old pods as new ones become ready. A bad change was contained to one pod instead of taking down the service. kubectl rollout undo deploy/svc put things back and the rollout completed.

The second case is a value that exists but is invalid. Set GREETING to a blank string, and the program's own startup validation catches it:

$ kubectl get pods
NAME                   READY   STATUS             RESTARTS      AGE
svc-789cd9b96-2v487    1/1     Running            0             103s
svc-789cd9b96-jvqbt    1/1     Running            0             104s
svc-7f4667dd48-fnswr   0/1     CrashLoopBackOff   1 (20s ago)   21s
$ kubectl logs svc-7f4667dd48-fnswr
config error: GREETING must not be empty
$ kubectl get pod svc-7f4667dd48-fnswr -o jsonpath='exit={.status.containerStatuses[0].lastState.terminated.exitCode}'
exit=78

This is where the choices in the Rust code pay off. The pod's status says CrashLoopBackOff, the logs hold one clear line, and the exit code is 78, which separates a configuration mistake from a crash. Again the old pods stayed up. Compare to a service that reads its configuration lazily: it would pass its readiness probe, receive traffic, and fail on the first request that needed the missing value, after the rollout had already replaced the working pods. Validating at startup turns that into a failed rollout that never reached users.

Resources: requests, limits, and what each one does

Requests and limits do different jobs. The request is what the scheduler uses to place the pod on a node, and what the pod is guaranteed. The limit is a ceiling enforced at runtime. They behave differently for CPU and memory.

Exceeding a CPU limit throttles the process: it is slowed, not killed. Exceeding a memory limit kills it with the out-of-memory killer, and the container restarts with the reason OOMKilled and exit code 137. For that reason this manifest sets a memory limit but no CPU limit. A CPU limit on a latency-sensitive service can throttle it during bursts even when the node has spare CPU, and many teams set CPU requests only. Memory is the reverse: set a limit, because an unbounded leak would otherwise take the node with it, and set the request equal to the limit if you want the pod to be the last to be evicted under pressure.

Rust makes sizing easy because memory use is flat and predictable. A service that uses 8 MiB at rest and peaks at 30 MiB under load can run with a request of 16 MiB and a limit of 64 MiB, as here. Measure the peak under realistic load, add headroom for the allocator and for spikes, and watch for OOMKilled in kubectl describe pod as the sign the limit is too low. Be aware of the arithmetic of unbounded work: a service that buffers request bodies or collects a whole query result into a Vec can exceed any limit on a large enough input, and the fix belongs in the code, with a body size limit and streaming, not in a bigger number.

Graceful shutdown on pod deletion

When a pod is deleted, whether by a rollout, a scale-down or a node drain, Kubernetes sets its state to Terminating, sends SIGTERM to the container, and gives it terminationGracePeriodSeconds (30 by default, set explicitly above) before SIGKILL. The test is to start a slow request on a specific pod and delete that pod while the request is in flight:

$ kubectl port-forward pod/svc-659b886675-gklfj 18091:8080 &
$ curl -s -m 20 -w " http=%{http_code}\n" localhost:18091/slow &
$ time kubectl delete pod svc-659b886675-gklfj
pod "svc-659b886675-gklfj" deleted from rh-lab namespace
real 2.27
finished the slow request http=200

The in-flight request completed with a 200, and the delete returned after 2.3 seconds, which is the time the program needed to finish the request and exit, far under the 30-second allowance. Without the SIGTERM handler, the program running as PID 1 would have ignored the signal and the kubelet would have waited the full grace period and killed it, cutting the request. The mechanics, including why PID 1 ignores a signal with no handler, are in the Fly guide, where the same image was measured with docker stop.

Rolling updates and the race at the start of shutdown

A rolling update is the case where graceful shutdown matters most, and where one subtle race lives. When a pod is deleted, Kubernetes does two things at about the same time: it sends SIGTERM, and it removes the pod from the Service's endpoints. Those propagate independently, and the endpoint removal can reach other components a moment after the signal reaches the process. If the process stops accepting connections immediately on SIGTERM, requests that were routed to it a moment ago can arrive at a closed port.

To see it, a loop of 400 requests, 50 ms apart, ran on the node against the Service's ClusterIP while the Deployment was restarted with kubectl rollout restart. The count of HTTP status codes:

baseline, run 1:  398 x 200, 2 x 000
baseline, run 2:  399 x 200, 1 x 000

A 000 is a request where curl got no response: a failed connection. So 3 of 800 requests failed during two rollouts, with a graceful shutdown handler already in place. The mitigation is a short preStop sleep, which delays SIGTERM so the endpoint removal has time to propagate. The image is scratch, which has no sleep binary to run, but Kubernetes 1.30 and later has a native sleep action, so no executable is needed:

          lifecycle:
            preStop:
              sleep:
                seconds: 5

The same test with the preStop sleep:

with preStop sleep 5, run 1:  399 x 200, 1 x 000
with preStop sleep 5, run 2:  400 x 200
with preStop sleep 5, run 3:  400 x 200
with preStop sleep 5, run 4:  400 x 200

Three of four rollouts were clean after the change, against zero of two before, which is consistent with the race but is a small sample on a single-node cluster, so treat it as suggestive and not as proof. The one failed request in the first run with the sleep shows the sleep does not remove every possible failure, and the loop was driven from the node itself, a different path from a cloud load balancer, which has its own deregistration delay. The reliable way to evaluate this in your own cluster is the same experiment: a request loop through the real path, several rollouts, and a count of failures. Keep the preStop delay short (a few seconds) and terminationGracePeriodSeconds longer than the sleep plus your slowest request, since the grace period clock includes the preStop time.

A ConfigMap change does not reach a running pod

Editing configuration is a common source of confusion, because the edit succeeds and nothing changes. Environment variables are read when a container starts, so a running process never sees a new value. The test is to change the ConfigMap and ask the running service:

$ curl -s localhost:18095/
hello from kubernetes
$ kubectl patch configmap svc-config --type=merge -p '{"data":{"greeting":"changed"}}'
configmap/svc-config patched
$ curl -s localhost:18095/
hello from kubernetes
$ kubectl rollout restart deploy/svc
deployment "svc" successfully rolled out
$ curl -s localhost:18095/
changed

Five seconds after the patch the service still answered with the old value, and only the rollout brought the new one in. That is the right behavior for a program that validates its configuration once at startup, and it means a configuration change goes through the same rolling update, probes and rollback as a code change. The practical rule is to roll the Deployment after changing a ConfigMap or Secret that it uses as environment variables. Tools such as Helm and Kustomize automate this by hashing the ConfigMap into a pod annotation, so a change in the data changes the pod template and triggers the rollout on its own. (Files mounted from a ConfigMap as volumes are refreshed in place after a delay, but a program that read the file at startup still holds the old contents unless it watches for changes.)

For credentials, use a Secret instead of a ConfigMap, with the same environment-variable mechanism. A Secret is base64-encoded, not encrypted, in the cluster's data store by default, so enable encryption at rest or use an external secret manager for anything sensitive, and restrict who can read Secrets in the namespace with RBAC. Never put a credential in the image or in the manifest committed to version control.

Other settings that decide whether a rollout is safe

Three Deployment settings shape a rollout. maxUnavailable and maxSurge control how many pods can be missing or extra at once; for a two-replica service, maxUnavailable: 0 with maxSurge: 1 guarantees full capacity throughout, at the cost of needing room for one extra pod. minReadySeconds makes a new pod stay ready for a number of seconds before it counts, which catches a service that passes its probe and then crashes. And a PodDisruptionBudget limits how many pods a voluntary disruption, such as a node drain, may take at once; without one, draining a node can evict all replicas of a service that happen to sit on it. None of these was exercised on this single-node cluster, so test them on a cluster with several nodes before trusting them.

What was not tested

This was a single-node local cluster. Ingress controllers, TLS, horizontal pod autoscaling, network policies, secrets management, node failure and multi-zone scheduling were not exercised. The readiness and shutdown behavior shown here does not depend on any of them, but the behavior of a load balancer in front of the cluster, which adds its own health checks and its own delay in removing a pod, can change the shutdown timing, as noted above. Read your provider's documentation on connection draining, and measure with the real path.

A pre-deploy checklist

• The container binds 0.0.0.0 and reads everything from environment variables.
• A bad setting makes the process exit at once with a distinct code and one log line.
• Readiness may check dependencies; liveness checks only that the process can respond.
• A memory limit is set from measured peak use; CPU limits are a deliberate choice.
• The process handles SIGTERM, drains in-flight requests, and exits 0, within terminationGracePeriodSeconds.
• A short preStop delay covers endpoint removal, and a rollout test under load shows no failures.
• At least two replicas run, with a PodDisruptionBudget, so a drain does not take the service down.
• Every kubectl command names its context, and the one aimed at production is typed deliberately.

When Kubernetes is more than you need

Kubernetes earns its complexity when you run many services, need to place workloads across machines, and have people to operate the cluster. For a single small service, a platform such as Fly.io or a single VM with systemd does the same supervising with far less to manage, and for event-driven work AWS Lambda removes the servers altogether. The advantage of writing the service as shown here is that it runs unchanged on all of them: a port, environment variables, a cheap health route and a signal handler are the whole contract.