Incident Response for Rust Services a Short Runbook
A one-page runbook for when a Rust service misbehaves: stabilize first, check recent changes, read the four golden signals, find the failing dependency, and use the Rust-specific evidence: panics, memory, executor stalls.
An incident is a bad time to design your process. A short runbook written in advance turns panic into a list. This one is for a Rust service running in containers behind a load balancer with a SQL database; adapt the commands to your platform. It assumes you already have logs, metrics and a way to roll back.
0. Stabilize before you diagnose
- Acknowledge and open a channel. One place for communication, one person coordinating.
- Roll back the most recent change if one is plausibly related: a deploy in the last hour, a config or flag change, a migration. Rolling back is cheap and informative: if it fixes things, you know the cause class.
- Reduce load or add capacity: scale out, shed non-essential traffic, disable expensive features through a flag, enable maintenance mode for a feature, raise rate limiting.
- Tell people: status page, support, stakeholders, with what is known and the next update time.
1. What changed?
| Source | Check |
|---|---|
| Deploys | Release history; compare the incident start with deployment times |
| Config and flags | Change log of the config service, feature flags, environment variables |
| Traffic | A spike, a new client, a retry storm, a crawler |
| Dependencies | Database, cache, queue, third-party APIs: their own status pages and metrics |
| Infrastructure | Node failures, autoscaling, network, DNS, certificates expiring |
| Time | A cron job, a daylight saving change, a leap second, a token that expired at midnight |
2. The four golden signals
- Latency: p50 and p99 per route (tail latency). A rise in p99 alone points to queueing or a slow dependency.
- Errors: the rate and the kind: 5xx, timeouts, panics, 4xx spikes from a bad client.
- Traffic: requests per second against normal.
- Saturation: CPU, memory, connection pool wait, file descriptors, executor busy ratio, queue depth.
Saturation tells you which resource ran out first, and a resource running out explains most "slow, then failing" incidents.
3. Rust-specific evidence
Panics and restarts
kubectl logs deploy/api --previous | grep -i -B2 -A12 panicked
kubectl get pods -l app=api # restart counts
kubectl describe pod api-xyz | grep -A5 "Last State"A panic in a request handler kills the task (and in Axum or Actix is caught per request, returning 500); a panic in a spawned background task silently stops that task; panic = "abort" kills the process. The message and backtrace say where. If there is no backtrace, set RUST_BACKTRACE=1 in the deployment for next time. "Last State: OOMKilled" means the container hit its memory limit (measuring memory), exit code 137 means killed by a signal, 101 is a panic exit.
High latency with low CPU
A classic async symptom: requests pile up, CPU is idle. Suspects: a blocked executor (blocking calls on async threads), an exhausted connection pool or a downstream that stopped responding with no timeout. Check pool wait metrics, Tokio busy ratio and scheduling delay (Tokio metrics), then take a task dump or tokio-console view if you built for it (task dumps, tokio-console). A thread backtrace (eu-stack -p PID) shows a blocked worker.
Resource exhaustion
ls /proc/$(pgrep api)/fd | wc -l # file descriptors vs ulimit
grep -E "VmRSS|Threads" /proc/$(pgrep api)/status # memory and threads
ss -s # socket summary, many in CLOSE_WAIT?Descriptors near the limit (too many open files), memory climbing steadily (a cache without eviction or an unbounded channel), thousands of sockets in CLOSE_WAIT (connections never dropped). Cross-check with database connections in use against max_connections (connection limits).
Crash loops
Read the first log lines after start. Typical causes: a missing or malformed environment variable, a secret that did not mount, a migration that fails, a dependency that is down and the service exits instead of retrying, a port already in use, a changed file permission (permission denied). Back off restarts to avoid hammering a recovering dependency.
4. Decide and act
- Form one hypothesis from the evidence, say it out loud in the channel and test it with one action.
- Prefer reversible actions: scale, rollback, flag off, raise a limit. Avoid risky fixes in production, such as hand-editing data without a second pair of eyes.
- Capture evidence before restarting: logs, metrics screenshots, a dump.
- Confirm recovery with the same metrics that showed the failure, and watch for a relapse before declaring it over.
5. After: a blameless review
Within a few days, write a short review: what happened (timeline), the impact, the cause, what went well, what was slow, and a small number of concrete follow-ups with owners. Typical ones: a missing alert, a missing timeout, a dashboard to build, a runbook step to automate, a test that would have caught it. Fix the process, not the person. Keep the runbook in the repository and update it after each incident (release checklist, Rust on Kubernetes basics).
Prepare ahead
- Dashboards for the golden signals and saturation, per service.
- Alerts on symptoms users feel (error rate, latency objectives) and on exhaustion (80 percent of a limit).
- One-command rollback and a tested way to disable features.
- Debug builds or flags you can enable safely in production: backtraces on, task dumps behind an admin signal.
- On-call practice: run a game day where you break something on purpose and follow this page.
