Testing Graceful Shutdown and Signal Handling in Rust Apps
Make shutdown testable by taking a shutdown future instead of listening for signals inside the server. Assert that in-flight requests finish, new ones are refused and the process exits within a deadline.
Shutdown code runs rarely, fails silently and matters on every deploy. Requests that die mid-flight look like random 502s after a release. Because it is hard to trigger from a test, it is often untested. The fix is a design change: make the trigger a parameter. This post shows the structure and the assertions. It compiles on Rust 1.90 with edition 2024, Axum 0.8 and Tokio 1.47.
Inject the shutdown trigger
use std::future::Future;
use axum::{routing::get, Router};
use tokio::net::TcpListener;
pub fn app() -> Router {
Router::new()
.route("/fast", get(|| async { "ok" }))
.route("/slow", get(|| async {
tokio::time::sleep(std::time::Duration::from_millis(300)).await;
"finished"
}))
}
pub async fn serve(listener: TcpListener, shutdown: impl Future<Output = ()> + Send + 'static) -> std::io::Result<()> {
axum::serve(listener, app()).with_graceful_shutdown(shutdown).await
}
// production wiring
pub async fn ctrl_c() {
tokio::signal::ctrl_c().await.expect("install handler");
}serve knows nothing about signals; main passes ctrl_c() (and a SIGTERM equivalent on Unix, see Axum graceful shutdown), and tests pass a channel receiver.
The behavior test
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
use tokio::{sync::oneshot, time::timeout};
async fn start() -> (String, oneshot::Sender<()>, tokio::task::JoinHandle<std::io::Result<()>>) {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let url = format!("http://{}", listener.local_addr().unwrap());
let (tx, rx) = oneshot::channel::<()>();
let handle = tokio::spawn(serve(listener, async move { let _ = rx.await; }));
(url, tx, handle)
}
#[tokio::test]
async fn in_flight_request_completes_after_shutdown_starts() {
let (url, shutdown, server) = start().await;
let client = reqwest::Client::new();
let slow = tokio::spawn({
let (client, url) = (client.clone(), url.clone());
async move { client.get(format!("{url}/slow")).send().await.unwrap().text().await.unwrap() }
});
tokio::time::sleep(Duration::from_millis(50)).await; // let the request reach the handler
shutdown.send(()).unwrap();
assert_eq!(slow.await.unwrap(), "finished"); // drained, not cut off
timeout(Duration::from_secs(2), server).await.expect("server exits").unwrap().unwrap();
}
#[tokio::test]
async fn new_connections_are_refused_after_shutdown() {
let (url, shutdown, server) = start().await;
shutdown.send(()).unwrap();
timeout(Duration::from_secs(2), server).await.expect("server exits").unwrap().unwrap();
let result = reqwest::Client::new().get(format!("{url}/fast")).send().await;
assert!(result.is_err(), "listener should be closed");
}
}The first test starts a slow request, triggers shutdown while it is in flight, and asserts two things: the response still arrives, and the server then exits. The 50 ms sleep ensures the request has reached the handler before shutdown starts; on a loaded CI machine increase it, or better, have the handler signal when it has started with a channel. The second test asserts that after shutdown completes, the port no longer accepts connections. Both use timeout so a server that never stops fails the test instead of hanging the job.
Testing the drain deadline
#[tokio::test(start_paused = true)]
async fn forced_exit_after_the_deadline() {
let result = tokio::time::timeout(
std::time::Duration::from_secs(30),
drain(std::time::Duration::from_secs(10)),
)
.await;
assert!(result.is_ok());
}
async fn drain(deadline: std::time::Duration) {
// wait for in-flight work, but never longer than the deadline
let work = tokio::time::sleep(std::time::Duration::from_secs(3600)); // a stuck job
let _ = tokio::time::timeout(deadline, work).await;
}With start_paused (Tokio's test-util feature) the clock jumps when all tasks are idle, so a ten-second deadline costs no real time (testing async with paused time). The assertion is that a stuck job does not block shutdown forever.
The one process-level test
#[cfg(unix)]
#[test]
fn exits_cleanly_on_sigterm() {
use std::{process::{Command, Stdio}, time::{Duration, Instant}};
let mut child = Command::new(env!("CARGO_BIN_EXE_myserver"))
.stdout(Stdio::null())
.spawn()
.unwrap();
std::thread::sleep(Duration::from_millis(500)); // wait for startup; poll a health endpoint for a robust version
let pid = child.id().to_string();
assert!(Command::new("kill").args(["-TERM", &pid]).status().unwrap().success());
let start = Instant::now();
let status = loop {
if let Some(s) = child.try_wait().unwrap() { break s; }
assert!(start.elapsed() < Duration::from_secs(5), "did not exit after SIGTERM");
std::thread::sleep(Duration::from_millis(20));
};
assert!(status.success());
}This covers what the injected-trigger tests cannot: that the binary really installs a SIGTERM handler and exits with status 0. Keep it as a single test, since it spawns a process and depends on timing. Cargo sets CARGO_BIN_EXE_<name> for integration tests so you can find the built binary. Poll a readiness endpoint instead of sleeping if startup time varies (health checks). What shutdown should do in the first place is in Tokio graceful shutdown.
