Debugging Async Backtraces and Task Dumps in Tokio

The worst production bug in an async service is the silent hang: requests stop completing, CPU is idle and logs are quiet. You attach a debugger or take a core dump and see ten threads parked in the runtime's idle loop. The tasks you care about are not on any thread; they are data on the heap, waiting. Tokio can print them. This post shows how. It was written against Rust 1.90, Tokio 1.47 on Linux x86-64, and the exact flags may change with versions, so check Tokio's documentation for Handle::dump.

Why ordinary tools fail

A future that is awaiting is not running, so it is not on a stack. Its state lives in the future object. A thread backtrace can only show what is currently executing, which for an idle runtime is the worker parking. To see "task 4,112 is waiting on mutex.lock() called from handle_checkout called from serve", the runtime has to walk the tasks' futures and reconstruct the chain of awaits. That is an async backtrace.

Enable it

# Cargo.toml
[dependencies]
tokio = { version = "1.47", features = ["full"] }

RUSTFLAGS="--cfg tokio_unstable --cfg tokio_taskdump" cargo build --release

Two flags are required because the feature is still unstable, and it is supported on Linux targets (x86, x86-64, aarch64). It works best with frame pointers or debug info available so the trace names are meaningful: set debug = 1 in the profile and consider -C force-frame-pointers=yes. Use a separate target directory for builds with these flags, since they rebuild every crate.

Capture a dump

use std::time::Duration;
use tokio::runtime::Handle;

async fn dump_tasks() {
    let handle = Handle::current();
    match tokio::time::timeout(Duration::from_secs(2), handle.dump()).await {
        Ok(dump) => {
            for (i, task) in dump.tasks().iter().enumerate() {
                let trace = task.trace();
                println!("TASK {i}:\n{trace}\n");
            }
        }
        Err(_) => eprintln!("task dump timed out: the runtime is blocked, so dump the threads instead"),
    }
}

The timeout matters: if a worker is blocked in synchronous code, the dump cannot finish, and that fact is itself the diagnosis. In that case, a thread backtrace (gdb -p PID -batch -ex "thread apply all bt", or eu-stack) points at the blocking call (blocking in async code).

Trigger it on demand

use tokio::signal::unix::{signal, SignalKind};

pub fn spawn_dump_on_signal() {
    tokio::spawn(async {
        let mut usr1 = signal(SignalKind::user_defined1()).expect("SIGUSR1 handler");
        while usr1.recv().await.is_some() {
            dump_tasks().await;
        }
    });
}

kill -USR1 $(pgrep myservice)

A signal needs no network exposure, which is convenient on a locked-down host. An admin-only HTTP endpoint is the alternative where you can reach the service but not its host (protect it with authentication and keep it off the public listener). Write the dump to stderr or a file; a large service can produce megabytes.

Reading the result

TASK 17:
╼ myservice::checkout::handle_checkout::{{closure}} at src/checkout.rs:88:42
  └╼ tokio::sync::mutex::Mutex<T>::lock::{{closure}} at tokio/src/sync/mutex.rs:...
     └╼ tokio::sync::batch_semaphore::Acquire::poll ...

The shape is illustrative; paths and line numbers will be yours. Each task's trace is a tree from the task root to the leaf future it is waiting on. What to look for:

• Many tasks with the same leaf (the same lock, the same channel receive, the same connection pool acquire): a bottleneck or a deadlock. Count them: 400 tasks waiting on the pool means the pool is exhausted.
• Tasks waiting on I/O with no timeout: a read or request that will never complete (why tasks never finish).
• A lock held across an await: one task shows an owned guard and waits on something slow, while others wait for the lock (mutex choices).
• A task waiting on a channel whose sender is alive but idle: a worker loop that never ends (mpsc channels).

Limits and alternatives

Decide before an incident which of these you can use in production, and rehearse it once: a first dump taken during an outage, with an unfamiliar build flag, goes badly. A shorter introduction to finding the sticking point is the deadlock debugging guide.