Debugging a Hung Async Rust Program Step by Step Explained

A hung async service is hard to read because nothing is running. The stack of a stuck thread shows the runtime's idle loop, not your code, and the process looks healthy to every tool that only checks whether it is alive. The method below finds the cause in a fixed order, cheapest check first, with the commands to use and the output I got from each one. Every program here was compiled and run on Rust 1.99 (edition 2024) with Tokio 1.47.1, on an Apple M1 Max under macOS. The Linux commands (eu-stack, top -H) behave the same way; only the tool names change.

By "hung" I mean a process that is up, accepts no new work or finishes none, and prints no error. A process that crashed, or one that is slow but moving, is a different problem. The first step exists to tell those apart before you spend an afternoon on the wrong one.

Step 1: is it busy or idle?

top -H -p "$(pgrep -f my-service)"

This lists the threads of the process, one row each, with their own CPU use. The shape of the answer splits the whole problem in two.

If one thread sits near 100 percent, some task never yields. That is a loop with no .await in it, a spin on try_recv, or a future that wakes itself on every poll and so never lets anything else run. The scheduler cannot take the thread away from a task in the middle of a poll, because Tokio's scheduling is cooperative: a task gives the thread back only at an await point that actually returns Pending. Jump to step 2 and look at where that thread is.

If every thread is idle, tasks are waiting on something that never arrives, or workers are blocked in the kernel on a call that never returns. Both look the same from top, which is why step 2 exists. The idle case is also the common one. A busy hang gets noticed by the fan; an idle hang gets noticed by a customer.

One more reading to take now: the thread count. A Tokio multi thread runtime starts one worker per core by default, plus a pool for spawn_blocking that grows up to 512 threads. A process with hundreds of threads all parked is saturating the blocking pool, which usually means something is waiting on a lock or a socket inside spawn_blocking with no deadline.

Step 2: look at the native threads

gdb -p "$(pgrep -f my-service)" -batch -ex "thread apply all bt" | head -80

You can use lldb on macOS or eu-stack -p on Linux for the same dump. The output is long, so read it for names rather than for every frame. Worker threads named tokio-runtime-worker should be parked in the scheduler: the innermost frames are things like park, epoll_wait or kevent. That is a healthy idle worker, and it tells you the runtime itself is fine.

A worker inside read, std::fs, std::thread::sleep, a synchronous database client or Mutex::lock is a blocked runtime thread. Each worker that is stuck takes capacity away from every task in the process. When all of them are stuck, nothing else runs at all, including timers, so even the timeouts you add later cannot fire (the guide to blocking the Tokio runtime covers why). The fix is spawn_blocking or an async API for the call.

The failure is easy to reproduce, and doing so shows what the symptom looks like from outside. The program below starts a multi thread runtime with two workers and a task that bumps a counter every 100 milliseconds. A watchdog thread, created with std::thread::spawn so that it lives outside the runtime it is watching, reads that counter once a second. After a second and a half the main task blocks both workers with std::thread::sleep.

use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use std::time::Duration;

#[tokio::main(flavor = "multi_thread", worker_threads = 2)]
async fn main() {
    let beat = Arc::new(AtomicU64::new(0));

    // A task that proves the runtime is still turning.
    let b = Arc::clone(&beat);
    tokio::spawn(async move {
        loop {
            b.fetch_add(1, Ordering::Relaxed);
            tokio::time::sleep(Duration::from_millis(100)).await;
        }
    });

    // The watchdog lives on a plain thread, outside the runtime it watches.
    let b = Arc::clone(&beat);
    std::thread::spawn(move || {
        let mut last = 0;
        loop {
            std::thread::sleep(Duration::from_secs(1));
            let now = b.load(Ordering::Relaxed);
            println!("heartbeat {now}");
            if now == last {
                println!("runtime stalled: no heartbeat for 1s");
                std::process::exit(2);
            }
            last = now;
        }
    });

    tokio::time::sleep(Duration::from_millis(1500)).await;
    // Two workers, two blocking calls: every worker is now stuck.
    let a = tokio::spawn(async { std::thread::sleep(Duration::from_secs(30)) });
    let c = tokio::spawn(async { std::thread::sleep(Duration::from_secs(30)) });
    let _ = tokio::join!(a, c);
}

The heartbeat task is the canary: if it runs, the runtime is turning. The watchdog is the part worth copying into a service. It has to be a plain OS thread, because a watchdog implemented as another async task shares the fate of the thing it watches. Running it:

heartbeat 10
heartbeat 15
heartbeat 15
runtime stalled: no heartbeat for 1s

The first line shows about ten beats in the first second. The second shows five more in the half second before the blocking calls start. The third shows the counter has not moved, and the watchdog exits the process with code 2. In a real service you would log the thread dump here instead of exiting, or trigger a core dump, and let your supervisor restart the process. The heartbeat is cheap enough to leave on permanently, and it converts "the service is up but dead" into a signal an alert can fire on.

The same program explains why the usual advice, "just add a timeout", does not always work. A timeout is a timer, and timers are driven by the runtime. If the runtime's threads are monopolised, the timer never gets polled. The next program runs on a single thread, spawns a task that spins in a loop with no await, and wraps a 100 millisecond sleep in a one second timeout:

use std::time::Duration;

#[tokio::main(flavor = "current_thread")]
async fn main() {
    let spin = tokio::spawn(async {
        let mut n = 0u64;
        loop {
            n = n.wrapping_add(1);
            if n == u64::MAX {
                break;
            }
        }
    });
    let ticker = tokio::spawn(async {
        tokio::time::sleep(Duration::from_millis(100)).await;
        println!("ticker ran");
    });
    let r = tokio::time::timeout(Duration::from_secs(1), ticker).await;
    println!("{r:?}");
    spin.abort();
}

I ran it and killed it after six seconds. It printed nothing. Neither "ticker ran" nor the timeout result appeared, because the one worker thread never got back to the scheduler to poll either of them. The lesson is narrow but important: timeouts localise a hang that is made of waiting, and they are blind to a hang that is made of running or blocking. Use the thread dump for the second kind and the timeout for the first.

Step 3: see the tasks

When the threads are idle and healthy, the question becomes which tasks are parked and on what. Tokio can export that, through the tokio-console tool and the console-subscriber crate. It needs one unstable cfg flag, because the instrumentation lives behind tokio_unstable.

[dependencies]
tokio = { version = "=1.47.1", features = ["full", "tracing"] }
console-subscriber = "=0.4.1"
tracing = "0.1"

fn main() {
    console_subscriber::init();
    tracing::info!("console subscriber installed");
}

RUSTFLAGS="--cfg tokio_unstable" cargo run

The cfg flag is easy to forget, and the failure is loud rather than silent. Without it, console_subscriber::init() panics at startup:

thread 'main' panicked at console-subscriber-0.4.1/src/lib.rs:262:9:
task tracing requires Tokio to be built with RUSTFLAGS="--cfg tokio_unstable"!

Changing RUSTFLAGS rebuilds every dependency, so set it in .cargo/config.toml for the profile you use for diagnostics instead of toggling it by hand. With the subscriber installed, connect the tokio-console binary from another terminal. It lists every task with its poll count, busy time, idle time and the resource it last waited on. Sort by idle time. A task with a very long idle time and a poll count that stopped growing is a stuck task, and the console shows the lock, channel or semaphore it is waiting on (finding stuck tasks with tokio-console walks through the screens). Name your tasks with tokio::task::Builder::new().name("ingest") so the list reads as your architecture instead of a column of spawn.rs:168.

If you cannot restart the process with a flag, Tokio can also print a trace of every task in a running process, an async backtrace per task, through its task dump support (task dumps). That one needs a build with the cfg flag too, but not a tracing subscriber.

Step 4: put timeouts around suspects

When you have a short list of awaits that might never complete, wrapping them in tokio::time::timeout converts a silent hang into an error with a label. This is the cheapest step that works on a live service, and it needs no tooling.

use std::time::Duration;
use tokio::sync::mpsc;
use tokio::time::timeout;

async fn guarded<F: std::future::Future>(label: &str, fut: F) -> Option<F::Output> {
    match timeout(Duration::from_secs(2), fut).await {
        Ok(v) => Some(v),
        Err(_) => {
            eprintln!("{label} did not complete in 2s");
            None
        }
    }
}

#[tokio::main]
async fn main() {
    let (tx, mut rx) = mpsc::channel::<u32>(8);
    let worker_tx = tx.clone();
    tokio::spawn(async move {
        for i in 0..3 {
            worker_tx.send(i).await.unwrap();
        }
    });

    let drain = async {
        let mut n = 0;
        while let Some(v) = rx.recv().await {
            println!("got {v}");
            n += 1;
        }
        n
    };
    // `tx` is still alive in this scope, so the loop never sees None.
    let r = guarded("drain loop", drain).await;
    println!("result: {r:?}");
    drop(tx);
}

The helper takes a label so that the log line names the call site. It returns an Option so the caller decides whether a timeout is fatal. The example then creates a channel, clones the sender into a worker that sends three numbers and exits, and drains the receiver in a loop. That looks right and it is not:

got 0
got 1
got 2
drain loop did not complete in 2s
result: None

All three values arrive, and then the loop waits for a fourth that will never come. recv() returns None only when every sender has been dropped, and the original tx is still alive in main. The worker's clone is gone; the one I kept is not. The label told me which await was stuck, and the cause is visible in the structure: the receiver loop and a live sender share a scope. The fix is to drop the original after cloning, before waiting.

use tokio::sync::mpsc;

#[tokio::main]
async fn main() {
    let (tx, mut rx) = mpsc::channel::<u32>(8);
    let worker_tx = tx.clone();
    drop(tx);
    tokio::spawn(async move {
        for i in 0..3 {
            worker_tx.send(i).await.unwrap();
        }
    });
    let mut n = 0;
    while let Some(v) = rx.recv().await {
        println!("got {v}");
        n += 1;
    }
    println!("drained {n}");
}

got 0
got 1
got 2
drained 3

The loop now ends on its own after the third message, since the only sender left is the worker's, which is dropped when the task finishes. This is the most common idle hang I see in channel based pipelines: a pipeline stage that holds a sender to its own input, usually because the sender was cloned into a handler struct that outlives the loop. The same shape appears with JoinSet, where a loop over join_next() ends only when the set is empty.

Pick the timeout carefully. A timeout around a legitimate long wait, such as a worker loop that is supposed to sit on a channel for days, will fire constantly. Put timeouts around operations that have an expected duration: a lock acquisition, a request, a semaphore permit, a shutdown step.

Step 5: a lock held across an await

The next most common cause is a std::sync::Mutex guard that lives across an .await. The compiler catches the version of this that happens inside tokio::spawn, which is the reason you often never see it. This is the program that triggers it:

use std::sync::{Arc, Mutex};
use std::time::Duration;

#[tokio::main]
async fn main() {
    let state = Arc::new(Mutex::new(0u32));
    let s = Arc::clone(&state);
    tokio::spawn(async move {
        let mut guard = s.lock().unwrap();
        *guard += 1;
        tokio::time::sleep(Duration::from_millis(10)).await;
        *guard += 1;
    });
}

And this is what rustc 1.99 says:

error: future cannot be sent between threads safely
   --> src/bin/h2_fail.rs:8:5
    |
  8 | /     tokio::spawn(async move {
  9 | |         let mut guard = s.lock().unwrap();
 10 | |         *guard += 1;
 11 | |         tokio::time::sleep(Duration::from_millis(10)).await;
 12 | |         *guard += 1;
 13 | |     });
    | |______^ future created by async block is not `Send`
    |
    = help: within `{async block@src/bin/h2_fail.rs:8:18: 8:28}`, the trait `Send` is not implemented for `std::sync::MutexGuard<'_, u32>`
note: future is not `Send` as this value is used across an await
   --> src/bin/h2_fail.rs:11:55
    |
  9 |         let mut guard = s.lock().unwrap();
    |             --------- has type `std::sync::MutexGuard<'_, u32>` which is not `Send`
 10 |         *guard += 1;
 11 |         tokio::time::sleep(Duration::from_millis(10)).await;
    |                                                       ^^^^^ await occurs here, with `mut guard` maybe used later

The message is exact about what is wrong: mut guard is still alive at the await on line 11, and a std guard is not Send, so the whole future cannot move between worker threads. The reason the guard is not Send is the hang itself. A std mutex must be unlocked by the thread that locked it on some platforms, and holding it while the task is parked means the lock stays taken for as long as the task sleeps, while other tasks that want it block their own threads.

The compiler only protects you when the future has to be Send. On a current_thread runtime, in block_on, inside join! or on a LocalSet, the same code compiles, and it deadlocks. This program joins two futures on one thread. The first takes the lock and awaits; the second tries to take the same lock:

use std::sync::{Arc, Mutex};
use std::time::Duration;

#[tokio::main(flavor = "current_thread")]
async fn main() {
    std::thread::spawn(|| {
        std::thread::sleep(Duration::from_secs(2));
        println!("watchdog: still stuck after 2s");
        std::process::exit(2);
    });

    let state = Arc::new(Mutex::new(0u32));
    let a = Arc::clone(&state);
    let b = Arc::clone(&state);

    let holder = async move {
        let mut guard = a.lock().unwrap();
        *guard += 1;
        println!("holder: took the lock, now awaiting");
        tokio::time::sleep(Duration::from_millis(10)).await;
        *guard += 1;
    };
    let waiter = async move {
        println!("waiter: asking for the lock");
        let guard = b.lock().unwrap();
        println!("waiter: got it, value {}", *guard);
    };
    tokio::join!(holder, waiter);
}

holder: took the lock, now awaiting
waiter: asking for the lock
watchdog: still stuck after 2s

The holder printed, went to sleep for 10 milliseconds, and the waiter ran. The waiter called lock(), which blocks the only thread, so the holder was never polled again and could never release the lock. The 10 millisecond sleep would have finished instantly if anything were around to notice. The watchdog thread, again outside the runtime, is what caught it. The thread dump for this case shows the one worker inside Mutex::lock, which is why step 2 comes before step 3.

There are two fixes, and the better one is not the one people reach for first. The first is to end the guard before the await, using a block so the borrow checker drops it:

use std::sync::{Arc, Mutex};
use std::time::Duration;

#[tokio::main]
async fn main() {
    let state = Arc::new(Mutex::new(0u32));
    let s = Arc::clone(&state);
    let task = tokio::spawn(async move {
        {
            let mut guard = s.lock().unwrap();
            *guard += 1;
        }
        tokio::time::sleep(Duration::from_millis(10)).await;
        *s.lock().unwrap() += 1;
    });
    task.await.unwrap();
    println!("value: {}", state.lock().unwrap());
}

value: 2

The second is tokio::sync::Mutex, whose guard may be held across an await because waiting on it yields instead of blocking. It is slower for the common case of a short critical section, and it invites the habit of holding a lock while doing I/O. Prefer the first form, and keep the std mutex, unless the critical section genuinely has to include an await (which mutex to pick, and the specific hazard in mutex guard across await).

Step 6: the rest of the causes

If steps 1 to 5 found nothing, the cause is one of a short list, and each has a way to confirm it.

1. A semaphore permit that never returns. A permit leaked with forget, or held by a task that is itself stuck, leaves every later acquire waiting. The test below reproduces a leak and proves the timeout catches it (limiting concurrency with a semaphore).
2. A select! that is not cancel safe. Dropping a half read message can leave a protocol out of step so the peer waits forever (cancellation safety).
3. A hand written future that returns Pending without storing a waker. Nothing will ever poll it again (writing a future by hand, and the smaller demonstration in building a tiny executor).
4. A task that did the work and never signalled it. A oneshot sender dropped without sending, or a result never delivered (task never finishes).
5. A remote peer that stopped answering. The hang is real but not in your process. A request with no deadline will wait for the kernel's TCP timeout, which can be hours.

use std::time::Duration;
use tokio::sync::Semaphore;
use tokio::time::timeout;

#[tokio::test(start_paused = true)]
async fn leaked_permit_is_caught() {
    let sem = Semaphore::new(1);
    sem.acquire().await.unwrap().forget();
    let r = timeout(Duration::from_secs(5), sem.acquire()).await;
    assert!(r.is_err(), "second acquire should time out");
}

fn main() {}

running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Two details make this test fast and honest. start_paused = true freezes Tokio's clock, and when every task is idle the runtime advances it to the next timer, so a five second timeout costs microseconds of wall time. And forget() on the permit is the bug: the permit is gone for good, so the second acquire can only time out. The test asserts the timeout, which is the shape every hang regression test should take.

Step 7: make it reproducible

Once you know the cause, write the test before fixing it. Paused time plus a timeout turns a hang that took hours to appear in production into a failing test that takes a few milliseconds, and it stays in the suite after the fix. The pattern is always the same: set up the stuck state, wrap the awaited operation in a timeout, and assert that it completes (or, as above, assert that it does not, while the bug is still present). Keep the console subscriber behind a Cargo feature so a staging build can enable it without shipping the instrumentation everywhere. The cost of the diagnostics belongs to a build you chose to make, not to every deploy.

Reading what you find

A few notes on interpreting the evidence, since the tools produce plenty of output and little of it is the answer. In a native thread dump, ignore the frames from the standard library and the runtime and read from the top until the first function you recognise as yours or as a dependency. A worker stuck in std::sys::sync::mutex beneath your handle_request is a std mutex taken inside a handler. A worker in std::io::stdio or write beneath a logging call can be a pipe that nobody is draining, so a process logging to a full pipe stops completely and the cause is the log reader. Both are blocked threads, found with step 2, and neither would show in the task view as anything but a task that stopped being polled.

In the console, the columns carry specific meanings. A task's busy time is the total spent inside poll; its idle time is the total spent waiting; its polls count how often the runtime entered it. A task with large busy time and few polls spends long stretches inside a single poll, which is the signature of blocking work in async code. A task with tiny busy time, huge idle time and an old last poll is waiting, and the resource column says on what. A task that has been polled ten million times with no progress is busy waiting, which is the self waking future of step 1.

Before concluding that a task is stuck, check how many of the same kind there are. One idle connection handler is normal; fifty thousand that never finish are a leak. Task count over time is a fine metric to export, because a hang often starts as a slow climb. A gauge of live tasks, from the runtime's metrics, and the age of the oldest in flight request, from your own middleware, cost almost nothing and turn "the service seems slow" into a graph (Tokio runtime metrics).

The cost of each tool, and when this method does not apply

Each step trades coverage for effort. The thread dump needs shell access to the host and debug symbols, and costs a short pause while the process is attached. It sees blocked threads and nothing about parked tasks. The console sees tasks and nothing about the kernel, and requires a rebuild with the unstable cfg flag plus a subscriber that buffers events for every task, so run it on a staging build before you try it on the loaded one. Timeouts need no tooling and cost one timer per guarded call, but they only localise waiting, as the spinning example showed. The heartbeat watchdog is the cheapest of all and catches exactly one failure, a stalled runtime.

Some hangs this order will not find:

• A process stuck in the kernel. A thread in uninterruptible sleep, usually on a dead network filesystem, shows state D in ps and will not respond to a debugger either. The problem is the mount, not the async code.
• Another runtime. tokio-console and Tokio task dumps cover Tokio only. A program on smol or async-std still gets steps 1, 2 and 4, and the heartbeat idea carries over to any executor.
• Livelock rather than deadlock. Tasks that keep waking each other and make no progress look busy in step 1, not idle, and the signal is a poll count that climbs while the application level counter stays flat.
• Non-async hangs. A deadlock between two std mutexes taken in opposite orders is not an async bug, though the thread dump still shows it clearly. Fix the lock order.

Two other posts cover the surrounding ground: how the Tokio runtime schedules work explains why a blocked worker starves the rest, and timeouts in Tokio covers how to choose the deadlines for step 4.

The order, in one pass

Check CPU per thread. Dump native threads and look for workers inside blocking calls. List tasks and find the ones with a stopped poll count. Add labelled timeouts around the awaits that remain. Then walk the causes in order of how often they bite: blocking calls, guards across await, unclosed channels, leaked permits, unsafe cancellation, missing wakers. Fix the cause, keep the regression test, and leave the watchdog on.