Tokio Console for Finding Stuck Tasks in Rust Services

When an async service hangs or slows and the logs say nothing, the question is always the same: which task is stuck, and on what? A debugger shows thread stacks, which are nearly useless for async code because a parked task has no stack at all. tokio-console is the instrument built for this: a terminal UI, like top for tasks, that shows every task in a running Tokio program with how long it has been alive, how much time it spent polling versus waiting, how many times it was woken, and a set of warnings for known bad patterns. This post sets it up, explains each column and warning, instruments a program with one stuck and one busy task, reads the symptoms and covers the costs and the alternatives when you cannot run it. The instrumented program compiled and ran on Rust 1.99 with tokio 1.54 and console-subscriber 0.4.

Setup: three things

The console needs the runtime to emit task events, which requires an unstable cfg flag and a feature, plus a subscriber in your binary that serves the data over a local connection.

[dependencies]
tokio = { version = "1", features = ["full", "tracing"] }
console-subscriber = "0.4"

[build]
rustflags = ["--cfg", "tokio_unstable"]

cargo install --locked tokio-console
cargo run
tokio-console

The cfg flag is why this is an opt-in tool: tokio_unstable enables instrumentation hooks that Tokio does not promise to keep stable, so put it in a development configuration and keep it out of release builds. In the program, one line starts the subscriber before the runtime does any work: console_subscriber::init();. It listens on 127.0.0.1:6669 by default, and the tokio-console client connects there. Remote debugging works through an SSH tunnel to that port; the subscriber has no authentication, so do not bind it to a public interface.

A program with two problems

The following program has a task that waits on a channel whose sender was dropped without sending (it can never wake), and a task that burns CPU for 40 ms between yields (it hogs a worker). Both are given names with tokio::task::Builder, which makes them findable in the console:

use std::time::Duration;

#[tokio::main]
async fn main() {
    console_subscriber::init();

    let stuck = tokio::task::Builder::new()
        .name("waits-forever")
        .spawn(async {
            let (_tx, rx) = tokio::sync::oneshot::channel::<()>();
            let _ = rx.await;
        })
        .unwrap();

    let busy = tokio::task::Builder::new()
        .name("busy-loop")
        .spawn(async {
            loop {
                let t = std::time::Instant::now();
                while t.elapsed() < Duration::from_millis(40) {}
                tokio::task::yield_now().await;
            }
        })
        .unwrap();

    tokio::time::sleep(Duration::from_secs(60)).await;
    println!("{} {}", stuck.is_finished(), busy.is_finished());
}

Run with the cfg flag and the program serves the console data for a minute. (The listing is shown with a 60 second sleep so you have time to attach the client; the version compiled for this article used half a second and printed false false, both tasks still alive.) In the console, the task list is sorted by the column you choose; the two tasks look nothing alike.

Reading the task list

For the program above, waits-forever has a long total time, almost all of it idle, one poll, and zero wakes. That combination is the signature of a task whose waker nobody holds: the future registered interest and nothing will ever notify it. busy-loop has high busy time and a large mean poll time (around 40 ms), polled repeatedly. The 40 ms figure is the diagnosis: no task should hold a worker that long between awaits, the rule from blocking the Tokio runtime. Select a task with the arrow keys and Enter to open its detail view: poll durations as a histogram, the waker history and the span of its activity.

The warnings

The console runs a handful of lints over the live data and flags tasks that match. The ones that matter most:

• Lost waker. The task was woken zero times and has no waker references left anywhere: it can never be resumed. That is the "forgot to store the waker" bug in a hand-written future, or a channel sender dropped while the receiver waits. It is the exact case of waits-forever and the quickest way to find a task that never finishes. The causes are catalogued in tasks that never finish.
• Never yielded. The task has been polled for a long time without returning to the scheduler. Usually blocking or a tight loop with no await, the case of busy-loop (when it runs without the periodic yield).
• Self-wakes. The task wakes itself repeatedly, which looks like yield_now in a loop or a hand-written future calling wake_by_ref and returning Pending, a busy loop wearing async clothing. It burns a core while looking idle.
• Large future / auto-boxed future. A task whose future is very large, or one that was boxed automatically because of its size. Often a big buffer held across an await, as in futures as state machines.

A debugging session, in order

1. Reproduce the slowdown or hang with the console attached. Sort by Sched descending: tasks that wait long to be polled indicate a blocked or overloaded runtime. Sort by Busy to find who is hogging.
2. For a hang, sort by Total and look at the oldest idle tasks with the Warn column set. A lost-waker flag is a direct answer.
3. Open the suspect's detail view and read its location. Jump to the code and look for the await it is parked on.
4. For a suspected blocker, check the poll-time histogram: a long tail beyond a millisecond confirms it.
5. Fix, then run again and confirm the warning is gone and the numbers moved.

Costs, limits and alternatives

Instrumentation adds overhead to every task event: expect a measurable slowdown (the project documents it as noticeable under heavy load), so do not leave it on in a latency-sensitive production service, and run the profile with and without it before trusting timings. Memory grows with the number of tasks tracked, and the retention window for completed tasks is configurable. The console shows tasks and resources (mutexes, semaphores, timers) but not your application's own logic or CPU samples; for hot code use a profiler as in profiling with flamegraphs.

When you cannot run the console, two cheaper tools answer part of the question. The runtime's own metrics (Handle::metrics()) report the number of alive tasks and, with tokio_unstable, busy duration per worker and queue depths, which you can export to a dashboard. And task dumps, in the post on task dumps and async backtraces, capture a snapshot of where every task is parked. Use the console on a developer machine or a staging replica; use metrics and dumps in production.