Tokio Metrics and Runtime Stats for Production Dashboards

A Tokio service can be healthy by every dashboard you already have and still be minutes from falling over, because CPU percentage and request latency both lag the thing that actually fails first: the runtime's queues. Tokio exposes a handful of numbers that move before users notice. This page shows how to read them, which ones need a compile flag, and how to turn them into something a dashboard can scrape. It compiles on Rust 1.99 with edition 2024, tokio 1.54 and tokio-metrics 0.5.

Three numbers available on stable Rust

Handle::current().metrics() returns a RuntimeMetrics. Without any flags it gives you three values that are enough to start.

use std::time::Duration;
use tokio::runtime::Handle;

#[tokio::main(flavor = "multi_thread", worker_threads = 2)]
async fn main() {
    let metrics = Handle::current().metrics();
    println!("workers: {}", metrics.num_workers());

    let mut tasks = Vec::new();
    for _ in 0..200 {
        tasks.push(tokio::spawn(tokio::time::sleep(Duration::from_millis(300))));
    }
    tokio::time::sleep(Duration::from_millis(50)).await;
    println!("alive tasks: {}", metrics.num_alive_tasks());
    println!("global queue: {}", metrics.global_queue_depth());

    for t in tasks {
        t.await.unwrap();
    }
    println!("alive after: {}", metrics.num_alive_tasks());
}

workers: 2
alive tasks: 200
global queue: 0
alive after: 0

num_alive_tasks counts the 200 sleepers and not the main future, which #[tokio::main] drives with block_on rather than as a spawned task. It drops back to 0 when they finish. Treat it as a leak detector: a count that only ever climbs means tasks are being spawned and never completing, usually because they await a channel nobody sends on. global_queue_depth is the number of tasks waiting in the shared injection queue. It stayed at 0 here because sleeping tasks are not runnable. A queue that is persistently above zero means workers cannot keep up, and it is the earliest warning you have that the runtime is saturated.

Make the problem visible

To see the queue move, give the runtime more runnable work than workers. Here two workers face 500 tasks that each spin for a few milliseconds without yielding.

use std::time::{Duration, Instant};
use tokio::runtime::Handle;

fn burn(ms: u64) {
    let start = Instant::now();
    while start.elapsed() < Duration::from_millis(ms) {}
}

#[tokio::main(flavor = "multi_thread", worker_threads = 2)]
async fn main() {
    let metrics = Handle::current().metrics();
    for _ in 0..500 {
        tokio::spawn(async { burn(5) });
    }
    // Spawned from the main task, so the tasks land in a worker queue or the injector.
    tokio::task::yield_now().await;
    println!("global queue right after spawning: {}", metrics.global_queue_depth());
    tokio::time::sleep(Duration::from_millis(100)).await;
    println!("global queue 100 ms later:        {}", metrics.global_queue_depth());
}

global queue right after spawning: 483
global queue 100 ms later:        226

Five hundred five-millisecond tasks on two workers need about 1.25 seconds of wall time, and after 100 ms the queue had drained from 483 to 226. Anything else you spawn in that window waits behind the backlog, and that is the symptom a user sees: a health check that normally takes 2 ms takes a quarter of a second. The counts vary from run to run because they depend on timing and on how many tasks land in worker-local queues, so read them as a trend, not a precise count.

Sample the metrics on an interval

A single reading is a snapshot. What you want is a rate over a window, and the tokio-metrics crate does that for you with RuntimeMonitor, which yields one struct of deltas per interval.

use std::time::Duration;
use tokio_metrics::RuntimeMonitor;

fn burn(ms: u64) {
    let start = std::time::Instant::now();
    while start.elapsed() < Duration::from_millis(ms) {}
}

#[tokio::main(flavor = "multi_thread", worker_threads = 2)]
async fn main() {
    let handle = tokio::runtime::Handle::current();
    let monitor = RuntimeMonitor::new(&handle);
    let mut intervals = monitor.intervals();

    for _ in 0..40 {
        tokio::spawn(async { burn(20) });
    }
    tokio::time::sleep(Duration::from_millis(300)).await;

    let i = intervals.next().unwrap();
    println!("workers:    {}", i.workers_count);
    println!("busy ratio: {:.2}", i.busy_ratio());
    println!("busy total: {:?}", i.total_busy_duration);
}

workers:    2
busy ratio: 2.00
busy total: 800.670167ms

Forty tasks of 20 ms is 800 ms of busy time, and that is what total_busy_duration reports. The surprise is busy_ratio: it printed 2.00, not 1.00. The ratio is the busy time of all workers added together divided by the wall-clock length of the interval, so it can reach the worker count. Divide by workers_count to get utilization per worker, which here is 1.00: both workers were saturated. Alert on busy_ratio() / workers_count above about 0.8 over a long window, because past that point there is no headroom for a burst.

A low utilization with a deep queue points somewhere else entirely: a blocking call holding a worker, or a task that never yields. The next tool finds those.

Find the slow task with TaskMonitor

Runtime metrics tell you the scheduler is unhappy. TaskMonitor wraps individual futures and records how long each poll took, which is how you catch code that blocks without an .await. A poll longer than 50 ms counts as slow by default.

use std::time::Duration;
use tokio_metrics::TaskMonitor;

#[tokio::main(flavor = "multi_thread", worker_threads = 2)]
async fn main() {
    let monitor = TaskMonitor::new();
    let mut intervals = monitor.intervals();

    let mut handles = Vec::new();
    for _ in 0..4 {
        handles.push(tokio::spawn(monitor.instrument(async {
            // blocks the worker for 60 ms with no await: a slow poll
            std::thread::sleep(Duration::from_millis(60));
            tokio::time::sleep(Duration::from_millis(10)).await;
        })));
    }
    for h in handles {
        h.await.unwrap();
    }
    let m = intervals.next().unwrap();
    println!("polls:              {}", m.total_poll_count);
    println!("slow polls:         {}", m.total_slow_poll_count);
    println!("mean poll duration: {:?}", m.mean_poll_duration());
    println!("slow poll ratio:    {:.2}", m.slow_poll_ratio());
}

polls:              8
slow polls:         4
mean poll duration: 31.034531ms
slow poll ratio:    0.50

Each task is polled twice, once up to the sleep and once after, so there are 8 polls. The four first polls blocked for 60 ms and counted as slow, while a healthy async task has a mean poll duration measured in microseconds. Wrap each class of handler in its own monitor and you can tell which route is starving the others.

Metrics that need tokio_unstable

Poll counts, steal counts, per-worker busy time and the blocking pool size are behind the tokio_unstable cfg. You enable it for the whole build, not as a feature, in .cargo/config.toml.

[build]
rustflags = ["--cfg", "tokio_unstable"]

Without the flag, calling an unstable method fails like this:

error[E0599]: no method named `num_blocking_threads` found for struct `tokio::runtime::RuntimeMetrics` in the current scope

"Unstable" here means the API may change between minor releases, not that it is unsound. Many teams enable it in production for exactly these counters. Note that changing rustflags rebuilds your entire dependency graph, and an environment variable RUSTFLAGS overrides the config file, so check that CI and the release build see the same value.

Export them for a dashboard

A background task can copy the values into whatever your metrics system expects. This one writes Prometheus text format by hand, which is enough to show the shape. In a real service you would register gauges with the prometheus or metrics crate (Prometheus metrics in Rust).

use tokio::runtime::Handle;

fn render(handle: &Handle) -> String {
    let m = handle.metrics();
    format!(
        "# TYPE tokio_workers gauge\ntokio_workers {}\n\
         # TYPE tokio_alive_tasks gauge\ntokio_alive_tasks {}\n\
         # TYPE tokio_global_queue_depth gauge\ntokio_global_queue_depth {}\n",
        m.num_workers(),
        m.num_alive_tasks(),
        m.global_queue_depth()
    )
}

#[tokio::main]
async fn main() {
    print!("{}", render(&Handle::current()));
}

# TYPE tokio_workers gauge
tokio_workers 10
# TYPE tokio_alive_tasks gauge
tokio_alive_tasks 0
# TYPE tokio_global_queue_depth gauge
tokio_global_queue_depth 0

Serve that string from a /metrics route and point Prometheus at it. Ten workers is what #[tokio::main] chose on the machine we ran this on, one per logical core.

What to alert on

• global_queue_depth above zero for more than a minute. A brief spike is a burst; a plateau is saturation.
• num_alive_tasks rising steadily with flat traffic. That is a task leak.
• A high busy_ratio with a low request rate, which points at CPU work running on the async workers that belongs on spawn_blocking (blocking the Tokio runtime).

Pick thresholds from your own baseline rather than from this page. Record the three stable gauges for a week of ordinary traffic, note the daily peak of global_queue_depth and the quietest hour's num_alive_tasks, and alert at a comfortable multiple of those. A service that idles at 40 tasks and peaks at 900 has a very different "normal" from one that holds 20,000 open connections, and a fixed threshold copied from a blog post will be wrong for one of them.

What these numbers cannot tell you

Runtime metrics describe the scheduler, not your handlers. They will not tell you which task is slow, which query is the culprit or why a request took 900 ms. For that you need per-task instrumentation, either tokio_metrics::TaskMonitor around a specific future or a profile (profiling Tokio services). Use runtime metrics to decide whether the runtime is the problem, and the other tools to find out which task is.