Profiling Async Rust Services With Tokio Metrics in Steps
Use Tokio runtime and task metrics to see worker busy time, scheduling delay, queue depth and slow polls. Export them, read the numbers that matter and tell blocking code from a genuine lack of capacity.
When an async service is slow, a CPU profile can look innocent: the process is not busy, yet latency is high. Often the executor itself is the bottleneck: tasks are ready but waiting for a worker that is busy with something long or blocked. Tokio exposes numbers for exactly this. This post shows how to collect and read them. It compiles on Rust 1.90 with edition 2024, Tokio 1.47 and tokio-metrics 0.4 (with rt feature).
Runtime metrics
[dependencies]
tokio = { version = "1.47", features = ["full"] }
tokio-metrics = { version = "0.4", features = ["rt"] }
futures = "0.3"use std::time::Duration;
use tokio_metrics::RuntimeMonitor;
#[tokio::main]
async fn main() {
let handle = tokio::runtime::Handle::current();
let monitor = RuntimeMonitor::new(&handle);
tokio::spawn(async move {
for interval in monitor.intervals() {
println!(
"busy_ratio={:.2} workers={} global_queue={} mean_poll_us={:.0}",
interval.busy_ratio(),
interval.workers_count,
interval.global_queue_depth,
interval.mean_poll_duration.as_secs_f64() * 1e6,
);
tokio::time::sleep(Duration::from_secs(1)).await;
}
});
// ... run the service
tokio::time::sleep(Duration::from_secs(5)).await;
}Each call to the intervals iterator yields metrics for the period since the last one. The ones that matter first:
| Metric | Meaning | Reading it |
|---|---|---|
busy_ratio() | Fraction of time workers spent running tasks | Near 1.0: out of executor capacity |
global_queue_depth | Tasks waiting in the shared queue | Sustained growth: tasks arrive faster than workers drain them |
mean_poll_duration | Average time inside a task poll | High: tasks run long without yielding |
total_noop_count, steal counts (unstable) | Parks with no work, steals between workers | Load balance behavior |
Task metrics
use tokio_metrics::TaskMonitor;
#[tokio::main]
async fn main() {
let monitor = TaskMonitor::new();
let m = monitor.clone();
tokio::spawn(async move {
for interval in m.intervals() {
println!(
"polls={} mean_poll_us={:.0} slow_polls={} mean_sched_delay_us={:.0}",
interval.total_poll_count,
interval.mean_poll_duration().as_secs_f64() * 1e6,
interval.total_slow_poll_count,
interval.mean_scheduled_duration().as_secs_f64() * 1e6,
);
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
}
});
for _ in 0..1000 {
tokio::spawn(monitor.instrument(async {
tokio::task::yield_now().await;
}));
}
tokio::time::sleep(std::time::Duration::from_secs(3)).await;
}instrument wraps a future so its polls are measured. Wrap the top-level request handler future or a specific operation, with one monitor per kind of work. The most useful number is scheduling delay (mean_scheduled_duration): the time from a task being woken to being polled. A healthy service has delays of microseconds; milliseconds mean tasks are queued behind something.
Reading the combinations
- High busy ratio, high queue depth, mean poll short: you simply need more capacity: more worker threads, more instances or less work per request.
- Low busy ratio but long scheduling delays or latency spikes: a few tasks hold workers for a long time (or block them), starving others; look at slow polls and for blocking calls (blocking in async code).
- High mean poll duration: CPU-heavy work inside async tasks (parsing, hashing, serialization of big bodies). Move it to
spawn_blockingor Rayon (spawn_blocking and Rayon), or chunk it withyield_now. - Everything looks fine, latency is still high: the wait is outside the executor: database, pool, downstream calls, locks. Add spans and pool metrics (pool sizing).
Exporting
use tokio_metrics::RuntimeMonitor;
fn export_runtime_metrics(handle: tokio::runtime::Handle) {
let monitor = RuntimeMonitor::new(&handle);
tokio::spawn(async move {
for m in monitor.intervals() {
metrics::gauge!("tokio_busy_ratio").set(m.busy_ratio());
metrics::gauge!("tokio_global_queue_depth").set(m.global_queue_depth as f64);
metrics::gauge!("tokio_mean_poll_seconds").set(m.mean_poll_duration.as_secs_f64());
tokio::time::sleep(std::time::Duration::from_secs(5)).await;
}
});
}Send the values to your metrics backend alongside request latency percentiles (tail latency). Alert on sustained busy ratio above about 0.8 and on scheduling delay above a few milliseconds. Correlate the alerts with deploys and traffic.
When metrics are not enough
Metrics tell you that the executor is struggling and roughly why; they do not name the offending function. Follow up with tokio-console for per-task views (tokio-console), a flamegraph under load (profiling with flamegraphs) and task dumps for hangs (task dumps). A shorter introduction to the built-in runtime counters is in Tokio runtime metrics.
