Tuning Tokio Worker Threads and When to Leave Defaults

Someone sets worker_threads = 64 on a 4-core container because "more threads, more throughput", and the service gets slower. A Tokio worker is not a request handler. It is a scheduler loop that runs tasks until they await, and extra loops beyond the number of cores can only fight over cores. This page covers the default, the two cases where it is wrong, and how to decide with a measurement. It compiles on Rust 1.99 with edition 2024 and tokio 1.54.

What the default is

The multi-thread runtime starts one worker per core the process can see. You can read what it chose.

fn main() {
    let rt = tokio::runtime::Runtime::new().expect("build runtime");
    let m = rt.metrics();
    println!("available_parallelism: {:?}", std::thread::available_parallelism());
    println!("tokio workers:         {}", m.num_workers());
}

available_parallelism: Ok(10)
tokio workers:         10

That value comes from std::thread::available_parallelism, which on Linux honors CPU affinity and cgroup quotas. On modern kernels and recent Rust that means a container limited to 4 CPUs reports 4. Older setups, some orchestrators and a few quota styles still report the host's core count, and then you have 64 workers competing for 4 CPUs worth of time.

Set the number explicitly

The builder lets you override it, and there is an environment variable that does the same without a rebuild: TOKIO_WORKER_THREADS.

use tokio::runtime::Builder;

fn main() {
    let rt = Builder::new_multi_thread()
        .worker_threads(4)
        .thread_name("api-worker")
        .enable_all()
        .build()
        .expect("build runtime");

    rt.block_on(async {
        let name = std::thread::current();
        println!("running on {:?}", name.name());
        println!("workers: {}", tokio::runtime::Handle::current().metrics().num_workers());
    });
}

running on Some("main")
workers: 4

block_on runs the top-level future on the calling thread, which is why the first line says main; tasks you spawn run on the api-worker threads. Naming them pays off the first time you read a profile or a top -H listing.

Zero is not allowed

An easy mistake is to compute the count from a setting that can be empty.

thread 'main' panicked at /Users/.../tokio-1.54.1/src/runtime/builder.rs:
Worker threads cannot be set to 0

Clamp it: .worker_threads(n.max(1)). A single worker is legal and behaves like a multi-thread runtime with no stealing, which is occasionally useful in tests.

Case one: the container lies

Check what your runtime thinks it has, from inside the container, then compare it with the limit you configured.

docker run --rm --cpus=2 my-service:latest
docker exec my-service cat /sys/fs/cgroup/cpu.max

200000 100000

We did not run Docker for this page, so the two lines above show the documented cgroup v2 format rather than a captured session. The two numbers in cpu.max are the quota and the period in microseconds: 200,000 over 100,000 is two CPUs of time per 100 ms. If a service in that container logs 16 workers at startup, the runtime has not seen the quota, and you can read it yourself and size the pool from it.

use std::fs;

fn cgroup_cpus() -> Option<usize> {
    let raw = fs::read_to_string("/sys/fs/cgroup/cpu.max").ok()?;
    let mut parts = raw.split_whitespace();
    let quota: f64 = parts.next()?.parse().ok()?; // "max" fails to parse: no limit
    let period: f64 = parts.next()?.parse().ok()?;
    Some((quota / period).ceil().max(1.0) as usize)
}

fn main() {
    let detected = std::thread::available_parallelism().map(|n| n.get()).unwrap_or(1);
    let workers = cgroup_cpus().unwrap_or(detected).min(detected);
    println!("detected {detected}, using {workers}");
}

detected 10, using 10

On this macOS host there is no cgroup file, so the function returns None and the detected value stands. Inside the 2-CPU container above it would print using 2 on a runtime that over-counted. Take the minimum of the two so you never ask for more than the machine has, and round the quota up, because a quota of 1.5 CPUs still wants two workers to use it.

If a container limited to 2 CPUs logs 16 workers, set worker_threads to match the quota. The symptom of getting this wrong is not a crash. It is throttling: the kernel's CFS scheduler pauses the whole container once it has used its quota for the period, so p99 latency grows in steps of 100 ms (the default period) even though average CPU looks modest. Check nr_throttled in /sys/fs/cgroup/cpu.stat; a number that keeps rising confirms it.

Case two: you share the machine on purpose

If the process also runs a rayon pool, a database engine or another busy runtime, giving Tokio every core starves the others. Reserving two cores for CPU-bound work and giving Tokio the rest is a legitimate configuration (spawn_blocking and Rayon). Measure both sides. An I/O-bound service that spends 90% of its time waiting uses a fraction of one core per thousand connections, so shrinking the worker count rarely hurts it.

Measuring instead of guessing

The experiment is cheap: run the same load at several worker counts and compare throughput and tail latency. This sketch runs a fixed CPU-light, I/O-shaped workload and prints time per setting.

use std::time::{Duration, Instant};
use tokio::runtime::Builder;

fn run(workers: usize) -> Duration {
    let rt = Builder::new_multi_thread()
        .worker_threads(workers)
        .enable_time()
        .build()
        .unwrap();
    let start = Instant::now();
    rt.block_on(async {
        let mut set = tokio::task::JoinSet::new();
        for _ in 0..20_000 {
            set.spawn(async {
                let mut x = 0u64;
                for i in 0..2_000 {
                    x = x.wrapping_add(i * i);
                }
                tokio::time::sleep(Duration::from_millis(1)).await;
                x
            });
        }
        while set.join_next().await.is_some() {}
    });
    start.elapsed()
}

fn main() {
    for w in [1, 2, 4, 8, 16, 64] {
        println!("{w:>2} workers: {:?}", run(w));
    }
}

 1 workers: 328.524666ms
 2 workers: 192.73075ms
 4 workers: 117.079959ms
 8 workers: 75.298333ms
16 workers: 69.008292ms
64 workers: 88.327917ms

On a 10-core machine the time fell almost linearly up to 8 workers (329 ms, 193, 117, 75), barely moved at 16 (69 ms), and got worse at 64 (88 ms), which is 28% slower than the best run. Threads past the core count add context switches and idle-worker wake-ups and buy nothing. The default of 10 sits near the bottom of the curve, which is the usual result and the reason to leave it alone. The shape is the lesson, not these exact milliseconds: your workload will bend it differently, and a service dominated by waiting will flatten much earlier. We ran each setting once on a warm machine, so repeat it before you trust a difference of a few percent.

One more input is hyper-threading. available_parallelism counts logical cores, so a 4-core CPU with two hardware threads each reports 8. For workloads that are mostly waiting that is harmless. For CPU-heavy tasks the second thread of a core adds maybe 20 to 30 percent rather than 100, so the sweep above is the honest way to find where your curve flattens. Run it on the instance type you deploy to, not on a laptop with a different core layout, and keep the result next to the deployment config so the next person to resize the instance knows why the number is what it is.

Signs the number is not your problem

• High latency with low CPU usually means blocked workers or a slow dependency, not too few workers. Look at global_queue_depth first (runtime metrics).
• A request that takes the same time at 4 and 64 workers is waiting on something outside the runtime.
• Blocking calls on a worker are fixed by moving them to spawn_blocking, which has its own pool, not by adding workers.

When the default stays

If the runtime sees the right number of CPUs, and you are not sharing them, do nothing. A tuned number that nobody remembers is a liability when the instance size changes. If you do set it, derive it from the environment (TOKIO_WORKER_THREADS or the cgroup limit) rather than hard-coding a literal, and confirm it with a load test before and after (load testing a Rust service). Background on what the workers actually do is in the runtime explainer.