Spawn Blocking in Tokio and When Rayon Is the Better Tool

Once you accept that CPU-heavy work does not belong on async worker threads, two tools offer a home for it: Tokio's own spawn_blocking, and the rayon thread pool. They look interchangeable for a single call and behave very differently for many calls. spawn_blocking is a thread-per-job escape hatch built for work that waits (a synchronous library, a file read). rayon is a work-stealing pool sized to your cores, built for work that computes. This post shows both used correctly, measures them on coarse and fine-grained jobs on the same machine, and gives the rule for choosing. Everything compiled on Rust 1.99, edition 2024, with tokio 1 and rayon 1.

Calling each from async code

For one job, the two are about equally easy. spawn_blocking returns a handle you await; for rayon you hand the job to its pool and wait on a channel:

fn checksum(data: &[u8]) -> u64 {
    data.iter().fold(0xcbf29ce484222325u64, |h, b| (h ^ *b as u64).wrapping_mul(0x100000001b3))
}

#[tokio::main]
async fn main() {
    let payload = vec![7u8; 1_000_000];

    let p = payload.clone();
    let via_blocking = tokio::task::spawn_blocking(move || checksum(&p)).await.unwrap();

    let p = payload.clone();
    let (tx, rx) = tokio::sync::oneshot::channel();
    rayon::spawn(move || {
        let _ = tx.send(checksum(&p));
    });
    let via_rayon = rx.await.unwrap();

    println!("{}", via_blocking == via_rayon);
    println!("{via_blocking:#018x}");
}

true
0x6e2f16c4cc4b3c65

The oneshot channel is the bridge: the rayon job runs on a rayon thread and sends its result, and the async side awaits the receiver without blocking a worker. If the rayon job panics, the sender is dropped without sending and rx.await returns an error, which you should map to your own error type. Both run the closure off the async workers; neither is faster for a single job.

Coarse jobs: the two tie

When each job is large (milliseconds or more) and the count is modest, the overhead per job vanishes. The benchmark computed a 2-million-iteration hash for each of 64 inputs on an Apple M1 Max with 10 cores, in a release build, best of three rounds:

Both scaled across the cores almost identically, and neither reached 10x because the performance cores outnumber the efficiency cores on this chip. For coarse jobs either is fine, and spawn_blocking is simpler to write because it needs no extra dependency.

Fine-grained jobs: the gap opens

Now change one thing: 100,000 small jobs of about 2.5 microseconds each. The result is the reason this post exists:

Spawning a blocking task costs on the order of microseconds (an allocation, a queue push, a thread wakeup), which is more than the work itself, so running 100,000 of them in parallel was slower than running them in a loop. Rayon splits the range into a few chunks per thread and processes each chunk in a tight loop, so its overhead is amortised across thousands of items. The lesson generalises: parallelism needs jobs coarse enough to repay the cost of dispatching them. With spawn_blocking you must batch by hand (chunk the input and spawn one job per chunk); rayon does it for you.

The thread explosion

The other difference is how many threads each can create. The blocking pool grows to 512 threads by default. A last measurement sends 1,000 blocking jobs that each sleep for 50 ms, as a stand-in for a synchronous call waiting on I/O, and counts the distinct threads that ran them: 512. The pool did what it is designed to do, which was to start a thread for each blocked job up to its cap, and the other 488 jobs queued behind. For waiting work, that is right: threads blocked on I/O cost little CPU. For computing work it is wrong, since 512 CPU-bound threads on 10 cores means thrashing and a lot of memory (each thread reserves a stack). Rayon's pool has one thread per core and cannot oversubscribe.

Both pools can be bounded: max_blocking_threads on the runtime builder, or a semaphore in front of the call. Rayon's size is set with ThreadPoolBuilder::num_threads, and a custom pool lets you keep a request-handling pool separate from a batch pool so one cannot starve the other:

use rayon::prelude::*;
use rayon::ThreadPoolBuilder;

fn main() {
    let pool = ThreadPoolBuilder::new().num_threads(3).thread_name(|i| format!("cpu-{i}")).build().unwrap();

    let names: Vec<String> = pool.install(|| {
        (0..6).into_par_iter().map(|_| std::thread::current().name().unwrap().to_string()).collect()
    });

    let distinct: std::collections::BTreeSet<_> = names.iter().collect();
    println!("{} threads in the pool, all named cpu-N: {}", pool.current_num_threads(), distinct.iter().all(|n| n.starts_with("cpu-")));
}

3 threads in the pool, all named cpu-N: true

install runs the closure inside the pool, so every parallel iterator in it uses those three threads instead of the global pool. The global pool is sized to the number of cores and is shared by every crate that uses rayon in your process, which is the right default for a CLI and a hazard in a server that also runs rayon-using libraries.

The benchmark, so you can run it on your machine

The numbers above come from this program. Build it with cargo run --release; a debug build changes every ratio, because the per-job overhead stays similar while the work gets many times slower. Your core count, your operating system's thread-wake latency and the size of the job all move the crossover point, so change ITERS to see where spawn_blocking stops losing:

use rayon::prelude::*;
use std::time::Instant;

const ITERS: u64 = 2_000;

fn small(seed: u64) -> u64 {
    let mut h = 0xcbf29ce484222325u64 ^ seed;
    for i in 0..ITERS {
        h ^= i;
        h = h.wrapping_mul(0x100000001b3);
    }
    h
}

#[tokio::main]
async fn main() {
    let n = 100_000u64;

    let t = Instant::now();
    let seq = (0..n).map(small).fold(0, |a, b| a ^ b);
    println!("sequential              {:>8.1} ms", t.elapsed().as_secs_f64() * 1e3);

    let t = Instant::now();
    let handles: Vec<_> = (0..n).map(|j| tokio::task::spawn_blocking(move || small(j))).collect();
    let mut blocking = 0;
    for h in handles {
        blocking ^= h.await.unwrap();
    }
    println!("spawn_blocking per item {:>8.1} ms", t.elapsed().as_secs_f64() * 1e3);

    let t = Instant::now();
    let (tx, rx) = tokio::sync::oneshot::channel();
    rayon::spawn(move || {
        let r = (0..n).into_par_iter().map(small).reduce(|| 0, |a, b| a ^ b);
        let _ = tx.send(r);
    });
    let parallel = rx.await.unwrap();
    println!("rayon par_iter          {:>8.1} ms", t.elapsed().as_secs_f64() * 1e3);

    assert!(seq == blocking && blocking == parallel);
}

The final assertion matters in a benchmark: it proves that all three strategies computed the same answer, and it stops the optimiser from deleting the work. Measure the way you will run: warm up once, take the best of several runs, and keep other heavy processes off the machine.

Choosing

A cheap mixed design is to put a bounded tokio::sync::mpsc channel in front of a rayon-backed worker so the async side feels backpressure when the CPU stage falls behind; channel sizing is in bounded channels and backpressure.

Cancellation and what neither can do

Neither tool can cancel a running closure. Aborting the Tokio handle of a spawn_blocking task only stops the await; the closure runs to the end on its thread, and its result is discarded. The same holds for rayon. If the work must be stoppable, check a shared AtomicBool at loop boundaries and return early. Also remember the other half of the blocking problem: a rayon job that calls back into async code with block_on can deadlock if the async side is waiting for that very job. Keep the boundary one-way. The reasoning behind moving work off the workers at all is in blocking the Tokio runtime.