Tokio Semaphores for Limiting Concurrency in Rust Services

An unbounded fan-out is a denial of service you wrote against yourself. A loop that spawns a task per URL, per row or per request opens as many sockets, file handles and database connections as there are items, and the operating system or the remote server refuses somewhere around the thousandth. A tokio::sync::Semaphore is the standard fix: a counter of permits that tasks must hold while they do the limited thing. It is small enough to hold in your head and subtle enough to misuse in four distinct ways. This post builds a limited fan-out, shows the permit moving with the task, and covers the variants (acquire_owned, try_acquire, closing, multiple permits) and the mistakes. Everything compiled on Rust 1.99, edition 2024, with tokio 1.

The shape

Create the semaphore with the number of permits you will allow. Before the limited section, acquire one; hold it for the duration; drop it to release. Spawn as many tasks as you like: those without a permit wait, parked and costing only their stack-less future, until one frees up.

use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use tokio::sync::Semaphore;
use tokio::time::{sleep, Duration, Instant};

#[tokio::main(flavor = "current_thread", start_paused = true)]
async fn main() {
    let limit = Arc::new(Semaphore::new(3));
    let running = Arc::new(AtomicUsize::new(0));
    let peak = Arc::new(AtomicUsize::new(0));
    let t = Instant::now();

    let mut handles = Vec::new();
    for job in 0..10u32 {
        let limit = limit.clone();
        let (running, peak) = (running.clone(), peak.clone());
        handles.push(tokio::spawn(async move {
            let _permit = limit.acquire().await.unwrap();
            let now = running.fetch_add(1, Ordering::SeqCst) + 1;
            peak.fetch_max(now, Ordering::SeqCst);
            sleep(Duration::from_millis(100)).await;
            running.fetch_sub(1, Ordering::SeqCst);
            job
        }));
    }
    for h in handles {
        h.await.unwrap();
    }

    println!("10 jobs, peak concurrency {}, finished in {:?}", peak.load(Ordering::SeqCst), t.elapsed());
}

10 jobs, peak concurrency 3, finished in 400ms

Ten jobs of 100 ms with three permits finish in four rounds (3 + 3 + 3 + 1), so 400 ms, and the peak never passed three. The permit is a guard value: _permit is dropped at the end of the block, which gives the permit back. Note the name _permit and not _. Writing let _ = limit.acquire().await drops the guard on the same line and the limit does nothing; an underscore alone is the single most common reason a semaphore appears to have no effect.

Permits that move: acquire_owned

The guard from acquire() borrows the semaphore, so it cannot be moved into a spawned task unless the semaphore outlives it. The usual pattern acquires before spawning, which also bounds the number of tasks that exist, and passes an owned permit into the task with acquire_owned() on an Arc<Semaphore>:

use std::sync::Arc;
use tokio::sync::Semaphore;
use tokio::time::{sleep, Duration, Instant};

#[tokio::main(flavor = "current_thread", start_paused = true)]
async fn main() {
    let limit = Arc::new(Semaphore::new(2));
    let t = Instant::now();
    let mut handles = Vec::new();

    for job in 0..6u32 {
        let permit = limit.clone().acquire_owned().await.unwrap();
        println!("job {job} admitted at {:>3} ms", t.elapsed().as_millis());
        handles.push(tokio::spawn(async move {
            sleep(Duration::from_millis(100)).await;
            drop(permit);
            job
        }));
    }
    for h in handles {
        h.await.unwrap();
    }
    println!("done at {} ms", t.elapsed().as_millis());
}

job 0 admitted at   0 ms
job 1 admitted at   0 ms
job 2 admitted at 100 ms
job 3 admitted at 100 ms
job 4 admitted at 200 ms
job 5 admitted at 200 ms
done at 300 ms

Here the loop itself waits at acquire_owned, so at most two tasks exist at any moment, which is the right behaviour for ten million inputs: the loop does not queue ten million tasks in front of the limit. If you acquire inside the task, as in the first example, the tasks all exist and wait in the semaphore's queue. Both limit concurrency; only the second pays memory for the waiting tasks. Prefer acquiring in the loop when the producer can wait.

Not waiting: try_acquire for load shedding

A request handler often prefers refusing to waiting. try_acquire returns an error immediately if no permit is free, which the handler turns into a 429 or 503:

use std::sync::Arc;
use tokio::sync::{Semaphore, TryAcquireError};

fn handle(limit: &Arc<Semaphore>, id: u32) -> String {
    match limit.clone().try_acquire_owned() {
        Ok(permit) => {
            std::mem::forget(permit);
            format!("request {id}: accepted")
        }
        Err(TryAcquireError::NoPermits) => format!("request {id}: busy, try later"),
        Err(TryAcquireError::Closed) => format!("request {id}: shutting down"),
    }
}

fn main() {
    let limit = Arc::new(Semaphore::new(2));
    for id in 1..=4 {
        println!("{}", handle(&limit, id));
    }
    println!("available permits: {}", limit.available_permits());

    limit.close();
    println!("{}", handle(&limit, 5));
}

request 1: accepted
request 2: accepted
request 3: busy, try later
request 4: busy, try later
available permits: 0
request 5: shutting down

The mem::forget keeps the permit consumed so the example shows the refusal; in a real handler you hold the permit until the request finishes. close() makes every pending and future acquire fail with Closed, which is the shutdown hook: close the semaphore, and waiting tasks wake with an error rather than hanging. That is a cleaner way to drain a pool than a flag, as in graceful shutdown.

Weighted permits

Work is rarely uniform. A semaphore can hand out several permits at once with acquire_many, so heavy jobs consume more of the budget than light ones. A budget of 100 "memory units" can run one 80-unit job beside four 5-unit jobs:

use std::sync::Arc;
use tokio::sync::Semaphore;
use tokio::time::{sleep, Duration, Instant};

#[tokio::main(flavor = "current_thread", start_paused = true)]
async fn main() {
    let budget = Arc::new(Semaphore::new(100));
    let t = Instant::now();
    let mut handles = Vec::new();

    for (name, cost) in [("render", 80u32), ("thumb-a", 20), ("thumb-b", 20), ("thumb-c", 20)] {
        let permit = budget.clone().acquire_many_owned(cost).await.unwrap();
        println!("{name} admitted at {:>3} ms using {cost} units, {} left", t.elapsed().as_millis(), budget.available_permits());
        handles.push(tokio::spawn(async move {
            sleep(Duration::from_millis(100)).await;
            drop(permit);
        }));
    }
    for h in handles {
        h.await.unwrap();
    }
}

render admitted at   0 ms using 80 units, 20 left
thumb-a admitted at   0 ms using 20 units, 0 left
thumb-b admitted at 100 ms using 20 units, 80 left
thumb-c admitted at 100 ms using 20 units, 60 left

Only the first thumbnail fits beside the 80-unit render (80 + 20 uses the whole budget), so the other two were admitted at 100 ms, when the first two jobs released their permits. Requests for more permits than the semaphore holds can never be satisfied and will wait forever, so validate the cost against the total before calling. The semaphore is also fair: waiters are served in order, so a large request at the head of the queue is not starved by a stream of small ones behind it, at the cost that small requests wait behind it even when they would fit.

Choosing the number

The limit should come from the thing you are protecting, not from a guess. For outbound HTTP it is the connection limit the remote service will tolerate from one client (often 10 to 100). For a database it is the pool size, minus a margin for other code paths, since a semaphore larger than the pool just moves the waiting from your semaphore into the pool. For CPU-bound blocking work it is the number of cores. For memory, it is the budget divided by the worst-case footprint of one job, which is where weighted permits help. A useful sanity check is Little's law: with a limit of N and a per-job time of W seconds, throughput tops out at N / W jobs per second. Ten permits and a 200 ms call give at most 50 calls a second no matter how many tasks you spawn; if you need 500 per second, the limit has to be 100 or the call has to get faster.

Make the limit configurable and export two numbers, the permits in use and the tasks waiting. A semaphore that is always at zero available with a growing queue is telling you the limit is below the demand; one that never gets above half used is a limit you could raise or drop. Without those gauges, a semaphore is a silent bottleneck that someone will rediscover during an incident.

Four ways it goes wrong

1. Dropping the permit immediately, with let _ = .... The limit becomes a no-op.
2. Holding the permit longer than needed, through a response write or a log flush. The permit should cover exactly the resource you are protecting (the connection), not the whole request.
3. Acquiring two semaphores in different orders in different tasks: a classic deadlock. Fix a global order, or acquire both through one weighted semaphore.
4. Panicking while holding it. The permit is released on unwind because the guard drops, so the limit survives; but state you guarded may be half-updated.

When a semaphore is the wrong tool

For a bounded window over a known batch, a JoinSet with join-before-spawn does the same job without a shared object, as in JoinSet. For a bounded stream, buffer_unordered(n) is one line, covered in buffer_unordered. For a rate (so many per second) rather than a concurrency (so many at once), a semaphore is the wrong shape: it limits simultaneity, not frequency, and a token bucket is the right model, built in a token-bucket rate limiter. And to limit how many blocking calls run at once, put a semaphore in front of spawn_blocking, as suggested in the runtime explained.