Skip to content
rusthacks.dev
Architecture

Fan Out and Fan In Work Pipelines in Rust Services Design

Split work across a bounded pool of workers and merge the results: a producer, bounded channels for backpressure, N workers, a collector and clean shutdown. A Tokio pipeline with ordering, errors and cancellation handled.

5 min read
Summarize in AI
Flat illustration for the post “Fan Out and Fan In Work Pipelines in Rust Services Design”, composed around threads.

Many jobs have the same shape: read a stream of items, do slow work on each, collect what comes out. Download and resize images, call an API for each record, enrich rows from a database. Doing it serially is slow; doing it with a task per item is unbounded. The fan-out fan-in pipeline sits between: a fixed number of workers share the input and feed one collector. This post builds it with Tokio channels and covers the design questions. Versions: Rust 1.90 with edition 2024, Tokio 1.47, futures 0.3.

The pipeline

Text
producer ──► [bounded queue] ──► worker 1 ─┐
                             ├─► worker 2 ─┼─► [bounded queue] ──► collector
                             └─► worker N ─┘

Implementation

rust
use std::sync::Arc;
use tokio::sync::{mpsc, Mutex};

#[derive(Debug)]
pub struct Job {
    pub index: usize,       // position in the input, to restore order if needed
    pub url: String,
}

#[derive(Debug)]
pub struct Outcome {
    pub index: usize,
    pub result: Result<usize, String>,    // bytes fetched, or an error message
}

async fn process(job: &Job) -> Result<usize, String> {
    // the slow work: a network call, a decode, a query
    tokio::time::sleep(std::time::Duration::from_millis(50)).await;
    Ok(job.url.len())
}

pub async fn run_pipeline(urls: Vec<String>, workers: usize) -> Vec<Outcome> {
    let (job_tx, job_rx) = mpsc::channel::<Job>(workers * 2);          // small buffer: backpressure
    let (out_tx, mut out_rx) = mpsc::channel::<Outcome>(workers * 2);
    let job_rx = Arc::new(Mutex::new(job_rx));                         // workers share one receiver

    // fan out: N workers pulling from the same queue
    let mut handles = Vec::new();
    for _ in 0..workers {
        let (rx, tx) = (job_rx.clone(), out_tx.clone());
        handles.push(tokio::spawn(async move {
            loop {
                let job = { rx.lock().await.recv().await };           // hold the lock only while waiting for a job
                let Some(job) = job else { break };                   // channel closed and drained
                let result = process(&job).await;
                if tx.send(Outcome { index: job.index, result }).await.is_err() {
                    break;                                            // collector gone: stop
                }
            }
        }));
    }
    drop(out_tx);                                                      // keep only the workers' clones

    // producer
    let producer = tokio::spawn(async move {
        for (index, url) in urls.into_iter().enumerate() {
            if job_tx.send(Job { index, url }).await.is_err() {
                break;
            }
        }
        // job_tx dropped here: workers see the end of input
    });

    // fan in: collect until every worker has dropped its sender
    let mut results = Vec::new();
    while let Some(o) = out_rx.recv().await {
        results.push(o);
    }
    let _ = producer.await;
    for h in handles {
        let _ = h.await;
    }
    results.sort_by_key(|o| o.index);                                  // restore input order
    results
}

The pieces to notice: the job channel has a small capacity, so if workers are slow the producer blocks on send and memory stays bounded. Workers share the receiver through a mutex held only while waiting for the next job, a common pattern for multi-consumer use of Tokio's single-consumer channel (the async-channel crate offers a native multi-consumer channel). Termination flows from the front: when the producer drops job_tx, the queue drains and each worker's recv returns None; workers end and drop their result senders; the collector's recv returns None when the last sender is gone. The explicit drop(out_tx) matters, since a leftover sender would keep the collector waiting forever (why tasks never finish).

The shorter version with streams

rust
use futures::{stream, StreamExt};

pub async fn run_with_streams(urls: Vec<String>, workers: usize) -> Vec<(usize, Result<usize, String>)> {
    stream::iter(urls.into_iter().enumerate())
        .map(|(i, url)| async move {
            tokio::time::sleep(std::time::Duration::from_millis(50)).await;
            (i, Ok::<usize, String>(url.len()))
        })
        .buffer_unordered(workers)        // at most `workers` in flight
        .collect()
        .await
}

buffer_unordered(n) keeps up to n futures running and yields results as they finish. For a single stage of independent work it is the least code and is usually the right first choice (batching and pipelining). Reach for explicit channels when you have several stages with different concurrency, long-lived workers with state (a connection each), or need to feed work continuously rather than from a finished list.

Design questions to answer up front

  1. Errors: does one failure stop everything (cancel the rest), or is each item's result recorded and the run continues? Carry Result in the outcome, as above, so the collector decides.
  2. Order: results arrive in completion order. Attach the input index and sort at the end, or use buffered (ordered, but a slow head item delays emission) instead of buffer_unordered.
  3. Cancellation: accept a shutdown signal; workers should stop pulling new jobs and finish or abort the current one (cancellation safety, graceful shutdown). A JoinSet aborts its tasks when dropped.
  4. Concurrency level: set by the scarcest downstream resource: the API's rate limit, the database pool size, file descriptors (semaphores).
  5. Retries: inside the worker, with backoff, bounded attempts and idempotent work (timeouts and retries).
  6. Panics: a panic in a worker task ends that worker silently; check the join results and respawn or fail the run.

Fan out in a service

Inside a request handler, a bounded pipeline is a way to enrich many records without flooding a dependency: the handler awaits the collector, and the concurrency limit protects the dependency from a single large request. For work that outlives the request, hand it to a durable queue instead of in-process channels, because a restart loses whatever was in memory (background job queues, SKIP LOCKED queues).

Test it

  • All inputs produce exactly one outcome (count and set equality).
  • The pipeline terminates when the input is empty and when the input is large (use paused time for slow workers, testing without sleeping).
  • A failing item is reported, and others still complete.
  • Maximum concurrency never exceeds the worker count (track an atomic high-water mark inside process).

Used with discipline, channels give you bounded, observable concurrency with few moving parts; the rules are bound every queue, end by dropping senders and decide error and order semantics before writing code (backpressure with bounded channels).

Taggedchannels
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.