Sharing State in Tokio With Arc Mutex and Message Passing

Spawned tasks cannot borrow from each other or from their parent, so the moment two tasks need the same data (a counter, a cache, a connection pool, a configuration), you must pick how they share it. Rust offers four answers: put it behind an Arc with a lock, use atomics for single numbers, give it to one task and talk to that task through a channel, or publish immutable snapshots. They trade simplicity, speed and the ability to hold the state across an .await. This post builds the same shared counter and shared map each way, measures the cost of 2 million contended updates on one machine, and ends with a rule for picking. Everything compiled on Rust 1.99, edition 2024, with tokio 1.

Why Arc, and why it is not enough

A task spawned with tokio::spawn must be 'static, so it cannot hold a reference to a local. Wrapping the data in an Arc gives every task its own owning handle to the same allocation. Arc alone gives shared read access; the contents are immutable. To change them you need interior mutability, which is a lock, an atomic, or a channel to an owner. The reasoning for why Arc is needed at all is in why spawn needs a static future.

Option 1: Arc around a Mutex

The default for shared mutable state:

use std::collections::HashMap;
use std::sync::{Arc, Mutex};

#[tokio::main]
async fn main() {
    let hits: Arc<Mutex<HashMap<String, u32>>> = Arc::new(Mutex::new(HashMap::new()));

    let mut handles = Vec::new();
    for worker in 0..4 {
        let hits = Arc::clone(&hits);
        handles.push(tokio::spawn(async move {
            for page in ["home", "docs", "home"] {
                let mut map = hits.lock().unwrap();
                *map.entry(page.to_string()).or_default() += 1;
            }
            worker
        }));
    }
    for h in handles {
        h.await.unwrap();
    }

    let map = hits.lock().unwrap();
    let mut rows: Vec<_> = map.iter().collect();
    rows.sort();
    println!("{rows:?}");
}

[("docs", 4), ("home", 8)]

This uses std::sync::Mutex, which is the right choice here because the guard is dropped before any .await and the critical section is tiny. The choice between that and tokio::sync::Mutex is the subject of which mutex, and the trap when it goes wrong is in holding a guard across an await.

Option 2: atomics, for a single number

A counter, a flag or a high-water mark needs no lock. AtomicU64 updates in one instruction and cannot deadlock, but it can hold exactly one machine word:

use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::Arc;

struct Stats {
    requests: AtomicU64,
    peak_size: AtomicU64,
    draining: AtomicBool,
}

#[tokio::main]
async fn main() {
    let stats = Arc::new(Stats { requests: AtomicU64::new(0), peak_size: AtomicU64::new(0), draining: AtomicBool::new(false) });

    let handles: Vec<_> = [120u64, 4096, 77, 900]
        .into_iter()
        .map(|size| {
            let s = stats.clone();
            tokio::spawn(async move {
                s.requests.fetch_add(1, Ordering::Relaxed);
                s.peak_size.fetch_max(size, Ordering::Relaxed);
            })
        })
        .collect();
    for h in handles {
        h.await.unwrap();
    }
    stats.draining.store(true, Ordering::Release);

    println!(
        "requests={} peak={} draining={}",
        stats.requests.load(Ordering::Relaxed),
        stats.peak_size.load(Ordering::Relaxed),
        stats.draining.load(Ordering::Acquire)
    );
}

requests=4 peak=4096 draining=true

Relaxed is correct for counters whose value is only reported; the flag uses release and acquire so that writes before the flag are visible to a reader who sees it set. Anything that needs two fields to change together (a count and a total) cannot be done with two atomics, because a reader can see one updated and the other not. Use a lock for that.

Option 3: one owner, many senders (the actor)

Instead of sharing the data, give it to one task and let everyone else send messages. There is no lock to hold, no guard to leak across an await, and the owner may freely await in the middle of handling a message. A request with an answer carries a oneshot for the reply:

use std::collections::HashMap;
use tokio::sync::{mpsc, oneshot};

enum Msg {
    Hit(String),
    Get(String, oneshot::Sender<u32>),
}

#[derive(Clone)]
struct Counters {
    tx: mpsc::Sender<Msg>,
}

impl Counters {
    fn spawn() -> Self {
        let (tx, mut rx) = mpsc::channel(64);
        tokio::spawn(async move {
            let mut map: HashMap<String, u32> = HashMap::new();
            while let Some(msg) = rx.recv().await {
                match msg {
                    Msg::Hit(k) => *map.entry(k).or_default() += 1,
                    Msg::Get(k, reply) => {
                        let _ = reply.send(map.get(&k).copied().unwrap_or(0));
                    }
                }
            }
        });
        Counters { tx }
    }

    async fn hit(&self, key: &str) {
        let _ = self.tx.send(Msg::Hit(key.to_string())).await;
    }

    async fn get(&self, key: &str) -> u32 {
        let (reply, answer) = oneshot::channel();
        self.tx.send(Msg::Get(key.to_string(), reply)).await.unwrap();
        answer.await.unwrap()
    }
}

#[tokio::main]
async fn main() {
    let counters = Counters::spawn();
    let mut handles = Vec::new();
    for _ in 0..4 {
        let c = counters.clone();
        handles.push(tokio::spawn(async move {
            c.hit("home").await;
            c.hit("home").await;
            c.hit("docs").await;
        }));
    }
    for h in handles {
        h.await.unwrap();
    }
    println!("home={} docs={} missing={}", counters.get("home").await, counters.get("docs").await, counters.get("nope").await);
}

home=8 docs=4 missing=0

The handle type Counters is cheap to clone and hides the channel, so callers see ordinary async methods. Because the owner processes messages one at a time, the map is consistent without any locking. The cost is a channel hop per operation, and a bounded channel makes producers wait when the owner is slow, a property covered in bounded channels and backpressure.

Option 4: share immutable snapshots

For read-mostly data such as configuration, do not lock at all: publish a new immutable value each time it changes. A watch channel carries the latest value to every task, and readers clone an Arc and keep working with a consistent snapshot while writers replace it:

use std::sync::Arc;
use tokio::sync::watch;

#[derive(Debug)]
struct Config {
    version: u32,
    limit: usize,
}

#[tokio::main(flavor = "current_thread")]
async fn main() {
    let (tx, rx) = watch::channel(Arc::new(Config { version: 1, limit: 10 }));

    let reader = {
        let rx = rx.clone();
        move || {
            let snapshot: Arc<Config> = rx.borrow().clone();
            snapshot
        }
    };

    let before = reader();
    tx.send(Arc::new(Config { version: 2, limit: 50 })).unwrap();
    let after = reader();

    println!("{before:?}");
    println!("{after:?}");
}

Config { version: 1, limit: 10 }
Config { version: 2, limit: 50 }

A task that took a snapshot keeps seeing version 1 for the duration of its work, which is what you want when a request must use one configuration throughout. The writer builds a whole new value, so there are no half-updated states. The crate arc-swap offers the same idea without a channel and with lock-free reads, for hot paths.

What it costs

The measurement used 8 tasks making 250,000 increments each (2 million in all) to one shared number, on an Apple M1 Max in a release build, best of three rounds. Every task ran its loop without yielding, which is the most contended pattern for the lock types:

The surprise is the atomic: it lost to the plain mutex, because eight cores hammering one cache line make the line bounce between them, and a mutex held for a few nanoseconds serialises them more cheaply. Atomics win when updates are rare or spread across many counters; for a single hot word, they are not magic. The async mutex and the channel cost five to six times as much per operation as the std mutex, which is the price of being able to wait without blocking a thread. All five are cheap next to the work they guard: 100 nanoseconds is a thousandth of the time of a network round trip. Pick for correctness and for whether you must hold the state across an await, and do not chase these numbers unless a profile points at the lock.

One more difference shows up only when something goes wrong. A std::sync::Mutex is poisoned when a thread panics while holding it: later lock() calls return an error, and the usual .lock().unwrap() then panics too, so one bug takes down every task that touches the lock. That is often the behaviour you want, since the protected data may be half-updated. The Tokio mutex does not poison, so a panic mid-update leaves the data in whatever state it reached and later tasks use it as if nothing happened. An actor restarts cleanly after a panic, because the owner task is the only place the state lives and you decide whether to rebuild it.

Choosing

Reach for the actor when the state has behaviour: invariants across fields, expiry, batching, or I/O in the middle of an update. Reach for a lock when it is a bag of data with trivial operations. And avoid the hybrid that appears in many codebases, a lock that guards a handle to another service: the lock then serialises calls to that service, and a channel to a task that owns the handle does the same job with queueing and backpressure built in.