Skip to content
rusthacks.dev
Tooling

Debugging Deadlocks in Rust With Thread Dumps and Traces

A Rust program that hangs is often a deadlock. Capture every thread's stack with gdb, parking_lot deadlock detection or tokio-console, and read who waits for what.

3 min read
Summarize in AI
Flat illustration for the post “Debugging Deadlocks in Rust With Thread Dumps and Traces”, composed around async.

A hung process makes no noise: no panic, no log, CPU idle. The diagnosis is to look at what every thread is doing. Everything applies to Rust 1.90 on Linux with gdb 15 and parking_lot 0.12.

A deadlock to find

rust
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;

fn main() {
    let a = Arc::new(Mutex::new(0));
    let b = Arc::new(Mutex::new(0));

    let (a1, b1) = (Arc::clone(&a), Arc::clone(&b));
    let t1 = thread::spawn(move || {
        let _ga = a1.lock().unwrap();
        thread::sleep(Duration::from_millis(100));
        let _gb = b1.lock().unwrap();
    });

    let (a2, b2) = (Arc::clone(&a), Arc::clone(&b));
    let t2 = thread::spawn(move || {
        let _gb = b2.lock().unwrap();
        thread::sleep(Duration::from_millis(100));
        let _ga = a2.lock().unwrap();
    });

    t1.join().unwrap();
    t2.join().unwrap();
}

Thread 1 takes a then wants b; thread 2 takes b then wants a. Both wait forever.

Step 1: confirm

Shell
top -H -p "$(pgrep app)"

Idle threads at 0% CPU, no I/O: a deadlock or a missing wakeup. A thread at 100% is spinning, a different bug.

Step 2: dump all stacks

Shell
gdb -p "$(pgrep app)" -batch -ex "thread apply all bt" > dump.txt

(rust-gdb for pretty printers.) Look for threads stopped in lock functions:

Text
Thread 3 (LWP 4242):
#0  futex_wait ...
#1  std::sys::sync::mutex::futex::Mutex::lock_contended
#2  std::sync::poison::mutex::Mutex<T>::lock
#3  app::main::{{closure}} at src/main.rs:20
Thread 2 (LWP 4241):
#0  futex_wait ...
#1  std::sys::sync::mutex::futex::Mutex::lock_contended
#3  app::main::{{closure}} at src/main.rs:12

Two threads blocked in lock, at different lines, each in code that holds the other's lock. The source lines tell you which locks. This view is the same for a core file (core dumps). On macOS, sample PID or lldb's thread backtrace all do the job.

Step 3: automatic detection with parking_lot

Cargo.tomlTOML
[dependencies]
parking_lot = { version = "0.12", features = ["deadlock_detection"] }
rust
use std::{thread, time::Duration};

fn spawn_deadlock_checker() {
    thread::spawn(|| loop {
        thread::sleep(Duration::from_secs(10));
        let deadlocks = parking_lot::deadlock::check_deadlock();
        if deadlocks.is_empty() {
            continue;
        }
        eprintln!("{} deadlock(s) detected", deadlocks.len());
        for (i, threads) in deadlocks.iter().enumerate() {
            for t in threads {
                eprintln!("deadlock #{i}, thread {:?}\n{:#?}", t.thread_id(), t.backtrace());
            }
        }
    });
}

Use parking_lot::Mutex in place of std::sync::Mutex (same API, no poisoning) and call the checker from main. It finds cycles among parking_lot locks only. Enable the feature in debug and test builds; it adds overhead.

Async hangs

A hung async service usually has no blocked thread at all: tasks are waiting for something that will never happen (a channel nobody sends on, a lock never released, a future never woken). Stack dumps show worker threads idle in the executor. Use tokio-console (with console-subscriber and RUSTFLAGS="--cfg tokio_unstable") to list tasks, their poll counts and the time since they were last polled:

Text
ID  STATE  TOTAL  BUSY  IDLE  POLLS  LOCATION
17  idle   42m    0ms   42m   2      src/worker.rs:51

A task idle for 42 minutes with two polls is the waiting one. A blocked runtime thread is the other cause: a std Mutex held across .await or CPU work on a worker (guard across await, blocking the runtime).

Prevention

1. Fix a global lock order and always acquire in it. 2. Hold locks for short, simple operations; copy data out and release before calling anything else. 3. Never call unknown code (callbacks) while holding a lock. 4. Prefer message passing and single owner tasks (sharing state). 5. Use try_lock_for with a timeout in places where you can recover.

Taggeddeadlock
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.