Flaky Tests in Rust and Seven Usual Causes to Check

A flaky test costs more than a failing one, because people learn to rerun it and then to ignore the red. In Rust the compiler prevents data races on memory, which leaves the races on everything else: files, ports, environment and time. These seven causes account for nearly all the flakes we expect to see, and each has a one-line check. This post is the diagnostic piece of the testing in Rust guide.

Rust 1.90, edition 2024.

First, confirm it

cargo test -- --test-threads=1
for i in $(seq 50); do cargo test my_test -q || break; done

If the failure only appears with parallel threads, a shared resource is involved. If it appears even alone, look at time, randomness and ordering.

1. Shared global state

use std::sync::atomic::{AtomicUsize, Ordering};
static COUNTER: AtomicUsize = AtomicUsize::new(0);

#[test]
fn a() { COUNTER.fetch_add(1, Ordering::SeqCst); assert_eq!(COUNTER.load(Ordering::SeqCst), 1); }
#[test]
fn b() { COUNTER.fetch_add(1, Ordering::SeqCst); assert_eq!(COUNTER.load(Ordering::SeqCst), 1); }

Whichever runs second sees 2. Fix: pass state in, do not use statics, or run tests in separate processes with nextest.

2. Fixed ports

Two tests binding 127.0.0.1:3000 collide with Address already in use. Bind port 0 and read the assigned port (address already in use).

3. The clock

Instant::now() comparisons with tight margins, SystemTime::now() at a day boundary, and sleep(10ms) expecting something to have happened all fail under load. Inject a clock trait for logic, and for async code use Tokio's paused time (#[tokio::test(start_paused = true)]).

4. Unordered output

let mut names: Vec<_> = map.keys().cloned().collect();
names.sort();
assert_eq!(names, ["a", "b"]);

HashMap and HashSet iterate in a different order in every process. Sort before asserting, or compare as sets.

5. Thread races

A test that spawns a thread and then checks a result after sleep is guessing. Join the thread, or wait on a channel with recv_timeout. For lock-free code, use Loom to explore interleavings.

6. Files and directories

Tests writing out.txt in the working directory collide. Use tempfile::tempdir() per test, which also cleans up. Fixtures should be read-only.

7. The environment

std::env::set_var changes the whole process; in edition 2024 it is unsafe, because it races with other threads reading the environment. Never set variables in parallel tests. Pass configuration explicitly, or serialize those tests with serial_test.

Make flakes visible

In CI, report retried tests (nextest marks FLAKY), keep a list and treat each as a bug. A random seed in a property test should be printed on failure so you can reproduce it; proptest stores it (property testing).

Limits

Some flakes come from real nondeterminism in the code under test, such as a race in production code. That is the useful case: the test found a bug. Do not "fix" it by adding a sleep. Rust's test harness behavior is described in the book under controlling how tests are run.