Benchmarking Pitfalls in Rust and Dead Code Removal
Benchmarks fail by measuring nothing: the optimizer deletes the work, the input is unrealistic, the machine is noisy or the harness measures setup. Here are the traps and how to avoid each.

Every benchmark is a claim about speed, and many are wrong in the same few ways. The optimizer is the first culprit: it sees that you throw the result away and deletes the computation, so your function runs in zero time. The rest are variations of measuring something other than the real workload. This post lists six traps with the fix for each. It follows profiling Rust programs in the performance cluster, and more tooling is in the testing posts.
Rust 1.90, criterion 0.5.
1. The work disappears
use std::time::Instant;
fn fib(n: u32) -> u64 { if n < 2 { n as u64 } else { fib(n - 1) + fib(n - 2) } }
fn main() {
let t = Instant::now();
for _ in 0..1000 {
fib(20);
}
println!("{:?}", t.elapsed());
}In release mode, fib(20) has a fixed argument and an unused result. LLVM may evaluate it at compile time, or remove the call. The elapsed time then says nothing about fib. The fix:
use std::hint::black_box;
for _ in 0..1000 {
black_box(fib(black_box(20)));
}black_box makes the compiler assume the value could be anything and could be used. It does not add a measurable cost of its own beyond forcing a store.
2. A debug build
cargo bench builds in release, but cargo run does not. A timing loop in main run with plain cargo run measures opt-level = 0. See release versus debug.
3. Setup inside the timed region
Building the input, opening a file or allocating a Vec inside the loop measures them too. Criterion's iter_batched separates setup from the routine:
use criterion::{criterion_group, criterion_main, BatchSize, Criterion};
use std::hint::black_box;
fn bench(c: &mut Criterion) {
c.bench_function("sort 10k", |b| {
b.iter_batched(
|| (0..10_000u32).rev().collect::<Vec<_>>(),
|mut v| { v.sort(); black_box(v) },
BatchSize::SmallInput,
)
});
}
criterion_group!(benches, bench);
criterion_main!(benches);The setup closure runs outside the timing. Reverse sorted data is one shape; use data that looks like production.
4. Unrealistic inputs
Tiny inputs stay in L1 cache and give branch predictors an easy time. Sorted or constant data hides branch mispredictions. If production sorts 10 million random-ish records, benchmarking 100 sorted ones tells you little. Parametrize benchmarks by size to see where behavior changes.
5. Noise
Another process, CPU frequency scaling, thermal limits and a laptop on battery all move numbers by more than the effect you are chasing. Close other programs, run several times, prefer a desktop or a dedicated machine, and trust the harness's confidence intervals. A change that Criterion reports as "no change detected" is no change.
6. Measuring the wrong thing
Benchmarking a function in isolation can be fast and the same function inside the program slow, because the cache is cold, the data is spread across memory or the allocator behaves differently. Use the benchmark to compare two implementations, and the profiler to understand the real program.
A checklist
- Release build with production settings
black_boxon inputs and results- Setup outside the timed region
- Realistic size and shape of data
- Several runs, a statistical harness, a quiet machine
- A profile or assembly check when a result looks too good
Limits
Microbenchmarks cannot tell you about contention, I/O or memory pressure. For services, load test the real thing. The tools are documented at Criterion.rs and in the standard library's page for black_box.


