Skip to content
rusthacks.dev
Performance

Benchmarking Pitfalls in Rust and Dead Code Removal

Benchmarks fail by measuring nothing: the optimizer deletes the work, the input is unrealistic, the machine is noisy or the harness measures setup. Here are the traps and how to avoid each.

3 min read
Summarize in AI
Flat illustration for the post “Benchmarking Pitfalls in Rust and Dead Code Removal”, composed around perf.

Every benchmark is a claim about speed, and many are wrong in the same few ways. The optimizer is the first culprit: it sees that you throw the result away and deletes the computation, so your function runs in zero time. The rest are variations of measuring something other than the real workload. This post lists six traps with the fix for each. It follows profiling Rust programs in the performance cluster, and more tooling is in the testing posts.

Rust 1.90, criterion 0.5.

1. The work disappears

rust
use std::time::Instant;

fn fib(n: u32) -> u64 { if n < 2 { n as u64 } else { fib(n - 1) + fib(n - 2) } }

fn main() {
    let t = Instant::now();
    for _ in 0..1000 {
        fib(20);
    }
    println!("{:?}", t.elapsed());
}

In release mode, fib(20) has a fixed argument and an unused result. LLVM may evaluate it at compile time, or remove the call. The elapsed time then says nothing about fib. The fix:

rust
use std::hint::black_box;
for _ in 0..1000 {
    black_box(fib(black_box(20)));
}

black_box makes the compiler assume the value could be anything and could be used. It does not add a measurable cost of its own beyond forcing a store.

2. A debug build

cargo bench builds in release, but cargo run does not. A timing loop in main run with plain cargo run measures opt-level = 0. See release versus debug.

3. Setup inside the timed region

Building the input, opening a file or allocating a Vec inside the loop measures them too. Criterion's iter_batched separates setup from the routine:

rust
use criterion::{criterion_group, criterion_main, BatchSize, Criterion};
use std::hint::black_box;

fn bench(c: &mut Criterion) {
    c.bench_function("sort 10k", |b| {
        b.iter_batched(
            || (0..10_000u32).rev().collect::<Vec<_>>(),
            |mut v| { v.sort(); black_box(v) },
            BatchSize::SmallInput,
        )
    });
}

criterion_group!(benches, bench);
criterion_main!(benches);

The setup closure runs outside the timing. Reverse sorted data is one shape; use data that looks like production.

4. Unrealistic inputs

Tiny inputs stay in L1 cache and give branch predictors an easy time. Sorted or constant data hides branch mispredictions. If production sorts 10 million random-ish records, benchmarking 100 sorted ones tells you little. Parametrize benchmarks by size to see where behavior changes.

5. Noise

Another process, CPU frequency scaling, thermal limits and a laptop on battery all move numbers by more than the effect you are chasing. Close other programs, run several times, prefer a desktop or a dedicated machine, and trust the harness's confidence intervals. A change that Criterion reports as "no change detected" is no change.

6. Measuring the wrong thing

Benchmarking a function in isolation can be fast and the same function inside the program slow, because the cache is cold, the data is spread across memory or the allocator behaves differently. Use the benchmark to compare two implementations, and the profiler to understand the real program.

A checklist

  • Release build with production settings
  • black_box on inputs and results
  • Setup outside the timed region
  • Realistic size and shape of data
  • Several runs, a statistical harness, a quiet machine
  • A profile or assembly check when a result looks too good

Limits

Microbenchmarks cannot tell you about contention, I/O or memory pressure. For services, load test the real thing. The tools are documented at Criterion.rs and in the standard library's page for black_box.

Also inTesting
Taggedbenchmark
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.