Benchmarking Rust CLIs Against Shell Pipelines Fairly

A Rust CLI that beats grep | sort | uniq -c by a factor is a good thing to claim and an easy thing to claim wrongly. The tool that makes honest comparison easy is hyperfine. This post shows how to use it and which mistakes turn a benchmark into marketing. The commands were written for hyperfine 1.19 on Linux and macOS; your timings will differ and are not quoted here.

The basic comparison

cargo install --locked hyperfine

hyperfine --warmup 3 \
  'grep -c ERROR big.log' \
  './target/release/mytool count ERROR big.log'

It prints, for each command, the mean time with standard deviation, the min and max, and the number of runs, then a summary such as "mytool ran N times faster than grep". Build the Rust side with --release: a debug build can be 10 to 50 times slower and makes the whole comparison meaningless.

Decide: warm or cold

• Warm cache (the default after warmup): the input is in memory, so you measure parsing and processing speed. This is the case for repeated runs and CI.
• Cold cache: the first read from disk. It measures I/O. On Linux:

hyperfine --runs 5 \
  --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' \
  'grep -c ERROR big.log' './target/release/mytool count ERROR big.log'

On macOS use sudo purge in the prepare step. Say which one you measured, because both are legitimate and answer different questions.

Do not time the shell

hyperfine -N --warmup 5 './target/release/mytool --version' 'grep --version'

By default hyperfine runs commands through a shell, which costs a millisecond or two. For fast commands that overhead is the measurement. -N (--shell=none) runs the program directly. When comparing a pipeline you need the shell, and hyperfine subtracts its estimate of the startup cost.

Sweeps and parameters

hyperfine --warmup 2 \
  --parameter-list threads 1,2,4,8 \
  './target/release/mytool --threads {threads} count ERROR big.log'

hyperfine --warmup 2 --export-markdown results.md --export-json results.json \
  'grep -c ERROR big.log' './target/release/mytool count ERROR big.log'

A parameter list shows scaling with threads or input size in one report. The export options give you a Markdown table for a README and JSON for charts. Keep the JSON with the commit that produced it so a later regression can be compared.

Make it fair

1. Same input, same output. Run both commands once and compare results. A faster tool that counts differently is just wrong. -i in grep versus case-sensitive search, Unicode versus bytes and line-ending handling all change work done.
2. Same features. Do not compare a regex engine with Unicode tables to one in ASCII mode without saying so.
3. Realistic data. A 10 KB file measures process startup. Use sizes your users have: hundreds of megabytes for a log tool.
4. Quiet machine. Close other work, plug in a laptop, and avoid thermal throttling: sort the runs and check the max.
5. Check variance. If the standard deviation is a large share of the mean, find the noise or increase runs; hyperfine warns about outliers.
6. Report the setup. Hardware, OS, file size, tool versions, command lines.

Pipelines versus a single tool

hyperfine --warmup 2 \
  'grep ERROR big.log | cut -d" " -f3 | sort | uniq -c | sort -rn | head' \
  './target/release/mytool top-field ERROR big.log --field 3 --limit 10'

A pipeline runs several processes concurrently and uses multiple cores, so the comparison is "one tool on one or more threads" against "five processes". That is fair as a user-facing result, but know what you are measuring. Also check memory (/usr/bin/time -v or -l on macOS): a fast tool that uses ten times the memory is a trade, not a win.

After the measurement

If the numbers disappoint, profile instead of guessing: a flamegraph shows where the time goes (profiling with flamegraphs), and the usual wins in text tools are buffered I/O, avoiding per-line allocation and memory-mapped reads (reading large files). For in-process timing of functions, use Criterion or Divan rather than hyperfine (benchmarking pitfalls).