Skip to content
rusthacks.dev
Web

Load Testing a Rust Service With Oha and Grafana k6

A load test is only useful if it models real traffic and reports percentiles. oha gives a quick check, and k6 scripts realistic user flows with thresholds.

14 min read
Summarize in AI
Flat illustration for the post “Load Testing a Rust Service With Oha and Grafana k6”, composed around testing.

"It handles 20,000 requests per second" is not a measurement until you can finish the sentence: which endpoint, what payload, how many connections, and what the 99th percentile latency was while it did. A load test that answers those questions tells you where your service breaks before your users do. A load test that does not can produce a comforting number that is wrong by a factor of twenty, and the rest of this guide shows how that happens with a program you can run.

The tools are oha for quick checks and Grafana k6 for scripted scenarios, run against an Axum 0.8 service built with Rust 1.99. The measurement-error demonstration uses only tokio 1.54.

Set the test up so the number means something

Most bad load tests fail before the first request, in the setup. Five rules cover nearly all of it.

  1. Test a release build. A debug build can be 10 to 50 times slower on hot paths, so the result describes the missing optimizer rather than your code.
  2. Use production-shaped data. An endpoint that scans an indexed table of a thousand rows is a different program from the same endpoint over fifty million.
  3. Run the generator on a different machine. A load generator sharing a host with the service steals its CPU, and on a laptop you are measuring the contention between the two.
  4. Warm up. The first seconds build connection pools, fill caches and fault in pages. Run one minute of load and throw it away.
  5. Hold everything else still. Same build, same configuration, same instance size, same data between two runs, or the comparison between them carries no information.

Also decide what you are asking. "How fast can it go" and "does it meet 300 requests per second at p99 under 150 ms" are different tests with different tools. The second one has a pass or fail answer, and it is the one worth automating.

The mistake that hides your worst latency

Here is the part that most guides skip and that matters more than any tool choice. There are two ways a load generator can send traffic.

A closed loop model runs N workers, and each sends its next request only after the previous one returns. An open loop model sends requests on a schedule, say 250 per second, whether or not earlier ones have finished. Real users behave like the second: they do not wait politely for your slow response before arriving.

The difference is invisible while the service is healthy and enormous when it stalls. In a closed loop, a stall also stalls the generator, so it stops sending requests exactly while the service is in trouble. The slow requests that should have been issued during the stall are never sent, so they never show up in the percentiles. This is called coordinated omission, and it makes the tail look far better than it is.

The program below simulates it. The "service" has four concurrent slots and answers in 10 ms, except that at request 100 the whole service freezes for 400 ms, like a garbage collection pause, a lock or a stuck disk. We measure it twice with the same service: once closed loop with four workers, once open loop at 250 requests per second, with each latency measured from the moment the request was due to be sent.

src/main.rsrust
use std::sync::{Arc, Mutex};
use std::sync::atomic::{AtomicU32, Ordering};
use std::time::{Duration, Instant};
use tokio::sync::Semaphore;
use tokio::time::{interval, sleep, MissedTickBehavior};

// A fake service: 4 concurrent slots, 10 ms per request, and a 400 ms freeze
// of the whole service that starts at request 100 (a GC pause, a lock, a slow disk).
struct Service {
    slots: Semaphore,
    count: AtomicU32,
    frozen_until: Mutex<Option<Instant>>,
}

impl Service {
    async fn call(&self) {
        let _permit = self.slots.acquire().await.unwrap();
        let n = self.count.fetch_add(1, Ordering::Relaxed);
        if n == 100 {
            *self.frozen_until.lock().unwrap() = Some(Instant::now() + Duration::from_millis(400));
        }
        let frozen = *self.frozen_until.lock().unwrap();
        if let Some(until) = frozen {
            tokio::time::sleep_until(until.into()).await;
        }
        sleep(Duration::from_millis(10)).await;
    }
}

fn percentile(sorted: &[Duration], p: f64) -> Duration {
    let i = ((sorted.len() as f64 - 1.0) * p).round() as usize;
    sorted[i]
}

fn report(label: &str, mut v: Vec<Duration>) {
    v.sort();
    println!(
        "{label:<12} n={:<5} p50={:>6.1?} p95={:>6.1?} p99={:>6.1?} max={:>6.1?}",
        v.len(), percentile(&v, 0.50), percentile(&v, 0.95), percentile(&v, 0.99), v[v.len() - 1]
    );
}

#[tokio::main]
async fn main() {
    // Closed loop: 4 workers, each sends the next request after the last returns.
    let svc = Arc::new(Service { slots: Semaphore::new(4), count: AtomicU32::new(0), frozen_until: Mutex::new(None) });
    let mut handles = Vec::new();
    for _ in 0..4 {
        let svc = svc.clone();
        handles.push(tokio::spawn(async move {
            let mut lat = Vec::new();
            for _ in 0..500 {
                let t = Instant::now();
                svc.call().await;
                lat.push(t.elapsed());
            }
            lat
        }));
    }
    let mut closed = Vec::new();
    for h in handles { closed.extend(h.await.unwrap()); }
    report("closed loop", closed);

    // Open loop: a request is due every 4 ms (250/s) no matter what, and
    // latency is measured from the moment it was due.
    let svc = Arc::new(Service { slots: Semaphore::new(4), count: AtomicU32::new(0), frozen_until: Mutex::new(None) });
    let mut tick = interval(Duration::from_millis(4));
    tick.set_missed_tick_behavior(MissedTickBehavior::Burst);
    let mut handles = Vec::new();
    for _ in 0..2000 {
        let due = tick.tick().await;
        let svc = svc.clone();
        handles.push(tokio::spawn(async move {
            svc.call().await;
            due.elapsed()
        }));
    }
    let mut open = Vec::new();
    for h in handles { open.push(h.await.unwrap()); }
    report("open loop", open);
}
Text
closed loop  n=2000  p50=12.0ms p95=13.1ms p99=15.9ms max=413.0ms
open loop    n=2000  p50=12.9ms p95=299.0ms p99=388.7ms max=414.5ms

Same service, same freeze, two stories. The closed loop reports a p99 of 15.9 ms. The open loop reports 388.7 ms, because during the 400 ms freeze it kept scheduling a request every 4 ms, about a hundred of them, and each one waited for the unfreeze. The closed loop recorded only the four requests that happened to be in flight. If your dashboards said p99 was 16 ms while users saw four-hundred-millisecond stalls, this is the reason.

The code worth reading carefully is due.elapsed(): the clock starts at the intended send time, not when the task finally ran. Tools that call themselves open loop but start the timer when the request is sent still omit the queueing delay in the generator. MissedTickBehavior::Burst matters for the same reason, since the default delay behavior would also shift the schedule when the generator falls behind.

The practical rule is simple. Prefer a fixed arrival rate when you test for latency, and use closed loop (fixed concurrency) only when you test for maximum throughput and understand that its tail is optimistic.

Quick checks with oha

oha is a single binary that sends HTTP load and prints a latency histogram. It is the right tool for "did my change make this endpoint slower". Install it with cargo install oha, build the service with cargo build --release, start it, and fire from another machine.

Shell
cargo install oha
oha -z 30s -c 50 --no-tui http://10.0.0.12:3000/search?q=rust

-z 30s is the duration and -c 50 is the number of concurrent connections, which makes this a closed loop run. The report has a summary with requests per second and success rate, a latency histogram, a latency distribution with the 50th, 95th and 99th percentiles, and a count of responses per status code. Copy those percentiles, not the requests per second line, into your notes.

To switch to a fixed arrival rate, add -q 2000 for 2,000 queries per second. To send a body, add -m POST -d '{"query":"rust"}' -H 'content-type: application/json'. Always read the status code distribution first: a run with 40 percent 503 responses can show wonderful latency because errors return instantly.

Scenarios with k6

oha hits one URL. Real traffic is a mix: most people read, some search, a few write. k6 scripts that in JavaScript and, importantly, lets you put pass/fail thresholds in the script so a regression fails a CI job.

load/search.jsJavaScript
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    steady: {
      executor: 'constant-arrival-rate',
      rate: 300,
      timeUnit: '1s',
      duration: '5m',
      preAllocatedVUs: 100,
      maxVUs: 400,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<150', 'p(99)<400'],
  },
};

const BASE = __ENV.BASE_URL || 'http://10.0.0.12:3000';

export default function () {
  const roll = Math.random();
  const res = roll < 0.8
    ? http.get(`${BASE}/posts/rust-ownership`)
    : http.get(`${BASE}/search?q=lifetimes`);
  check(res, { 'status is 200': (r) => r.status === 200 });
}

constant-arrival-rate is the open loop executor: k6 starts 300 iterations per second and allocates more virtual users when the service slows, up to maxVUs. If k6 runs out of virtual users it prints a warning about dropped iterations, which is the generator admitting that it could no longer keep the schedule. Treat that warning as a failed test, since the same coordinated omission has crept back in.

Run it with k6 run -e BASE_URL=http://10.0.0.12:3000 load/search.js. The thresholds turn the run into a verdict: k6 exits non-zero when p95 exceeds 150 ms or more than 1 percent of requests fail, so a staging pipeline can gate a release on it.

Test shapes for different questions

TestQuestion it answersHow
SmokeDoes it work at allOne virtual user, a minute
LoadDoes it hold the expected peakArrival rate at forecast peak, 10 to 30 minutes
StressWhere does it break, and howRamp the rate until errors or latency cross your limit
SoakDoes it leak or driftModerate load for hours, watch memory and file descriptors
SpikeHow does it recover from a burstJump from 50 to 500 per second, then back

Run a ramp with ramping-arrival-rate and note the rate at which p99 first crosses your limit. That is the capacity figure you report, with the limit stated. Then look at how it fails. A graceful rise in latency followed by 503 responses from a concurrency limit is good design. A crash and restart loop is a finding.

Read the result against the server's own metrics

Latency tells you something is slow. The server's metrics say why, and the order you look in saves time.

  • CPU pinned near 100 percent across cores means the service is compute bound. Take a profile under load with the method in profiling a slow endpoint.
  • CPU low but latency high means waiting: on the database, on a lock, on a downstream call. Check how long requests wait for a pooled connection.
  • Tokio worker busy time or poll durations climbing means something blocks the async runtime. See runtime metrics.
  • Memory climbing steadily during a soak means a leak or an unbounded cache.

A worked reading: suppose a 300 per second run holds p99 at 90 ms, but at 450 per second p99 jumps to 2 seconds while CPU sits at 35 percent. Low CPU with exploding latency is the signature of a pool limit. If the database pool has 10 connections and each request holds one for 25 ms, the ceiling is 10 / 0.025 = 400 requests per second, and above that requests queue for a connection. The fix is not more CPU, it is a bigger pool, a faster query, or releasing the connection before the slow non-database work. Capacity arithmetic like this predicts where the knee in your graph will be before you run the test, and a test whose knee appears elsewhere is telling you about a second bottleneck.

Size the test with Little's law

Before choosing a concurrency number for -c or maxVUs, calculate it. Little's law says the average number of requests in the system equals the arrival rate multiplied by the average time each spends there. It needs no assumptions about the distribution, and it tells you how many connections your test needs and how many your service must tolerate.

src/main.rsrust
// Little's law: concurrency = arrival rate x time in system.
fn main() {
    // (label, arrival rate per second, mean service time in seconds)
    for (label, rate, service) in [("search", 300.0_f64, 0.045_f64), ("report", 40.0, 0.900)] {
        let in_flight = rate * service;
        println!("{label:<7} {rate:>5.0}/s x {service:.3}s = {in_flight:>5.1} requests in flight");
    }
    // A pool of 10 connections, each request holding one for 25 ms:
    let pool = 10.0_f64;
    let hold = 0.025_f64;
    println!("pool ceiling: {:.0} requests/s", pool / hold);
}
Text
search    300/s x 0.045s =  13.5 requests in flight
report     40/s x 0.900s =  36.0 requests in flight
pool ceiling: 400 requests/s

Two readings come out of those three lines. The slow, rare report endpoint holds more requests in flight (36) than the fast, busy search endpoint (13.5), so a thread pool, a semaphore or a database pool sized only for the hot endpoint will be exhausted by a modest number of reports. And the pool ceiling is the same division run backwards: ten connections held for 25 ms each cannot serve more than 400 requests a second, however many CPU cores sit idle. When a load test shows latency exploding at a rate close to that figure, you have found the bottleneck without a profiler.

For the generator, the same arithmetic sets maxVUs. At 300 arrivals per second with a 150 ms worst case, you need at least 45 virtual users in flight, and a margin of two or three times that so a slow period does not exhaust them and drop iterations.

Percentiles are not additive

A trap that catches careful people: you cannot average percentiles. If one load generator reports a p99 of 80 ms and another reports 120 ms, the combined p99 is not 100 ms. It depends on the full distributions, and it can be higher than both. The same goes for dashboards that average a per-minute p99 into an hourly p99.

When you run distributed load, export the raw histograms or the raw samples and merge those. A histogram with logarithmic buckets, such as an HDR histogram, merges exactly and stays compact, which is why serious tools use it. In k6 you can stream every sample to a time series store with --out and compute percentiles over the union afterward. In your own Rust service, the metrics or prometheus crates record durations in fixed buckets; choose bucket boundaries around your objective (for a 150 ms goal, include 100, 125, 150, 200 and 300 ms) so you can see the fraction under the limit directly instead of estimating it.

The other consequence is about sample size. A p99 computed from 200 requests is the second slowest request and nothing more. For a stable p99 you want at least several thousand samples, and for p99.9 tens of thousands. A thirty second run at 300 per second gives 9,000 samples, enough for p99, thin for p99.9.

What changes between a laptop run and production

Even with a separate generator machine, a few environmental differences quietly change results, and they are worth checking before you trust a capacity number.

  • TLS. If production terminates TLS at a load balancer but your test hits the service over plain HTTP, you have skipped real CPU work, and if you test over TLS from a single source address you can overwhelm the handshake path instead of the handler. Match production's termination point, and reuse connections the way real clients do.
  • Connection reuse. Keep-alive makes a request cheap. A test that opens a new connection per request measures connection setup, which may be your actual problem or may be irrelevant. Know which one you chose.
  • Payload distribution. Real request bodies and response sizes vary. Replaying only the smallest payload hides the memory and serialization cost of the large ones. Sample real, anonymized requests if you can.
  • Instance noise. On shared cloud instances, burstable CPU credits make the first ten minutes faster than the next hour. A short test on a burstable instance reports a capacity the instance cannot sustain.

Put the check in CI, with honest limits

A threshold-gated k6 run belongs in the pipeline, against an environment that resembles production. Keep it short, around two minutes at the expected peak, and treat it as a regression alarm rather than a capacity study.

.github/workflows/load.ymlYAML
name: load
on:
  workflow_dispatch:
  schedule:
    - cron: '0 3 * * *'
jobs:
  k6:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: grafana/setup-k6-action@v1
      - run: k6 run -e BASE_URL=${{ secrets.STAGING_URL }} load/search.js

Shared CI runners are noisy, so the thresholds should be loose enough not to flake and strict enough to catch a real regression, such as a p95 that doubles. Run it nightly and on demand rather than on every commit, and keep the long soak and stress tests for a scheduled job with its own dedicated environment. When a threshold fails, the useful artifact is the profile from the same build under the same load, so wire the failing commit to the method in profiling a slow endpoint.

Mistakes that produce confident wrong answers

  • Testing one cached URL. The load balancer or an application cache serves it from memory and you measure the cache.
  • Reporting the average. A mean of 20 ms is compatible with one request in a hundred taking two seconds. Always show p50, p95 and p99, and the maximum.
  • Letting the generator become the bottleneck. Watch its CPU. If it sits near 100 percent, the numbers describe the generator.
  • Ignoring the error rate. Fast failures flatter latency.
  • Running against production data stores shared with real users, or against a staging database that the other team is migrating at the same time.
  • Comparing runs across days without pinning the instance type, since a noisy neighbor changes the baseline.

When this approach does not fit

Load tests of a service that fronts a third party you pay per call, such as a payments API or an SMS gateway, should stub the third party, or you will spend money and trip their abuse detection. Put a fake behind the same trait your handler uses and give it a realistic latency distribution. Tests of authenticated flows need test accounts and a way to mint tokens that does not hit your identity provider thousands of times a second. And a single endpoint benchmark inside the process, with criterion, answers a different question, about CPU cost per call, that a network load test blurs with everything else. Use both: the benchmark finds a slow function, and the load test tells you whether it matters at your traffic.

Finish every load test the same way: write down the build hash, the instance type, the data size, the arrival rate, the percentile figures and the first thing that saturated. Six months from now, when someone asks whether a new release is slower, a one-paragraph record of what "normal" looked like is worth more than any dashboard, because it carries the conditions that made the numbers true.

Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.