Build a Log Analyzer in Rust With Regex and Iterators

The first question after an incident is usually "which endpoint was slow, and for how long?" The answer is sitting in an access log, a text file with one request per line, and the tools people reach for are grep and awk pipelines that nobody can read the next week. A log analyzer in Rust takes about a hundred lines, runs through a million lines in well under a second, and is a good place to learn three things that carry over everywhere: parsing with a regular expression, aggregating with iterators and a HashMap, and the borrow checker's opinion about references into data that goes away.

The analyzer built here parses an access log, counts responses by status, and reports the median, 95th and 99th percentile latency per path. It ran on Rust 1.99 with edition 2024, regex 1.13.1 and rayon 1.12.0. The log is synthetic, generated by the program itself with a fixed seed so you can reproduce it, and the numbers shown are what it printed.

The log format, field by field

Every line in this log looks like this:

10.0.0.14 - - [06/Nov/2026:10:00:00 +0000] "GET /static/app.js HTTP/1.1" 200 3464 29
10.0.3.83 - - [06/Nov/2026:10:00:01 +0000] "GET /api/items HTTP/1.1" 200 783 28

It is the common access log layout plus a trailing field. Reading left to right: the client address, two fields that are almost always a dash (identity and user), the timestamp in brackets, the request in quotes (method, path and protocol), the status code, the response size in bytes, and a final number that this log adds, the time the server spent in milliseconds. A parser is a function from one such line to a struct of the fields you care about, and it has to say "this line is not valid" without crashing, because real logs contain truncated lines, health-check noise and the occasional binary junk.

A regex that parses it

The pattern below names each field, so the code reads fields by name and not by position. Named groups are worth the extra characters: when the log format gains a field, the names keep working and the numbered groups all shift.

use std::collections::HashMap;
use std::sync::LazyLock;
use std::time::Instant;

use regex::Regex;

static LINE: LazyLock<Regex> = LazyLock::new(|| {
    Regex::new(r#"^(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] "(?P<method>[A-Z]+) (?P<path>\S+) [^"]*" (?P<status>\d{3}) (?P<bytes>\d+|-) (?P<ms>\d+)$"#).unwrap()
});

#[derive(Debug)]
struct Entry<'a> {
    path: &'a str,
    status: u16,
    ms: u32,
}

fn parse(line: &str) -> Option<Entry<'_>> {
    let caps = LINE.captures(line)?;
    Some(Entry {
        path: caps.name("path")?.as_str(),
        status: caps["status"].parse().ok()?,
        ms: caps["ms"].parse().ok()?,
    })
}

/// The same fields without a regex: split on the quotes, then on spaces.
fn parse_manual(line: &str) -> Option<Entry<'_>> {
    let (_, rest) = line.split_once('"')?;
    let (request, tail) = rest.split_once('"')?;
    let path = request.split_whitespace().nth(1)?;
    let mut tail = tail.split_whitespace();
    let status = tail.next()?.parse().ok()?;
    let _bytes = tail.next()?;
    let ms = tail.next()?.parse().ok()?;
    Some(Entry { path, status, ms })
}

fn percentile(sorted: &[u32], p: f64) -> u32 {
    sorted[((sorted.len() - 1) as f64 * p).round() as usize]
}

fn synthetic_log(n: usize) -> String {
    let paths = ["/", "/search", "/posts/ownership", "/api/items", "/static/app.js"];
    let mut out = String::with_capacity(n * 90);
    let mut seed = 12345u64;
    for i in 0..n {
        seed = seed.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
        let r = (seed >> 33) as u32;
        let path = paths[(r % 5) as usize];
        let status = match r % 50 { 0 => 500, 1..=3 => 404, _ => 200 };
        let ms = 5 + (r % 40) + if r % 97 == 0 { 900 } else { 0 };
        out.push_str(&format!(
            "10.0.{}.{} - - [06/Nov/2026:10:{:02}:{:02} +0000] \"GET {path} HTTP/1.1\" {status} {} {ms}\n",
            r % 4, r % 250, (i / 60) % 60, i % 60, 200 + r % 5000
        ));
    }
    out
}

fn main() {
    let log = synthetic_log(200_000);
    log.lines().take(2).for_each(|l| println!("{l}"));

    let start = Instant::now();
    let mut by_status: HashMap<u16, usize> = HashMap::new();
    let mut by_path: HashMap<&str, Vec<u32>> = HashMap::new();
    let mut bad = 0;
    for line in log.lines() {
        match parse(line) {
            Some(e) => {
                *by_status.entry(e.status).or_default() += 1;
                by_path.entry(e.path).or_default().push(e.ms);
            }
            None => bad += 1,
        }
    }
    let elapsed = start.elapsed();

    let mut statuses: Vec<_> = by_status.into_iter().collect();
    statuses.sort();
    println!("\nstatus counts: {statuses:?}, unparsed: {bad}");

    let mut rows: Vec<_> = by_path.into_iter().collect();
    rows.sort_by_key(|(path, _)| *path);
    println!("{:<20} {:>7} {:>6} {:>6} {:>6}", "path", "count", "p50", "p95", "p99");
    for (path, mut times) in rows {
        times.sort_unstable();
        println!(
            "{path:<20} {:>7} {:>4}ms {:>4}ms {:>4}ms",
            times.len(), percentile(&times, 0.5), percentile(&times, 0.95), percentile(&times, 0.99)
        );
    }
    println!("\nregex:  parsed {} lines in {elapsed:.2?}", log.lines().count());

    let start = Instant::now();
    let total: u64 = log.lines().filter_map(parse_manual).map(|e| u64::from(e.ms) + u64::from(e.status)).sum();
    println!("manual: parsed {} lines in {:.2?} (checksum {total})", log.lines().count(), start.elapsed());
}

Run in release mode over 200,000 generated lines, the program printed:

status counts: [(200, 184075), (404, 12037), (500, 3888)], unparsed: 0
path                   count    p50    p95    p99
/                      39710   25ms   40ms   40ms
/api/items             39805   28ms   43ms  908ms
/posts/ownership       40125   27ms   42ms  907ms
/search                40250   26ms   41ms  906ms
/static/app.js         40110   29ms   44ms   44ms

regex:  parsed 200000 lines in 95.18ms
manual: parsed 200000 lines in 23.76ms (checksum 50314718)

Read the table before the code. The median and 95th percentile look healthy for every path, between 25 and 44 ms. The 99th percentile tells a different story for three paths: about 900 ms. About one request in a hundred is slow, and the median would never show it. This is the whole argument for percentiles over averages: the mean of the /search timings would be around 35 ms, a number that describes no actual request. In this synthetic data the slow requests happen to land on three of the five paths, which is itself a finding you would chase: is it those endpoints, or something they share?

Reading the pattern

The expression is long, so here is how it breaks down. ^ and $ anchor it to the whole line, which matters: without them the regex happily matches a fragment in the middle of a corrupted line and returns nonsense. \S+ means "one or more non-space characters" and is the workhorse for fields with no spaces inside. \[(?P<time>[^\]]+)\] captures everything between the brackets, using a negated character class instead of a lazy .*?, which is both clearer and faster. "(?P<method>[A-Z]+) (?P<path>\S+) [^"]*" reads the quoted request, taking the path as a run of non-spaces and discarding the protocol. (?P<status>\d{3}) insists on exactly three digits. And (?P<bytes>\d+|-) allows the dash that servers write when no body was sent.

Two properties of Rust's regex crate are worth knowing because they differ from regex engines in many other languages. It guarantees matching time linear in the length of the input, so a hostile or accidental pathological line cannot hang the analyzer, which is not true of backtracking engines. The price is that it does not support backreferences or lookaround. For a log parser you will not miss either.

The pattern is built once inside a LazyLock. Compiling a regex is orders of magnitude more expensive than running it, and building it inside parse would repeat that cost for every line. That is the same mistake discussed in profiling a slow endpoint, and it is the first thing to check when a regex-based tool is slow.

Is the regex worth it?

The last two lines of the output contain a measurement that deserves a decision. The same fields, extracted by splitting on the quotes and then on whitespace, took 23.76 ms against 95.18 ms for the regex: four times faster. The manual version is eight lines.

fn parse_manual(line: &str) -> Option<Entry<'_>> {
    let (_, rest) = line.split_once('"')?;
    let (request, tail) = rest.split_once('"')?;
    let path = request.split_whitespace().nth(1)?;
    let mut tail = tail.split_whitespace();
    let status = tail.next()?.parse().ok()?;
    let _bytes = tail.next()?;
    let ms = tail.next()?.parse().ok()?;
    Some(Entry { path, status, ms })
}

So which should you use? For a fixed format you control, and a file of a few million lines, either finishes in under a second and the regex is easier to read, easier to change and self-documenting. Choose the regex while the format is in flux or the analysis is one-off. Switch to the manual parser, or a purpose-built one such as the nom combinator library, when the volume reaches hundreds of millions of lines or the parser sits in a service that runs all day. The manual version also has weaker validation: it accepts a line with a path containing a space as long as the quotes are balanced, and it never checks that the status is three digits. The regex is stricter by construction, which is sometimes what you want and sometimes the reason a few lines land in unparsed.

Whatever you pick, count the lines that fail to parse and print the count. A parser that silently drops 8 percent of the log because of a format change produces beautiful, wrong numbers. The bad counter in the program exists for that reason, and on this clean synthetic input it reads zero. On a real log, anything above a small fraction of a percent is a sign the format has drifted.

Aggregating with iterators and a HashMap

The aggregation is two maps filled in one pass: a count per status code and a list of latencies per path. The entry API is the idiom to learn here.

let mut by_status: HashMap<u16, usize> = HashMap::new();
let mut by_path: HashMap<&str, Vec<u32>> = HashMap::new();

for entry in log.lines().filter_map(parse) {
    *by_status.entry(entry.status).or_default() += 1;
    by_path.entry(entry.path).or_default().push(entry.ms);
}

entry(key).or_default() looks up the key once, inserts a default value if it is missing, and returns a mutable reference to the value. The alternative, if let Some(v) = map.get_mut(..) { ... } else { map.insert(..) }, hashes the key twice and is longer. filter_map(parse) turns the parser's Option into skipped lines, which is the natural place to keep the unparsed count if you replace it with an explicit loop.

The by_path key is a &str, a slice that points into the original log text. That is why each Entry has a lifetime parameter, Entry<'a>: the path borrows from the line it came from. It saves a heap allocation per line, which is real money at millions of lines, and it has a consequence that surprises people the first time.

The error you get when the data does not stay alive

The program above holds the entire log in one String, so the borrowed paths stay valid until main ends. A file of 20 GB cannot do that, and the natural fix is to stream it with BufReader::lines(), which gives you one owned String per line and drops it when the loop body ends. If you try to keep the borrowed Entry values across iterations, the compiler stops you:

error[E0597]: `line` does not live long enough
  --> src/bin/q543x.rs:18:36
   |
17 |         let line = line.unwrap();
   |             ---- binding `line` declared here
18 |         if let Some(entry) = parse(&line) {
   |                                    ^^^^^ borrowed value does not live long enough
...
21 |     }
   |     - `line` dropped here while still borrowed
22 |     println!("{}", entries.len());
   |                    ------- borrow later used here

Read it as a sentence: entry borrows from line, line is dropped at the end of each iteration, and entries outlives that. The compiler has caught what would be a dangling pointer in C. There are two honest fixes and they have different costs. Aggregate inside the loop, so nothing borrowed outlives the iteration, and store owned data in the maps (HashMap<String, Vec<u32>>, cloning the path only the first time you see it with entry(path.to_string()), or better, checking get_mut(path) first). Or read the whole file into one String and keep the borrows, which is simple and costs memory equal to the file. A 2 GB log fits comfortably on a laptop and the borrowed version is faster; a 200 GB log needs streaming and owned keys.

Percentiles done properly

The percentile function sorts the latencies and indexes into them: for p between 0 and 1, the element at round((n - 1) * p). This is the nearest-rank method, simple and good enough for operations work. Three details keep it honest. Sort each list once, with sort_unstable, which is faster than the stable sort when equal elements are interchangeable. Do not compute percentiles of percentiles: if you shard the log, merge the raw samples or use a histogram, as explained in load testing a Rust service. And mind the sample size, because the 99th percentile of 50 requests is simply the slowest one or two.

Memory is the cost of exact percentiles: a Vec<u32> per path holds every latency, 4 bytes each, so a billion requests is 4 GB. For long logs, replace the vector with a fixed-bucket histogram, say 1 ms buckets up to 10 seconds, and read the percentile from the cumulative counts. The error is bounded by the bucket width and the memory is constant.

Using all the cores

Parsing lines is independent work per line, so it parallelizes. rayon's par_lines splits a &str into chunks and processes them on a thread pool. Each worker builds its own HashMap with fold, and reduce merges them, so there is no shared map and no lock.

use rayon::prelude::*;

let parallel: HashMap<u16, usize> = log
    .par_lines()
    .filter_map(parse)
    .fold(HashMap::new, |mut acc: HashMap<u16, usize>, e| {
        *acc.entry(e.status).or_default() += 1;
        acc
    })
    .reduce(HashMap::new, |mut a, b| {
        for (k, v) in b {
            *a.entry(k).or_default() += v;
        }
        a
    });

On a million lines the serial loop took 412 ms and the parallel one 215 ms, on a machine with 10 cores. That is a 1.9 times speedup, far less than the core count, and the reason is instructive. The work per line is small, so splitting the text, scheduling the chunks and merging the maps are a large share of the total. The regex also draws a scratch cache from a shared pool on every match, and allocating the captures costs time. Parallelism pays when the per-item work is heavy; for light work, reducing per-item cost (the manual parser) beats adding cores. Try the manual parser inside par_lines to see the two compound.

Time windows and top-N

Two more questions come up immediately. "When did the slow period happen?" needs a time bucket per minute. Take the HH:MM part of the timestamp, which is a fixed-width slice of the captured string, and use it as a map key; the buckets with the highest 99th percentile are your incident window. And "what are the ten slowest paths?" does not need a full sort. Keep a BinaryHeap of at most ten items wrapped in std::cmp::Reverse, push each candidate, and pop the smallest whenever the heap exceeds ten. It is O(n log 10) and holds ten items.

A histogram instead of a list of every latency

The exact percentile needs every sample in memory. For a log with hundreds of millions of lines you want constant memory, and a histogram gives it: count how many requests fell into each 1 ms bucket, then walk the buckets until the cumulative count passes the percentile you want. The error is at most one bucket width, which is 1 ms here and far below the noise in a latency measurement.

/// Latency histogram with 1 ms buckets up to a cap. Memory is fixed at (cap + 1) counters,
/// however many requests you feed it.
struct Histogram {
    buckets: Vec<u64>,
    total: u64,
}

impl Histogram {
    fn new(cap_ms: usize) -> Self {
        Self { buckets: vec![0; cap_ms + 1], total: 0 }
    }

    fn record(&mut self, ms: u32) {
        let i = (ms as usize).min(self.buckets.len() - 1);
        self.buckets[i] += 1;
        self.total += 1;
    }

    /// Smallest bucket whose cumulative count reaches p of the total.
    fn percentile(&self, p: f64) -> usize {
        let target = (self.total as f64 * p).ceil() as u64;
        let mut seen = 0;
        for (ms, count) in self.buckets.iter().enumerate() {
            seen += count;
            if seen >= target {
                return ms;
            }
        }
        self.buckets.len() - 1
    }
}

fn main() {
    let mut h = Histogram::new(10_000);
    // 990 fast requests and 10 slow ones.
    for i in 0..990 {
        h.record(20 + (i % 15));
    }
    for _ in 0..10 {
        h.record(900);
    }
    println!("p50 = {} ms, p95 = {} ms, p99 = {} ms, p99.9 = {} ms", h.percentile(0.50), h.percentile(0.95), h.percentile(0.99), h.percentile(0.999));
    println!("memory: {} counters", h.buckets.len());
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn percentiles_match_the_sorted_answer() {
        let mut h = Histogram::new(100);
        for ms in 1..=100 {
            h.record(ms);
        }
        assert_eq!(h.percentile(0.5), 50);
        assert_eq!(h.percentile(0.95), 95);
        assert_eq!(h.percentile(1.0), 100);
    }

    #[test]
    fn values_over_the_cap_land_in_the_last_bucket() {
        let mut h = Histogram::new(10);
        h.record(5_000);
        assert_eq!(h.percentile(0.5), 10);
    }
}

p50 = 27 ms, p95 = 34 ms, p99 = 34 ms, p99.9 = 900 ms
memory: 10001 counters

The program feeds in 990 fast requests and 10 slow ones, exactly 1 percent. The output teaches two things at once. The memory is 10,001 counters whether you record a thousand requests or a billion. And the 99th percentile is 34 ms, not 900: with exactly 1 percent slow requests, the 99th percentile is the slowest of the fast group, and the slow tail only appears at p99.9. A report that stops at p99 would have said everything was fine. Which percentile to watch depends on your traffic and your promise to users; for a service handling a million requests a day, one request in a thousand is a thousand unhappy users, and p99.9 is the number that sees them. The two unit tests pin the behavior at the edges, including values past the cap, which land in the last bucket rather than panicking on an out-of-range index.

Unit-test the parser against the lines that hurt

Logs are hostile input, and the parser deserves tests built from real troublemakers rather than the happy path. Keep a handful of lines that broke you in the past as fixtures.

#[cfg(test)]
mod tests {
    use super::*;

    const GOOD: &str = r#"10.0.0.1 - - [06/Nov/2026:10:00:00 +0000] "GET /search?q=a%20b HTTP/1.1" 200 512 31"#;

    #[test]
    fn parses_a_normal_line() {
        let e = parse(GOOD).unwrap();
        assert_eq!((e.path, e.status, e.ms), ("/search?q=a%20b", 200, 31));
    }

    #[test]
    fn a_dash_for_bytes_is_accepted() {
        let line = GOOD.replace("512", "-");
        assert!(parse(&line).is_some());
    }

    #[test]
    fn truncated_lines_return_none_instead_of_panicking() {
        assert!(parse(&GOOD[..40]).is_none());
        assert!(parse("").is_none());
    }

    #[test]
    fn a_four_digit_status_is_rejected() {
        assert!(parse(&GOOD.replace(" 200 ", " 2000 ")).is_none());
    }
}

The truncated-line test is the important one. A log file being written while you read it ends mid-line, and rotating a file can leave half a record at the top. A parser that returns None for those and a counter that surfaces them is correct; a parser that unwraps is an outage in the analysis tool during the incident you are analyzing.

Turn it into a command

The program so far generates its own data, which is right for an article and wrong for a tool. To make it useful, take a path (or standard input) from the command line and stream the file with a BufReader, aggregating inside the loop so memory stays flat, as the E0597 discussion above prescribes. Write the report to standard output and the unparsed count to standard error, exit with status 1 if more than a configurable fraction of lines failed to parse (a tolerance of 1 percent is a reasonable default), and add --json for scripts. Those four decisions are the difference between a script that works on your laptop and a tool that survives being put into a cron job. The structure is the same as in a command line tool from clap to a release: a thin main, a run that returns Result, and tests that call run with a temporary file.

When this approach does not fit

If you search or filter logs interactively, a purpose-built tool such as ripgrep for text or jq for JSON lines is faster to use than any program you write. If you need dashboards and alerts over time, structured logs shipped to a metrics system serve you better than parsing text after the fact; emitting JSON from the application removes the regex entirely and makes every field typed. And if your log has multi-line records, such as stack traces, a line-at-a-time parser must first group lines into records, usually by recognizing which lines start with a timestamp. Choose the tool by how often you will ask the question: a hundred-line Rust program earns its keep when the question recurs. For streaming the same data without loading it, see how the iterator and I/O pieces fit together in building a file finder.

Before you trust a number from it

Check the analyzer against the log with one cross-check that does not use your code. Count the lines with wc -l, count the 500 responses with grep -c '" 500 ', and compare to the totals the tool printed. If the counts disagree, the parser is dropping or mangling lines, and the percentile table above it is not worth reading. Do this once per log format, and again whenever the server or proxy configuration changes the format. A tool that has been reconciled against a trivial one is a tool whose output you can quote in an incident review without apologizing for it.