Reading Large Files in Rust With Buffered IO and Memory Maps

The first version of a file-reading tool is always fs::read_to_string, and it works until someone points it at a 40 GB log. Three techniques cover the large-file cases: buffered line reads, chunked reads and memory maps. This post compares them with working code. It compiles on Rust 1.90 with edition 2024 and memmap2 0.9; timings are not quoted because they depend on your disk and file.

Line by line, without allocating per line

use std::fs::File;
use std::io::{self, BufRead, BufReader};

fn count_matching(path: &str, needle: &str) -> io::Result<usize> {
    let mut reader = BufReader::with_capacity(1 << 20, File::open(path)?);
    let mut line = String::new();
    let mut hits = 0;
    loop {
        line.clear();
        if reader.read_line(&mut line)? == 0 {
            break;
        }
        if line.contains(needle) {
            hits += 1;
        }
    }
    Ok(hits)
}

reader.lines() is convenient but allocates a new String for each line. Clearing and reusing one buffer removes that cost. The 1 MiB buffer reduces system calls compared with the 8 KiB default. Memory use is constant regardless of file size.

Bytes, not text

use std::fs::File;
use std::io::{self, BufRead, BufReader};

fn count_lines_bytes(path: &str) -> io::Result<u64> {
    let mut reader = BufReader::with_capacity(1 << 20, File::open(path)?);
    let mut buf = Vec::new();
    let mut n = 0;
    loop {
        buf.clear();
        if reader.read_until(b'\n', &mut buf)? == 0 {
            break;
        }
        n += 1;
    }
    Ok(n)
}

read_line fails with an InvalidData error on invalid UTF-8, which kills a run over a log with one stray byte (invalid UTF-8 when reading). Working with Vec<u8> and regex::bytes avoids both the validation cost and the failure.

Chunks for custom parsers

use std::fs::File;
use std::io::{self, Read};

fn checksum(path: &str) -> io::Result<u64> {
    let mut f = File::open(path)?;
    let mut buf = vec![0u8; 1 << 20];
    let mut sum = 0u64;
    loop {
        let n = f.read(&mut buf)?;
        if n == 0 {
            break;
        }
        sum = buf[..n].iter().fold(sum, |a, &b| a.wrapping_add(b as u64));
    }
    Ok(sum)
}

When records are fixed-size or you parse a binary format, read fixed chunks into a reused buffer. A single read may return fewer bytes than the buffer holds, so always use the returned count (n), not the buffer length.

Memory maps

use memmap2::Mmap;
use std::fs::File;

fn count_newlines(path: &str) -> std::io::Result<usize> {
    let file = File::open(path)?;
    // SAFETY: we assume no other process modifies or truncates the file while mapped.
    let map = unsafe { Mmap::map(&file)? };
    Ok(map.iter().filter(|&&b| b == b'\n').count())
}

The map presents the file as a &[u8] the OS pages in on demand. You get the whole file as a slice, which makes searching with memchr or splitting into chunks for threads simple, and no copy into user space:

use rayon::prelude::*;

fn parallel_count(map: &[u8]) -> usize {
    map.par_chunks(8 << 20)
        .map(|c| c.iter().filter(|&&b| b == b'\n').count())
        .sum()
}

The unsafe contract

• If another process truncates the file, touching the missing pages raises SIGBUS and kills your process.
• If it modifies the file, the slice's contents change while Rust assumes immutability, which is formally undefined behavior even if usually benign.
• On network filesystems and some platforms, mmap semantics are weaker or slower.

Use mmap on files your tool controls or that are known to be static (a rotated log, a dataset), take a lock if you can, and handle the failure mode. If you cannot make that promise, use buffered reads.

Choosing

Measure the real tool with a representative file (benchmarking with hyperfine). Often the parse or the regex is the bottleneck, not the read (regex performance).