Skip to content
rusthacks.dev
Performance

Memory Usage in Rust Services and Finding Leaks Step by Step

Find where a Rust service's memory goes: read RSS and allocator stats, profile allocations with dhat or heaptrack, tell a leak from growth, and fix the usual causes.

18 min read
Summarize in AI
Flat illustration for the post “Memory Usage in Rust Services and Finding Leaks Step by Step”, composed around tooling.

"Rust cannot leak memory" is one of the most repeated half-truths about the language. What the compiler guarantees is memory safety: no use-after-free, no double free, no data race. It does not promise that memory is freed. The standard library says so plainly: leaking is safe, and mem::forget, Box::leak and a reference-counted cycle are all ordinary safe code. A long-running Rust service can and does grow without bound, and when it does the cause is nearly always one of five patterns, none of which is a bug in the borrow checker. This guide shows each pattern with a program you can run, the tool that finds it (macOS leaks, the dhat heap profiler, resident-memory sampling and Tokio's task metrics), and the fix, then covers the case that looks like a leak and is not: memory the allocator has kept after you freed it. Everything ran on Rust 1.99, dhat 0.3 and Tokio 1.54 on an Apple M1 Max; the commands for Linux are named where they differ.

Three different things people call a leak

KindWhat it isDoes a leak detector report it?Typical cause
True leakMemory that is allocated and unreachable, so nothing can free itYesReference cycles, mem::forget, forgotten FFI allocations
Unbounded growthMemory that is still reachable but only ever added toNo: it is "in use"Caches without eviction, queues without limits, tasks that never finish, registries that are never cleaned
Allocator retentionMemory freed by the program but not returned to the operating systemNo: nothing is allocatedFragmentation, per-thread arenas, high-water-mark behaviour

The distinction decides the tool. A leak detector finds the first kind and says nothing about the other two, which in practice cause most production incidents: the service whose resident memory climbs 20 MB an hour has a reachable data structure that is too big, not a leak in the strict sense. Always begin by watching the process's resident set size over time and the size of its major collections, then reach for a profiler.

Cause 1: reference-counted cycles

Rc and Arc free their contents when the count reaches zero. If two values hold strong references to each other, neither count can reach zero, and both stay allocated until the process exits. This program builds one cycle and one correct structure, and counts drops:

src/bin/cycle.rsrust
use std::cell::RefCell;
use std::rc::{Rc, Weak};
use std::sync::atomic::{AtomicUsize, Ordering};

static DROPPED: AtomicUsize = AtomicUsize::new(0);

struct Node {
    name: &'static str,
    next: RefCell<Option<Rc<Node>>>,
    prev: RefCell<Option<Weak<Node>>>,
    payload: Vec<u8>,
}

impl Drop for Node {
    fn drop(&mut self) {
        DROPPED.fetch_add(1, Ordering::SeqCst);
        println!("dropping {}", self.name);
    }
}

fn node(name: &'static str) -> Rc<Node> {
    Rc::new(Node { name, next: RefCell::new(None), prev: RefCell::new(None), payload: vec![0; 1024] })
}

fn leaky() {
    let a = node("leaky a");
    let b = node("leaky b");
    *a.next.borrow_mut() = Some(Rc::clone(&b));
    *b.next.borrow_mut() = Some(Rc::clone(&a));
    println!("leaky: strong counts a={} b={}", Rc::strong_count(&a), Rc::strong_count(&b));
}

fn fixed() {
    let a = node("fixed a");
    let b = node("fixed b");
    *a.next.borrow_mut() = Some(Rc::clone(&b));
    *b.prev.borrow_mut() = Some(Rc::downgrade(&a));
    println!("fixed: strong counts a={} b={}, weak count a={}", Rc::strong_count(&a), Rc::strong_count(&b), Rc::weak_count(&a));
}

fn main() {
    leaky();
    println!("after leaky(): {} nodes dropped", DROPPED.load(Ordering::SeqCst));
    fixed();
    println!("after fixed(): {} nodes dropped", DROPPED.load(Ordering::SeqCst));
}
Text
leaky: strong counts a=2 b=2
after leaky(): 0 nodes dropped
fixed: strong counts a=1 b=2, weak count a=1
dropping fixed a
dropping fixed b
after fixed(): 2 nodes dropped

After leaky() returned, no node had been dropped: both strong counts were 2, so when the local variables went out of scope each count fell to 1 and stopped. fixed() made the back-edge a Weak reference, which does not keep its target alive (strong count 1 for a, weak count 1), and both nodes dropped as soon as the function ended. The rule is to make references that point "up" or "back" in a structure weak: a child's pointer to its parent, an observer's pointer to the subject it watches, a cache entry's pointer to its owner. The macOS tool leaks reports the first function as a real leak, with the cycle:

Text
Process 85216: 194 nodes malloced for 15 KB
Process 85216: 4 leaks for 2752 total leaked bytes.

That is two nodes of 96 bytes plus their 1,024-byte payload vectors, reported with the allocation site (Rc::new in the Node constructor) and the words "ROOT CYCLE". On Linux, valgrind --leak-check=full or heaptrack give the same information, and a build with the leak sanitizer (RUSTFLAGS=-Zsanitizer=leak on nightly) finds cycles in tests. Cycles are not exotic: a graph with parent pointers, a GUI with callbacks that capture the widget that owns them, and an event bus whose subscribers hold the bus all form them.

Cause 2: a collection that only grows

A HashMap used as a cache, with no eviction, is reachable memory by definition, so no leak detector will flag it and every allocation looks intentional. It is the most common cause of a service that "leaks". The program fills two caches with 1 KiB values, 20,000 at a time, and prints resident memory as reported by ps. The first has no limit; the second evicts the oldest entry once it holds 10,000. They run as separate processes, because (as the last line of the first shows) a process that has freed memory does not always shrink:

src/bin/growth.rsrust
use std::collections::HashMap;
use std::process::Command;

fn rss_mb() -> f64 {
    let out = Command::new("ps").args(["-o", "rss=", "-p", &std::process::id().to_string()]).output().unwrap();
    String::from_utf8_lossy(&out.stdout).trim().parse::<f64>().unwrap() / 1024.0
}

struct UnboundedCache {
    map: HashMap<u64, Vec<u8>>,
}

struct BoundedCache {
    map: HashMap<u64, Vec<u8>>,
    order: std::collections::VecDeque<u64>,
    cap: usize,
}

impl BoundedCache {
    fn insert(&mut self, k: u64, v: Vec<u8>) {
        if self.map.insert(k, v).is_none() {
            self.order.push_back(k);
            if self.order.len() > self.cap {
                if let Some(old) = self.order.pop_front() {
                    self.map.remove(&old);
                }
            }
        }
    }
}

fn main() {
    let mode = std::env::args().nth(1).unwrap_or_default();
    println!("{mode}: start {:.0} MB resident", rss_mb());

    if mode == "unbounded" {
        let mut cache = UnboundedCache { map: HashMap::new() };
        for round in 1..=5 {
            for k in 0..20_000u64 {
                cache.map.insert(round * 1_000_000 + k, vec![1u8; 1024]);
            }
            println!("  after {:>6} requests: {:>6} entries, {:>4.0} MB resident", round * 20_000, cache.map.len(), rss_mb());
        }
        drop(cache);
        println!("  after dropping the cache:        {:>4.0} MB resident", rss_mb());
    } else {
        let mut cache = BoundedCache { map: HashMap::new(), order: Default::default(), cap: 10_000 };
        for round in 1..=5 {
            for k in 0..20_000u64 {
                cache.insert(round * 1_000_000 + k, vec![1u8; 1024]);
            }
            println!("  after {:>6} requests: {:>6} entries, {:>4.0} MB resident", round * 20_000, cache.map.len(), rss_mb());
        }
    }
}
Text
unbounded: start 2 MB resident
  after  20000 requests:  20000 entries,   23 MB resident
  after  40000 requests:  40000 entries,   45 MB resident
  after  60000 requests:  60000 entries,   69 MB resident
  after  80000 requests:  80000 entries,   89 MB resident
  after 100000 requests: 100000 entries,  108 MB resident
  after dropping the cache:         106 MB resident

bounded: start 2 MB resident
  after  20000 requests:  10000 entries,   13 MB resident
  after  40000 requests:  10000 entries,   14 MB resident
  after  60000 requests:  10000 entries,   14 MB resident
  after  80000 requests:  10000 entries,   14 MB resident
  after 100000 requests:  10000 entries,   14 MB resident

The unbounded cache grew linearly: about 21 MB per 20,000 entries, which is about 1.07 KB per entry (the 1,024-byte payload plus the key, hash table slot and allocator overhead), reaching 108 MB at 100,000 entries and with no end in sight. The bounded cache stopped at 10,000 entries and stayed at 14 MB for the whole run. Real caches differ in the eviction rule (least recently used, time to live, size-weighted) and the right answer is to use a library that implements it, such as moka or lru, instead of the hand-rolled queue here, but the principle is the one in the numbers: every cache needs a bound, and the bound needs to be measured in the unit that costs you, bytes, not entries. Registries have the same disease: a global map of connections, sessions or subscriptions that is inserted into on connect and forgotten on disconnect is a cache with no eviction. A useful discipline is to expose the size of every long-lived collection as a metric; a graph that rises and never falls is the diagnosis.

Cause 3: tasks that never finish

Every spawned Tokio task holds its future and everything the future captured. A task waiting for something that will never happen is a small permanent allocation, and a service that spawns one per request accumulates them. Tokio exposes the count, which makes this one easy to see:

src/bin/tasks.rsrust
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::{mpsc, oneshot};

async fn handle_request(id: u32, reply: oneshot::Receiver<u32>) -> u32 {
    reply.await.unwrap_or(id)
}

#[tokio::main]
async fn main() {
    let metrics = tokio::runtime::Handle::current().metrics();
    println!("alive tasks at start: {}", metrics.num_alive_tasks());

    // The sender is kept alive in a Vec, so the receiver never resolves and the task never ends.
    let mut senders = Vec::new();
    for id in 0..1000 {
        let (tx, rx) = oneshot::channel();
        senders.push(tx);
        tokio::spawn(handle_request(id, rx));
    }
    tokio::time::sleep(Duration::from_millis(100)).await;
    println!("after 1000 requests whose replies never come: {} alive tasks", metrics.num_alive_tasks());

    drop(senders);
    tokio::time::sleep(Duration::from_millis(100)).await;
    println!("after dropping the senders: {} alive tasks", metrics.num_alive_tasks());

    // An unbounded channel with a slow consumer.
    let queued = Arc::new(AtomicUsize::new(0));
    let (tx, mut rx) = mpsc::unbounded_channel::<Vec<u8>>();
    let consumer = tokio::spawn({
        let queued = queued.clone();
        async move {
            let mut n = 0u64;
            while let Some(_msg) = rx.recv().await {
                queued.fetch_sub(1, Ordering::Relaxed);
                tokio::time::sleep(Duration::from_millis(2)).await;
                n += 1;
            }
            n
        }
    });
    for _ in 0..5000 {
        queued.fetch_add(1, Ordering::Relaxed);
        tx.send(vec![0u8; 1024]).unwrap();
    }
    tokio::time::sleep(Duration::from_millis(200)).await;
    let depth = queued.load(Ordering::Relaxed);
    println!("unbounded channel after 5000 sends and 200 ms: {depth} messages still queued (about {} MiB)", depth * 1024 / 1048576);
    drop(tx);
    let _ = consumer.await;
}
Text
alive tasks at start: 0
after 1000 requests whose replies never come: 1000 alive tasks
after dropping the senders: 0 alive tasks
unbounded channel after 5000 sends and 200 ms: 4944 messages still queued (about 4 MiB)

A thousand tasks were each waiting on a reply whose sender was kept alive in a vector, so they sat in the runtime as 1,000 alive tasks. Dropping the senders resolved the waits (with an error, which the task handled by returning), and the count fell to 0. In a real service the sender is held by a pending request table that is never cleaned, or by the other end of a connection that never closed. Export num_alive_tasks as a metric, add timeouts to every wait that crosses a boundary (timeouts), and use tokio-console to find which tasks are old and idle (tokio-console). The causes of tasks that wait forever are listed in tasks that never finish.

Cause 4: queues with no limit

The last lines of the same program send 5,000 one-kilobyte messages into an unbounded channel whose consumer takes about two milliseconds per message. After 200 ms about 4,947 messages were still queued, roughly 4 MiB for a burst that lasted no time at all. If producers run faster than the consumer indefinitely, the queue is a memory leak with a deadline: the out-of-memory killer. A bounded channel makes the producer wait instead, which is the backpressure from bounded channels. The same applies to Vec buffers that collect log lines or metrics for a flush that stops happening when the downstream service is down; give every buffer a capacity and a policy for overflow (drop oldest, drop newest or block).

Cause 5: deliberate leaks that multiply

Box::leak turns a box into a &'static mut and never frees it. It is the right tool for configuration loaded once at startup that must live for the whole program, and the wrong one anywhere that runs per request. mem::forget and ManuallyDrop have the same shape, and so does a static container that is pushed to from a handler. The dhat profiler makes these visible by counting allocations at any moment. In testing mode it lets a unit test assert that a function released everything:

src/bin/dhat_demo.rsrust
use std::collections::HashMap;

#[global_allocator]
static ALLOC: dhat::Alloc = dhat::Alloc;

fn build_cache(n: usize) -> HashMap<u64, Vec<u8>> {
    (0..n as u64).map(|k| (k, vec![0u8; 1024])).collect()
}

fn session(n: usize) -> usize {
    let cache = build_cache(n);
    cache.values().map(|v| v.len()).sum()
}

fn main() {
    let _profiler = dhat::Profiler::builder().testing().build();

    let before = dhat::HeapStats::get();
    let total = session(1000);
    let after = dhat::HeapStats::get();

    println!("session used {total} bytes of payload");
    println!("allocations during session: {} blocks, {} bytes total", after.total_blocks - before.total_blocks, after.total_bytes - before.total_bytes);
    println!("peak during session: {} blocks, {} bytes", after.max_blocks, after.max_bytes);
    println!("still allocated afterwards: {} blocks, {} bytes", after.curr_blocks - before.curr_blocks, after.curr_bytes - before.curr_bytes);

    let leaked: &'static mut Vec<u8> = Box::leak(Box::new(vec![0u8; 4096]));
    leaked.push(1);
    let end = dhat::HeapStats::get();
    println!("after Box::leak of a 4 KiB vec: {} blocks, {} bytes still allocated", end.curr_blocks - before.curr_blocks, end.curr_bytes - before.curr_bytes);
}
Text
session used 1024000 bytes of payload
allocations during session: 1001 blocks, 1091592 bytes total
peak during session: 1001 blocks, 1091592 bytes
still allocated afterwards: 0 blocks, 0 bytes
after Box::leak of a 4 KiB vec: 4 blocks, 9304 bytes still allocated

Building a thousand-entry cache inside session allocated 1,001 blocks and 1.09 MB, and after the function returned, 0 blocks were still allocated: nothing leaked. The leak at the end is visible as blocks and bytes that remain. (Of the four blocks counted there, the leaked vector's grown buffer and its box are two; the standard output buffer that println! allocates lazily accounts for the others, a reminder to snapshot the stats before printing when you are counting precisely.) The same HeapStats API in a test, with an assert_eq!(stats.curr_blocks, 0) after the code under test, turns "does this function release everything?" into a regression test. Without testing mode, dhat::Profiler::new_heap() writes a dhat-heap.json that you open in the DHAT viewer to see which call sites hold the live bytes at the peak, which is the answer to "what is using the memory" for the unbounded-growth case. Install the profiler only in a build you use for diagnosis: it replaces the global allocator and slows the program several times.

The FFI leak, and a test that catches it

When Rust hands memory to C, ownership crosses a boundary the compiler cannot follow. CString::into_raw gives up ownership and returns a raw pointer; the memory is freed only if someone gives the pointer back to CString::from_raw, and the type system does not remember to ask. Forgetting it is silent, small and repeated on every call, which is the ideal profile for a leak that takes a week to notice. The dhat test harness turns it into a failing test:

src/bin/ffi_leak.rsrust
use std::ffi::{c_char, CString};

#[global_allocator]
static ALLOC: dhat::Alloc = dhat::Alloc;

/// Stand-in for a C function that only reads the string it is given.
fn c_api_that_borrows(_name: *const c_char) {}

fn leaks_a_string() {
    let name = CString::new("sensor-1").unwrap();
    c_api_that_borrows(name.into_raw());
}

fn does_not_leak() {
    let name = CString::new("sensor-1").unwrap();
    let raw = name.into_raw();
    c_api_that_borrows(raw);
    drop(unsafe { CString::from_raw(raw) });
}

fn assert_no_leak(label: &str, f: fn()) {
    let before = dhat::HeapStats::get();
    f();
    let after = dhat::HeapStats::get();
    let blocks = after.curr_blocks - before.curr_blocks;
    let bytes = after.curr_bytes - before.curr_bytes;
    println!("{label}: {blocks} block(s), {bytes} byte(s) still allocated");
    dhat::assert_eq!(blocks, 0, "{label} leaked");
}

fn main() {
    let _profiler = dhat::Profiler::builder().testing().build();
    assert_no_leak("does_not_leak", does_not_leak);
    assert_no_leak("leaks_a_string", leaks_a_string);
}
Text
does_not_leak: 0 block(s), 0 byte(s) still allocated
leaks_a_string: 1 block(s), 9 byte(s) still allocated
thread 'main' panicked at src/bin/ffi_leak.rs:28:5:
dhat: assertion failed: `(left == right)`
  left: `1`,
 right: `0`: leaks_a_string leaked

The correct function allocated and freed (0 blocks left). The leaking one left 1 block of 9 bytes, the string "sensor-1" and its terminating zero byte, and the assertion failed with the name of the function. A helper like assert_no_leak is worth keeping in the test suite of any crate that wraps a C library or uses unsafe to manage memory: call each public function in a loop, in a test with dhat in testing mode, and assert that the block count did not move. The two rules for FFI memory are to free with the same allocator that allocated (a pointer from CString::into_raw must come back through from_raw, never through C's free), and to document, for every function that returns a pointer, who frees it. The conventions for the boundary are in FFI ownership and free.

The leak that is only a label

Metrics libraries and tracing layers keep one time series, or one span statistic, per distinct combination of label values. If one of the labels is a path that includes an identifier, or a user id, or an error message with a request id in it, the number of series grows with traffic, not with the number of routes, and each series costs memory for as long as the process lives. The program counts keys for 10,000 requests to /users/<id>/orders/<id> under two labelling schemes:

src/bin/cardinality.rsrust
use std::collections::HashMap;

fn raw_label(path: &str) -> String {
    path.to_string()
}

fn route_label(path: &str) -> String {
    path.split('/')
        .map(|seg| if !seg.is_empty() && seg.chars().all(|c| c.is_ascii_digit()) { ":id" } else { seg })
        .collect::<Vec<_>>()
        .join("/")
}

fn main() {
    let mut by_raw: HashMap<String, u64> = HashMap::new();
    let mut by_route: HashMap<String, u64> = HashMap::new();

    for i in 0..10_000u32 {
        let path = format!("/users/{}/orders/{}", i % 2_500, i);
        *by_raw.entry(raw_label(&path)).or_default() += 1;
        *by_route.entry(route_label(&path)).or_default() += 1;
    }

    println!("keys when the label is the raw path:   {}", by_raw.len());
    println!("keys when the label is the route:      {}", by_route.len());
    println!("example route label: {}", route_label("/users/42/orders/9001"));
}
Text
keys when the label is the raw path:   10000
keys when the label is the route:      1
example route label: /users/:id/orders/:id

Labelling by the raw path produced 10,000 distinct keys from 10,000 requests, one per request, and the structure never shrank. Labelling by the route template produced one. In a real service the key is a Prometheus label or a span field rather than a map key, but the memory behaves identically, in your process and in the metrics database that ingests it. Use the route pattern the router matched (axum exposes it as MatchedPath), never the concrete URL; keep user ids, emails and request ids out of labels entirely and in logs, where they are not aggregated. The dashboards side of the same discipline is in Grafana dashboards for Rust services.

When it is not a leak: the allocator keeps memory

The last line of the unbounded run is the surprising one: after drop(cache) freed all 100,000 values, the process still held 106 MB resident. The program did free the memory; the allocator did not give it back to the operating system. Allocators keep freed pages for reuse because asking the OS again is slow, and they can only return memory that forms a contiguous free region at the right end of an arena; one long-lived small allocation in the middle pins a whole page. The effect is that resident memory follows the peak, not the current need. How to deal with it:

  • Compare live data with resident memory. If the sum of your collections' sizes is small and RSS is large and flat, you have retention, not a leak. The test is whether RSS grows without bound or levels off.
  • Try another allocator. jemalloc (via tikv-jemallocator) and mimalloc behave differently from the system allocator under multi-threaded workloads with many allocation sizes, and often return memory more eagerly or fragment less. It is a two-line change, and the right move is to measure both under a replay of production traffic. The comparison is in allocators compared.
  • On Linux with glibc, per-thread arenas can multiply the retained memory by the thread count; MALLOC_ARENA_MAX=2 or switching allocators is the usual remedy, and malloc_trim(0) asks the allocator to release what it can.
  • Cap peaks, not only steady state. A job that reads a 2 GB file once leaves the process at 2 GB afterwards. Stream instead of loading whole.

What to monitor so the next one is caught in a day

Every cause above has a number that would have shown it early, and almost none of those numbers is collected by default. Add them to the service before you need them:

MetricWhat it catchesA sensible alert
Resident set size, per processEverything, eventuallyRising for hours after the last deploy, or above 80 percent of the container limit
Size and eviction count of each cacheUnbounded growthSize at its cap and evictions per second climbing (the cache is too small), or size still rising with no cap
num_alive_tasks from the Tokio runtimeTasks that never finishA count that does not fall back when traffic drops
Queue and channel depthProducers outrunning consumersDepth above a fraction of the bound for a minute
Open connections and file descriptorsRegistries never cleaned, sockets never closedA count that rises with uptime, not traffic
Metric series count (cardinality)Labels that grow with trafficAny rise in series per route after a release

The pattern across the table is the same: a number that should plateau or follow traffic, and an alert for the case when it follows uptime instead. A memory graph that climbs with the age of the process, not the load on it, is the signature of a leak or an unbounded structure, and the way to tell the two apart is the next measurement: if a heap profile at the peak shows live bytes in one data structure, it is unbounded growth; if the profiler shows nothing live where RSS is high, it is retention and the allocator is the lever. Spend an afternoon adding the table's metrics to a service and the next memory incident becomes a graph you read instead of a week you lose.

One last operational note. A memory limit in a container turns every kind of growth into a restart: the kernel's out-of-memory killer ends the process, the orchestrator starts a new one, and the graph looks like a saw. The restarts hide the leak from users and from the dashboard that shows only availability. Alert on the restart count and on the memory trend, not on downtime alone, or the leak will run for months at a rate nobody notices.

A diagnosis procedure

  1. Graph resident memory over hours, per process, with the deploy markers. Does it level off (retention or a bounded cache filling up) or climb forever (growth or a leak)?
  2. Graph the sizes of your long-lived collections and the Tokio task count. A flat line in one and a rising line in RSS points at something you are not measuring; a rising line in one is the answer.
  3. Take a heap profile at the peak, not at startup, with dhat or heaptrack, and read the call sites holding the most live bytes.
  4. Run the leak detector on a test, a short run of the program that creates and discards objects, to catch cycles (macOS leaks, Linux valgrind). If it reports nothing but memory still grows, the cause is reachable.
  5. Fix, then prove it: add the metric you were missing, and a regression test with dhat in testing mode for the function that leaked.

The profiling side of the same workflow is in profiling with flamegraphs and measuring memory with heaptrack and dhat; the language-level story, why safe Rust permits leaks, is in memory leaks in safe code.

Also inTooling
Taggedmemory
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.