eBPF and Rust With Aya for Observability Tools Step by Step

Observing a running Linux system usually means changing it (adding logging, restarting under a debugger) or accepting the cost of tools that slow it down. eBPF turns that around. You write a very small program, the kernel proves it safe, and it runs inside the kernel at a hook point such as "a process called exec", reporting what it sees into a shared table that an ordinary program reads. Aya lets you write both halves in Rust: the kernel program and the user-space loader. This guide builds the smallest useful example, a counter of how many times each program name was executed, and it was actually loaded into a Linux kernel and run. It also shows the most important thing about eBPF, the verifier, by deliberately writing a program the kernel refuses and reading its refusal. Versions: aya 0.16.0, aya-ebpf 0.3.0, bpf-linker 0.13.0, rustc 1.99.0 for user space and nightly 1.101.0 for the kernel program, on Linux 7.0.14 (the kernel Docker Desktop runs in its virtual machine) on an Apple M1 Max.

What eBPF is, in terms of what you write

An eBPF program is a function compiled to a restricted instruction set and loaded into the kernel with the bpf() system call. Before it runs, the kernel's verifier walks every possible path through it and rejects it unless it can prove the program terminates, never reads memory it should not, and never dereferences a pointer it has not checked. Once accepted, it is compiled to native code and attached to a hook: a tracepoint, a function entry, a network interface. Whenever the hook fires, the program runs. Programs share data with user space through maps, kernel-resident tables (hash maps, arrays, ring buffers) that both sides can read and write.

That gives a three-part shape to every Aya project. A kernel crate compiled for the bpfel-unknown-none target holds the programs and declares the maps. A user-space crate loads the compiled object, attaches the programs to hooks, and reads the maps. Optionally a shared crate holds types both sides use. The restricted nature of the kernel side is why it is #![no_std] and #![no_main]: there is no allocator, no standard library, no threads, only the helper functions the kernel provides.

Setting up the toolchain

The kernel crate needs two things most Rust projects do not. It needs the nightly compiler, because the eBPF target has no prebuilt standard library and must be built from source with -Z build-std=core, which is a nightly flag, plus the rust-src component. And it needs bpf-linker, a linker that turns Rust's LLVM output into a loadable eBPF object. Both went into one Docker image, because Linux is the only place this runs and Docker Desktop's virtual machine provides a Linux kernel on a Mac:

FROM rust:1.99-slim-bookworm
RUN apt-get update && apt-get install -y --no-install-recommends build-essential pkg-config git curl ca-certificates zstd bpftool iproute2 \
 && rm -rf /var/lib/apt/lists/*
RUN rustup toolchain install nightly --profile minimal --component rust-src
RUN curl -sL https://github.com/aya-rs/bpf-linker/releases/latest/download/bpf-linker-aarch64-unknown-linux-musl.tar.zst -o /tmp/bl.tar.zst \
 && tar -xpf /tmp/bl.tar.zst -C /usr/local/bin && rm /tmp/bl.tar.zst

Two notes on that file. The linker is installed from the project's release tarball and not with cargo install bpf-linker: the project's own README advises against building it from source, because a source build is tied to one specific LLVM version, and the nightly compiler used here carries LLVM 23.1.5. The prebuilt binary reported bpf-linker 0.13.0 and worked on the first try. And the tarball name contains the CPU architecture (aarch64 here); on an x86-64 machine you would fetch the x86_64 one.

The kernel program

The kernel crate is small. It declares one map, a hash from a 16-byte process name to a 64-bit counter, and one program attached to a tracepoint:

#![no_std]
#![no_main]

use aya_ebpf::{
    helpers::bpf_get_current_comm,
    macros::{map, tracepoint},
    maps::HashMap,
    programs::TracePointContext,
};

#[map]
static EXEC_COUNTS: HashMap<[u8; 16], u64> = HashMap::with_max_entries(1024, 0);

#[tracepoint]
pub fn count_exec(_ctx: TracePointContext) -> u32 {
    let comm = bpf_get_current_comm().unwrap_or([0u8; 16]);
    unsafe {
        match EXEC_COUNTS.get_ptr_mut(&comm) {
            Some(count) => *count += 1,
            None => {
                let _ = EXEC_COUNTS.insert(&comm, &1u64, 0);
            }
        }
    }
    0
}

#[panic_handler]
fn panic(_info: &core::panic::PanicInfo) -> ! {
    unsafe { core::hint::unreachable_unchecked() }
}

Read it as the kernel will. #[map] places the static into an ELF section the loader recognises as a map definition, with room for 1,024 distinct names. #[tracepoint] marks count_exec as a program of type tracepoint; its argument is the context the kernel passes (unused here, so the underscore). The body asks the kernel for the name of the current process with bpf_get_current_comm, a 16-byte buffer, then looks it up in the map. If the name is present, it increments the counter; if not, it inserts a 1. The lookup returns a raw pointer, which is why that block is unsafe, and the match on Some and None is not just style: the kernel's verifier insists on exactly that null check, as the failure test below proves. The panic handler must exist because there is no standard library; it tells the compiler that reaching it is impossible, which is acceptable because the verifier would have rejected any path that could panic.

The manifest sets the profile for size and speed, with debug information on for the loader:

[package]
name = "lab-ebpf"
version = "0.1.0"
edition = "2021"

[dependencies]
aya-ebpf = "0.3"

[[bin]]
name = "lab-ebpf"
path = "src/main.rs"

[profile.release]
opt-level = 3
lto = true
panic = "abort"
codegen-units = 1
debug = 2

It is built for the eBPF target with the nightly flag described above:

cd ebpf
cargo +nightly build --target=bpfel-unknown-none -Z build-std=core --release

The build finished in 13.6 seconds, most of it compiling core and the dependencies. The output is not a native executable. It is an ELF object of 2,192 bytes, and reading its section table shows what the loader will find:

readelf -S -W target/bpfel-unknown-none/release/lab-ebpf | grep -E "tracepoint|maps"

  [ 3] tracepoint        PROGBITS        0000000000000000 0001a0 000300 00  AX  0   0  8
  [ 4] .reltracepoint    REL             0000000000000000 0005a0 000050 10   I  7   3  8
  [ 5] maps              PROGBITS        0000000000000000 0004a0 00001c 00  WA  0   0  4

The program lives in a section named tracepoint, 0x300 (768) bytes of eBPF instructions. The maps section is 0x1c (28) bytes, the definition of the one map. The .reltracepoint section holds relocations: places in the program that must be patched with the map's real file descriptor at load time. This is what "the loader" does, and why a compiled eBPF program is a file you load and not a binary you execute.

The user-space half

The loader is an ordinary Rust program using the aya crate. It embeds the object file into its own binary at compile time, loads the program into the kernel, attaches it to the sched_process_exec tracepoint, does some work to observe, and then reads the map:

use anyhow::{Context, Result};
use aya::{Ebpf, maps::HashMap, programs::TracePoint};
use std::process::{Command, Stdio};

fn main() -> Result<()> {
    let mut ebpf = Ebpf::load(aya::include_bytes_aligned!(concat!(
        env!("CARGO_MANIFEST_DIR"),
        "/../ebpf/target/bpfel-unknown-none/release/lab-ebpf"
    )))
    .context("loading the eBPF object")?;

    let program: &mut TracePoint = ebpf.program_mut("count_exec").context("no such program")?.try_into()?;
    program.load().context("the kernel verifier rejected the program")?;
    program.attach("sched", "sched_process_exec")?;
    println!("attached count_exec to sched:sched_process_exec");

    // Generate some work to observe: three commands, a few times each.
    for (cmd, times) in [("true", 5), ("ls", 3), ("uname", 2)] {
        for _ in 0..times {
            Command::new(cmd).stdout(Stdio::null()).status()?;
        }
    }

    let counts: HashMap<_, [u8; 16], u64> = HashMap::try_from(ebpf.map("EXEC_COUNTS").context("no map")?)?;
    let mut rows: Vec<(String, u64)> = counts
        .iter()
        .map(|kv| kv.map(|(k, v)| (String::from_utf8_lossy(&k).trim_end_matches('\0').to_string(), v)))
        .collect::<Result<_, _>>()?;
    rows.sort();
    for (comm, n) in rows {
        println!("{comm:>8}  {n}");
    }
    Ok(())
}

The steps map one to one onto the concepts above. Ebpf::load parses the object. program_mut("count_exec") finds the program by its function name, and load() is the point at which the kernel's verifier runs. attach("sched", "sched_process_exec") connects it to the tracepoint named by category and event. The loader then launches known commands (five true, three ls, two uname) so the answer is predictable, reads the typed map and prints it. The key type is the same 16-byte array on both sides, and the typed HashMap<_, [u8; 16], u64> wrapper checks that at the boundary.

Running it

Loading eBPF needs privileges, and attaching a tracepoint needs the kernel's tracing file system mounted. In a container, that means --privileged and mounting tracefs. Here is the run:

docker run --rm --privileged -v "$PWD:/w" -w /w/app aya-lab bash -c \
  'mount -t tracefs nodev /sys/kernel/tracing; ./target/release/lab'

attached count_exec to sched:sched_process_exec
      ls  3
    true  5
   uname  2

The counts are exactly what the loader ran: three ls, five true, two uname. They are the kernel program's counts, not the loader's. Every exec in that window passed through count_exec and landed in the map. The total ran inside the kernel with no changes to ls and no tracing of the loader itself. (The loader's own start-up is not in the table because it started before the attach.)

Looking at the loaded program from outside

Once a program is in the kernel, it belongs to the system, not to your loader, and the standard tool for inspecting it is bpftool. I changed the loader to wait a few seconds before reading the map (an environment variable, HOLD_SECS) and ran bpftool from the same container while it was waiting:

bpftool prog show | grep -A2 count_exec
bpftool map show | grep -A1 EXEC_COUNTS
bpftool prog dump xlated id "$PID"

1597: tracepoint  name count_exec  tag 7cd66e66b72eecde  gpl
	loaded_at 2026-11-07T19:30:42+0000  uid 0
	xlated 848B  jited 600B  memlock 4096B  map_ids 106,105
106: hash  name EXEC_COUNTS  flags 0x0
	key 16B  value 8B  max_entries 1024  memlock 91744B
   0: (bf) r6 = r10
   1: (07) r6 += -16
   2: (bf) r1 = r6
   3: (b7) r2 = 0
   4: (b7) r3 = 16
   5: (85) call pc+92#bpf_prog_4f9302ba65b90ccd_F

The first line is the kernel's record of your program: id 1597, type tracepoint, the name you gave the function, a hash of its instructions (tag) and the licence string gpl. That licence matters: the kernel only lets a program call certain helper functions if it declares a GPL-compatible licence, and Aya sets it for you. The size line is a useful number. The program the verifier accepted is 848 bytes of translated eBPF instructions, and the kernel compiled it to 600 bytes of native arm64 code (jited). The map is a hash with 16-byte keys, 8-byte values and room for 1,024 entries, and the kernel charged 91,744 bytes of locked memory for it. Those 91,744 bytes are the cost of reserving space for the full table, not for the three entries in use, which is why max_entries deserves a thought.

The dump of the translated program starts the way the verifier trace did: the stack pointer copy, the call that zeroes the name buffer (a call to a subprogram, call pc+92), and then bpf_get_current_comm. It is the same code in a more compact form. Being able to read it is mostly useful for one purpose: when the verifier rejects something, the instruction numbers in its message refer to exactly this listing, so you can find the line that failed.

It watches the whole machine

The run with the wait was also instructive for what it counted. The first run, with no pause, saw only my three commands. This one printed the same three plus a number of other names:

attached count_exec to sched:sched_process_exec
       6  16
 bpftool  4
     cut  1
    grep  3
    head  3
      ls  3
mysqladmin  8
redis-cli  7
    runc  16
      sh  1
    true  5
   uname  2
    wget  1

The additional names are the shell commands I used to inspect (bpftool, grep, head, cut, sh) and, more tellingly, programs from other containers on the same virtual machine: mysqladmin, redis-cli, runc and wget, which are what health checks of unrelated containers execute. Even true and ls appear because the loader ran them. (The row whose name is just 6 is a process whose name is the digit 6; I did not identify it, and the row shows that names are whatever the kernel stored, not a vetted label.) That is the point and the hazard of this kind of tool in one table: a tracepoint is global. It sees every exec on the kernel, in every container and namespace, not just yours, which is why it needs privilege and why anything built on it should be treated as a security-sensitive component. It is also why filtering by cgroup or PID namespace inside the kernel program is a standard early improvement.

What the verifier refuses

The verifier is the part of eBPF that Rust developers meet first, because it is a second type checker that you cannot argue with. To see it, I removed the null check and wrote what looks like reasonable code: look up the counter and increment it.

unsafe {
    // Missing the null check: the lookup can fail.
    *EXEC_COUNTS.get_ptr_mut(&comm).unwrap_unchecked() += 1;
}

It compiles. Rust has no objection, because the unsafe block says to trust the programmer. The kernel does not. Loading it printed a long verifier trace, instruction by instruction with the state of each register, and at the end:

80: (85) call bpf_map_lookup_elem#1   ; R0=map_value_or_null(id=4,map=EXEC_COUNTS,ks=16,vs=8)
81: (79) r1 = *(u64 *)(r0 +0)
R0 invalid mem access 'map_value_or_null'
verification time 155 usec
stack depth 48+0
processed 159 insns (limit 1000000)

Read the last lines like a compiler error. Instruction 80 called the map lookup helper, and the verifier tracked its return value as map_value_or_null: a pointer that might be null. Instruction 81 then read through it, and the verifier stopped: "R0 invalid mem access". The register R0 holds the helper's result, and the verifier will not allow a dereference until a null comparison has narrowed its type. The trace before this point (about 160 instructions in all) also contains an unrelated detail worth seeing once: the compiler unrolled the zeroing of the 16-byte name buffer into a small loop, and the verifier walked it iteration by iteration, which is how it proves loops terminate. Programs that loop over data of unknown size are the ones that get rejected for exceeding the verifier's limits, which is why eBPF code favours bounded loops and fixed-size buffers.

The error text is long and the fix is short: restore the match. The lesson is that the kernel enforces a property Rust's types do not express here, and the error arrives at load time, not at compile time. So test loading in CI, not just compiling.

A mistake in the build loop

When I introduced the broken program I rebuilt only the kernel crate and ran the loader, and it happily printed the correct counts. The loader had embedded the old object when it was compiled: include_bytes_aligned! copies the file into the user-space binary at build time, so changing the kernel crate does nothing until the loader is rebuilt. After touch app/src/main.rs and a rebuild the verifier error appeared. If a change to your kernel program seems to have no effect, this is the first thing to check, and a build script (or the aya-build crate) that rebuilds the kernel crate and triggers the loader is the usual cure. The same trap runs in the other direction: the loader reported a correct result from a program I had already broken.

Reading the result safely

Three details of the map access matter in real tools. First, map reads from user space are not atomic with respect to the kernel program, so a counter can be read while it is being incremented; for monotonic counts that is fine, for pairs of values that must agree it is not. Second, per-CPU variants of the map types (PerCpuHashMap) avoid contention by giving each CPU its own slot, at the cost of the loader summing them. Third, the map has a fixed capacity, 1,024 here; when it is full, an insert fails, and the kernel program above silently ignores that failure with let _ =. A real tool would count the failures in a second map so lost events are visible.

Where this stops

• This ran in a virtual machine, not on bare metal. Docker Desktop on a Mac runs containers inside a Linux VM with kernel 7.0.14. That kernel had BPF, JIT and BTF enabled, and the tracepoint worked, but a production Linux host differs in kernel version, configuration, security modules and lockdown settings.
• Only one hook type was used. Tracepoints are the stable, easiest hook. Kprobes, uprobes, XDP, tc and cgroup hooks have different contexts, different helper sets and different stability promises. None was run here.
• aya-log and ring buffers were not used. Printing from the kernel and streaming events to user space are the next things most tools need, and they are the more complex part. A hash map polled from user space is the simplest pattern, not the most scalable.
• Privilege is the real cost. Loading eBPF needs CAP_BPF and CAP_PERFMON (or root) and, in the container, a privileged run. That is a large grant; scope it carefully and never expose a tool like this to untrusted users.
• The arm64 toolchain only. The linker tarball, the object and the numbers come from aarch64. The source is portable, but I did not build for x86-64.
• Cleanup is the loader's job. A program stays loaded only while something holds a reference to it. When the loader exits, Aya detaches the tracepoint and the kernel frees the program and the map; a tool that pins them to the BPF file system instead keeps them alive after exit, and then someone has to remove them. I only ran the first kind.
• The counting was a toy. It counts by process name, so two different programs with the same name merge, and a program that renames itself escapes. A real tool keys on PID and executable path.

The measured outcome is short. A 2,192-byte object built in 13.6 seconds was accepted by a real kernel, attached to sched_process_exec, and counted ten process launches exactly (three, five and two). Removing one null check produced a rejection at instruction 81 with a precise reason. Aya made both halves ordinary Rust, and the verifier made one of them stricter than Rust ever will be.