Writing a Linux Daemon in Rust With Systemd Notify

A Linux service that systemd starts, supervises and stops has to follow a small set of rules, and getting them right is the difference between a daemon that restarts cleanly at three in the morning and one that leaves a half-started process behind. Rust makes the program side easy and leaves the contract with systemd to you. This guide builds the contract piece by piece: how systemd learns that your service is ready (the notify protocol), how it learns that you are still alive (the watchdog), how you reload without dropping connections, how you shut down on SIGTERM, how to take a listening socket from systemd instead of binding one, and which unit-file options turn a normal process into a locked-down one. The notify protocol is a datagram sent to a Unix socket, so it can be tested without systemd; the programs below were run on macOS against a stand-in socket that records the same messages systemd would receive, and compiled with Rust 1.99, tokio 1.54, sd-notify 0.5 and listenfd 1. The unit files are the standard directives from systemd.service(5); check them with systemd-analyze verify on your machine, since this guide could not run systemd itself.

What systemd needs to know about your process

The default service type, Type=simple, treats the service as started the moment the process is forked. That is almost never true: the process is still reading configuration, opening ports and warming caches. Anything ordered After= your service may start while you are not ready, and a deployment script that runs systemctl start returns before the port accepts connections. With Type=notify, systemd waits until the service sends READY=1 over the notification socket. The same channel carries status text, the watchdog heartbeat and reload and stop announcements. The protocol is a short list of KEY=VALUE lines in one datagram sent to the path in the NOTIFY_SOCKET environment variable:

The daemon, with every signal handled

A daemon is a loop that waits for one of three things: a termination signal, a reload signal, or time for the next watchdog ping. In async Rust, that is a select! over signal streams and an interval. This is the whole daemon, with the "work" left out so the lifecycle is visible:

async fn daemon() {
    let mut term = signal(SignalKind::terminate()).unwrap();
    let mut hup = signal(SignalKind::hangup()).unwrap();

    tokio::time::sleep(Duration::from_millis(300)).await;
    sd_notify::notify(&[NotifyState::Ready, NotifyState::Status("accepting connections")]).unwrap();

    let mut watchdog = sd_notify::watchdog_enabled().map(|d| d / 2).map(tokio::time::interval);
    loop {
        tokio::select! {
            _ = term.recv() => {
                sd_notify::notify(&[NotifyState::Stopping, NotifyState::Status("draining")]).unwrap();
                tokio::time::sleep(Duration::from_millis(100)).await;
                return;
            }
            _ = hup.recv() => {
                let usec = sd_notify::NotifyState::monotonic_usec_now().unwrap();
                sd_notify::notify(&[NotifyState::Reloading, usec]).unwrap();
                tokio::time::sleep(Duration::from_millis(50)).await;
                sd_notify::notify(&[NotifyState::Ready, NotifyState::Status("reloaded")]).unwrap();
            }
            _ = async { match watchdog.as_mut() { Some(i) => { i.tick().await; } None => std::future::pending().await } } => {
                sd_notify::notify(&[NotifyState::Watchdog]).unwrap();
            }
        }
    }
}

Walk through what it does. It registers handlers for SIGTERM and SIGHUP first, so a signal arriving during startup is queued rather than killing the default way. The 300 ms sleep stands in for startup work. READY=1 and a status line are sent together in one datagram, which is what the crate's slice argument does. The watchdog interval is half the period systemd asked for, which is the documented rule of thumb: sd_notify::watchdog_enabled() returns Some(duration) only if WATCHDOG_USEC is set (and, with systemd, if WATCHDOG_PID names this process), so the daemon works unchanged outside systemd, where it simply does not ping. On SIGHUP it announces RELOADING=1 with the current monotonic time, which systemd requires for the notify-reload type, does the reload, and announces READY=1 again. On SIGTERM it announces STOPPING=1, drains, and returns, so the process exits with status 0.

Seeing the conversation without systemd

To check that these messages are what systemd would see, a test harness plays systemd's part: it binds a datagram socket, starts the daemon as a child with NOTIFY_SOCKET pointing at it and a 400 ms watchdog, sends SIGHUP at 900 ms and SIGTERM at 1,500 ms, and logs everything it receives:

fn harness() {
    let path = std::env::temp_dir().join(format!("notify-{}.sock", std::process::id()));
    let _ = std::fs::remove_file(&path);
    let sock = UnixDatagram::bind(&path).unwrap();
    sock.set_read_timeout(Some(Duration::from_millis(200))).unwrap();

    let exe = std::env::current_exe().unwrap();
    let mut child = Command::new(exe)
        .arg("daemon")
        .env("NOTIFY_SOCKET", &path)
        .env("WATCHDOG_USEC", "400000")
        .env("WATCHDOG_PID", "")
        .stdout(Stdio::null())
        .spawn()
        .unwrap();
    let pid = child.id() as i32;
    let t0 = Instant::now();

    let mut log: Vec<(u128, String)> = Vec::new();
    let mut buf = [0u8; 512];
    let mut sent_hup = false;
    let mut sent_term = false;
    loop {
        match sock.recv(&mut buf) {
            Ok(n) => log.push((t0.elapsed().as_millis() / 100 * 100, String::from_utf8_lossy(&buf[..n]).replace('\n', " | "))),
            Err(_) => {}
        }
        let ms = t0.elapsed().as_millis();
        if ms > 900 && !sent_hup {
            unsafe { libc::kill(pid, libc::SIGHUP) };
            sent_hup = true;
        }
        if ms > 1500 && !sent_term {
            unsafe { libc::kill(pid, libc::SIGTERM) };
            sent_term = true;
        }
        if let Some(status) = child.try_wait().unwrap() {
            log.push((t0.elapsed().as_millis() / 100 * 100, format!("process exited: {status}")));
            break;
        }
    }
    let _ = std::fs::remove_file(&path);
    let mut last = String::new();
    for (ms, line) in log {
        if line.starts_with("WATCHDOG=1") && last == line {
            continue;
        }
        println!("{ms:>5} ms  {line}");
        last = line;
    }
}

fn main() {
    if std::env::args().nth(1).as_deref() == Some("daemon") {
        tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap().block_on(daemon());
    } else {
        harness();
    }
}

300 ms  READY=1 | STATUS=accepting connections | 
  300 ms  WATCHDOG=1 | 
  900 ms  RELOADING=1 | MONOTONIC_USEC=776077409717 | 
  900 ms  READY=1 | STATUS=reloaded | 
 1100 ms  WATCHDOG=1 | 
 1500 ms  STOPPING=1 | STATUS=draining | 
 1700 ms  process exited: exit status: 0

The timeline is the lifecycle. At 300 ms the service reports ready with a status, then pings the watchdog (the repeated pings every 200 ms are collapsed in the log). At 900 ms the SIGHUP produced the two-message reload sequence, and the monotonic timestamp is a microsecond count that differs on each run. At 1,500 ms SIGTERM produced STOPPING=1, and 200 ms later the process exited with status 0. A harness like this is worth keeping as an integration test: it asserts the protocol without root, without a VM and without waiting for a real systemd. The one thing it cannot check is systemd's reaction, for instance that a missed watchdog really kills and restarts the service; that belongs in a staging test with systemd-run.

The unit file

The unit file turns those behaviours on and adds the supervision policy. This one is a production-shaped starting point; the comments say what each line is for:

[Unit]
Description=My Rust service
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
NotifyAccess=main
ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.toml
ExecReload=/bin/kill -HUP $MAINPID

# Supervision
Restart=on-failure
RestartSec=2
WatchdogSec=10
TimeoutStartSec=30
TimeoutStopSec=30

# Identity and state
DynamicUser=yes
StateDirectory=myapp
ConfigurationDirectory=myapp
LogsDirectory=myapp

# Hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
CapabilityBoundingSet=
MemoryMax=512M

[Install]
WantedBy=multi-user.target

Read it in three groups. Supervision. Type=notify with NotifyAccess=main means only the main process may send notifications, which stops a stray child from claiming readiness. Restart=on-failure restarts after a non-zero exit, a signal death, or a watchdog timeout, but not after a clean stop. WatchdogSec=10 promises systemd a ping at least every 10 seconds (the daemon sends one every five); if none arrives, systemd kills the service and the restart policy applies, which turns a deadlocked process into a restarted one. The two timeouts bound startup and shutdown, after which systemd sends SIGKILL. Identity. DynamicUser=yes allocates a transient unprivileged user for the service, and the three *Directory= options create directories under /var/lib, /etc and /var/log that the user owns, so the service never needs root and never needs a pre-created account. Hardening. NoNewPrivileges blocks privilege escalation via setuid binaries; ProtectSystem=strict makes the whole file system read-only except the directories you declare; PrivateTmp gives the service its own /tmp; CapabilityBoundingSet= with nothing after it drops every Linux capability; RestrictAddressFamilies forbids exotic socket types; MemoryMax turns a leak into an out-of-memory kill of this service only, instead of the host. Check how exposed a unit is with systemd-analyze security myapp.service, which scores each directive. Binding a port below 1024 under these rules needs either socket activation (next section) or AmbientCapabilities=CAP_NET_BIND_SERVICE.

Socket activation: take the socket instead of binding it

Letting systemd create the listening socket and pass it to your process solves three problems at once. The service can run unprivileged yet listen on port 80, because systemd (root) bound it. Connections that arrive while the service is restarting queue in the kernel's accept backlog instead of being refused, so a restart or deploy drops nothing. And the service can start on demand. The unit pair is a socket unit and a service unit with the same name:

[Socket]
ListenStream=0.0.0.0:8080
Accept=no

[Install]
WantedBy=sockets.target

On start, systemd passes the socket as file descriptor 3 and sets LISTEN_FDS=1 and LISTEN_PID to your pid. The listenfd crate reads those and hands you a listener, with a fallback to binding yourself when the process was not started by systemd, so the same binary works in development:

use listenfd::ListenFd;
use std::net::TcpListener;

fn listener() -> std::io::Result<TcpListener> {
    let mut fds = ListenFd::from_env();
    match fds.take_tcp_listener(0)? {
        Some(l) => {
            println!("using the socket systemd passed in (fd 3)");
            Ok(l)
        }
        None => {
            println!("no inherited socket, binding 127.0.0.1:0 myself");
            TcpListener::bind("127.0.0.1:0")
        }
    }
}

fn main() -> std::io::Result<()> {
    let l = listener()?;
    println!("listening on {}", l.local_addr()?.ip());
    Ok(())
}

Run outside systemd, it takes the fallback path, as the output shows when it ran here:

no inherited socket, binding 127.0.0.1:0 myself
listening on 127.0.0.1

Under systemd the first branch runs and prints the message about the inherited socket, with the address configured in the socket unit. Convert the listener to Tokio with tokio::net::TcpListener::from_std after calling set_nonblocking(true). Because the socket outlives the process, a systemctl restart myapp never closes the port.

Reload that cannot take the service down

The daemon announces a reload, but the interesting part is what the reload does. Three rules make it safe: parse and validate the new configuration completely before touching the live one; replace the live configuration atomically, so no request ever sees half of the old and half of the new; and keep the old configuration if the new one is bad, reporting the error instead of exiting. A small shared structure delivers all three. Readers take an Arc snapshot and keep using it for the whole request, and the reload swaps the pointer under a write lock held for a few nanoseconds:

use std::sync::{Arc, RwLock};

#[derive(Debug, Clone, PartialEq)]
struct Config {
    workers: usize,
    greeting: String,
}

fn parse(text: &str) -> Result<Config, String> {
    let mut workers = None;
    let mut greeting = None;
    for (n, line) in text.lines().enumerate().filter(|(_, l)| !l.trim().is_empty()) {
        let (key, value) = line.split_once('=').ok_or_else(|| format!("line {}: expected key=value", n + 1))?;
        match key.trim() {
            "workers" => {
                let w: usize = value.trim().parse().map_err(|e| format!("line {}: workers: {e}", n + 1))?;
                if !(1..=64).contains(&w) {
                    return Err(format!("line {}: workers must be 1..=64, got {w}", n + 1));
                }
                workers = Some(w);
            }
            "greeting" => greeting = Some(value.trim().to_string()),
            other => return Err(format!("line {}: unknown key {other:?}", n + 1)),
        }
    }
    Ok(Config { workers: workers.ok_or("missing workers")?, greeting: greeting.ok_or("missing greeting")? })
}

struct Shared {
    current: RwLock<Arc<Config>>,
}

impl Shared {
    fn snapshot(&self) -> Arc<Config> {
        self.current.read().unwrap().clone()
    }

    fn reload(&self, text: &str) -> Result<(), String> {
        let new = parse(text)?;
        *self.current.write().unwrap() = Arc::new(new);
        Ok(())
    }
}

fn main() {
    let shared = Shared { current: RwLock::new(Arc::new(parse("workers = 4\ngreeting = hello").unwrap())) };

    let in_flight = shared.snapshot();
    println!("start:  {:?}", shared.snapshot());

    println!("reload ok:     {:?}", shared.reload("workers = 8\ngreeting = hello again"));
    println!("now serving:   {:?}", shared.snapshot());
    println!("old request still sees: {:?}", in_flight);

    println!("reload broken: {:?}", shared.reload("workers = 800\ngreeting = x"));
    println!("reload typo:   {:?}", shared.reload("worker = 2"));
    println!("still serving: {:?}", shared.snapshot());
}

start:  Config { workers: 4, greeting: "hello" }
reload ok:     Ok(())
now serving:   Config { workers: 8, greeting: "hello again" }
old request still sees: Config { workers: 4, greeting: "hello" }
reload broken: Err("line 1: workers must be 1..=64, got 800")
reload typo:   Err("line 1: unknown key \"worker\"")
still serving: Config { workers: 8, greeting: "hello again" }

Read the output as a sequence of promises. The snapshot taken before the reload (in_flight) still sees four workers after the swap, so a request that started under the old configuration finishes under it; no request ever observes an inconsistent mix. The valid reload took effect for new snapshots. The two bad reloads returned errors that name the line and the problem, and the service kept serving the previous configuration. In the daemon, the SIGHUP branch reads the file, calls reload, and on Err logs the message and sends STATUS=reload failed: ... followed by READY=1, so the operator sees the failure in systemctl status while production traffic continues. The lock is a plain RwLock because the critical section is a pointer copy; for a hot path with millions of reads per second, the arc-swap crate makes reads lock-free, a trade measured in sharing state between tasks.

What a reload cannot change is anything bound to a resource: the listening address, a TLS key already loaded into a handshake context, the number of worker threads in a started runtime. For those, the honest options are to document that they need a restart, or to implement the change by starting new resources, switching traffic and draining the old ones, which is a mini deployment inside the process. Keep the set of reloadable settings small and write it down in the unit's documentation; an operator who sends SIGHUP and discovers that the port did not change has been misled by the signal's name. A related system-level detail: systemd does not run ExecReload when the main process is not active, and treats a non-zero exit from the reload command as a failed reload, which is why the unit above uses kill -HUP (it exits 0 once the signal is delivered) and relies on the notify protocol for the outcome.

Knowing you are running under systemd

A binary that runs both on laptops and under systemd should adapt without flags. Systemd sets a few environment variables you can read. INVOCATION_ID exists for every service started by systemd and is unique per run, so it doubles as a correlation id in logs. JOURNAL_STREAM is set when stdout or stderr is connected to the journal, and its presence is the signal to drop timestamps and colour from your log format. NOTIFY_SOCKET and WATCHDOG_USEC, as above, enable the lifecycle messages, and the sd-notify crate already treats their absence as "do nothing", so the same code path is safe everywhere. LISTEN_FDS enables socket activation, handled by listenfd. Checking these is cheaper and more accurate than checking whether the parent process is pid 1.

Environment is also the right place for the deployment's secrets and paths, with EnvironmentFile=/etc/myapp/env in the unit (mode 0600, owned by root), or better LoadCredential=, which hands the service a file in a private directory named by $CREDENTIALS_DIRECTORY that is never visible to other users or to systemctl show. Reading a credential is a plain file read, so the Rust side stays simple, and it keeps keys out of the process environment where /proc/PID/environ and crash reporters can see them.

Logging: write to stdout, let the journal do the rest

A service under systemd should not manage log files or rotate them. Write to standard output and standard error; the journal captures both, attaches the unit name, user and timestamps, and rotates by size. Read with journalctl -u myapp -f. Two refinements are worth making. Do not print timestamps yourself, since the journal adds them (configure your tracing subscriber with .without_time() when it detects JOURNAL_STREAM in the environment). And preserve severity: the journal understands a syslog-style prefix, so a line starting with <3> is stored as an error and <6> as info (the SyslogLevelPrefix default is on). The tracing-journald crate sends structured fields directly instead, which makes journalctl -o json output queryable by field. Keep one rule for secrets: nothing the service prints may contain them, because the journal is world-readable to members of the systemd-journal group; see logging without leaking secrets.

Shutting down properly

When you run systemctl stop, systemd sends SIGTERM to the main process, waits up to TimeoutStopSec, and then sends SIGKILL to everything left. Your job in that window is the drain: stop accepting, finish in-flight work, flush, exit with status 0. The pattern that does it with cancellation tokens and a deadline is in graceful shutdown, and the daemon above is its skeleton. Two details specific to systemd: the stop is announced with STOPPING=1 so that tools showing the state see "deactivating" with your status text, and exit status 0 after SIGTERM is what makes Restart=on-failure leave the service stopped. A process that exits non-zero because it caught the signal and returned an error would be restarted by the supervisor, which is rarely what someone typing systemctl stop wanted.

What goes wrong

Checking the unit before you deploy

systemd-analyze verify /etc/systemd/system/myapp.service
systemd-analyze security myapp.service
systemd-run --user --collect --wait --property=Type=notify ./target/release/myapp
journalctl -u myapp --since "5 minutes ago"

The first catches typos and unknown directives, the second scores the sandbox and lists what is still exposed, and the third runs your binary under a transient unit with the same contract, a quick way to try the notify handshake on a laptop that runs systemd. In CI, run the harness from this guide to cover the part that is pure Rust.

Where this does not apply

Inside a container, the container runtime is the supervisor: Docker and Kubernetes restart containers, send SIGTERM and read exit codes, but they ignore NOTIFY_SOCKET unless you run systemd inside the container, and their health model is probes, not the watchdog. The shutdown and signal handling carry over directly (see graceful shutdown on Kubernetes), the unit file does not. On macOS, launchd has its own plist format and no equivalent of the notify protocol, and on Windows a service uses the Service Control Manager API. If you target several, put the lifecycle behind a small trait (ready(), status(), stopping()) with a no-op implementation for platforms without it. For the systemd socket and watchdog features in more depth, see systemd sockets and watchdog.