Rust and Systemd Services Sockets and Watchdogs Explained

If you deploy to a VM without Kubernetes, systemd is your process supervisor, and it does a lot more than ExecStart. A few features turn it into a good production environment for a Rust binary: readiness notification, watchdogs, socket activation and sandboxing. This post wires them up. Systemd 252 or later, Rust 1.90 with edition 2024, sd-notify 0.4 and Tokio 1.47.

A baseline unit

# /etc/systemd/system/api.service
[Unit]
Description=Example API
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/opt/api/api
Restart=on-failure
RestartSec=2
TimeoutStopSec=30
WatchdogSec=30
EnvironmentFile=/etc/api/api.env

# sandboxing
DynamicUser=yes
StateDirectory=api
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
CapabilityBoundingSet=
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

DynamicUser=yes creates a transient unprivileged user for the service, and StateDirectory=api gives it a writable /var/lib/api. ProtectSystem=strict mounts the rest of the filesystem read-only for the service. These cost nothing and limit damage if the service is compromised (security checklist). LimitNOFILE raises the descriptor limit (too many open files). Run systemd-analyze security api.service to see a score and what else to restrict.

Readiness: Type=notify

[dependencies]
sd-notify = "0.4"
tokio = { version = "1.47", features = ["full"] }

use sd_notify::NotifyState;

#[tokio::main]
async fn main() -> std::io::Result<()> {
    let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await?;
    // ... run migrations, warm caches, connect to the database ...

    let _ = sd_notify::notify(false, &[NotifyState::Ready]);   // READY=1: now systemd marks us active
    serve(listener).await;
    let _ = sd_notify::notify(false, &[NotifyState::Stopping]);
    Ok(())
}

async fn serve(_l: tokio::net::TcpListener) {}

With Type=simple systemd considers the service started as soon as the process forks, so units that depend on it (After=api.service) may start before it listens. Type=notify delays "active" until you send READY=1. The call is harmless outside systemd (no NOTIFY_SOCKET set, it returns without effect), so the same binary runs everywhere. Send STATUS=... messages too for systemctl status to display.

The watchdog

use sd_notify::NotifyState;
use std::time::Duration;

fn spawn_watchdog() {
    let mut usec = 0;
    if !sd_notify::watchdog_enabled(false, &mut usec) {
        return;
    }
    let interval = Duration::from_micros(usec / 2);   // ping at half the timeout
    tokio::spawn(async move {
        let mut tick = tokio::time::interval(interval);
        loop {
            tick.tick().await;
            if app_is_healthy().await {
                let _ = sd_notify::notify(false, &[NotifyState::Watchdog]);
            }
        }
    });
}

async fn app_is_healthy() -> bool { true }

With WatchdogSec=30, systemd expects a WATCHDOG=1 message at least every 30 seconds and kills and restarts the service if none arrives. A crashed process is already restarted by Restart=; the watchdog catches the process that is alive but stuck: a deadlock, a blocked executor, an infinite loop. Make the ping depend on real health, not just "the timer task runs": if the executor is blocked, the timer task will not run either, which is the signal you want, and for a deeper check have the ping task verify that a request-path canary still works. Do not ping from a separate OS thread unconditionally, or a hung service will keep pinging.

Socket activation

# /etc/systemd/system/api.socket
[Socket]
ListenStream=3000
Backlog=1024

[Install]
WantedBy=sockets.target

use std::os::fd::FromRawFd;

fn listener_from_systemd() -> Option<std::net::TcpListener> {
    let pid_ok = std::env::var("LISTEN_PID").ok()?.parse::<u32>().ok()? == std::process::id();
    let fds: i32 = std::env::var("LISTEN_FDS").ok()?.parse().ok()?;
    if !pid_ok || fds < 1 {
        return None;
    }
    // SAFETY: systemd passes the first socket as fd 3 (SD_LISTEN_FDS_START) when LISTEN_FDS >= 1
    let l = unsafe { std::net::TcpListener::from_raw_fd(3) };
    l.set_nonblocking(true).ok()?;
    Some(l)
}

#[tokio::main]
async fn main() -> std::io::Result<()> {
    let std_listener = match listener_from_systemd() {
        Some(l) => l,
        None => std::net::TcpListener::bind("0.0.0.0:3000")?,
    };
    std_listener.set_nonblocking(true)?;
    let _listener = tokio::net::TcpListener::from_std(std_listener)?;
    Ok(())
}

Systemd opens and holds the socket and starts the service on the first connection (or at boot, depending on configuration), passing the listening descriptor as fd 3 and telling you through LISTEN_FDS and LISTEN_PID. The listenfd crate wraps exactly this check. The benefit: during systemctl restart api the socket stays open, so clients connecting in the gap queue in the backlog instead of being refused, which gives near-zero-downtime restarts for simple services. It also allows binding privileged ports without running as root. Keep a fallback to bind so local runs without systemd work.

Graceful stop and logging

• Systemd sends SIGTERM, waits TimeoutStopSec, then SIGKILL. Handle SIGTERM and stop accepting work, finish in-flight requests and exit (Axum graceful shutdown). Send STOPPING=1 when you begin.
• Log to stdout and stderr; the journal collects them with the unit name and priority. View with journalctl -u api -f. With tracing, the tracing-journald layer attaches structured fields.
• Set Restart=on-failure with a RestartSec and StartLimitBurst so a crash loop does not hammer a dependency.

Check it works

systemctl daemon-reload && systemctl enable --now api.socket api.service
systemctl status api
systemd-analyze security api.service
kill -STOP $(systemctl show -p MainPID --value api)   # simulate a hang; the watchdog should restart it

Testing the watchdog by stopping the process confirms the whole chain. For containerized equivalents of readiness and liveness, see health checks and Rust on Kubernetes.