Monitoring Rust Services With Grafana Dashboards Explained

A dashboard is worth building only if it answers one question fast: is the service healthy right now, and if not, where does it hurt? Most Rust services need five panels, not fifty, and the difficult part is not drawing them. It is choosing metric names, histogram buckets and query windows that make the panels tell the truth. This guide builds the whole chain with real software: a Rust service that exports Prometheus metrics, a Prometheus server scraping it, and a Grafana instance whose datasource and dashboard are provisioned from files. Then it asks Grafana's own query API what each panel would show, and prints the answers. Along the way, a histogram bucket choice makes one panel report a 99th-percentile latency that is double the real worst case, and the guide shows the fix and the corrected numbers. Versions: rustc 1.99.0, prometheus crate 0.14, Prometheus 3.5.0, Grafana 12.2.0, run in Docker on an Apple M1 Max. The traffic is simulated by a deterministic generator, so the figures describe this experiment and not a production service.

What the service exports

The four golden signals reduce, for a request-serving service, to three numbers per route, known as RED: the rate of requests, the errors among them and their duration. Add one saturation gauge, requests in flight, and a dashboard can say whether the service is busy, failing or slow. Those map to three Prometheus metric types: a counter labelled by path and status, a histogram of latency labelled by path, and a gauge. In Rust with the prometheus crate:

use prometheus::{Encoder, HistogramOpts, HistogramVec, IntCounterVec, IntGauge, Opts, Registry, TextEncoder};
use std::sync::Arc;

struct Metrics {
    requests: IntCounterVec,
    duration: HistogramVec,
    in_flight: IntGauge,
}

fn build() -> (Registry, Arc<Metrics>) {
    let registry = Registry::new();
    let requests = IntCounterVec::new(Opts::new("http_requests_total", "Requests by path and status"), &["path", "status"]).unwrap();
    // Buckets are chosen around a 250 ms latency target, not left at the defaults.
    let duration = HistogramVec::new(
        HistogramOpts::new("http_request_duration_seconds", "Request latency")
            .buckets(vec![0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5]),
        &["path"],
    )
    .unwrap();
    let in_flight = IntGauge::new("http_requests_in_flight", "Requests being served").unwrap();
    registry.register(Box::new(requests.clone())).unwrap();
    registry.register(Box::new(duration.clone())).unwrap();
    registry.register(Box::new(in_flight.clone())).unwrap();
    (registry, Arc::new(Metrics { requests, duration, in_flight }))
}

Three choices in that code are worth a sentence each. The counter and histogram carry a path label whose values are the route templates such as /api/items, never raw URLs with ids in them: every distinct label value is a separate time series, and an unbounded label (a user id, a full URL) is the usual way a metrics system runs out of memory. The status label holds the code, so errors can be separated from successes without a second metric. And the histogram buckets are set explicitly, because the defaults assume a different kind of service. I chose them around a 250 ms latency target at first; the sections below show how that choice backfires.

To exercise the dashboard, a background thread simulates traffic against three routes with different personalities: a fast, rarely failing /api/items (about 60% of requests), a mid-weight /api/search (30%) and a slow, flakier /api/report (10%). The generator is a fixed linear congruential sequence, so the traffic is repeatable. The service serves the registry over a tiny HTTP listener on port 9464, using the text exposition format. A few seconds after start, the metrics endpoint returned lines like these:

http_request_duration_seconds_sum{path="/api/items"} 6.333625532500451
http_request_duration_seconds_count{path="/api/items"} 260
http_request_duration_seconds_sum{path="/api/report"} 26.767241645185273
http_request_duration_seconds_count{path="/api/report"} 35
http_requests_in_flight 0
http_requests_total{path="/api/items",status="200"} 259
http_requests_total{path="/api/items",status="500"} 1
http_requests_total{path="/api/report",status="200"} 33
http_requests_total{path="/api/report",status="500"} 2

A histogram appears as a family: a _count, a _sum and one _bucket series per boundary. Each counter only ever goes up, which is the property that lets Prometheus compute rates by subtracting two samples and survive a restart of the service.

Prometheus: scrape it

Prometheus pulls. A four-line configuration tells it where the service is and how often to look. The target is host.docker.internal because Prometheus ran in a container and the Rust binary ran on the host:

global:
  scrape_interval: 5s
scrape_configs:
  - job_name: app
    static_configs:
      - targets: ["host.docker.internal:9464"]

After starting the container, the targets API reported app up http://host.docker.internal:9464/metrics. A five-second interval is aggressive and chosen so the experiment fills in quickly; fifteen seconds is the usual default. The interval matters for queries: a rate() range must contain at least two samples, and a good rule is a window of four times the scrape interval. With a 5 second interval, the 1 minute windows used below hold twelve samples. In production with 15 seconds, the common [5m] window holds twenty.

Grafana: provision, do not click

A dashboard built by hand in the browser lives in Grafana's database and is lost with it, cannot be reviewed and drifts between environments. Provisioning loads the datasource and the dashboard from files at start-up, so both can live in version control. Three files do it. The datasource has a fixed uid that dashboards refer to, which is the detail that stops panels from losing their datasource when it is recreated:

apiVersion: 1
datasources:
  - name: Prometheus
    uid: prom
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

apiVersion: 1
providers:
  - name: rust
    type: file
    options:
      path: /var/lib/grafana/dashboards

The provider points at a directory; every JSON file in it becomes a dashboard. Grafana, started with those mounted, answered the API with the datasource Prometheus prometheus prom and the dashboard Rust service: RED with 5 panels, with provisioned: true, which means the UI will refuse to overwrite it and tell you to change the file instead. That flag is the behaviour you want for something that is supposed to be reviewed.

The dashboard file

A dashboard is a JSON document. Most of the bulk is layout; the part that matters is each panel's targets, the PromQL. This is the latency panel in full, the one with three queries:

{
  "id": 3,
  "type": "timeseries",
  "title": "Latency p50 / p95 / p99",
  "datasource": { "type": "prometheus", "uid": "prom" },
  "gridPos": { "x": 16, "y": 0, "w": 8, "h": 8 },
  "targets": [
    { "refId": "A", "legendFormat": "p50",
      "expr": "histogram_quantile(0.5, sum by (le) (rate(http_request_duration_seconds_bucket{job=\"app\"}[1m])))" },
    { "refId": "B", "legendFormat": "p95",
      "expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job=\"app\"}[1m])))" },
    { "refId": "C", "legendFormat": "p99",
      "expr": "histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job=\"app\"}[1m])))" }
  ],
  "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] }
}

The expression is the one to understand. rate(...[1m]) turns each cumulative bucket counter into a per-second rate over the last minute. sum by (le) adds the rates across every label except the bucket boundary, which merges all routes and all instances into one distribution; the le must be kept, because the quantile function needs the boundaries. histogram_quantile(0.99, ...) then estimates the value below which 99% of observations fall. Averaging per-instance percentiles is a common mistake and is wrong, because percentiles do not average. Aggregate the buckets first, then take the quantile. The other four panels use the same pieces: sum by (path) (rate(http_requests_total{job="app"}[1m])) for traffic, the 5xx rate divided by the total rate for errors, and the gauge as it is.

What each panel shows

Grafana has an HTTP API that evaluates a panel's queries the same way the browser does. I called it for each panel in the provisioned dashboard, about two minutes after the traffic started, using POST /api/ds/query with the panel's expression and an instant evaluation. This is a more reliable check than a screenshot, because it proves the datasource uid, the query and the data all line up, and it can run in CI. The first run, with the buckets chosen around a 250 ms target, returned the following (tidied into columns):

Requests per second by path -> /api/items 87.0, /api/report 16.0, /api/search 43.9
Error ratio                 -> 0.0167
Latency p50 / p95 / p99     -> 0.0423 s, 0.7891 s, 1.9202 s
In flight                   -> 0
p99 by path                 -> /api/items 0.0495, /api/report 2.437, /api/search 0.2481

Read it as an operator would. Traffic is about 147 requests per second, split 60/30/10 across the routes, as the generator intended. The error ratio is 1.67%, which would be yellow on a panel with a 1% warning threshold. Median latency is 42 ms and healthy. But the p99 is 1.92 seconds, and the per-path panel points at /api/report at 2.44 seconds. The dashboard did its job: one glance names the slow route and the error rate, without a single log line.

Now check the number against the truth, which this experiment can do because the generator is known. A /api/report request takes 0.6 * (0.5 + u * 1.5) seconds, where u is uniform between 0 and 1. The slowest possible request is therefore 1.2 seconds, and the real 99th percentile is about 1.19. The dashboard said 2.44. It was off by a factor of two, in the direction that would send an engineer looking for a problem that does not exist.

The histogram bucket trap

The cause is the bucket list. The configured boundaries ran 0.5, 1.0, 2.5 and all the slow observations landed between 1.0 and 1.2 seconds, inside the single wide bucket from 1.0 to 2.5. Prometheus knows how many observations fell in each bucket but not where inside it, so histogram_quantile assumes they are spread evenly across the bucket. The 99th percentile sits near the top of that bucket's counts, and the interpolation places it near the top of the range: 2.4 seconds. The estimate is only as good as the bucket, and a wide bucket around the value you care about makes it useless. The fix is to add boundaries around the latencies that matter, here 0.75, 1.25 and 1.5, and restart the service:

HistogramOpts::new("http_request_duration_seconds", "Request latency")
    .buckets(vec![0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 2.5]),

Here a second lesson appeared. I queried right after the restart and the per-path p99 for /api/report still read 2.33 seconds. The new boundaries (0.75, 1.25, 1.5) are new time series that start at zero when the process starts, and a one-minute rate() window that straddles the restart mixes series that did and did not exist. After waiting for the window to fill with data from the new process, the same query was stable. That is also what a rolling deploy looks like on a dashboard, so do not trust a quantile in the first minutes after a bucket change. Once settled, the whole dashboard returned:

Requests per second by path -> /api/items 87.7, /api/report 15.7, /api/search 43.7
Error ratio                 -> 0.0183
Latency p50 / p95 / p99     -> 0.042 s, 0.79 s, 1.1418 s
In flight                   -> 0
p99 by path                 -> /api/report 1.2385, /api/search 0.2481, /api/items 0.0495

The per-path p99 for /api/report fell from 2.437 to 1.2385 seconds, against a true value of about 1.19, and the overall p99 fell from 1.92 to 1.14 seconds against a true value near 1.11. The remaining error, about 4%, is the interpolation inside the 1.0 to 1.25 bucket and is what a bucket-based estimate always costs. Everything else is unchanged within noise: the 95th percentile moved from 0.79 to 0.79 seconds, because it already sat in narrow buckets. The lesson generalises: put bucket boundaries densely around your latency objective and around the values you alert on, and check the estimate against a known number at least once. A histogram with defaults is a convenient way to produce confident, wrong percentiles.

Recording rules and a first alert, run for real

The p99 query is the most expensive on the dashboard, and an alert needs the same expression. Prometheus can evaluate it once and store the result as a new series, called a recording rule, so panels and alerts read the cheap precomputed value. The naming convention is level:metric:operations. I added two recording rules and one alert, checked the file with promtool check rules, which reported SUCCESS: 3 rules found, and mounted it into the Prometheus container:

groups:
  - name: red
    interval: 5s
    rules:
      - record: path:http_request_duration_seconds:p99_1m
        expr: histogram_quantile(0.99, sum by (le, path) (rate(http_request_duration_seconds_bucket{job="app"}[1m])))
      - record: job:http_requests:error_ratio_1m
        expr: sum(rate(http_requests_total{job="app",status=~"5.."}[1m])) / sum(rate(http_requests_total{job="app"}[1m]))
      - alert: HighErrorRatio
        expr: job:http_requests:error_ratio_1m > 0.01
        for: 30s
        labels:
          severity: page
        annotations:
          summary: "More than 1% of requests are failing"

The rules API then reported all three rules healthy, and querying the recorded series returned the same per-path values the dashboard had shown: 0.0495 seconds for /api/items, 1.2379 for /api/report and 0.2481 for /api/search. The alert's expression held at about 0.015 to 0.018, above the 0.01 threshold, and the alert sat in the pending state, which is what the for: 30s clause means: the condition must stay true for the whole period before the alert fires. I stopped checking while it was still pending, so this guide has not observed it reach firing, and the error ratio hovering near the threshold in a simulated run is a reminder that thresholds need headroom. Two design points follow. The for clause is what keeps a single bad scrape from paging anyone, and it is also a delay, so choose it against how long an outage you can tolerate. And the alert reads the recorded ratio, not the raw query, so the dashboard panel, the alert and any report all agree on one definition of "error ratio".

A last practical point on cost. Recording rules trade storage for query time: each rule adds one series per label combination it outputs, here three for the p99 rule and one for the error ratio. That is cheap. The mistake to avoid is recording a rule for every query "in case", which fills the database with series nobody reads. Record the expressions that more than one consumer needs, or that are slow enough to make a panel lag, and leave the rest as ordinary queries. Review the list when a dashboard is retired, since an orphaned rule keeps evaluating every few seconds for as long as the file lists it.

Rules that keep the dashboard honest

• Ratios for errors, not counts. The error panel divides the 5xx rate by the total rate. Fifty errors a second means opposite things at 100 and at 100,000 requests a second.
• Aggregate buckets, then take the quantile. histogram_quantile(q, sum by (le) (rate(...))). Never average percentiles from instances.
• Window at least four scrape intervals. One minute at a 5 second interval gives twelve samples; a window with fewer than two returns nothing, which looks like a healthy zero.
• Bound label values. Route templates, not URLs; status codes or classes, not messages. Count the series a label will create before you ship it.
• Put units on every panel. The panel above sets "unit": "s" and the requests panel uses reqps, so the axis reads in seconds and requests per second instead of bare numbers.
• Order panels by the question. Top row for rate, errors and duration (is it broken?), second row for saturation (is it about to break?), detail and per-instance breakdowns below.
• Keep the dashboard in version control. Provisioned files are reviewed like code, and the API check above can run in CI to catch a panel whose query returns nothing.

Variables, thresholds and alerts

Three more features turn a static page into a usable one, and the guide did not run them, so treat these as documented Grafana behaviour and not as observed results. A template variable such as $path, defined by the query label_values(http_requests_total, path), lets one set of panels filter to a route, with queries written as path=~"$path". Thresholds on the error-ratio panel colour it yellow at 1% and red at 5%; choose the numbers from your service-level objective, not from the example. And alert rules belong in the same repository as the dashboard: the error ratio above a threshold for several minutes is the natural first alert, and a burn-rate alert on the same query is the better second one. Alert on symptoms users feel (errors, latency) and keep causes (CPU, in flight) as dashboard panels.

Where this stops

• The traffic was simulated. A deterministic generator wrote the metrics straight into the registry; no HTTP requests were served, so real concerns such as handler overhead, the cost of recording a metric on a hot path and label lookup time were not measured.
• The Grafana UI was not looked at. I verified provisioning and every panel's data through the API. Layout, colours and how a panel renders in a browser were not inspected, and an API result does not prove that a visualisation is readable.
• One instance. The aggregation advice (sum the buckets across instances) is standard and is why the queries are written as they are, but this experiment ran a single target and did not exercise multi-instance behaviour or restarts of Prometheus itself.
• Short windows. One-minute rates suit an experiment. Production dashboards usually use five-minute windows for stability and show faster signals on separate panels or in alerts.
• Not a replacement for traces and logs. A dashboard says which route is slow. Finding why takes spans and logs, and the dashboard is the place that tells you which to open.
• Cost. Every series is stored. A histogram with twelve buckets, three routes and a status label is manageable; the same histogram with a label of user ids is an outage of your monitoring.

The measured outcome in one paragraph: a Rust service exporting three metrics, scraped every five seconds and provisioned into Grafana from files, produced a five-panel RED dashboard whose every query returned data through the API. The first version overstated the slowest route's 99th percentile at 2.44 seconds against a true 1.19 because of one wide histogram bucket, and adding three boundaries brought it to 1.24. The lesson is in the buckets, not in Grafana.