Observability for Rust Services With OpenTelemetry

A log line tells you what one process said at one moment. It cannot tell you why a checkout took 2.4 seconds when the checkout service, the payment service and the inventory service all report success in under 300 ms each. For that you need a trace, a record of the whole request that follows it across process boundaries, and metrics that summarize thousands of such requests so you can see the 2.4 second outlier in context. OpenTelemetry is the vendor-neutral standard for emitting all three, and in Rust it plugs into the tracing crate you probably already use.

This guide takes a service from no telemetry to traces in a collector, with context propagating between two services and metrics beside them. The versions are the ones compiled for this article: Rust 1.99 (edition 2024), opentelemetry 0.33.1, opentelemetry_sdk 0.33.1, opentelemetry-otlp 0.33.1 and tracing-opentelemetry 0.34.0. These crates change their APIs between minor releases, so pin the four together and read the changelog before upgrading any one.

How the pieces fit together

OpenTelemetry in Rust is a stack, and most confusion comes from not knowing which layer a type belongs to.

The design choice that matters is that your application code only ever touches tracing. You do not write OpenTelemetry spans by hand. If you later change backends, or stop exporting, none of your instrumented functions change. Everything specific to OpenTelemetry lives in one initialization function.

Dependencies

[dependencies]
anyhow = "1"
opentelemetry = "0.33"
opentelemetry_sdk = "0.33"
opentelemetry-otlp = { version = "0.33", features = ["grpc-tonic"] }
tracing = "0.1"
tracing-opentelemetry = "0.34"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tokio = { version = "1.54", features = ["full"] }

The grpc-tonic feature is easy to miss. Without it, SpanExporter::builder().with_tonic() does not exist, and the compiler reports "no method named with_tonic found", which does not hint at the feature. Use http-proto instead if you would rather send OTLP over HTTP, which passes through more proxies.

Initialize once, at the top of main

use opentelemetry::trace::TracerProvider as _;
use opentelemetry::KeyValue;
use opentelemetry_otlp::SpanExporter;
use opentelemetry_sdk::trace::{Sampler, SdkTracerProvider};
use opentelemetry_sdk::Resource;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;
use tracing_subscriber::EnvFilter;

pub fn init_tracing() -> anyhow::Result<SdkTracerProvider> {
    let exporter = SpanExporter::builder().with_tonic().build()?;

    let provider = SdkTracerProvider::builder()
        .with_batch_exporter(exporter)
        .with_sampler(Sampler::ParentBased(Box::new(Sampler::TraceIdRatioBased(0.1))))
        .with_resource(
            Resource::builder()
                .with_service_name("orders-api")
                .with_attribute(KeyValue::new("deployment.environment", "staging"))
                .build(),
        )
        .build();

    let tracer = provider.tracer("orders-api");
    tracing_subscriber::registry()
        .with(EnvFilter::from_default_env())
        .with(tracing_subscriber::fmt::layer())
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .init();

    Ok(provider)
}

This compiles as shown. Several lines carry decisions.

The exporter reads its destination from the environment: OTEL_EXPORTER_OTLP_ENDPOINT, defaulting to http://localhost:4317 for gRPC. Keeping the endpoint out of the code means the same binary talks to a collector on your laptop and to one in the cluster.

with_batch_exporter buffers spans and sends them in the background. The alternative, a simple exporter, makes a network call on every span end, which puts exporter latency in your request path and is only appropriate for tests.

The sampler is ParentBased(TraceIdRatioBased(0.1)). At the edge of your system, where no parent exists, it keeps 10 percent of traces, decided by a hash of the trace ID so the decision is consistent. Once a trace is sampled in, every downstream service honors the parent's decision, so you get complete traces rather than a patchwork. Without ParentBased, each service samples independently and ten percent at each of three hops leaves one trace in a thousand complete.

The resource attributes (service.name, deployment.environment) are what let your backend group spans. A service without a name shows up as unknown_service, which is the first sign someone skipped this step.

The function returns the provider, and main must keep it and call shutdown() on the way out. Batched spans sit in a buffer; a process that exits without flushing loses the last few seconds, which are usually the failure you were trying to capture.

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let provider = telemetry::init_tracing()?;
    run_server().await?;
    provider.shutdown()?;
    Ok(())
}

Instrument functions, not exporters

Because the bridge sits under tracing, adding a span to a function is one attribute. This program runs as pasted and prints the span structure through the fmt layer:

use opentelemetry::trace::TracerProvider as _;
use opentelemetry::KeyValue;
use opentelemetry_otlp::SpanExporter;
use opentelemetry_sdk::trace::{Sampler, SdkTracerProvider};
use opentelemetry_sdk::Resource;
use tracing::instrument;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;
use tracing_subscriber::EnvFilter;

fn init_tracing() -> anyhow::Result<SdkTracerProvider> {
    let exporter = SpanExporter::builder().with_tonic().build()?;

    let provider = SdkTracerProvider::builder()
        .with_batch_exporter(exporter)
        .with_sampler(Sampler::ParentBased(Box::new(Sampler::TraceIdRatioBased(0.1))))
        .with_resource(
            Resource::builder()
                .with_service_name("orders-api")
                .with_attribute(KeyValue::new("deployment.environment", "staging"))
                .build(),
        )
        .build();

    let tracer = provider.tracer("orders-api");
    tracing_subscriber::registry()
        .with(EnvFilter::from_default_env())
        .with(tracing_subscriber::fmt::layer())
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .init();

    Ok(provider)
}

#[instrument(skip(items), fields(item_count = items.len()))]
async fn place_order(user_id: u64, items: Vec<u32>) -> Result<(), String> {
    charge(user_id, items.len() as u64 * 1999).await
}

#[instrument]
async fn charge(user_id: u64, cents: u64) -> Result<(), String> {
    tracing::info!("charging");
    Ok(())
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let provider = init_tracing()?;
    place_order(42, vec![1, 2, 3]).await.unwrap();
    provider.shutdown()?;
    Ok(())
}

Running it with RUST_LOG=info prints the event with its enclosing spans, exactly as the exporter would see them:

INFO place_order{user_id=42 item_count=3}:charge{user_id=42 cents=5997}: q511: charging

Read the prefix as a stack: the event happened inside charge, which ran inside place_order. In a trace viewer this is a tree with place_order as the parent span and charge as its child, with user_id, item_count and cents as attributes. Three habits make these spans useful:

• Use skip for arguments that are large or sensitive. #[instrument(skip(items))] keeps the vector out of the span, and fields(item_count = items.len()) records the one fact you wanted from it. Spans are data leaving your service, so a card number or a token in a field is a leak.
• Record the outcome. Add err to #[instrument(err)] on a function that returns Result, and an error return marks the span as failed with the error text, which is what lets you filter for failed requests in the backend.
• Do not instrument tiny hot functions. A span costs an allocation and a few atomic operations; wrapping a function called a million times a second adds up. Instrument boundaries: handlers, database calls, outbound HTTP, queue operations.

Propagation: how a trace crosses services

Everything so far stays inside one process. A trace spans services only if the caller sends its context and the callee adopts it. The W3C standard puts this in a header called traceparent: a version, a 16-byte trace ID, an 8-byte parent span ID and a flags byte. The two functions below inject the current span's context into outgoing headers and extract it from incoming ones, and the main runs both sides to prove the trace ID survives the hop.

use axum::http::{HeaderMap, HeaderName, HeaderValue};
use opentelemetry::propagation::{Extractor, Injector};
use opentelemetry::trace::{TraceContextExt, TracerProvider as _};
use opentelemetry::global;
use opentelemetry_sdk::propagation::TraceContextPropagator;
use opentelemetry_sdk::trace::SdkTracerProvider;
use tracing::info_span;
use tracing_opentelemetry::OpenTelemetrySpanExt;
use tracing_subscriber::layer::SubscriberExt;

struct HeaderInjector<'a>(&'a mut HeaderMap);

impl Injector for HeaderInjector<'_> {
    fn set(&mut self, key: &str, value: String) {
        if let (Ok(name), Ok(val)) = (HeaderName::from_bytes(key.as_bytes()), HeaderValue::from_str(&value)) {
            self.0.insert(name, val);
        }
    }
}

struct HeaderExtractor<'a>(&'a HeaderMap);

impl Extractor for HeaderExtractor<'_> {
    fn get(&self, key: &str) -> Option<&str> {
        self.0.get(key).and_then(|v| v.to_str().ok())
    }
    fn keys(&self) -> Vec<&str> {
        self.0.keys().map(|k| k.as_str()).collect()
    }
}

/// Client side: copy the current span's context into outgoing headers.
fn inject_current_context(headers: &mut HeaderMap) {
    let cx = tracing::Span::current().context();
    global::get_text_map_propagator(|p| p.inject_context(&cx, &mut HeaderInjector(headers)));
}

/// Server side: make the incoming context the parent of this request's span.
fn adopt_remote_parent(span: &tracing::Span, headers: &HeaderMap) {
    let parent = global::get_text_map_propagator(|p| p.extract(&HeaderExtractor(headers)));
    let _ = span.set_parent(parent);
}

fn main() {
    global::set_text_map_propagator(TraceContextPropagator::new());
    let provider = SdkTracerProvider::builder().build();
    let tracer = provider.tracer("demo");
    let subscriber = tracing_subscriber::registry().with(tracing_opentelemetry::layer().with_tracer(tracer));
    tracing::subscriber::set_global_default(subscriber).unwrap();

    // Service A makes a call.
    let mut wire = HeaderMap::new();
    let a_trace_id = {
        let span = info_span!("checkout");
        let _guard = span.enter();
        inject_current_context(&mut wire);
        span.context().span().span_context().trace_id()
    };
    println!("traceparent: {}", wire["traceparent"].to_str().unwrap());

    // Service B receives it.
    let span = info_span!("charge_card");
    adopt_remote_parent(&span, &wire);
    let b_trace_id = span.context().span().span_context().trace_id();
    println!("A trace id: {a_trace_id}");
    println!("B trace id: {b_trace_id}");
    println!("same trace: {}", a_trace_id == b_trace_id);
}

traceparent: 00-3f9483ac00466553165b6bf05af69625-4a0d83d5c19a34ac-01
A trace id: 3f9483ac00466553165b6bf05af69625
B trace id: 3f9483ac00466553165b6bf05af69625
same trace: true

The header splits as 00 (version), 3f94...9625 (trace ID), 4a0d...34ac (the caller's span ID, which becomes the callee's parent) and 01 (the sampled flag). Service B created its span knowing nothing about service A except what that header said, and the trace IDs match, so a backend can draw both on one timeline.

Two practical notes. The propagator must be registered once with global::set_text_map_propagator(TraceContextPropagator::new()); forget it and inject_context writes nothing, silently, and your traces never join. And the injection must run while the right span is current. If you spawn a task and make the outbound call inside it, the task does not inherit the span unless you instrument the future with .instrument(span) or in_current_span(). A trace that breaks into disconnected fragments at an await or a spawn is almost always this.

In an Axum service, put the extraction in a middleware or in the TraceLayer that creates the request span, so every handler is already inside a span with the right parent. For reqwest, inject into the HeaderMap before sending, or use the reqwest-tracing middleware, which does the same thing.

Run a collector and look at a trace

You can send spans straight to a backend, but a collector in between is worth the extra container. It lets you add batching, redact attributes, tail-sample and change backends without redeploying your services. A local setup with Jaeger:

services:
  jaeger:
    image: jaegertracing/all-in-one:1.76
    environment:
      COLLECTOR_OTLP_ENABLED: "true"
    ports:
      - "16686:16686"
      - "4317:4317"

Run docker compose up -d, start the service with RUST_LOG=info cargo run, send a few requests, and open http://localhost:16686. Pick the service name you set, press Find Traces, and open one. You should see the place_order span with charge nested under it and the attributes you recorded. If the list is empty, check in this order: the endpoint variable, whether shutdown() ran, whether the sampler dropped the trace (at 10 percent, send twenty requests), and whether the exporter logged a connection error, which tracing will not print unless you enable the opentelemetry target in RUST_LOG.

The server side in Axum

The extraction function from the propagation example belongs in the layer that creates the request span, so no handler has to remember it. With tower-http, TraceLayer::make_span_with takes a closure that builds the span for each request, and that closure is the place to call the extractor.

use axum::{Router, http::Request, routing::get};
use tower_http::trace::TraceLayer;

fn app() -> Router {
    Router::new()
        .route("/orders/{id}", get(get_order))
        .layer(TraceLayer::new_for_http().make_span_with(|req: &Request<axum::body::Body>| {
            let span = tracing::info_span!(
                "http.request",
                http.request.method = %req.method(),
                url.path = %req.uri().path(),
                http.response.status_code = tracing::field::Empty,
            );
            adopt_remote_parent(&span, req.headers());
            span
        }))
}

Declaring http.response.status_code as tracing::field::Empty reserves the field so the layer's on_response hook can fill it in after the handler returns. Use the stable semantic convention names (http.request.method, url.path, http.response.status_code) rather than inventing your own, because backends build their dashboards and service maps from those names. One more habit: record the route template, /orders/{id}, not the concrete path, as the span name. A span named after each distinct URL gives you ten million unique names and a useless latency chart.

Errors, events and what belongs where

Spans measure work with a start and an end. Events mark moments inside a span. Use a span when you would want its duration, and an event when you would want a timestamp. A cache miss is an event on the handler's span, the database query that followed it is a child span. Getting this right keeps traces readable: a trace with 400 spans for one request is as hard to read as no trace.

For failures, make the span carry the status. With #[instrument(err)] an Err return records the error and marks the span as failed. For an HTTP handler that maps errors to responses, set the status in the response hook, and treat only server errors (500 and above) as span failures; a 404 is the client's mistake, and marking it red floods the error view with noise.

Testing the instrumentation

Telemetry is code, and the failure mode is silence, so test that the spans exist. The SDK ships an in-memory exporter in its testing module that collects finished spans instead of sending them. A test builds a provider with that exporter, runs the function, flushes, and asserts on the spans.

use opentelemetry::trace::TracerProvider as _;
use opentelemetry_sdk::trace::{InMemorySpanExporter, SdkTracerProvider};
use tracing_subscriber::layer::SubscriberExt;

#[test]
fn place_order_creates_a_charge_child_span() {
    let exporter = InMemorySpanExporter::default();
    let provider = SdkTracerProvider::builder()
        .with_simple_exporter(exporter.clone())
        .build();
    let subscriber = tracing_subscriber::registry()
        .with(tracing_opentelemetry::layer().with_tracer(provider.tracer("test")));

    tracing::subscriber::with_default(subscriber, || {
        let rt = tokio::runtime::Builder::new_current_thread().build().unwrap();
        rt.block_on(place_order(42, vec![1, 2, 3])).unwrap();
    });

    provider.force_flush().unwrap();
    let spans = exporter.get_finished_spans().unwrap();
    let names: Vec<_> = spans.iter().map(|s| s.name.as_ref()).collect();
    assert_eq!(names, ["charge", "place_order"]);
    let charge = &spans[0];
    let parent = &spans[1];
    assert_eq!(charge.parent_span_id, parent.span_context.span_id());
}

Children finish before parents, so charge comes first in the list. The final assertion checks the thing that matters: the parent link. A refactor that moves a call into a tokio::spawn without instrumenting it breaks that link, and this test catches it in CI instead of on the day you need the trace. The simple exporter is correct here, since it exports synchronously on span end and the test wants determinism. Check the exact module path of the in-memory exporter against your opentelemetry_sdk version, as it sits behind the testing feature and has moved between releases.

Debugging an empty or broken trace

When traces are missing or fragmented, work down this list, which is ordered by how often each is the cause.

1. No data at all: the endpoint variable is wrong, the collector is not reachable from the service's network namespace, or shutdown() never ran. In a container, localhost is the container, not the host.
2. Data but wrong names: the service.name resource attribute is missing, so everything lands under unknown_service.
3. Traces that stop at a service boundary: the propagator was never registered, or the outbound client does not inject. Print the headers of an outbound request and look for traceparent.
4. Traces fragmented inside one service: a spawned task or a blocking closure lost the span. Add .instrument(tracing::Span::current()) to the future, or capture the span before the spawn and enter it inside.
5. Spans appear only sometimes: that is the sampler working as configured. Temporarily switch to Sampler::AlwaysOn in staging to confirm.
6. Spans missing their attributes: the field name has a dot and was written as an identifier. Quote it in the macro or use the dotted-name syntax that the tracing macros allow.

Enable the SDK's own diagnostics by adding opentelemetry=debug,opentelemetry_sdk=debug to RUST_LOG. Exporter failures, such as a refused connection or a rejected payload, are logged there and nowhere else.

Metrics beside the traces

Traces show individual requests. Metrics show rates and distributions over all of them, which is where you set alerts. The meter API mirrors the trace one: a provider, a global registration, and instruments created from a named meter.

use opentelemetry::{global, KeyValue};
use opentelemetry_otlp::MetricExporter;
use opentelemetry_sdk::metrics::SdkMeterProvider;
use opentelemetry_sdk::Resource;

fn init_metrics() -> anyhow::Result<SdkMeterProvider> {
    let exporter = MetricExporter::builder().with_tonic().build()?;
    let provider = SdkMeterProvider::builder()
        .with_periodic_exporter(exporter)
        .with_resource(Resource::builder().with_service_name("orders-api").build())
        .build();
    global::set_meter_provider(provider.clone());
    Ok(provider)
}

fn main() -> anyhow::Result<()> {
    let provider = init_metrics()?;
    let meter = global::meter("orders-api");
    let placed = meter.u64_counter("orders.placed").with_description("orders accepted").build();
    let latency = meter.f64_histogram("orders.charge.duration").with_unit("s").build();

    placed.add(1, &[KeyValue::new("payment.method", "card")]);
    latency.record(0.231, &[KeyValue::new("payment.method", "card")]);

    provider.shutdown()?;
    Ok(())
}

A counter only goes up and is for events (orders placed). A histogram records a distribution and is for durations and sizes, from which the backend computes percentiles. Choose attributes carefully: every distinct combination of attribute values becomes its own time series. payment.method with four values is fine. A user.id attribute with a million values creates a million series, and your metrics bill and your backend's memory grow with it. Put high-cardinality identifiers on spans, where they are free, and keep metric attributes to a small enumerated set.

If you already export Prometheus metrics with a pull endpoint and want to keep it, that works alongside this; see Prometheus metrics in Rust. OpenTelemetry's metrics shine when you want one pipeline for all signals.

Join logs to traces

The last piece is being able to jump from a log line to its trace. Because the fmt layer sits in the same subscriber, it already prints the span stack. To make it searchable, switch to JSON output and include the trace ID, which you can read from the current span's context as in the propagation example. With the ID in every log line, an error in your log search becomes a link into the trace that shows what led to it. Without it you are correlating by timestamp and guessing.

Cost and operating rules

Telemetry can cost more than the service it watches, so a few rules keep it in check.

1. Sample at the head for volume and in the collector for value. Head sampling at a fixed ratio is simple and cheap. Tail sampling, configured in the collector, can keep every trace with an error or with a duration over a threshold and a fraction of the rest, but the collector must hold whole traces in memory while it decides.
2. Bound cardinality on metrics, as above.
3. Redact in one place. Add an attribute-processor rule to the collector that removes http.request.header.authorization and similar keys, so one leaked field in application code does not reach the vendor. See logging without leaking secrets.
4. Test the pipeline in staging by sending a known request and finding its trace by ID. A misconfigured exporter fails silently; the service runs fine and the dashboard stays empty.
5. Measure the overhead. Run your load test with telemetry on and off at the same rate. Batch export with sampling typically costs a small single-digit percentage of throughput; if you see more, look for hot-path spans.

When OpenTelemetry is more than you need

A single binary with a database, where logs and a /metrics endpoint already answer your questions, does not need a collector and a tracing backend. Start with structured tracing logs and request IDs (see request IDs in Axum), and add the exporter when a request starts crossing process boundaries. Equally, if your platform already provides tracing for you, such as a managed service that injects its own agent, do not run two systems that both claim the traceparent header. And treat the SDK's churn as a real cost: in a long-lived service, budget a few hours each quarter to bump the four crates together. For the architecture view that explains which signal answers which question, read observability architecture.

A rollout plan that does not hurt

Adopt this in three stages rather than all at once. First, ship the tracing initialization with the sampler at zero and the exporter pointed at a staging collector, which proves the build and the dependency versions without touching production traffic. Second, turn sampling up to one percent in production and watch the service's own CPU, memory and p99 for a day against the previous week. Third, add propagation service by service, starting at the edge and moving inward, so a partially instrumented chain never produces misleading half-traces. At each stage the rollback is a single environment variable, which is the other reason to keep the endpoint and sampler ratio out of the code: OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG let you turn the volume up or down without a deploy.