Observability: Correlating Logs, Metrics, and Traces in Anger

To correlate logs, metrics and traces, make every signal carry the same trace ID: logs get trace_id and span_id fields, latency histograms get exemplars pointing at trace IDs, and traces use it as their key. Then an alert jumps to an exemplar, the exemplar to a trace, and the trace to its log lines.

The "three pillars of observability" framing is everywhere, and it's misleading. The pillars aren't independent; the value is in moving between them. A high-latency request in metrics points to a trace, the trace points to a span, the span points to a log line. This is what makes that chain work and the discipline that keeps it working.

The mental model: same identifier across all three

The single most important practice: every signal carries the same trace ID. A request comes in, gets a trace ID, and every log line, every exemplar and every span uses it.

• Logs include trace_id (and span_id) as structured fields.
• Metrics carry the trace ID as an exemplar, not as a label (more below).
• Traces use the trace ID as their primary identifier.

This sounds trivial. Operationally, it's where most observability efforts fall down: the instrumentation must propagate trace context (W3C traceparent) across service boundaries, queue handoffs, batch jobs and retries.

Tooling we'd pick

• Tracing: OpenTelemetry SDK and Collector, exporting to Tempo or Jaeger.
• Metrics: Prometheus for service metrics; OpenTelemetry through the Collector into Mimir for app and business metrics.
• Logs: structured JSON to stdout, shipped to Loki or OpenSearch.
• Grafana as the shared UI, so one dashboard can pivot from a metric to a trace to a log.

OpenTelemetry is the common thread: instrument once, route to multiple backends. If you are weighing that against proprietary agents, we cover the tradeoff in OpenTelemetry vs vendor agents.

Step 1: structured logs with trace context

Every log line carries the trace ID. With an integrated logging library, the logger picks it up from the OpenTelemetry context automatically.

{
  "ts": "2026-06-18T14:32:01.123Z",
  "level": "error",
  "msg": "payment failed",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "service": "payments",
  "user_id": "u_12345",
  "amount_cents": 4500
}

The trace_id (32 hex characters, the W3C format) is what makes this line cross-referenceable. The other fields are context for the event.

• Use a structured logger. Pino, Zap, structlog, slog: anything that emits JSON natively. String-formatted lines are unparseable at scale.
• Inject trace context once per request so every subsequent line includes it.
• Redact secrets in logger config. Don't rely on developers remembering.

Step 2: metrics with exemplars

Exemplars are the bridge from metrics to traces. An exemplar is a sample observation, with its trace ID, attached to a metric series.

You alert on "p95 latency over 1s." The chart shows the spike, and exemplar dots on it show specific trace IDs that fell in the slow buckets. Click one and you are in the trace. Without exemplars, you go from "something is slow" to searching for slow traces by hand, which wastes minutes when they matter.

Prometheus stores exemplars only with --enable-feature=exemplar-storage, and they arrive through OpenMetrics scrapes or OTLP. Client libraries and OpenTelemetry SDKs attach them when a histogram is recorded inside an active span. Grafana renders them on time series panels once the data source has an exemplar link to your trace backend.

Step 3: traces with semantic conventions

A trace is a tree of spans with names, timing and attributes. The attributes hold most of the debugging value, and they're easy to get wrong.

Use the OpenTelemetry semantic conventions. As of version 1.44.0 the HTTP and database span conventions are stable: http.request.method and http.response.status_code (older instrumentation emitted http.status_code), db.system.name and db.query.text (formerly db.statement). Backends can then group errors by status code and surface slow database calls without custom mapping. Set OTEL_SEMCONV_STABILITY_OPT_IN during migration if dashboards still query the old names.

Beyond conventions we'd add tenant.id in multi-tenant systems (almost every debug starts with "which tenant?"), a user identifier for bug reports, and one service-specific request category.

The debugging flow

1. Alert fires. "p95 latency over 1s for service X."
2. Check the dashboard. Which metrics moved? CPU, memory, queue depth?
3. Pick an exemplar from a slow bucket.
4. Open the trace. Which span is slow: a query, an external call?
5. Filter logs by trace_id and read the error and request context.
6. Form a hypothesis. Slow query, check pg_stat_statements; external timeout, check the downstream service; cache misses, check hit rate.

Metric, exemplar, trace, log: that chain is what makes this fast.

Sampling: the cost question

Head-based sampling decides at trace start (keep 1%, drop the rest). Simple and cheap, but errors are rare, so most error traces get dropped.

Tail-based sampling decides after the trace completes: keep errors and slow traces, sample the rest. Better quality, but the Collector must buffer spans, and the README is explicit that all spans of a trace must reach the same Collector instance. Put a load-balancing exporter layer in front.

# otel-collector/config.yaml (processors section)
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 1000}
      - name: priority-tenants
        type: string_attribute
        string_attribute: {key: tenant.tier, values: [enterprise]}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 1}

A trace is kept if any policy matches. decision_wait defaults to 30s; shorter waits use less memory but can cut off long traces.

Monitor the monitoring

Observability infrastructure breaks like anything else. Watch ingestion latency, metric cardinality, actual versus expected sampling rate, and storage cost (logs usually cost the most). Add a synthetic alert that fires when no logs have arrived for N minutes; a silent pipeline looks exactly like a healthy system. CI deserves the same treatment; see pipeline observability for CI failures.

The mistakes that cost the most

Logging everything. Every request, query and state change at INFO drowns signal and blows up storage. Keep INFO for events, DEBUG for diagnostics you enable per service when investigating.

user_id as a metric label. Unbounded label values explode Prometheus cardinality. High-cardinality identifiers belong in trace attributes and log fields.

Uninstrumented batch jobs. Web requests get tracing; cron jobs and queue consumers don't, and those become the hardest incidents. Propagate context through message headers.

Trusting auto-instrumentation entirely. It misses application-specific work. Add manual spans around feature-flag evaluation and expensive business logic, and reach for bpftrace when the slowness is below the application.

Bake OpenTelemetry into service scaffolding, check trace propagation in code review for new service-to-service calls, and periodically replay a recent incident through the metric-to-log chain to find where it breaks. The OpenTelemetry tracing setup we ship goes deeper on the tracing layer, and burn-rate alerting covers what to alert on.

Questions people ask

How do I link logs to traces?

Write structured JSON logs that include the active trace_id and span_id, pulled from the OpenTelemetry context by your logging library or a log bridge. Then configure your log backend's data source (for example Loki in Grafana) with a derived field that turns trace_id into a link to your trace backend. Every log line becomes one click from its trace.

What are exemplars in Prometheus?

Exemplars are sample observations attached to a metric series, usually carrying a trace ID, so a latency histogram can point at a real request that landed in a slow bucket. Prometheus stores them only when started with --enable-feature=exemplar-storage, and Grafana draws them as dots on time series panels that link to the trace.

Should I use head-based or tail-based sampling?

Use tail-based sampling when you need every error and slow trace, which is most production systems. It decides after the trace completes, so you keep 100% of failures and a small percentage of normal traffic. The cost is Collector memory and a load-balancing tier, because all spans of a trace must reach one instance. Head-based sampling is fine for low-traffic or cost-capped setups.

The call we'd make

Standardize on OpenTelemetry for all three signals, put trace_id in every log line, enable exemplars on your latency histograms, and tail-sample at the Collector. That chain matters more than which backend stores each pillar. The caveat: tail sampling is beta in the Collector and adds a stateful tier you have to scale, so start with head sampling plus error logging if you can't run it yet.

Sources

• OpenTelemetry semantic conventions
• OpenTelemetry semantic conventions: HTTP spans
• Prometheus docs: Feature flags (exemplar storage)
• OpenTelemetry Collector contrib: Tail sampling processor