Distributed Tracing with OpenTelemetry: What We Ship, What We Skip

Ship OpenTelemetry auto-instrumentation for every HTTP, database, cache and queue call, add manual spans only at business boundaries, and tail-sample in a Collector gateway so every error and slow trace is kept. Skip per-function spans, logs-as-spans and high-cardinality attributes. Tracing tells you where a request is slow; it does not replace metrics.

We've run OpenTelemetry tracing across a few dozen production services for about two years. The promise of tracing (see where time goes inside a request, end to end) is real and we use it daily. The pitch that "tracing replaces metrics" is mostly wrong. This is the working version: what we instrument, what we skip, how we sample, and what tracing actually catches.

What tracing buys you over metrics

Metrics tell you that a request was slow. Traces tell you where it was slow.

A typical example: p99 latency on the checkout API jumps overnight. Metrics show the spike. Logs show successful 200s. The trace view shows most of every slow request sitting in one downstream call to the address-validation service, whose own spans point at a slow DNS lookup inside it. That is minutes of reading a flame graph instead of an afternoon bisecting code paths and grepping log lines.

That's the value: traces collapse "which service is slow, and which call inside it" into a flame graph you read in seconds. For services with deep call graphs (anything customer-facing in a microservices architecture), it's the most useful debugging tool we have.

What it does not do: long-term dashboards, alerting, capacity planning. Traces are too high-volume and high-cardinality for that. Metrics and logs do that work, and correlating logs, metrics and traces is how the three fit together.

What we instrument

Automatic, framework-level instrumentation covers most useful spans for free:

• HTTP servers (Express, FastAPI, Gin): a span per incoming request
• HTTP clients (requests, axios, net/http): a span per outgoing request
• Database drivers (pg, mysql2, Prisma): a span per query
• Redis and Memcached clients, gRPC calls, and queue produce/consume (SQS, RabbitMQ, Kafka)

We use the OpenTelemetry SDK with the auto-instrumentation packages for each language, wired in through init code.

Manual instrumentation covers what auto can't see:

• Business operations (order.placement, payment.charge, agent.run), named after the domain action, not the function
• Long blocking work that isn't a single library call (loops, batches, computation)
• Retry loops, with a retry.attempt span per attempt

from opentelemetry import trace

tracer = trace.get_tracer("orders")

with tracer.start_as_current_span("order.placement") as span:
    span.set_attributes({
        "order.total_usd": total,
        "order.item_count": len(items),
    })
    place_order(items)

The attributes matter more than the span itself. When something is slow, "this slow span had order.item_count=4203" is what makes the issue obvious.

What we deliberately don't instrument

Per-function spans inside a service. Tempting, and wrong. Spans dominate the cost, traces fill with hundreds of irrelevant spans, and the useful ones drown. Manual spans only at meaningful boundaries; rely on auto for I/O.

Logs as spans. Logs are events, spans are durations. Mixing them muddies both. Keep logs as logs with trace_id as a field so you can correlate.

Every third-party auto-instrumentation. Some are noisy (every internal SDK call shows up). We disable the ones with poor signal-to-noise per language.

Sampling: tail-based, in a gateway

Tracing every request is expensive: most APMs bill on ingested or indexed spans, and self-hosted backends pay in storage. We use tail-based sampling with four rules: keep 100% of traces with an ERROR status, 100% of traces over 2 seconds, 100% of high-value endpoints (payments, auth), and 10% of everything else.

Every interesting trace is captured, and routine traffic is sampled enough for aggregate stats, at a fraction of the cost of keeping everything. Head-based sampling (decide at request start) is simpler but commits before you know whether the request errors. We started there and switched after too many "the trace was sampled out exactly when something interesting happened" moments.

Tail sampling runs in the OpenTelemetry Collector's tail_sampling processor, which buffers spans for decision_wait (default 30s) before deciding:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 300
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 2000}
      - name: high-value
        type: string_attribute
        string_attribute: {key: http.route, values: ["/payments.*", "/auth.*"], enabled_regex_matching: true}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

The trap we fell into: the processor README states that all spans of a trace must reach the same Collector instance. Tail sampling in a per-node DaemonSet silently makes wrong decisions for any trace that crosses nodes. The fix is two tiers: node agents export through the loadbalancing exporter keyed on trace ID, and a gateway Deployment does the sampling, the same agent and gateway Collector layout on Kubernetes we use everywhere.

What tracing has actually caught

A retry storm in a third-party SDK. Its backoff loop retried immediately instead of waiting. The trace showed seven retry spans of the same call back to back. Without tracing, the symptom was just "this endpoint is slow."

An N+1 query from an ORM lazy-load. A new code path triggered the lazy-loader per list item: 200+ identical query spans in one request. Metrics showed elevated database CPU; the trace showed the call path.

Accidental cross-AZ placement. Two chatty services ended up in different availability zones, and the network span dominated every request between them.

An undersized connection pool. Requests spent hundreds of milliseconds waiting on "pool acquire." Bigger pool, problem gone.

In every case the issue was visible in metrics but ambiguous. Traces made the cause obvious.

What broke at scale

Context propagation across custom protocols. Auto-instrumentation only propagates traceparent for protocols it knows. An internal binary RPC and an old SOAP integration dropped the context, so we carried trace IDs through them by hand.

Cardinality in span attributes. user_id or order_id as attributes helps debugging but can blow up cost on backends that index or bill by attribute values. Check your vendor's pricing model before tagging every span with an ID.

Collector memory under burst. In a spike, the queue filled faster than it could send. The memory_limiter processor, placed first in the pipeline as its README recommends, now applies backpressure instead of letting the Collector get OOM-killed.

SDK initialization order. In Python, initialize the SDK at the top of the entry file before importing instrumented libraries, or use the opentelemetry-instrument wrapper. Getting this wrong silently disables auto-instrumentation.

The stack we run

• Instrumentation: OpenTelemetry SDKs (Python, Node, Go) with auto-instrumentation packages
• Collector: agent DaemonSet per Kubernetes node, plus a gateway tier for tail sampling
• Backend: Datadog APM (we also evaluated Honeycomb; Datadog won on existing integration depth)
• Trace-log correlation: trace_id and span_id injected into structured logs by the SDKs

We considered self-hosting Jaeger or Tempo. The operational cost was real and we'd have lost our existing dashboards and alerts. If vendor lock-in worries you more than that, OpenTelemetry vs vendor agents covers the trade.

Questions people ask

Should I use head-based or tail-based sampling in OpenTelemetry?

Head-based sampling decides at the start of a request, so it is cheap and simple but drops errors at the same rate as healthy traffic. Tail-based sampling decides after the trace completes, so you can keep every error and slow request. It costs memory and a Collector tier where every span of a trace reaches the same instance. Use head sampling for low-stakes services, tail sampling where incidents matter.

Can I run the tail sampling processor in a DaemonSet?

Not on its own. The processor needs every span of a trace on the same Collector instance, and a DaemonSet spreads spans across nodes, so traces crossing nodes get partial, wrong decisions. Run node agents that forward through the loadbalancing exporter routed by trace ID, and put tail_sampling in a separate gateway Deployment that you scale and monitor like any other production service.

Does distributed tracing replace metrics?

No. Traces answer "why was this specific request slow" by showing where time went across services. They are too high-volume and usually sampled, so they are a poor base for alerting, long-term dashboards or capacity planning. Keep metrics for alerts and trends, logs for event detail, and link all three with trace_id.

What we'd tell a team starting

Auto-instrumentation first, manual spans for business boundaries. Don't instrument every function.

Tail-based sampling in a gateway. A 10% baseline plus 100% of errors and slow requests captures everything interesting at a fraction of the cost.

Correlate logs and traces from day one. Get trace_id into structured logs.

Treat the Collector as production infrastructure. It's in the hot path and needs the same monitoring and capacity planning as any other service.

Tracing is the third leg alongside metrics and logs, not a replacement for them. For services with deep call graphs and latency that matters per request, it's the most useful of the three.

Sources

• OpenTelemetry: Sampling concepts
• OpenTelemetry Collector Contrib: Tail sampling processor README
• OpenTelemetry Collector: Memory limiter processor README
• OpenTelemetry: Python instrumentation