# Distributed Tracing with OpenTelemetry: What We Ship, What We Skip URL: https://www.devopsness.com/blog/distributed-tracing-opentelemetry-what-we-ship Author: Kiril Urbonas Published: 2026-04-27 Updated: 2026-10-31 Categories: DevOps Tags: Opentelemetry, Tracing, Observability, DevOps, Datadog Summary: How we run OpenTelemetry tracing across dozens of services. The instrumentation that earns its place, the patterns we abandoned, and what tracing actually catches that metrics don't. --- # Distributed Tracing with OpenTelemetry: What We Ship, What We Skip Ship OpenTelemetry auto-instrumentation for every HTTP, database, cache and queue call, add manual spans only at business boundaries, and tail-sample in a Collector gateway so every error and slow trace is kept. Skip per-function spans, logs-as-spans and high-cardinality attributes. Tracing tells you where a request is slow; it does not replace metrics. > [!VERIFIED] > OpenTelemetry Collector v0.164.0 (tail sampling processor at beta stability), OpenTelemetry Python 1.45, against the opentelemetry.io sampling and Python instrumentation docs and the Collector processor READMEs. Checked October 2026. We've run OpenTelemetry tracing across a few dozen production services for about two years. The promise of tracing (see where time goes inside a request, end to end) is real and we use it daily. The pitch that "tracing replaces metrics" is mostly wrong. This is the working version: what we instrument, what we skip, how we sample, and what tracing actually catches. ## What tracing buys you over metrics Metrics tell you *that* a request was slow. Traces tell you *where* it was slow. A typical example: p99 latency on the checkout API jumps overnight. Metrics show the spike. Logs show successful 200s. The trace view shows most of every slow request sitting in one downstream call to the address-validation service, whose own spans point at a slow DNS lookup inside it. That is minutes of reading a flame graph instead of an afternoon bisecting code paths and grepping log lines. That's the value: traces collapse "which service is slow, and which call inside it" into a flame graph you read in seconds. For services with deep call graphs (anything customer-facing in a microservices architecture), it's the most useful debugging tool we have. What it does not do: long-term dashboards, alerting, capacity planning. Traces are too high-volume and high-cardinality for that. Metrics and logs do that work, and [correlating logs, metrics and traces](/blog/observability-correlating-logs-metrics-traces) is how the three fit together. ## What we instrument **Automatic, framework-level instrumentation** covers most useful spans for free: - HTTP servers (Express, FastAPI, Gin): a span per incoming request - HTTP clients (requests, axios, net/http): a span per outgoing request - Database drivers (pg, mysql2, Prisma): a span per query - Redis and Memcached clients, gRPC calls, and queue produce/consume (SQS, RabbitMQ, Kafka) We use the OpenTelemetry SDK with the auto-instrumentation packages for each language, wired in through init code. **Manual instrumentation** covers what auto can't see: - Business operations (`order.placement`, `payment.charge`, `agent.run`), named after the domain action, not the function - Long blocking work that isn't a single library call (loops, batches, computation) - Retry loops, with a `retry.attempt` span per attempt ```python # app/orders.py from opentelemetry import trace tracer = trace.get_tracer("orders") with tracer.start_as_current_span("order.placement") as span: span.set_attributes({ "order.total_usd": total, "order.item_count": len(items), }) place_order(items) ``` The attributes matter more than the span itself. When something is slow, "this slow span had `order.item_count=4203`" is what makes the issue obvious. ## What we deliberately don't instrument **Per-function spans inside a service.** Tempting, and wrong. Spans dominate the cost, traces fill with hundreds of irrelevant spans, and the useful ones drown. Manual spans only at meaningful boundaries; rely on auto for I/O. **Logs as spans.** Logs are events, spans are durations. Mixing them muddies both. Keep logs as logs with `trace_id` as a field so you can correlate. **Every third-party auto-instrumentation.** Some are noisy (every internal SDK call shows up). We disable the ones with poor signal-to-noise per language. ## Sampling: tail-based, in a gateway Tracing every request is expensive: most APMs bill on ingested or indexed spans, and self-hosted backends pay in storage. We use **tail-based sampling** with four rules: keep 100% of traces with an `ERROR` status, 100% of traces over 2 seconds, 100% of high-value endpoints (payments, auth), and 10% of everything else. Every interesting trace is captured, and routine traffic is sampled enough for aggregate stats, at a fraction of the cost of keeping everything. Head-based sampling (decide at request start) is simpler but commits before you know whether the request errors. We started there and switched after too many "the trace was sampled out exactly when something interesting happened" moments. Tail sampling runs in the OpenTelemetry Collector's `tail_sampling` processor, which buffers spans for `decision_wait` (default 30s) before deciding: ```yaml # otel-collector/gateway.yaml processors: memory_limiter: check_interval: 1s limit_mib: 1500 spike_limit_mib: 300 tail_sampling: decision_wait: 10s num_traces: 50000 policies: - name: errors type: status_code status_code: {status_codes: [ERROR]} - name: slow type: latency latency: {threshold_ms: 2000} - name: high-value type: string_attribute string_attribute: {key: http.route, values: ["/payments.*", "/auth.*"], enabled_regex_matching: true} - name: baseline type: probabilistic probabilistic: {sampling_percentage: 10} ``` The trap we fell into: the processor README states that all spans of a trace must reach the same Collector instance. Tail sampling in a per-node DaemonSet silently makes wrong decisions for any trace that crosses nodes. The fix is two tiers: node agents export through the `loadbalancing` exporter keyed on trace ID, and a gateway Deployment does the sampling, the same [agent and gateway Collector layout on Kubernetes](/blog/opentelemetry-collector-kubernetes-setup) we use everywhere. ## What tracing has actually caught **A retry storm in a third-party SDK.** Its backoff loop retried immediately instead of waiting. The trace showed seven retry spans of the same call back to back. Without tracing, the symptom was just "this endpoint is slow." **An N+1 query from an ORM lazy-load.** A new code path triggered the lazy-loader per list item: 200+ identical query spans in one request. Metrics showed elevated database CPU; the trace showed the call path. **Accidental cross-AZ placement.** Two chatty services ended up in different availability zones, and the network span dominated every request between them. **An undersized connection pool.** Requests spent hundreds of milliseconds waiting on "pool acquire." Bigger pool, problem gone. In every case the issue was visible in metrics but ambiguous. Traces made the cause obvious. ## What broke at scale **Context propagation across custom protocols.** Auto-instrumentation only propagates `traceparent` for protocols it knows. An internal binary RPC and an old SOAP integration dropped the context, so we carried trace IDs through them by hand. **Cardinality in span attributes.** `user_id` or `order_id` as attributes helps debugging but can blow up cost on backends that index or bill by attribute values. Check your vendor's pricing model before tagging every span with an ID. **Collector memory under burst.** In a spike, the queue filled faster than it could send. The `memory_limiter` processor, placed first in the pipeline as its README recommends, now applies backpressure instead of letting the Collector get OOM-killed. **SDK initialization order.** In Python, initialize the SDK at the top of the entry file before importing instrumented libraries, or use the `opentelemetry-instrument` wrapper. Getting this wrong silently disables auto-instrumentation. ## The stack we run - **Instrumentation:** OpenTelemetry SDKs (Python, Node, Go) with auto-instrumentation packages - **Collector:** agent DaemonSet per Kubernetes node, plus a gateway tier for tail sampling - **Backend:** Datadog APM (we also evaluated Honeycomb; Datadog won on existing integration depth) - **Trace-log correlation:** `trace_id` and `span_id` injected into structured logs by the SDKs We considered self-hosting Jaeger or Tempo. The operational cost was real and we'd have lost our existing dashboards and alerts. If vendor lock-in worries you more than that, [OpenTelemetry vs vendor agents](/blog/opentelemetry-vs-vendor-agents) covers the trade. ## Questions people ask ### Should I use head-based or tail-based sampling in OpenTelemetry? Head-based sampling decides at the start of a request, so it is cheap and simple but drops errors at the same rate as healthy traffic. Tail-based sampling decides after the trace completes, so you can keep every error and slow request. It costs memory and a Collector tier where every span of a trace reaches the same instance. Use head sampling for low-stakes services, tail sampling where incidents matter. ### Can I run the tail sampling processor in a DaemonSet? Not on its own. The processor needs every span of a trace on the same Collector instance, and a DaemonSet spreads spans across nodes, so traces crossing nodes get partial, wrong decisions. Run node agents that forward through the `loadbalancing` exporter routed by trace ID, and put `tail_sampling` in a separate gateway Deployment that you scale and monitor like any other production service. ### Does distributed tracing replace metrics? No. Traces answer "why was this specific request slow" by showing where time went across services. They are too high-volume and usually sampled, so they are a poor base for alerting, long-term dashboards or capacity planning. Keep metrics for alerts and trends, logs for event detail, and link all three with `trace_id`. ## What we'd tell a team starting **Auto-instrumentation first, manual spans for business boundaries.** Don't instrument every function. **Tail-based sampling in a gateway.** A 10% baseline plus 100% of errors and slow requests captures everything interesting at a fraction of the cost. **Correlate logs and traces from day one.** Get `trace_id` into structured logs. **Treat the Collector as production infrastructure.** It's in the hot path and needs the same monitoring and capacity planning as any other service. Tracing is the third leg alongside metrics and logs, not a replacement for them. For services with deep call graphs and latency that matters per request, it's the most useful of the three. ## Sources - [OpenTelemetry: Sampling concepts](https://opentelemetry.io/docs/concepts/sampling/) - [OpenTelemetry Collector Contrib: Tail sampling processor README](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md) - [OpenTelemetry Collector: Memory limiter processor README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md) - [OpenTelemetry: Python instrumentation](https://opentelemetry.io/docs/languages/python/instrumentation/)