OpenTelemetry Collector on Kubernetes: Agent vs Gateway, With Configs That Work

Run two tiers: an agent DaemonSet on every node that receives OTLP from local pods and adds Kubernetes metadata, and a gateway Deployment that batches, samples and exports to your backend. Install both with the open-telemetry/opentelemetry-collector Helm chart, and put memory_limiter first in every pipeline. Metadata at the edge, secrets and sampling in one place.

Agent, gateway, or both

The Collector docs describe two deployment patterns, and on Kubernetes they map to two chart modes:

We default to both. The agent is where k8s_attributes is cheapest: with a node filter, each agent watches only the pods on its own node. The gateway is where tail sampling has to live, because every span of a trace must reach the same instance, which the docs solve with the load_balancing exporter routing on traceID.

Install the agent DaemonSet

The chart refuses to render without two values. Leave either out and you get [ERROR] 'image.repository' must be set. or [ERROR] 'mode' must be set. The docs point at the Kubernetes distribution image, otel/opentelemetry-collector-k8s, which ships k8s_attributes, load_balancing and the tail sampling processor.

# otel-agent-values.yaml
mode: daemonset
image:
  repository: otel/opentelemetry-collector-k8s
presets:
  kubernetesAttributes:
    enabled: true       # adds RBAC and a node-filtered k8s_attributes processor
resources:
  limits:
    memory: 512Mi       # the chart derives GOMEMLIMIT from this
config:
  processors:
    memory_limiter:
      check_interval: 1s
      limit_percentage: 80
      spike_limit_percentage: 25
    batch: {}
  exporters:
    otlp_grpc:
      endpoint: otel-gateway-opentelemetry-collector.observability.svc.cluster.local:4317
      tls:
        insecure: true  # in-cluster hop; use mTLS if your policy requires it
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, batch]
        exporters: [otlp_grpc]
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, batch]
        exporters: [otlp_grpc]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, k8s_attributes, batch]
        exporters: [otlp_grpc]

Two details in that file matter more than they look. First, we list k8s_attributes ourselves. The preset prepends the processor to any pipeline that doesn't already name it, which would put it ahead of memory_limiter; naming it explicitly keeps the limiter first. In daemonset mode the preset also sets filter.node_from_env_var: K8S_NODE_NAME and injects that variable, so each agent queries only its own node's pods.

Second, the chart sets hostPort: 4317 on the OTLP port by default, so applications reach their node's agent through the node IP. Point the SDK at it with the downward API:

# deployment.yaml (app container)
env:
  - name: HOST_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://$(HOST_IP):4317"

Install the gateway Deployment

# otel-gateway-values.yaml
mode: deployment
replicaCount: 2
image:
  repository: otel/opentelemetry-collector-k8s
resources:
  limits:
    memory: 2Gi
config:
  processors:
    memory_limiter:
      check_interval: 1s
      limit_percentage: 80
      spike_limit_percentage: 25
    batch: {}
  exporters:
    otlp_http:
      endpoint: https://otlp.example-backend.com
      headers:
        authorization: "Bearer ${env:OTLP_TOKEN}"
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp_http]
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp_http]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp_http]

$ helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
$ helm upgrade --install otel-gateway open-telemetry/opentelemetry-collector \
    -n observability --create-namespace -f otel-gateway-values.yaml
$ helm upgrade --install otel-agent open-telemetry/opentelemetry-collector \
    -n observability -f otel-agent-values.yaml
$ kubectl -n observability logs ds/otel-agent-opentelemetry-collector-agent | head

The gateway needs no k8s_attributes: the agents already attached k8s.pod.name, k8s.namespace.name and the deployment name. If you ever skip the agent tier, the processor README says a gateway can't see the original pod IP behind an agent, which is why agents in that setup run the processor in passthrough mode and forward the IP instead.

The processor order is the whole game

The memory_limiter README is explicit: put it first in every pipeline so it can push back on receivers before data piles up, and set GOMEMLIMIT to 80% of the container's hard limit. The chart does the second part for you (useGOMEMLIMIT: true by default). In containers, use limit_percentage rather than limit_mib so the limit follows the pod's memory limit. The README recommends a 1 second check_interval; the chart's default is 5 seconds, which is why we override it.

When the limiter trips you will see data refused due to high memory usage. That is the system working. The SDK or the upstream agent is supposed to retry; the README warns that data is permanently lost if the component in front of the limiter doesn't.

The errors you will hit first

• unknown type: "otlp_grpc" for id: "otlp_grpc" (valid values: [...]): the otlp exporter's type has been otlp_grpc since core v0.144.0 (otlphttp is now otlp_http), and contrib renamed k8sattributes to k8s_attributes in v0.146.0 (loadbalancing is now load_balancing), with the old names kept as deprecated aliases. An image older than that doesn't know the new names. Pin the image tag to the chart's appVersion, or set rewriteDeprecatedComponentNames: false on old images.
• service::pipelines::traces: references processor "k8s_attributes" which is not configured: you listed it in a pipeline without enabling the preset or defining it under processors.
• [ERROR] The opentelemetry-collector chart requires that the health_check extension...: you replaced service.extensions and dropped health_check; the probes depend on it.
• Pod metadata missing on every span: the ServiceAccount can't list pods. Check the agent logs for a forbidden error, and enable the preset instead of hand-writing the ClusterRole.

When something is defined but nothing flows, the troubleshooting docs list the usual cause first: a receiver configured but not enabled in any pipeline.

Where this breaks

Helm values merge maps but replace lists, so overriding receivers in a pipeline silently drops the chart's defaults (Jaeger, Zipkin, the self-scrape). That is usually what you want; know it is happening. The two-tier setup also doubles your upgrade surface, and the gateway is now on the critical path for all telemetry, so give it a PodDisruptionBudget and at least two replicas. If you need auto-instrumentation injected into pods, add the OpenTelemetry Operator, which requires cert-manager, on top of this; it doesn't replace the chart.

Questions people ask

Should the OpenTelemetry Collector run as a DaemonSet or Deployment?

Both, in most production clusters. A DaemonSet agent receives from local pods, adds Kubernetes metadata cheaply because it watches only its own node, and collects node metrics. A Deployment gateway holds backend credentials, runs tail sampling and handles retries in one place. Small clusters with one backend can run the DaemonSet alone and export directly.

Helm chart or OpenTelemetry Operator?

Use the Helm chart for the Collector itself; it is simpler to review and diff. Add the Operator when you want its OpenTelemetryCollector resource, sidecar injection, or auto-instrumentation of workloads through its Instrumentation resource. The Operator installs from a single manifest but needs cert-manager in the cluster first, which is an extra dependency to own.

Why does my collector say data refused due to high memory usage?

The memory_limiter processor hit its soft limit and is rejecting new data so the process doesn't get OOMKilled. Short spikes are normal. If it happens constantly, the Collector is undersized or the exporter is too slow: raise the pod's memory limit, add gateway replicas, or check that the backend isn't throttling you. Make sure SDKs retry so refused data isn't lost.

The call we'd make

Ship the agent DaemonSet plus a two-replica gateway, both from the official chart with the otel/opentelemetry-collector-k8s image, memory_limiter first and the new component names. Pin the image tag. Our OpenTelemetry tracing setup covers the SDK side, and if you are still deciding whether to run vendor agents at all, read OpenTelemetry vs vendor agents first. The caveat: two tiers is overkill below a handful of nodes.

Sources

• OpenTelemetry docs: OpenTelemetry Collector Helm chart
• OpenTelemetry docs: Gateway deployment pattern
• Memory limiter processor README
• OpenTelemetry docs: Collector troubleshooting