# OpenTelemetry Collector on Kubernetes: Agent vs Gateway, With Configs That Work URL: https://www.devopsness.com/blog/opentelemetry-collector-kubernetes-setup Author: Kiril Urbonas Published: 2026-10-31 Updated: 2026-10-31 Categories: DevOps Tags: Opentelemetry, Kubernetes, Observability, Monitoring, Helm Summary: Run a DaemonSet agent on every node and a small gateway Deployment behind it. Two Helm values files, the processor order that keeps the Collector alive, and the errors you will hit first. --- # OpenTelemetry Collector on Kubernetes: Agent vs Gateway, With Configs That Work Run two tiers: an agent DaemonSet on every node that receives OTLP from local pods and adds Kubernetes metadata, and a gateway Deployment that batches, samples and exports to your backend. Install both with the `open-telemetry/opentelemetry-collector` Helm chart, and put `memory_limiter` first in every pipeline. Metadata at the edge, secrets and sampling in one place. > [!VERIFIED] > OpenTelemetry Collector 0.164.0, Helm chart opentelemetry-collector 0.178.1, OpenTelemetry Operator 0.162.0, against the opentelemetry.io Kubernetes and deployment-pattern docs, the chart's values.yaml and the component READMEs. Checked October 2026. ## Agent, gateway, or both The Collector docs describe two deployment patterns, and on Kubernetes they map to two chart modes: | Pattern | Chart `mode` | What it is good at | What it costs | Run it when | |---|---|---|---|---| | Agent only | `daemonset` | Node-local receive, cheap pod metadata lookup, host and kubelet metrics | Every node holds backend credentials and does its own export retries | Small clusters, one backend | | Gateway only | `deployment` | One place for secrets, sampling and routing | Pod metadata by connection IP only; one hop for every span in the cluster | No per-node collection needed | | Agent plus gateway | both | Enrichment at the edge, policy in the middle | Two configs to keep in sync | Anything production-sized | We default to both. The agent is where `k8s_attributes` is cheapest: with a node filter, each agent watches only the pods on its own node. The gateway is where tail sampling has to live, because every span of a trace must reach the same instance, which the docs solve with the `load_balancing` exporter routing on `traceID`. ## Install the agent DaemonSet The chart refuses to render without two values. Leave either out and you get `[ERROR] 'image.repository' must be set.` or `[ERROR] 'mode' must be set.` The docs point at the Kubernetes distribution image, `otel/opentelemetry-collector-k8s`, which ships `k8s_attributes`, `load_balancing` and the tail sampling processor. ```yaml # otel-agent-values.yaml mode: daemonset image: repository: otel/opentelemetry-collector-k8s presets: kubernetesAttributes: enabled: true # adds RBAC and a node-filtered k8s_attributes processor resources: limits: memory: 512Mi # the chart derives GOMEMLIMIT from this config: processors: memory_limiter: check_interval: 1s limit_percentage: 80 spike_limit_percentage: 25 batch: {} exporters: otlp_grpc: endpoint: otel-gateway-opentelemetry-collector.observability.svc.cluster.local:4317 tls: insecure: true # in-cluster hop; use mTLS if your policy requires it service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, k8s_attributes, batch] exporters: [otlp_grpc] metrics: receivers: [otlp] processors: [memory_limiter, k8s_attributes, batch] exporters: [otlp_grpc] logs: receivers: [otlp] processors: [memory_limiter, k8s_attributes, batch] exporters: [otlp_grpc] ``` Two details in that file matter more than they look. First, we list `k8s_attributes` ourselves. The preset prepends the processor to any pipeline that doesn't already name it, which would put it ahead of `memory_limiter`; naming it explicitly keeps the limiter first. In `daemonset` mode the preset also sets `filter.node_from_env_var: K8S_NODE_NAME` and injects that variable, so each agent queries only its own node's pods. Second, the chart sets `hostPort: 4317` on the OTLP port by default, so applications reach their node's agent through the node IP. Point the SDK at it with the downward API: ```yaml # deployment.yaml (app container) env: - name: HOST_IP valueFrom: fieldRef: fieldPath: status.hostIP - name: OTEL_EXPORTER_OTLP_ENDPOINT value: "http://$(HOST_IP):4317" ``` ## Install the gateway Deployment ```yaml # otel-gateway-values.yaml mode: deployment replicaCount: 2 image: repository: otel/opentelemetry-collector-k8s resources: limits: memory: 2Gi config: processors: memory_limiter: check_interval: 1s limit_percentage: 80 spike_limit_percentage: 25 batch: {} exporters: otlp_http: endpoint: https://otlp.example-backend.com headers: authorization: "Bearer ${env:OTLP_TOKEN}" service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, batch] exporters: [otlp_http] metrics: receivers: [otlp] processors: [memory_limiter, batch] exporters: [otlp_http] logs: receivers: [otlp] processors: [memory_limiter, batch] exporters: [otlp_http] ``` ```bash $ helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts $ helm upgrade --install otel-gateway open-telemetry/opentelemetry-collector \ -n observability --create-namespace -f otel-gateway-values.yaml $ helm upgrade --install otel-agent open-telemetry/opentelemetry-collector \ -n observability -f otel-agent-values.yaml $ kubectl -n observability logs ds/otel-agent-opentelemetry-collector-agent | head ``` The gateway needs no `k8s_attributes`: the agents already attached `k8s.pod.name`, `k8s.namespace.name` and the deployment name. If you ever skip the agent tier, the processor README says a gateway can't see the original pod IP behind an agent, which is why agents in that setup run the processor in `passthrough` mode and forward the IP instead. ## The processor order is the whole game The `memory_limiter` README is explicit: put it first in every pipeline so it can push back on receivers before data piles up, and set `GOMEMLIMIT` to 80% of the container's hard limit. The chart does the second part for you (`useGOMEMLIMIT: true` by default). In containers, use `limit_percentage` rather than `limit_mib` so the limit follows the pod's memory limit. The README recommends a 1 second `check_interval`; the chart's default is 5 seconds, which is why we override it. When the limiter trips you will see `data refused due to high memory usage`. That is the system working. The SDK or the upstream agent is supposed to retry; the README warns that data is permanently lost if the component in front of the limiter doesn't. ## The errors you will hit first - `unknown type: "otlp_grpc" for id: "otlp_grpc" (valid values: [...])`: the `otlp` exporter's type has been `otlp_grpc` since core v0.144.0 (`otlphttp` is now `otlp_http`), and contrib renamed `k8sattributes` to `k8s_attributes` in v0.146.0 (`loadbalancing` is now `load_balancing`), with the old names kept as deprecated aliases. An image older than that doesn't know the new names. Pin the image tag to the chart's `appVersion`, or set `rewriteDeprecatedComponentNames: false` on old images. - `service::pipelines::traces: references processor "k8s_attributes" which is not configured`: you listed it in a pipeline without enabling the preset or defining it under `processors`. - `[ERROR] The opentelemetry-collector chart requires that the health_check extension...`: you replaced `service.extensions` and dropped `health_check`; the probes depend on it. - Pod metadata missing on every span: the ServiceAccount can't list pods. Check the agent logs for a `forbidden` error, and enable the preset instead of hand-writing the ClusterRole. When something is defined but nothing flows, the troubleshooting docs list the usual cause first: a receiver configured but not enabled in any pipeline. ## Where this breaks Helm values merge maps but replace lists, so overriding `receivers` in a pipeline silently drops the chart's defaults (Jaeger, Zipkin, the self-scrape). That is usually what you want; know it is happening. The two-tier setup also doubles your upgrade surface, and the gateway is now on the critical path for all telemetry, so give it a PodDisruptionBudget and at least two replicas. If you need auto-instrumentation injected into pods, add the OpenTelemetry Operator, which requires cert-manager, on top of this; it doesn't replace the chart. ## Questions people ask ### Should the OpenTelemetry Collector run as a DaemonSet or Deployment? Both, in most production clusters. A DaemonSet agent receives from local pods, adds Kubernetes metadata cheaply because it watches only its own node, and collects node metrics. A Deployment gateway holds backend credentials, runs tail sampling and handles retries in one place. Small clusters with one backend can run the DaemonSet alone and export directly. ### Helm chart or OpenTelemetry Operator? Use the Helm chart for the Collector itself; it is simpler to review and diff. Add the Operator when you want its `OpenTelemetryCollector` resource, sidecar injection, or auto-instrumentation of workloads through its `Instrumentation` resource. The Operator installs from a single manifest but needs cert-manager in the cluster first, which is an extra dependency to own. ### Why does my collector say data refused due to high memory usage? The `memory_limiter` processor hit its soft limit and is rejecting new data so the process doesn't get OOMKilled. Short spikes are normal. If it happens constantly, the Collector is undersized or the exporter is too slow: raise the pod's memory limit, add gateway replicas, or check that the backend isn't throttling you. Make sure SDKs retry so refused data isn't lost. ## The call we'd make Ship the agent DaemonSet plus a two-replica gateway, both from the official chart with the `otel/opentelemetry-collector-k8s` image, `memory_limiter` first and the new component names. Pin the image tag. Our [OpenTelemetry tracing setup](/blog/distributed-tracing-opentelemetry-what-we-ship) covers the SDK side, and if you are still deciding whether to run vendor agents at all, read [OpenTelemetry vs vendor agents](/blog/opentelemetry-vs-vendor-agents) first. The caveat: two tiers is overkill below a handful of nodes. ## Sources - [OpenTelemetry docs: OpenTelemetry Collector Helm chart](https://opentelemetry.io/docs/platforms/kubernetes/helm/collector/) - [OpenTelemetry docs: Gateway deployment pattern](https://opentelemetry.io/docs/collector/deploy/gateway/) - [Memory limiter processor README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md) - [OpenTelemetry docs: Collector troubleshooting](https://opentelemetry.io/docs/collector/troubleshooting/)