# Fix Prometheus High Cardinality: 6 Commands to Find and Drop Bad Series URL: https://www.devopsness.com/blog/prometheus-high-cardinality-fix Author: Kiril Urbonas Published: 2026-10-31 Updated: 2026-10-31 Categories: DevOps Tags: Prometheus, Monitoring, Observability, Troubleshooting, Performance Summary: When Prometheus runs out of memory, one label is almost always to blame. Six commands to find it, the relabel rules that drop it, and the scrape limits that stop it coming back. --- # Fix Prometheus High Cardinality: 6 Commands to Find and Drop Bad Series High cardinality in Prometheus is almost always one label holding unbounded values: a user ID, a request ID, a raw URL path. Each distinct value is a new time series, and the head block keeps every one of them in memory. Find it with `/api/v1/status/tsdb`, drop it with `metric_relabel_configs`, and cap each job with `sample_limit`. > [!VERIFIED] > Prometheus 3.15.1 (released 2026-10-21), promtool 3.15, against the prometheus.io configuration, HTTP API, promtool and instrumentation docs, and the scrape error strings in the v3.15.1 source. Checked October 2026. ## What it looks like before anyone says "cardinality" Nobody files a ticket titled "high cardinality". The symptoms show up first: - The Prometheus pod gets `OOMKilled` after a deploy, restarts, replays the WAL, and gets killed again. - Dashboards that loaded in a second now time out, and `count()` queries across a metric take minutes. - A target turns red on the Targets page with `sample limit exceeded` or `label_limit exceeded (metric: ..., number of labels: 41, limit: 30)`. Those two strings come straight from the scrape code and are good news: a limit you set earlier did its job. - `prometheus_tsdb_head_series` climbs in a straight line instead of flattening out. One error that gets blamed on cardinality and isn't: `out of bounds`. That one comes from the storage layer and means a sample's timestamp falls outside the window the head block accepts, which points at clock skew or backfilled data, not label explosion. Don't relabel your way out of a timestamp problem. ## Find the offending metric in five minutes Start with the TSDB status endpoint. It returns cardinality statistics for the head block: series count per metric name, value count per label name, memory per label name, and series count per label pair. It returns 10 items per list by default, and `limit` goes up to 10,000. ```bash # 1. Which metric names own the most series? $ curl -s 'http://localhost:9090/api/v1/status/tsdb?limit=15' \ | jq '.data.seriesCountByMetricName' # 2. Which label names have the most distinct values? $ curl -s 'http://localhost:9090/api/v1/status/tsdb?limit=15' \ | jq '.data.labelValueCountByLabelName' # 3. Which labels cost the most memory? $ curl -s 'http://localhost:9090/api/v1/status/tsdb?limit=15' \ | jq '.data.memoryInBytesByLabelName' ``` If Prometheus is too sick to answer HTTP, analyze the blocks on disk instead. `promtool tsdb analyze` reports churn, label pair cardinality and compaction efficiency. It defaults to the last block, shows 20 items per list, and takes `--match` to narrow to one selector: ```bash # 4. Offline analysis from inside the pod; the path is your --storage.tsdb.path $ kubectl exec -n monitoring prometheus-0 -c prometheus -- \ promtool tsdb analyze --limit=15 /prometheus # 5. Narrow to one suspect metric $ kubectl exec -n monitoring prometheus-0 -c prometheus -- \ promtool tsdb analyze --match='{__name__="http_requests_total"}' /prometheus ``` Once you have a suspect, confirm it with PromQL. These four belong in your runbook: ```promql # 6a. Series per metric name across the server (expensive, run it once) topk(10, count by (__name__) ({__name__=~".+"})) # 6b. Distinct values of one label on one metric count(count by (path) (http_requests_total)) # 6c. Which targets are adding new series right now topk(10, scrape_series_added) # 6d. Which targets ship the most samples after relabeling topk(10, scrape_samples_post_metric_relabeling) ``` `scrape_series_added` and `scrape_samples_post_metric_relabeling` are written automatically for every target, so 6c and 6d are cheap. A target whose `scrape_series_added` never drops toward zero is minting fresh series on every scrape, which is the signature of an ID in a label. ## Drop it at ingestion with metric_relabel_configs Metric relabeling runs on samples as the last step before ingestion, so it is where you remove cardinality you can't fix in the exporter today. Three moves cover nearly every case: ```yaml # prometheus.yml scrape_configs: - job_name: api static_configs: - targets: ["api:8080"] metric_relabel_configs: # Drop a whole metric family nobody queries - source_labels: [__name__] regex: "http_request_duration_seconds_bucket" action: drop # Collapse raw paths into a template instead of dropping the label - source_labels: [path] regex: "/users/[0-9]+(.*)" target_label: path replacement: "/users/:id${1}" # Remove labels that should never have been labels - regex: "user_id|request_id|session_id" action: labeldrop ``` Regexes are RE2 and anchored on both ends, so `user_id` matches only that exact label name. The docs carry one warning worth repeating: with `labeldrop` and `labelkeep`, make sure the remaining labels still identify each series uniquely. Drop `pod` from a metric that differs only by pod and you collapse several series into one, and the samples start colliding. Rewriting is usually better than dropping. Templating `/users/8812/orders` into `/users/:id/orders` keeps the dimension your dashboards group by and throws away only the unbounded part. ## Cap every job so the next mistake fails loudly Relabeling fixes the label you know about. Limits catch the one somebody ships next week. All of these default to 0, meaning no limit: ```yaml # prometheus.yml scrape_configs: - job_name: api sample_limit: 50000 # samples per scrape, counted after relabeling label_limit: 30 # labels per sample label_value_length_limit: 200 # bytes per label value static_configs: - targets: ["api:8080"] ``` The trade-off is blunt and documented: if a scrape exceeds `sample_limit` after metric relabeling, the entire scrape is treated as failed. You lose all of that target's data until someone fixes it, including the samples under the limit. Set it on every job anyway, because a red target with `sample limit exceeded` is a five-minute fix, while an OOM loop on the shared server is an outage for every team. Set `sample_limit` at roughly twice what `scrape_samples_post_metric_relabeling` shows today, and turn on `extra_scrape_metrics: true` so `scrape_sample_limit` is stored next to it for alerting: ```promql scrape_samples_post_metric_relabeling / (scrape_sample_limit > 0) > 0.8 ``` ## Prevention lives in the instrumentation code Relabeling is a tourniquet. The fix that sticks is in the code that creates the metric. The Prometheus instrumentation guide says to keep the cardinality of most metrics below 10, and to look for another approach once a metric has the potential to pass 100. The naming guide is more direct: never use labels for user IDs, email addresses or other unbounded sets. ```go // Bad: one series per user and per raw URL requests.WithLabelValues(r.URL.Path, userID).Inc() // Good: route template from your router, bounded status class requests.WithLabelValues(routeTemplate, statusClass(code)).Inc() ``` If you need per-user detail, it belongs in logs or traces, where a unique value per event costs nothing extra. We cover how teams split that work in [Prometheus vs Datadog](/blog/prometheus-vs-datadog), and the label budget gets much easier once you have picked [the SLIs that actually matter](/blog/sli-design-metrics-that-correlate-with-user-experience). ## Where this breaks `metric_relabel_configs` runs after the scrape, so Prometheus still downloads and parses the full exposition; it saves memory and disk, not the target's CPU or the network. Query 6a scans every series and can itself push a struggling server over the edge, so prefer the status API. And none of this shrinks data already written: dropped series age out with retention, they don't disappear on reload. ## Questions people ask ### How do I find high cardinality metrics in Prometheus? Call `GET /api/v1/status/tsdb?limit=20` and read `seriesCountByMetricName` and `labelValueCountByLabelName`; the top entries are your suspects. If the server can't answer HTTP, run `promtool tsdb analyze` against the data directory, which reports label pair cardinality and churn per block. Confirm with `count(count by (label) (metric))` in PromQL before you change anything. ### What happens when sample_limit is exceeded? The whole scrape fails, including the samples under the limit. Prometheus counts samples after metric relabeling, and if the total passes `sample_limit` the target is marked down with `sample limit exceeded`, so `up` drops to 0 for that target. That is deliberate: a loud, scoped failure on one target beats a slow memory leak that takes the shared server down. ### Does labeldrop reduce Prometheus memory usage? Yes for new data, once the dropped label actually merges series. Memory follows the number of active series in the head block, so removing a label with 50,000 values can shrink a metric from 50,000 series to a handful. Existing series stay until they go stale and are compacted out, and dropping a label that made series unique causes collisions instead. ## The call we'd make Put `sample_limit` and `label_limit` on every scrape job today, before you have a cardinality problem, and alert at 80% of the limit. When something does blow up, find it with the TSDB status API, template or drop it with `metric_relabel_configs`, then fix the instrumentation so the relabel rule can be deleted. The caveat: limits fail the whole scrape, so tell service owners the rule exists before the first target goes red. ## Sources - [Prometheus docs: Configuration (relabel_config, sample_limit, label_limit)](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) - [Prometheus docs: HTTP API, TSDB stats](https://prometheus.io/docs/prometheus/latest/querying/api/) - [Prometheus docs: promtool tsdb analyze](https://prometheus.io/docs/prometheus/latest/command-line/promtool/) - [Prometheus docs: Instrumentation, do not overuse labels](https://prometheus.io/docs/practices/instrumentation/)