Fix Prometheus High Cardinality: 6 Commands to Find and Drop Bad Series

High cardinality in Prometheus is almost always one label holding unbounded values: a user ID, a request ID, a raw URL path. Each distinct value is a new time series, and the head block keeps every one of them in memory. Find it with /api/v1/status/tsdb, drop it with metric_relabel_configs, and cap each job with sample_limit.

What it looks like before anyone says "cardinality"

Nobody files a ticket titled "high cardinality". The symptoms show up first:

• The Prometheus pod gets OOMKilled after a deploy, restarts, replays the WAL, and gets killed again.
• Dashboards that loaded in a second now time out, and count() queries across a metric take minutes.
• A target turns red on the Targets page with sample limit exceeded or label_limit exceeded (metric: ..., number of labels: 41, limit: 30). Those two strings come straight from the scrape code and are good news: a limit you set earlier did its job.
• prometheus_tsdb_head_series climbs in a straight line instead of flattening out.

One error that gets blamed on cardinality and isn't: out of bounds. That one comes from the storage layer and means a sample's timestamp falls outside the window the head block accepts, which points at clock skew or backfilled data, not label explosion. Don't relabel your way out of a timestamp problem.

Find the offending metric in five minutes

Start with the TSDB status endpoint. It returns cardinality statistics for the head block: series count per metric name, value count per label name, memory per label name, and series count per label pair. It returns 10 items per list by default, and limit goes up to 10,000.

# 1. Which metric names own the most series?
$ curl -s 'http://localhost:9090/api/v1/status/tsdb?limit=15' \
    | jq '.data.seriesCountByMetricName'

# 2. Which label names have the most distinct values?
$ curl -s 'http://localhost:9090/api/v1/status/tsdb?limit=15' \
    | jq '.data.labelValueCountByLabelName'

# 3. Which labels cost the most memory?
$ curl -s 'http://localhost:9090/api/v1/status/tsdb?limit=15' \
    | jq '.data.memoryInBytesByLabelName'

If Prometheus is too sick to answer HTTP, analyze the blocks on disk instead. promtool tsdb analyze reports churn, label pair cardinality and compaction efficiency. It defaults to the last block, shows 20 items per list, and takes --match to narrow to one selector:

# 4. Offline analysis from inside the pod; the path is your --storage.tsdb.path
$ kubectl exec -n monitoring prometheus-0 -c prometheus -- \
    promtool tsdb analyze --limit=15 /prometheus

# 5. Narrow to one suspect metric
$ kubectl exec -n monitoring prometheus-0 -c prometheus -- \
    promtool tsdb analyze --match='{__name__="http_requests_total"}' /prometheus

Once you have a suspect, confirm it with PromQL. These four belong in your runbook:

# 6a. Series per metric name across the server (expensive, run it once)
topk(10, count by (__name__) ({__name__=~".+"}))

# 6b. Distinct values of one label on one metric
count(count by (path) (http_requests_total))

# 6c. Which targets are adding new series right now
topk(10, scrape_series_added)

# 6d. Which targets ship the most samples after relabeling
topk(10, scrape_samples_post_metric_relabeling)

scrape_series_added and scrape_samples_post_metric_relabeling are written automatically for every target, so 6c and 6d are cheap. A target whose scrape_series_added never drops toward zero is minting fresh series on every scrape, which is the signature of an ID in a label.

Drop it at ingestion with metric_relabel_configs

Metric relabeling runs on samples as the last step before ingestion, so it is where you remove cardinality you can't fix in the exporter today. Three moves cover nearly every case:

# prometheus.yml
scrape_configs:
  - job_name: api
    static_configs:
      - targets: ["api:8080"]
    metric_relabel_configs:
      # Drop a whole metric family nobody queries
      - source_labels: [__name__]
        regex: "http_request_duration_seconds_bucket"
        action: drop
      # Collapse raw paths into a template instead of dropping the label
      - source_labels: [path]
        regex: "/users/[0-9]+(.*)"
        target_label: path
        replacement: "/users/:id${1}"
      # Remove labels that should never have been labels
      - regex: "user_id|request_id|session_id"
        action: labeldrop

Regexes are RE2 and anchored on both ends, so user_id matches only that exact label name. The docs carry one warning worth repeating: with labeldrop and labelkeep, make sure the remaining labels still identify each series uniquely. Drop pod from a metric that differs only by pod and you collapse several series into one, and the samples start colliding.

Rewriting is usually better than dropping. Templating /users/8812/orders into /users/:id/orders keeps the dimension your dashboards group by and throws away only the unbounded part.

Cap every job so the next mistake fails loudly

Relabeling fixes the label you know about. Limits catch the one somebody ships next week. All of these default to 0, meaning no limit:

# prometheus.yml
scrape_configs:
  - job_name: api
    sample_limit: 50000             # samples per scrape, counted after relabeling
    label_limit: 30                 # labels per sample
    label_value_length_limit: 200   # bytes per label value
    static_configs:
      - targets: ["api:8080"]

The trade-off is blunt and documented: if a scrape exceeds sample_limit after metric relabeling, the entire scrape is treated as failed. You lose all of that target's data until someone fixes it, including the samples under the limit. Set it on every job anyway, because a red target with sample limit exceeded is a five-minute fix, while an OOM loop on the shared server is an outage for every team. Set sample_limit at roughly twice what scrape_samples_post_metric_relabeling shows today, and turn on extra_scrape_metrics: true so scrape_sample_limit is stored next to it for alerting:

scrape_samples_post_metric_relabeling / (scrape_sample_limit > 0) > 0.8

Prevention lives in the instrumentation code

Relabeling is a tourniquet. The fix that sticks is in the code that creates the metric. The Prometheus instrumentation guide says to keep the cardinality of most metrics below 10, and to look for another approach once a metric has the potential to pass 100. The naming guide is more direct: never use labels for user IDs, email addresses or other unbounded sets.

// Bad: one series per user and per raw URL
requests.WithLabelValues(r.URL.Path, userID).Inc()

// Good: route template from your router, bounded status class
requests.WithLabelValues(routeTemplate, statusClass(code)).Inc()

If you need per-user detail, it belongs in logs or traces, where a unique value per event costs nothing extra. We cover how teams split that work in Prometheus vs Datadog, and the label budget gets much easier once you have picked the SLIs that actually matter.

Where this breaks

metric_relabel_configs runs after the scrape, so Prometheus still downloads and parses the full exposition; it saves memory and disk, not the target's CPU or the network. Query 6a scans every series and can itself push a struggling server over the edge, so prefer the status API. And none of this shrinks data already written: dropped series age out with retention, they don't disappear on reload.

Questions people ask

How do I find high cardinality metrics in Prometheus?

Call GET /api/v1/status/tsdb?limit=20 and read seriesCountByMetricName and labelValueCountByLabelName; the top entries are your suspects. If the server can't answer HTTP, run promtool tsdb analyze against the data directory, which reports label pair cardinality and churn per block. Confirm with count(count by (label) (metric)) in PromQL before you change anything.

What happens when sample_limit is exceeded?

The whole scrape fails, including the samples under the limit. Prometheus counts samples after metric relabeling, and if the total passes sample_limit the target is marked down with sample limit exceeded, so up drops to 0 for that target. That is deliberate: a loud, scoped failure on one target beats a slow memory leak that takes the shared server down.

Does labeldrop reduce Prometheus memory usage?

Yes for new data, once the dropped label actually merges series. Memory follows the number of active series in the head block, so removing a label with 50,000 values can shrink a metric from 50,000 series to a handful. Existing series stay until they go stale and are compacted out, and dropping a label that made series unique causes collisions instead.

The call we'd make

Put sample_limit and label_limit on every scrape job today, before you have a cardinality problem, and alert at 80% of the limit. When something does blow up, find it with the TSDB status API, template or drop it with metric_relabel_configs, then fix the instrumentation so the relabel rule can be deleted. The caveat: limits fail the whole scrape, so tell service owners the rule exists before the first target goes red.

Sources

• Prometheus docs: Configuration (relabel_config, sample_limit, label_limit)
• Prometheus docs: HTTP API, TSDB stats
• Prometheus docs: promtool tsdb analyze
• Prometheus docs: Instrumentation, do not overuse labels