Burn-Rate Alerting: The SLO Discipline That Prevents Alert Fatigue

The first version of every alerting setup is the same: "page if error rate > 1% for 5 minutes," plus a pile of cause alerts like "CPU > 80%", "disk > 90%", "pod restarted", "queue depth > 1000". It works for a week, then the pages start firing on blips that resolve on their own. We were paged at 3am for a CPU spike that auto-resolved, and in the same month we missed a 40-minute checkout degradation because no single resource threshold tripped. By month three, the real outage doesn't get a response.

Burn-rate alerting fixes both directions. The question changes from "is a number elevated right now?" to "are we consuming our error budget faster than we can sustain?" That second question is calibrated to user impact over time, and it is the only thing we let wake a human.

Causes lie, symptoms tell the truth

A cause-based alert assumes you know in advance which resource problem will hurt users. You don't. High CPU might be fine. Low CPU with a deadlocked thread pool might be an outage. The only thing that reliably means "users are in pain" is measuring what users experience: request success rate, latency, freshness.

So the alerts hang off SLIs defined at the user boundary, for example availability as the fraction of non-5xx requests and latency as the fraction served under 300ms. Picking those well is its own discipline; we cover it in SLI design: metrics that correlate with user experience.

If the SLO is "99.9% of requests succeed over a 30-day window," the error budget is 0.1% of requests over 30 days. With 1 million requests a month, that is 1,000 allowed errors. The budget is a quota: a sustained 0.1% error rate spends it linearly, a 30-second outage spikes it but may leave plenty. What matters is the rate of spend.

The burn rate, in one line of math

burn_rate = observed_error_rate / (1 - SLO_target)

# 99.9% SLO -> budget = 0.001
# 5% errors   -> 0.05  / 0.001 = 50   (page now)
# 0.2% errors -> 0.002 / 0.001 = 2    (not urgent)

A burn rate of 1 spends the budget exactly at the end of the window, which is the SLO limit, sustainably. A burn rate of 10 spends the month's budget in 3 days. A burn rate of 14.4 spends it in roughly 2 days, and 100 spends it in about 7 hours.

Multi-window, multi-burn-rate

A single window is either too twitchy (pages on a 30-second blip) or too slow (a 24-hour window barely moves during a 30-minute outage). The Google SRE workbook's fix is pairs of windows. Its starting point for a 99.9% SLO pages on 14.4x over 1h/5m and 6x over 6h/30m, and tickets on 1x over 3 days/6h. Our adapted ladder, with how much of the month's budget is gone by the time each rule fires:

The two-window AND is the key trick. The long window confirms the problem is sustained and not a transient. The short window confirms it is still happening right now, so the alert resolves within minutes of the fix instead of staying red for an hour after recovery. You get a low false-positive rate and a fast reset.

To see the smoothing at work: a 5-minute spike to 5% errors puts the 1-hour average around 0.4%, a burn rate of 4. Below threshold, no page. If the spike persists for 30 minutes, the 1-hour average hits 2.5%, a burn rate of 25, and it fires.

A concrete Prometheus rule

For an HTTP service with the SLO "99.9% non-5xx over 30 days":

# prometheus/rules/api-slo.yaml
- alert: APIErrorBudgetFastBurn
  expr: |
    (
      sum(rate(http_requests_total{service="api",status=~"5.."}[1h]))
      /
      sum(rate(http_requests_total{service="api"}[1h]))
    ) > 0.001 * 14.4
    and
    (
      sum(rate(http_requests_total{service="api",status=~"5.."}[5m]))
      /
      sum(rate(http_requests_total{service="api"}[5m]))
    ) > 0.001 * 14.4
  for: 5m
  labels:
    severity: page
  annotations:
    summary: "API error budget burning at >14.4x sustained rate (fast)"
    runbook: https://runbooks.internal/api-error-budget-burn

0.001 * 14.4 is the budget (0.1%) times the burn-rate threshold, so the rule reads "error rate above 1.44% on both windows." The for: 5m keeps instantaneous scrape blips from firing. The ticket and info rules have the same shape with [6h]/[30m] and * 6, and [24h]/[2h] and * 3. We use one rule template across every service with an SLO; only the SLI selector and the target change.

What we page on, and what we don't

The taxonomy that survived a year of tuning:

Page (wake someone up). Fast burn above 14.4 on both windows for 5 minutes. A 2% budget hit in an hour means a real outage is underway.

Ticket (daytime work item). Slow burn above 6. Degradation that needs investigation, not a hair-on-fire incident. The workbook pages on this row; we demoted it after a quarter of reviews showed it never needed a 3am response.

Info (Slack). Burn above 3 on the 24-hour window. A heads-up that something is trending; it often resolves on its own.

We tuned these by replaying the last quarter of incidents and asking which rules would have woken us correctly and which would have woken us for nothing.

We did not delete every resource alert. A handful survive as leading indicators at ticket severity, never page: "disk will fill in 4 hours at current rate", "cert expires in 7 days". They predict a future symptom and buy lead time. The rule: cause-based alerts may inform, only SLO alerts may page. Everything that wakes a human must trace back to a user having a bad time.

The result after the switch: total pages dropped roughly 70%, because the CPU/disk/restart noise was almost all cause-based and almost none of it user-impacting. The pages that remained correlated with real user pain nearly every time, so people stopped ignoring them. And the checkout-degradation class of incident, where success rate quietly drops while every resource looks fine, now pages within minutes because the symptom is what's measured.

Latency SLOs use the same mechanics

If the SLO is "99% of requests served in under 1s":

• Slow ratio: rate(http_requests_above_1s_total[window]) / rate(http_requests_total[window]).
• Burn rate against the 1% slow-request budget: slow_ratio / 0.01.
• Same window pairs, same thresholds, same severity tiers.

Same plumbing, different SLI. Our latency-sensitive services all run it.

Mistakes we made along the way

Forgetting the for: clause. Without it, a 200ms blip in metrics scraping could fire a page.

One very long window. "Alert if burn rate > 5 over 24 hours" catches the slow bleed but sleeps through a 30-minute outage. The window pairs cover both.

A global slo_target variable. Services have different targets. We pinned the target per rule.

Paging on cause metrics. CPU, memory pressure and disk space tell you something might fail later. SLI pages tell you users are failing now. Treat them differently.

Runbook-free pages. Every SLO alert links a runbook explaining how to investigate, what to roll back and who else to wake. For the broader alert, dashboard and runbook setup, see monitoring that actually helps on-call.

Keeping the discipline honest

Quarterly SLO review. Are we meeting targets with lots of margin (too loose) or violating them constantly (too tight)? The wider pattern sits in our deep dive on SLO-based monitoring for APIs.

After every page, grade the alert. Was it useful, too late, or a false alarm? Tune on real incident data, and feed the answer into postmortems that prevent repeat failures.

Don't alert on SLOs nobody is committed to. If product won't slow shipping when the budget is blown, the SLO is decorative.

Questions people ask

What is a good burn rate threshold for paging?

14.4 over a 1-hour window, confirmed by a 5-minute window, is the standard fast-burn threshold for a 30-day SLO. It fires after about 2% of the monthly budget is gone, which means the month's budget would be exhausted in roughly two days at that pace. Pair it with a slower 6x/6h rule for gradual degradation.

Why use two windows instead of one?

The long window proves the problem is sustained, so transient blips don't page. The short window proves it is still happening, so the alert clears within minutes of the fix instead of staying red for an hour. Either window alone gives you false positives or slow resets.

Should I delete CPU and disk alerts?

Not all of them. Keep the ones that predict a user-facing failure with useful lead time, like disk fill projections or certificate expiry, and route them to tickets. Delete the rest. Nothing cause-based should page.

The decision, concretely

• Still paging on CPU, memory or restarts? Move them to ticket severity or delete them. Only SLO burn alerts page.
• One window per alert? Switch to long-plus-short pairs: 14.4 on 1h/5m, 6 on 6h/30m, 3 on 24h/2h.
• Five SLOs per service? Cut to one per critical user journey. SLO sprawl makes the discipline incoherent.
• Page without a runbook? Write the runbook or delete the page.

The call we'd make

Start with one availability SLO per critical service, a 14.4x fast-burn page on 1h and 5m windows with for: 5m, and a 6x slow-burn ticket. Delete or demote every cause-based page the same week. It takes about a week to set up and pays back every on-call shift after it, because people start trusting the pager again. The caveat: the thresholds are only as good as the SLI under them, so if the SLI doesn't move when users complain, fix that first.

Sources

• Google SRE Workbook: Alerting on SLOs
• Prometheus documentation: Alerting rules