Burn-Rate Alerting: The SLO Discipline That Prevents Alert Fatigue

Burn-rate alerting pages only when you are spending the error budget faster than the SLO window can sustain. Burn rate is the observed error rate divided by the budget (1 minus the SLO target). For a 99.9% SLO, page at a 14.4x burn over 1 hour, confirmed over 5 minutes, and route slower burns to tickets.

The first version of every alerting setup is the same: "page if error rate > 1% for 5 minutes," plus a pile of cause alerts like "CPU > 80%", "disk > 90%", "pod restarted", "queue depth > 1000". It works for a week, then the pages start firing on blips that resolve on their own, while a slow degradation in checkout trips no single resource threshold. By month three, the real outage doesn't get a response.

The question has to change from "is a number elevated right now?" to "are we consuming our error budget faster than we can sustain?" That second question is calibrated to user impact over time, and it is the only thing we let wake a human.

Causes lie, symptoms tell the truth

A cause-based alert assumes you know in advance which resource problem will hurt users. You don't: high CPU might be fine, and low CPU with a deadlocked thread pool might be an outage. Only what users experience (success rate, latency, freshness) reliably means users are in pain.

So the alerts hang off SLIs defined at the user boundary, for example availability as the fraction of non-5xx requests and latency as the fraction served under 300ms. Picking those well is its own discipline; we cover it in SLI design: metrics that correlate with user experience.

If the SLO is "99.9% of requests succeed over a 30-day window," the error budget is 0.1% of requests over 30 days. With 1 million requests a month, that is 1,000 allowed errors. What matters is the rate of spend.

The burn rate, in one line of math

burn_rate = observed_error_rate / (1 - SLO_target)

# 99.9% SLO -> budget = 0.001
# 5% errors   -> 0.05  / 0.001 = 50   (page now)
# 0.2% errors -> 0.002 / 0.001 = 2    (not urgent)

A burn rate of 1 spends the budget exactly at the end of the window, which is the SLO limit, sustainably. A burn rate of 10 spends the month's budget in 3 days. A burn rate of 14.4 spends it in roughly 2 days, and 100 spends it in about 7 hours.

Multi-window, multi-burn-rate

A single window is either too twitchy (pages on a 30-second blip) or too slow (a 24-hour window barely moves during a 30-minute outage). The Google SRE workbook's fix is pairs of windows. Its starting point for a 99.9% SLO pages on 14.4x over 1h/5m (2% of budget) and 6x over 6h/30m (5%), and tickets on 1x over 3 days/6h (10%), with each short window 1/12 of its long window. Our adapted ladder, with how much of the month's budget is gone by the time each rule fires:

The two-window AND is the key trick. The long window confirms the problem is sustained; the short window confirms it is still happening, so the alert resolves within minutes of the fix. Low false positives, fast reset.

A concrete Prometheus rule

For an HTTP service with the SLO "99.9% non-5xx over 30 days":

- alert: APIErrorBudgetFastBurn
  expr: |
    (
      sum(rate(http_requests_total{service="api",status=~"5.."}[1h]))
      /
      sum(rate(http_requests_total{service="api"}[1h]))
    ) > 0.001 * 14.4
    and
    (
      sum(rate(http_requests_total{service="api",status=~"5.."}[5m]))
      /
      sum(rate(http_requests_total{service="api"}[5m]))
    ) > 0.001 * 14.4
  for: 5m
  labels:
    severity: page
  annotations:
    summary: "API error budget burning at >14.4x sustained rate (fast)"
    runbook: https://runbooks.internal/api-error-budget-burn

0.001 * 14.4 is the budget (0.1%) times the burn-rate threshold, so the rule reads "error rate above 1.44% on both windows." The for: 5m keeps instantaneous scrape blips from firing. The ticket and info rules have the same shape with [6h]/[30m] and * 6, and [24h]/[2h] and * 3. We use one rule template across every service with an SLO; only the SLI selector and the target change.

What we page on, and what we don't

The taxonomy we settled on:

Page (wake someone up). Fast burn above 14.4 on both windows for 5 minutes. A 2% budget hit in an hour means a real outage is underway.

Ticket (daytime work item). Slow burn above 6. Degradation that needs investigation, not a hair-on-fire incident. The workbook pages on this row; we demoted it after a quarter of reviews showed it never needed a 3am response.

Info (Slack). Burn above 3 on the 24-hour window. A heads-up that something is trending; it often resolves on its own.

Tune these by replaying the last quarter of incidents and asking which rules would have woken you correctly and which would have woken you for nothing.

We did not delete every resource alert. A handful survive as leading indicators at ticket severity, never page: "disk will fill in 4 hours at current rate", "cert expires in 7 days". They predict a future symptom and buy lead time. The rule: cause-based alerts may inform, only SLO alerts may page. Everything that wakes a human must trace back to a user having a bad time.

The effect after the switch: page volume fell sharply, because the CPU, disk and restart noise was almost all cause-based and almost none of it user-impacting. The pages that remained correlated with real user pain, so people stopped ignoring them, and the checkout-degradation class of incident now pages within minutes because the symptom is what's measured.

Latency SLOs use the same mechanics

If the SLO is "99% of requests served in under 1s":

• Slow ratio: rate(http_requests_above_1s_total[window]) / rate(http_requests_total[window]).
• Burn rate against the 1% slow-request budget: slow_ratio / 0.01.
• Same window pairs, same thresholds, same severity tiers.

Same plumbing, different SLI.

Mistakes we made along the way

Forgetting the for: clause. Without it, a 200ms blip in metrics scraping could fire a page.

Paging on cause metrics. CPU, memory pressure and disk space tell you something might fail later. SLI pages tell you users are failing now. Treat them differently.

Runbook-free pages. Every SLO alert links a runbook explaining how to investigate, what to roll back and who else to wake. For rotations and runbook setup, see on-call without burnout.

Keeping the discipline honest

Quarterly SLO review. Lots of margin means too loose; constant violations mean too tight. The wider pattern sits in our deep dive on SLO-based monitoring for APIs.

After every page, grade the alert. Useful, too late, or a false alarm? Feed the answer into post-mortems that drive change.

Don't alert on SLOs nobody is committed to. If product won't slow shipping when the budget is blown, the SLO is decorative.

Questions people ask

What is a good burn rate threshold for paging?

14.4 over a 1-hour window, confirmed by a 5-minute window, is the standard fast-burn threshold for a 30-day SLO. It fires after about 2% of the monthly budget is gone, which means the month's budget would be exhausted in roughly two days at that pace. Pair it with a slower 6x/6h rule for gradual degradation.

Why use two windows instead of one?

The long window proves the problem is sustained, so transient blips don't page. The short window proves it is still happening, so the alert clears within minutes of the fix instead of staying red for an hour. Either window alone gives you false positives or slow resets.

Should I delete CPU and disk alerts?

Not all of them. Keep the ones that predict a user-facing failure with useful lead time, like disk fill projections or certificate expiry, and route them to tickets. Delete the rest. Nothing cause-based should page.

The decision, concretely

• Still paging on CPU, memory or restarts? Move them to ticket severity or delete them. Only SLO burn alerts page.
• One window per alert? Switch to long-plus-short pairs: 14.4 on 1h/5m, 6 on 6h/30m, 3 on 24h/2h.
• Five SLOs per service? Cut to one per critical user journey. SLO sprawl makes the discipline incoherent.
• Page without a runbook? Write the runbook or delete the page.

For the thresholds at your own target, the uptime and SLA calculator computes the 1-hour, 6-hour and 3-day burn-rate alert levels for any SLO.

The call we'd make

Start with one availability SLO per critical service, a 14.4x fast-burn page on 1h and 5m windows with for: 5m, and a 6x slow-burn ticket. Delete or demote every cause-based page the same week. It takes about a week to set up and pays back every on-call shift after it, because people start trusting the pager again. The caveat: the thresholds are only as good as the SLI under them, so if the SLI doesn't move when users complain, fix that first.

Sources

• Google SRE Workbook: Alerting on SLOs
• Prometheus documentation: Alerting rules