Alert on Symptoms, Not Causes — SLO Burn-Rate Alerting in Practice

Alert on Symptoms, Not Causes — SLO Burn-Rate Alerting in Practice#

Our old alerting was a museum of causes: "CPU > 80%", "disk > 90%", "pod restarted", "queue depth > 1000". Each one fired regularly. Almost none of them correlated with users actually having a bad time. We were paged at 3am for a CPU spike that auto-resolved, and we missed a 40-minute checkout degradation because no single resource threshold tripped. Rebuilding around SLO burn rates fixed both directions of that problem.

Causes lie; symptoms tell the truth#

A cause-based alert (CPU, memory, restarts) assumes you know in advance which resource problem will hurt users. You don't. High CPU might be fine. Low CPU with a deadlocked thread pool might be an outage. The only thing that reliably means "users are in pain" is measuring the thing users experience: request success rate, latency, freshness.

So the first move was to define SLIs at the user boundary:

• Availability: fraction of requests that return non-5xx
• Latency: fraction of requests served under 300ms

Then SLOs: 99.9% availability over 30 days. That 0.1% is the error budget — the amount of failure we're allowed before it's a problem.

Burn rate: how fast you're spending the budget#

A burn rate of 1 means you're spending error budget exactly fast enough to exhaust it at the end of the window. A burn rate of 14.4 means you'll exhaust a 30-day budget in roughly 2 days if it continues. Alerting on burn rate means alerting on how fast users are being hurt — which is exactly what should determine urgency.

burn_rate = (error_rate_observed) / (1 - SLO_target)

# For 99.9% SLO, budget = 0.001
# 5% errors → burn_rate = 0.05 / 0.001 = 50  → page NOW
# 0.2% errors → burn_rate = 0.2  → not urgent, ticket

Multi-window, multi-burn-rate: fast and slow#

A single threshold is either too twitchy (pages on a 30-second blip) or too slow (misses a steady slow burn). The Google SRE pattern uses two burn-rate conditions at two windows:

# Fast burn: big problem, page immediately
- alert: ErrorBudgetFastBurn
  expr: |
    burn_rate_1h > 14.4 and burn_rate_5m > 14.4
  for: 2m
  severity: page

# Slow burn: chronic bleed, page during business hours
- alert: ErrorBudgetSlowBurn
  expr: |
    burn_rate_6h > 6 and burn_rate_30m > 6
  severity: page

- alert: ErrorBudgetTicket
  expr: |
    burn_rate_24h > 3 and burn_rate_2h > 3
  severity: ticket

The two-window AND is the key trick. The long window (1h, 6h) confirms the problem is sustained — it's not a transient. The short window (5m, 30m) confirms it's still happening right now — so the alert auto-resolves quickly once you fix it, instead of staying red for an hour after recovery. You get both low false-positive rate and fast reset.

Tuning the thresholds to budget consumption#

The burn-rate/window pairs map to "what fraction of the month's budget would this consume before we notice":

Fast burn pages because a 2% budget hit in an hour means a real outage is underway. The ticket-level slow bleed doesn't justify waking someone — but it does need fixing before it accumulates.

What changed on call#

• Total pages dropped roughly 70%. The CPU/disk/restart noise was almost all cause-based and almost all not user-impacting.
• The pages that did fire correlated with real user pain nearly every time, so people stopped ignoring them.
• The checkout-degradation class of incident — no single resource tripped, but success rate quietly dropped — now pages within minutes because the symptom is what's measured.

Keep a few cause-based alerts — as predictors#

We didn't delete every resource alert. A handful survive as leading indicators at ticket severity, not page: "disk will fill in 4 hours at current rate", "cert expires in 7 days". These predict a future symptom and give lead time to act. The rule: cause-based alerts may inform, but only symptom-based (SLO) alerts may page. Everything that wakes a human must trace back to a user having a bad time.

For the concrete Prometheus rules behind these thresholds and how we split severity into page, ticket, and info tiers, see burn-rate alerting: the SLO discipline that prevents alert fatigue.