Error Budgets to Roadmap: Turning Reliability Into Prioritization

Every quarter we had the same fight. SRE wanted to spend a sprint hardening the payments service. Product wanted three new features and argued reliability was "already good enough." Both sides had vibes and neither had numbers, so the loudest person in the room won, which was usually whoever had the VP's ear that week.

The error budget ended that fight. Not because everyone suddenly agreed, but because we replaced opinions with a shared number that either had room in it or didn't.

What the budget actually is #

If your SLO is 99.9% success over 30 days, your error budget is the 0.1% you're allowed to fail. That's not a rounding error, it's a real allowance you can spend. Over 30 days, 0.1% is about 43 minutes of full downtime, or the equivalent spread across degraded requests.

The reframe that made it click for product managers: the budget is a resource, like a compute budget. Spend it on shipping fast and taking risks, or hoard it and move cautiously. Your call. But when it's gone, it's gone.

code

budget_total   = (1 - 0.999) * total_requests
budget_spent   = failed_requests
budget_remaining = budget_total - budget_spent

We compute remaining budget as a percentage and put it on a wall in the team room. At the start of a 30-day window you have 100%. The only interesting question each week is how fast it's dropping.

Burn rate is the signal, not the raw number #

Remaining budget alone misleads. 40% left sounds healthy until you notice you burned 60% in the first three days. Burn rate, how fast you're spending relative to your allowance, is what tells you whether you're actually in trouble.

promql.promql

# 1-hour burn rate: current error ratio divided by the budget ratio
(
  sum(rate(http_requests_total{code=~"5..",service="payments"}[1h]))
  /
  sum(rate(http_requests_total{service="payments"}[1h]))
) / 0.001

A burn rate of 1 means you'll spend exactly your budget over the window, no more. A burn rate of 14.4 means you'll blow a whole month's budget in about two days. We alert on multi-window burn (fast and slow both firing) so a single bad minute doesn't page anyone.

The policy that turns the number into a decision #

Here's the part most teams skip and it's the part that matters. A budget with no policy attached is just another dashboard. We wrote down, and got leadership to sign, what happens at each threshold. This is the whole point, the number has to bind before an incident, not get argued about after.

yaml.yaml

error_budget_policy:
  service: payments
  slo: 99.9
  window_days: 30
  thresholds:
    - remaining_gt: 50
      action: "Ship normally. Feature work proceeds."
    - remaining_between: [10, 50]
      action: "New features require a reliability review before merge."
    - remaining_lt: 10
      action: "Feature freeze. All hands on reliability until budget recovers."
    - exhausted: true
      action: "Freeze plus incident review. No deploys except fixes."

The feature freeze at 10% is the teeth. The first time we hit it, product pushed back hard. But the policy was signed, so instead of a debate we had a rule, and the team spent the next week on the retry storm that had been eating the budget. Budget recovered, features resumed. Nobody had to win an argument.

Making it visible where decisions get made #

The budget only works if it shows up in planning, not just in Grafana. We pipe remaining budget into the sprint planning doc automatically, so when a PM opens the board on Monday, the number is right there next to the feature list.

python.python

remaining = fetch_budget_remaining("payments")
status = "FREEZE" if remaining < 10 else "review" if remaining < 50 else "ok"
post_to_planning_doc(f"Payments budget: {remaining:.0f}% remaining ({status})")

When the number is in the room, the conversation changes. "Should we ship the risky migration this sprint?" stops being a personality contest and becomes "we have 62% budget left, yes, and we'll watch burn rate during rollout."

What it doesn't fix #

An error budget is only as honest as the SLO behind it. If your SLI doesn't reflect what users feel, you'll happily ship while customers suffer, budget intact. We learned this the hard way when a data-freshness bug hurt users for a week without touching our success-rate SLO at all. Garbage SLI in, false confidence out.

The call we'd make #

Don't roll out error budgets until you've written the policy and gotten a real signature on the freeze rule. The budget without the policy is theater. The policy is what converts a metric into a decision that survives contact with a pushy stakeholder. Start with your single most critical service, prove the freeze works once, then expand.

Error Budgets to Roadmap: Turning Reliability Into Prioritization

What the budget actually is#

Burn rate is the signal, not the raw number#

The policy that turns the number into a decision#

Making it visible where decisions get made#

#Error Budgets

Error Budgets to Roadmap: Turning Reliability Into Prioritization