On-Call Without Burnout: Rotations, Runbooks, and Escalation

On-Call Without Burnout: Rotations, Runbooks, and Escalation#

The resignation email said it plainly: "I can't do another quarter of on-call like this." That was our strongest infrastructure engineer, and losing her forced a hard look at a rotation we'd let rot. She'd been paged 31 times in her last week-long shift, most of them at night, most of them for alerts that resolved themselves before she finished logging in. We'd built a system that punished people for being reliable enough to trust with the pager.

Fixing on-call isn't about paying a bigger stipend. It's rotation design, alert hygiene, runbooks, and an escalation culture that treats asking for help as normal.

Cut the pages before you touch the schedule#

No rotation is humane if the pager never stops. Before rescheduling anything, we audited a month of alerts and sorted them by "did a human do anything useful in response." The answer for 44% of pages was no, the alert self-resolved or the fix was "restart and it comes back." Those aren't incidents, they're noise trained to wake people.

We killed or auto-remediated them. A pod that OOMs and gets rescheduled by Kubernetes does not need a human at 3am; it needs a ticket in the morning and a memory-limit review.

# page only if it's still broken after the system tried to self-heal
- alert: PodCrashLoopingPersistent
  expr: |
    rate(kube_pod_container_status_restarts_total[15m]) > 0
    and
    kube_pod_container_status_ready == 0
  for: 15m          # give auto-healing a chance first
  labels:
    severity: page

The for: 15m is deliberate. If it heals itself in that window, nobody wakes up. Page volume dropped from an average of 5.2 pages per night to 0.9. That single change did more for burnout than any schedule tweak.

Rotation math that respects sleep#

Once the noise was gone, we redesigned the schedule. A few rules we now hold:

Follow-the-sun where headcount allows. We had engineers in Berlin and Austin, so a night page in Europe became a daytime page in the Americas. Where a team is single-region and can't do this, we split the week so nobody carries a full seven days of nights.

Never schedule back-to-back weeks, and enforce a recovery day after any shift with a nighttime page. We encode it in the scheduler rather than trusting goodwill.

Cap the blast radius: primary handles the page, but if they've been up past 2am on an incident, the policy explicitly says hand off to secondary and sleep. We measured "sleep-hours interrupted per shift" and made it a real metric alongside MTTR, because the human cost was invisible before we counted it.

Runbooks written for 3am, not for a demo#

A runbook you can only follow while fully awake is useless during the incident it exists for. Ours used to be architecture essays. We rewrote them to a strict shape: symptom at the top, exact commands in the middle, escalation at the bottom.

## Alert: CheckoutLatencyBurn

### You'll see
p99 on /checkout > 800ms, burn-rate alert firing.

### Check first (copy-paste)
1. Is it the DB pool?
   kubectl exec -n payments deploy/pgbouncer -- \
     psql -c "SHOW POOLS;" | grep cl_waiting
2. Any recent deploy? (last 30m)
   kubectl rollout history deploy/checkout -n payments

### Most likely fix
If a deploy in the last 30m: roll it back.
   kubectl rollout undo deploy/checkout -n payments

### If that doesn't work
Escalate to #payments-oncall secondary. Do not keep debugging alone past 20 min.

Copy-pasteable commands with real namespaces. No "investigate the database," which is not an instruction anyone half-asleep can act on. The bottom line matters as much as the top: an explicit permission to escalate at 20 minutes.

Escalation as a feature, not a failure#

The cultural rot in our old rotation was that escalating felt like admitting you weren't good enough. So people ground away alone at 3am on things a specialist could fix in five minutes. We made escalation the expected move, not the last resort.

escalation_policy:
  - level: 1
    target: primary_oncall
    timeout: 10m           # no ack in 10m -> auto-escalate
  - level: 2
    target: secondary_oncall
    timeout: 10m
  - level: 3
    target: eng_manager     # yes, the manager gets paged

Two things here changed behavior. Auto-escalation on no-ack means a primary who sleeps through a page isn't a catastrophe, the system routes onward. And putting the manager at level 3 signals that a bad night is an organizational problem, not one engineer's failure. We stopped tracking "how often did you escalate" as anything negative. Escalating early is good judgment.

Measure the human cost#

MTTR alone will happily hide a team burning out. We added metrics that see the people: pages per shift, percentage of pages at night, sleep-hours interrupted, and how many shifts hit the recovery-day rule. When any of those trend the wrong way, it's a planning topic, same as a reliability regression.

The call we'd make#

Start with alert hygiene, not the schedule. A humane rotation on top of noisy alerts is still misery, and half your pages probably shouldn't exist. Then fix runbooks so they work half-asleep, and make escalation the celebrated default instead of the walk of shame. We didn't just lower attrition, page volume down 80% meant the incidents that did fire got real attention. The engineer who quit? We'd have kept her if we'd done this a year earlier.