Incident Post-Mortems That Drive Change (Not Theater)

Incident Post-Mortems That Drive Change (Not Theater)#

Post-mortems are easy to do badly. The output is a document. Documents are easy to write and forget. The incident happens again 6 months later and someone says "didn't we have a post-mortem on this?" The post-mortem succeeded as a ritual and failed as an improvement loop.

After ~80 post-mortems across two years, here's what makes the difference between theater and change.

The premise: post-mortems exist to change the system#

Not to satisfy management, not to assign blame, not to produce a document. They exist to find changes that prevent or limit the next incident.

Everything below follows from this premise. If a format, meeting, or cultural norm doesn't move the system toward improvement, drop it.

The template we use#

The template is opinionated about what gets discussed:

1. Summary. 2-3 sentences. What broke, who was affected, how long.
2. Timeline. Time-stamped, terse. From first signal to mitigation. Includes both real events ("alert fired at 14:32") and decisions ("at 14:45 we decided to roll back").
3. Impact. Users affected, requests failed, SLO budget burned, financial impact if known. Specific.
4. Root causes (plural). Almost never one. The technical cause; the process gap that let it through; the alerting gap that delayed detection. Each is a separate root cause.
5. What went well. Genuinely — what saved time, what was a good call, what the team should keep doing.
6. What didn't go well. Specific, named, no blame. "The runbook was out of date" not "Alice wrote a bad runbook."
7. Action items. Concrete, owned, dated. Each tied to a root cause.
8. Lessons. What the team learned that should propagate. Often a deeper architectural insight that doesn't fit elsewhere.

The template is 1-2 pages. Anything longer doesn't get read.

Blamelessness as a practice, not a slogan#

"We don't blame people" is easy to say and hard to do. The mechanics that make it real:

• Names appear only in the timeline. "At 14:34, oncall paged Alice; she joined at 14:36." After that, the prose is "the team" or "the system." No personal pronouns in root causes or action items.
• The author is not the responder. A separate engineer writes the post-mortem from notes. The responder reviews. This decouples the writing from the defensiveness.
• "Why didn't X do Y?" is reframed as "What made Y hard to do?" "Alice didn't roll back" becomes "The rollback procedure required steps that weren't obvious from the runbook." Always.
• The review meeting is moderated by someone outside the incident. A neutral party can interrupt blame and refocus on systems.

We've had incidents where the obvious answer was "engineer made a typo." The post-mortem turns that into "deployment system accepted a config without sufficient validation; type system didn't catch the typo; no canary detected the regression before full rollout." Three action items, each addressing a layer.

The engineer who made the typo is fine. The system that allowed the typo to cause an outage gets fixed.

Action items: the make-or-break#

Most post-mortem failure is here. The pattern that fails:

• 10 action items.
• All marked priority "high."
• Assigned to "the platform team."
• No date.
• Six months later, none done.

The pattern that works:

• 3-5 action items max. More = nothing gets done. Pick the highest-leverage.
• Each owned by a specific person, not a team. The person can delegate but is accountable.
• Each has a due date. Not "Q3"; not "soon." A specific date, like "2026-07-15."
• Each is small enough to finish in 2 weeks. "Rewrite the deployment pipeline" is not an action item. "Add a config validation step to the deployment pipeline" is.
• Each tracked as a ticket in the same tracker as feature work. Visible to everyone, with the same status discipline.

Discipline check: we have a weekly post-mortem action item review. 15 minutes. Walk through open items; what's the status? If something hasn't moved in two weeks, escalate. If something has been pushed three times, close it as "won't do" rather than letting it rot.

Letting action items rot is the actual problem. Closing them is fine if they don't earn their place; pretending they're tracked when they're not is corrosive.

The 5-Why isn't enough#

The 5-Why technique (keep asking "why?" until you reach a root cause) is taught everywhere. It produces a single chain of causes. Real incidents have multiple chains.

We use a fishbone / Ishikawa approach: identify the categories of cause (people / process / tools / environment) and probe each separately. Then for each cause, ask "why?" until you bottom out.

A typical incident has:

• A technical cause (the actual bug).
• A detection gap (we found out from a customer, not monitoring).
• A response gap (the runbook was wrong).
• A prevention gap (the bug should have been caught in CI).
• A communication gap (status page wasn't updated).

Five separate threads, five separate action items.

Cultural moves that compound#

A few practices that have outsized impact:

Publish post-mortems internally to everyone. Not just the team involved. Other teams learn from incidents they didn't experience. The post-mortem becomes shared organizational learning.

Pattern reviews. Every quarter, a senior engineer reads the last quarter's post-mortems and writes a "patterns" doc. "We had 4 incidents related to autoscaling configuration this quarter." That pattern gets a strategic action item — not a per-incident one.

Postmortem of the post-mortem. Once a year, look at the action item completion rate, the time-to-mitigate trend, the recurrence rate. Are post-mortems actually preventing repeats? If not, the process needs adjustment.

Practice incidents. Run game days that trigger an incident in a controlled way. Run the response, write the post-mortem, run through action items. Build the muscle outside of real pressure.

What we used to do that we stopped#

Long meetings. A 90-minute "post-mortem review" with 15 people. Almost no one was prepared; discussion was shallow. Now: doc circulated 24 hours before, meeting is 30 minutes, focused on action items.

Severity-based formality. SEV1 incidents got a full post-mortem; SEV3 got a brief note. Result: pattern-finding was hard because SEV3 patterns weren't captured. Now everything ≥ SEV3 gets the same template (which is short enough that this isn't burdensome).

Action items as resolution criteria. "We can close this incident when action items are done." Action items often took months; incidents stayed "open" forever. Now: incident is closed when mitigated; action items live in their own tracker.

Naming-and-shaming any individual. Even mild — "X should have done Y." Erodes psychological safety. The org becomes worse at incident response when responders are afraid of post-mortem fallout.

Metrics that matter#

• Mean time to mitigate. Trends downward over time as the team learns.
• Action item completion rate. Aim for ~80% within their due date.
• Recurrence rate. Same root cause appearing twice within 6 months. Investigate why action items didn't prevent.
• Incidents per quarter. Useful at scale. Increasing without explanation = something is degrading.

Things that surprised us#

The "what went well" section matters. When all you discuss is what broke, the team feels defeated. Documenting what saved time builds confidence and reinforces good practice. Don't skip it.

Action items written by the people who'll do them are better. When the post-mortem author writes generic action items ("improve monitoring"), the assignee doesn't own them. When the assignee writes their own action item ("add a Prometheus alert for queue depth > 5000"), they ship it.

The trend is harder to measure than the individual. Each post-mortem feels productive. The pattern across all post-mortems is where the real insight is. The quarterly review is more impactful than any single post-mortem.

What to read next#

• Chaos engineering — game days for platform teams — the controlled-incident practice
• Burn-rate alerting and SLO discipline — the detection side
• SLI design — metrics that correlate with user experience — the measurement side
• Pipeline observability — CI failures and alerts — adjacent reliability discipline

Post-mortems are operationally simple and culturally hard. The template above is the easy part. Sustaining blamelessness, completing action items, and reading the long-term pattern is where the work is. Done well, every incident is tuition in a system that learns. Done badly, they're a ritual that consumes time and changes nothing.