Infrastructure Documentation as Code

Two years ago we had documentation in four places: Confluence (the largest, most stale), a "engineering-handbook" GitHub repo (kept up by one person who'd recently left), individual repo READMEs (varying quality), and Slack pinned messages (sometimes the most accurate, ironically). The result was that engineers looking for "how do I do X" usually couldn't find the right doc, or found multiple conflicting docs.

This post is about the move to making infrastructure documentation live alongside the code that creates the infrastructure. Six months in, the docs are mostly correct because the only people who can update them are the ones who change the system, and they have to.

Where docs actually live now#

Three places, by category:

• Per-service runbooks: in the service's own repo, under docs/runbook.md.
• Cross-cutting infrastructure docs (network design, IAM model, observability conventions): in our central infra repo under docs/.
• Onboarding / handbook content (how to set up your laptop, who's on call, who owns what): one repo called engineering-handbook, also in markdown.

That's it. Confluence still exists for non-engineering content (HR, legal, meeting notes), but nothing engineering-relevant lives there anymore.

The principle that drove the move#

Documentation and the system it documents drift if they're maintained by different people in different tools. The closer the doc is to the code, the smaller the gap. If "update the runbook" is the next line in the same PR that changes the system, the runbook stays correct. If it's a separate Confluence edit that needs to happen tomorrow, it doesn't.

So: the doc lives in the repo, gets updated in the same PR as the code change, and shows up in code review.

Per-service runbooks#

Every service repo has a docs/runbook.md with a fixed structure:

# Runbook: <service-name>

## What this service does
One-paragraph summary. New on-call should understand within 30 seconds.

## Owner
Team: <team-name>. Slack: <#channel>. PagerDuty rotation: <link>.

## Dependencies
- Depends on: <list of upstream services + their critical paths>
- Depended on by: <list of downstream services>
- External dependencies: <list of third-party APIs, with status pages>

## Alerts and what to do

### <Alert name 1>
**What it means**: <plain-English explanation>
**Customer impact**: <when this fires, what does the user see>
**First-5-minute steps**:
  1. <specific command or link>
  2. <specific command or link>
**Common causes**: <last 6 months of incidents>

### <Alert name 2>
... etc

## Common operations
- How to deploy: <link to deploy.sh + notes>
- How to rollback: <specific steps>
- How to scale up/down: <command>
- How to enable verbose logging: <command>

## Architecture notes
Anything non-obvious about the design that on-call would need to know.

## Known gotchas
The footguns. The "if you do X, also do Y" rules.

The structure is fixed across all services so on-call engineers know where to look. The "Alerts and what to do" section is the most-read part during incidents; we put it near the top.

How we keep them updated#

Three forces, each modest, that compound:

1. PR template asks. When you submit a PR that touches app/ or Dockerfile or infrastructure/, the PR template includes a checkbox: "I have updated docs/runbook.md if applicable." It's not auto-enforced, but the box is there.

2. Quarterly drill. Once a quarter, the on-call rotation runs through every alert in every runbook and asks: "If this fired right now, would the runbook lead me to the right action?" Anything that doesn't gets a Jira ticket against the owning team.

3. Last-modified shaming. We have a tiny script that runs weekly and lists every runbook that hasn't been touched in 90+ days. It posts to a Slack channel. There's no consequence beyond visibility — but visibility is enough to nudge the team to either update or explicitly mark "this runbook is stable, no changes needed" with a date.

Cross-cutting docs in the infra repo#

Things that span multiple services live in infra/docs/:

• network-design.md: VPC layout, subnet purposes, security group conventions
• iam-model.md: how roles are structured, who has what
• observability.md: metrics naming conventions, log formats, dashboard ownership
• incident-response.md: severity definitions, communication templates, escalation paths
• cloud-account-layout.md: which account is which, what lives where

These are the docs that a new engineer reads in their first week. They change less frequently than service runbooks but matter more for the big picture.

We use mkdocs to render them as a browsable site (deployed to a static hosting bucket gated behind SSO). The same markdown source serves both GitHub viewing and the rendered site. People read whichever is convenient.

What we don't do in the docs#

• Diagrams in proprietary formats (Lucidchart, draw.io, Visio). They rot fastest because nobody can edit them without the tool. We use Mermaid embedded in markdown — version-controllable, diff-able, editable in any text editor.
• Step-by-step screenshots. UI changes break them. We describe what to do in text and link to the relevant tool/page.
• Detailed user-facing docs (API references, customer-facing guides). Those are part of the product and live elsewhere. The runbook is for the team operating the service.

The culture change#

The shift from "docs are someone else's job" to "docs are part of the change" took about two months. The first month everyone said "yes I'll update the docs" and didn't. The second month, code reviewers started leaving comments like "you didn't update the runbook for the new alert" and PRs sat until the docs were updated. By month three, it was internalized.

The forcing function was code review. Without reviewers asking, the docs would have drifted again. With reviewers asking, the doc updates landed in the same PR and the gap stayed small.

What we measured#

A few metrics, more for amusement than enforcement:

• Runbook last-modified vs service code last-modified. The closer these are, the better. Our average gap is now ~3 days (was months). 90% of services have a runbook within 14 days of code change.
• First-page-to-action time during incidents. How long from "alert fires" to "engineer takes a meaningful action." We track this in incident reviews. Has dropped from ~9 minutes to ~3 since the runbook quality improved.
• New-engineer time-to-first-incident-response. New engineers' first independent on-call shift used to be a struggle. With clearer runbooks, the gap between "got hired" and "could effectively respond to alerts" has shortened from ~6 weeks to ~2.

These aren't strict KPIs but they're the trend we wanted.

Common pitfalls#

People starting with this approach hit a few patterns:

Documenting too much. A runbook that describes every internal data structure of a service is unreadable during an incident. The runbook is for operations, not architecture. Keep it short.

Documenting too little. "See the source code" isn't a runbook. The runbook should let an on-call engineer who has never looked at the service take useful action.

Letting one person own all the runbook updates. That person becomes a bottleneck and burns out. Distributing the responsibility (one team per service) is essential.

Using fancy doc generators that nobody on the team understands. mkdocs is fine. Sphinx is fine. Pick something the team already knows. Don't introduce a new tool just for documentation.

What this isn't#

This isn't a replacement for tribal knowledge transfer. The runbook tells you what to do when an alert fires; pairing with a senior engineer tells you why. Both are needed.

This isn't a substitute for proper incident response training. The runbook helps during the moment; the training is what builds the engineer's ability to handle the situation when the runbook isn't quite right.

This isn't going to fix a culture where documentation is genuinely undervalued. If the team doesn't see writing docs as part of the work, no tooling fixes that. The shift from Confluence to git-tracked markdown helped us, but the deeper shift was treating docs as code: review, version, maintain.

What I'd tell a team starting#

Pick the worst-documented service. Put a runbook template in its repo. Have one engineer fill it out — pairing with the team that owns the service. The first runbook is the slowest. Subsequent ones get easier because the template is established.

Don't try to migrate everything from your wiki. The wiki has 90% bit-rot anyway. Migrate the 10% that's still correct, and let the rest die. The new docs are written from current truth, not from a stale source.

Tie doc updates to PR review for the corresponding service. Make it cultural. The technical part of this is trivial; the cultural part is everything.

The first time on-call thanks you for the runbook saving them at 3am — that's when you know it stuck.