Best Practices: Infrastructure Documentation as Code
Wikis rot. We moved every operational doc into the repo it describes. Six months in, the docs are mostly correct because the only people who can update them are the ones who change the system.
On this page
Infrastructure Documentation as Code
Two years ago we had documentation in four places: Confluence (the largest, most stale), a "engineering-handbook" GitHub repo (kept up by one person who'd recently left), individual repo READMEs (varying quality), and Slack pinned messages (sometimes the most accurate, ironically). The result was that engineers looking for "how do I do X" usually couldn't find the right doc, or found multiple conflicting docs.
This post is about the move to making infrastructure documentation live alongside the code that creates the infrastructure. Six months in, the docs are mostly correct because the only people who can update them are the ones who change the system, and they have to.
Where docs actually live now
Three places, by category:
- Per-service runbooks: in the service's own repo, under
docs/runbook.md. - Cross-cutting infrastructure docs (network design, IAM model, observability conventions): in our central
infrarepo underdocs/. - Onboarding / handbook content (how to set up your laptop, who's on call, who owns what): one repo called
engineering-handbook, also in markdown.
That's it. Confluence still exists for non-engineering content (HR, legal, meeting notes), but nothing engineering-relevant lives there anymore.
The principle that drove the move
Documentation and the system it documents drift if they're maintained by different people in different tools. The closer the doc is to the code, the smaller the gap. If "update the runbook" is the next line in the same PR that changes the system, the runbook stays correct. If it's a separate Confluence edit that needs to happen tomorrow, it doesn't.
So: the doc lives in the repo, gets updated in the same PR as the code change, and shows up in code review.
Per-service runbooks
Every service repo has a docs/runbook.md with a fixed structure:
# Runbook: <service-name>
## What this service does
One-paragraph summary. New on-call should understand within 30 seconds.
## Owner
Team: <team-name>. Slack: <#channel>. PagerDuty rotation: <link>.
## Dependencies
- Depends on: <list of upstream services + their critical paths>
- Depended on by: <list of downstream services>
- External dependencies: <list of third-party APIs, with status pages>
## Alerts and what to do
### <Alert name 1>
**What it means**: <plain-English explanation>
**Customer impact**: <when this fires, what does the user see>
**First-5-minute steps**:
1. <specific command or link>
2. <specific command or link>
**Common causes**: <last 6 months of incidents>
### <Alert name 2>
... etc
## Common operations
- How to deploy: <link to deploy.sh + notes>
- How to rollback: <specific steps>
- How to scale up/down: <command>
- How to enable verbose logging: <command>
## Architecture notes
Anything non-obvious about the design that on-call would need to know.
## Known gotchas
The footguns. The "if you do X, also do Y" rules.
The structure is fixed across all services so on-call engineers know where to look. The "Alerts and what to do" section is the most-read part during incidents; we put it near the top.
How we keep them updated
Three forces, each modest, that compound:
-
PR template asks. When you submit a PR that touches
app/orDockerfileorinfrastructure/, the PR template includes a checkbox: "I have updateddocs/runbook.mdif applicable." It's not auto-enforced, but the box is there. -
Quarterly drill. Once a quarter, the on-call rotation runs through every alert in every runbook and asks: "If this fired right now, would the runbook lead me to the right action?" Anything that doesn't gets a Jira ticket against the owning team.
-
Last-modified shaming. We have a tiny script that runs weekly and lists every runbook that hasn't been touched in 90+ days. It posts to a Slack channel. There's no consequence beyond visibility — but visibility is enough to nudge the team to either update or explicitly mark "this runbook is stable, no changes needed" with a date.
Cross-cutting docs in the infra repo
Things that span multiple services live in infra/docs/:
network-design.md: VPC layout, subnet purposes, security group conventionsiam-model.md: how roles are structured, who has whatobservability.md: metrics naming conventions, log formats, dashboard ownershipincident-response.md: severity definitions, communication templates, escalation pathscloud-account-layout.md: which account is which, what lives where
These are the docs that a new engineer reads in their first week. They change less frequently than service runbooks but matter more for the big picture.
We use mkdocs to render them as a browsable site (deployed to a static hosting bucket gated behind SSO). The same markdown source serves both GitHub viewing and the rendered site. People read whichever is convenient.
What we don't do in the docs
- Diagrams in proprietary formats (Lucidchart, draw.io, Visio). They rot fastest because nobody can edit them without the tool. We use Mermaid embedded in markdown — version-controllable, diff-able, editable in any text editor.
- Step-by-step screenshots. UI changes break them. We describe what to do in text and link to the relevant tool/page.
- Detailed user-facing docs (API references, customer-facing guides). Those are part of the product and live elsewhere. The runbook is for the team operating the service.
The culture change
The shift from "docs are someone else's job" to "docs are part of the change" took about two months. The first month everyone said "yes I'll update the docs" and didn't. The second month, code reviewers started leaving comments like "you didn't update the runbook for the new alert" and PRs sat until the docs were updated. By month three, it was internalized.
The forcing function was code review. Without reviewers asking, the docs would have drifted again. With reviewers asking, the doc updates landed in the same PR and the gap stayed small.
What to measure
A few metrics, more for amusement than enforcement:
- Runbook last-modified vs service code last-modified. The closer these are, the better. Our average gap is now ~3 days (was months). 90% of services have a runbook within 14 days of code change.
- First-page-to-action time during incidents. How long from "alert fires" to "engineer takes a meaningful action." We track this in incident reviews. Has dropped from ~9 minutes to ~3 since the runbook quality improved.
- New-engineer time-to-first-incident-response. New engineers' first independent on-call shift used to be a struggle. With clearer runbooks, the gap between "got hired" and "could effectively respond to alerts" has shortened from ~6 weeks to ~2.
These aren't strict KPIs but they're the trend we wanted.
Common pitfalls
People starting with this approach hit a few patterns:
Documenting too much. A runbook that describes every internal data structure of a service is unreadable during an incident. The runbook is for operations, not architecture. Keep it short.
Documenting too little. "See the source code" isn't a runbook. The runbook should let an on-call engineer who has never looked at the service take useful action.
Letting one person own all the runbook updates. That person becomes a bottleneck and burns out. Distributing the responsibility (one team per service) is essential.
Using fancy doc generators that nobody on the team understands. mkdocs is fine. Sphinx is fine. Pick something the team already knows. Don't introduce a new tool just for documentation.
What this isn't
This isn't a replacement for tribal knowledge transfer. The runbook tells you what to do when an alert fires; pairing with a senior engineer tells you why. Both are needed.
This isn't a substitute for proper incident response training. The runbook helps during the moment; the training is what builds the engineer's ability to handle the situation when the runbook isn't quite right.
This isn't going to fix a culture where documentation is genuinely undervalued. If the team doesn't see writing docs as part of the work, no tooling fixes that. The shift from Confluence to git-tracked markdown helped us, but the deeper shift was treating docs as code: review, version, maintain.
What I'd tell a team starting
Pick the worst-documented service. Put a runbook template in its repo. Have one engineer fill it out — pairing with the team that owns the service. The first runbook is the slowest. Subsequent ones get easier because the template is established.
Don't try to migrate everything from your wiki. The wiki has 90% bit-rot anyway. Migrate the 10% that's still correct, and let the rest die. The new docs are written from current truth, not from a stale source.
Tie doc updates to PR review for the corresponding service. Make it cultural. The technical part of this is trivial; the cultural part is everything.
The first time on-call thanks you for the runbook saving them at 3am — that's when you know it stuck.
Cite this article
Kiril Urbonas (2025). Best Practices: Infrastructure Documentation as Code. DevOpsNess. Updated 2026-11-20. https://www.devopsness.com/blog/best-practices-infrastructure-documentation-as-code
[Best Practices: Infrastructure Documentation as Code](https://www.devopsness.com/blog/best-practices-infrastructure-documentation-as-code) (Kiril Urbonas, DevOpsNess, updated 2026-11-20)
More from Infrastructure
Explore more articles in this category
Database Sharding: The Choices We Wish We'd Made Earlier
Sharding is more than "split the table": the shard key choice cascades through queries, joins, rebalancing, and operations. The decisions that pay off and the ones we redid.
50 Terraform Lessons Most Teams Learn Too Late
Terraform codebases rarely fail on day one. They fail on day 400. Fifty lessons from Terraform in Depth, fact-checked against current Terraform and OpenTofu, with code and a 20-point checklist.
Terraform State Management Strategies
How we organize Terraform state across 12 AWS accounts and 40+ services. Backends, locking, partitioning, and the migration we got wrong twice.
You might have missed
Evergreen posts worth revisiting.