How We Stopped Terraform Drift from Surprising On-Call
A real story of removing console-only changes, adding drift detection, and getting Terraform back in charge.
Key takeaways
A real story of removing console-only changes, adding drift detection, and getting Terraform back in charge.
On this page
How We Stopped Terraform Drift from Surprising On-Call#
Our worst incident of last year started with a simple question: “Why is there an EC2 instance we can't find in Terraform?”
The Incident#
- An engineer had manually patched a production ASG during an outage.
- Months later, we scaled out and Terraform attempted to “fix” the ASG back to its old shape.
- That change reduced capacity during a peak event.
Fix 1: Guardrails in Terraform#
- Enabled Terraform Cloud with VCS-driven runs only.
- Turned off direct access to production AWS credentials except via break-glass.
Fix 2: Regular Drift Detection#
- Added a nightly job:
```bash terraform plan -detailed-exitcode || echo "Drift detected" ```
- Wired the exit code into Slack with a short summary.
Fix 3: Runbook and Culture#
- Wrote a runbook for “emergency changes” that includes:
- ticket link,
- time-bounded console access,
- follow-up PR to codify the change in Terraform.
Drift still happens, but on-call no longer learns about it at the worst possible moment.
Get the DevOps Troubleshooting Cheat Sheet
Subscribe and get our free one-page reference for the errors that eat an afternoon — CrashLoopBackOff, OOMKilled, Terraform state locks, and more — plus new guides as we publish them.
Systemd Tricks We Use to Keep Services Boring
Concrete systemd unit patterns that reduced flakiness: restart policies, resource limits, and structured logs.
GitOps with Argo CD: Best Practices for 2025
The Argo CD patterns that held up under real traffic: repo layout, sync policies you can trust, secrets that stay out of git, and the anti-patterns we stopped repeating.
More from Infrastructure
Explore more articles in this category
Cloudflare's Clef: When a Decision Model Beats an LLM Call
Most moderation, routing, and triage calls aren't generation problems. Clef proves it by answering in 38.8 milliseconds instead of seconds.
Perplexity Left DynamoDB for CobbleDB: When Should You?
Perplexity built its own key-value store because DynamoDB's read path and bill stopped fitting 50 KB search items. Here is the checklist for when leaving is justified.
The Terraform Lock File Is Code: Review It Before You Init
A DPRK-linked group is mailing DevOps candidates Terraform take-home repos whose lock file points at a fake registry. terraform init then runs the attacker's provider.
You might have missed
Evergreen posts worth revisiting.
