How We Stopped Terraform Drift from Surprising On-Call
Last reviewed . Tool versions and defaults may have changed since, so check the commands against current docs.
A real story of removing console-only changes, adding drift detection, and getting Terraform back in charge.
On this page
Our worst incident of last year started with a simple question: “Why is there an EC2 instance we can't find in Terraform?”
The Incident
- An engineer had manually patched a production ASG during an outage.
- Months later, we scaled out and Terraform attempted to “fix” the ASG back to its old shape.
- That change reduced capacity during a peak event.
Fix 1: Guardrails in Terraform
- Enabled Terraform Cloud with VCS-driven runs only.
- Turned off direct access to production AWS credentials except via break-glass.
Fix 2: Regular Drift Detection
- Added a nightly job:
```bash terraform plan -detailed-exitcode || echo "Drift detected" ```
- Wired the exit code into Slack with a short summary.
Fix 3: Runbook and Culture
- Wrote a runbook for “emergency changes” that includes:
- ticket link,
- time-bounded console access,
- follow-up PR to codify the change in Terraform.
Drift still happens, but on-call no longer learns about it at the worst possible moment.
Cite this article
Kiril Urbonas (2025). How We Stopped Terraform Drift from Surprising On-Call. DevOpsNess. https://www.devopsness.com/blog/how-we-stopped-terraform-drift-from-surprising-on-call
[How We Stopped Terraform Drift from Surprising On-Call](https://www.devopsness.com/blog/how-we-stopped-terraform-drift-from-surprising-on-call) (Kiril Urbonas, DevOpsNess, 2025-03-07)
More from Infrastructure
Explore more articles in this category
Terraform Plan Wants to Recreate Your Production Database? Read This First
The plan says -/+ next to your RDS instance and one attribute ends in "forces replacement". Why it happens, how to undo it without data loss, and the CI gate that stops it reaching apply.
How to Stop Anyone From Deleting Your Production Database
GitLab lost six hours of data to one command on the wrong host. Linear lost five hours to one TRUNCATE CASCADE in a reviewed migration. The six guardrails that make that kind of mistake hard to make and cheap to recover from.
Kubelet Won't Start After Upgrading to Kubernetes 1.35? It's cgroup v1
Since Kubernetes 1.35 the kubelet refuses to start on cgroup v1 nodes by default. How to check every node in one command, the safe migration, and the temporary override with an expiry date.
You might have missed
Evergreen posts worth revisiting.