How We Stopped Terraform Drift from Surprising On-Call
A real story of removing console-only changes, adding drift detection, and getting Terraform back in charge.
On this page
Our worst incident of last year started with a simple question: “Why is there an EC2 instance we can't find in Terraform?”
The Incident
- An engineer had manually patched a production ASG during an outage.
- Months later, we scaled out and Terraform attempted to “fix” the ASG back to its old shape.
- That change reduced capacity during a peak event.
Fix 1: Guardrails in Terraform
- Enabled Terraform Cloud with VCS-driven runs only.
- Turned off direct access to production AWS credentials except via break-glass.
Fix 2: Regular Drift Detection
- Added a nightly job:
```bash terraform plan -detailed-exitcode || echo "Drift detected" ```
- Wired the exit code into Slack with a short summary.
Fix 3: Runbook and Culture
- Wrote a runbook for “emergency changes” that includes:
- ticket link,
- time-bounded console access,
- follow-up PR to codify the change in Terraform.
Drift still happens, but on-call no longer learns about it at the worst possible moment.
Systemd Tricks We Use to Keep Services Boring
Concrete systemd unit patterns that reduced flakiness: restart policies, resource limits, and structured logs.
GitOps with Argo CD: Best Practices for 2026
The Argo CD 3.x patterns that held up under real traffic in 2026: repo layout, sync policies you can trust, secrets that stay out of git, and the anti-patterns we stopped repeating.
More from Infrastructure
Explore more articles in this category
AWS EU-SOUTH-2 Packet Loss: The Single-AZ Lesson Nobody Learned
A single Availability Zone in AWS's Spain region degraded for roughly 2.5 hours, and most "multi-AZ" architectures would have failed anyway.
Cloudflare's Clef: When a Decision Model Beats an LLM Call
Most moderation, routing, and triage calls aren't generation problems. Clef proves it by answering in 38.8 milliseconds instead of seconds.
Perplexity Left DynamoDB for CobbleDB: When Should You?
Perplexity built its own key-value store because DynamoDB's read path and bill stopped fitting 50 KB search items. Here is the checklist for when leaving is justified.
You might have missed
Evergreen posts worth revisiting.
