# How We Stopped Terraform Drift from Surprising On-Call

URL: https://www.devopsness.com/blog/how-we-stopped-terraform-drift-from-surprising-on-call
Author: Kiril Urbonas
Published: 2025-03-07
Updated: 2025-03-07
Categories: Infrastructure
Tags: AWS, Terraform, Monitoring

Summary: A real story of removing console-only changes, adding drift detection, and getting Terraform back in charge.

---
# How We Stopped Terraform Drift from Surprising On-Call

Our worst incident of last year started with a simple question: “Why is there an EC2 instance we can't find in Terraform?”

## The Incident

- An engineer had **manually patched** a production ASG during an outage.
- Months later, we scaled out and Terraform attempted to “fix” the ASG back to its old shape.
- That change reduced capacity during a peak event.

## Fix 1: Guardrails in Terraform

- Enabled **Terraform Cloud** with VCS-driven runs only.
- Turned off direct access to production AWS credentials except via break-glass.

## Fix 2: Regular Drift Detection

- Added a nightly job:

\`\`\`bash
terraform plan -detailed-exitcode || echo "Drift detected"
\`\`\`

- Wired the exit code into Slack with a short summary.

## Fix 3: Runbook and Culture

- Wrote a runbook for “emergency changes” that includes:
  - ticket link,
  - time-bounded console access,
  - follow-up PR to codify the change in Terraform.

Drift still happens, but on-call no longer learns about it at the worst possible moment.