How We Stopped Terraform Drift from Surprising On-Call

How We Stopped Terraform Drift from Surprising On-Call#

Our worst incident of last year started with a simple question: “Why is there an EC2 instance we can't find in Terraform?”

The Incident#

• An engineer had manually patched a production ASG during an outage.
• Months later, we scaled out and Terraform attempted to “fix” the ASG back to its old shape.
• That change reduced capacity during a peak event.

Fix 1: Guardrails in Terraform#

• Enabled Terraform Cloud with VCS-driven runs only.
• Turned off direct access to production AWS credentials except via break-glass.

Fix 2: Regular Drift Detection#

• Added a nightly job:

```bash terraform plan -detailed-exitcode || echo "Drift detected" ```

• Wired the exit code into Slack with a short summary.

Fix 3: Runbook and Culture#

• Wrote a runbook for “emergency changes” that includes:
  • ticket link,
  • time-bounded console access,
  • follow-up PR to codify the change in Terraform.

Drift still happens, but on-call no longer learns about it at the worst possible moment.