Skip to main content
How We Stopped Terraform Drift from Surprising On-Call

How We Stopped Terraform Drift from Surprising On-Call

KU
· Founder & Chief Editor
• 1 min read•688 views

Last reviewed . Tool versions and defaults may have changed since, so check the commands against current docs.

A real story of removing console-only changes, adding drift detection, and getting Terraform back in charge.

Our worst incident of last year started with a simple question: “Why is there an EC2 instance we can't find in Terraform?”

The Incident

  • An engineer had manually patched a production ASG during an outage.
  • Months later, we scaled out and Terraform attempted to “fix” the ASG back to its old shape.
  • That change reduced capacity during a peak event.

Fix 1: Guardrails in Terraform

  • Enabled Terraform Cloud with VCS-driven runs only.
  • Turned off direct access to production AWS credentials except via break-glass.

Fix 2: Regular Drift Detection

  • Added a nightly job:

```bash terraform plan -detailed-exitcode || echo "Drift detected" ```

  • Wired the exit code into Slack with a short summary.

Fix 3: Runbook and Culture

  • Wrote a runbook for “emergency changes” that includes:
    • ticket link,
    • time-bounded console access,
    • follow-up PR to codify the change in Terraform.

Drift still happens, but on-call no longer learns about it at the worst possible moment.

React
Share this post
Cite this article
Plain text

Kiril Urbonas (2025). How We Stopped Terraform Drift from Surprising On-Call. DevOpsNess. https://www.devopsness.com/blog/how-we-stopped-terraform-drift-from-surprising-on-call

Markdown

[How We Stopped Terraform Drift from Surprising On-Call](https://www.devopsness.com/blog/how-we-stopped-terraform-drift-from-surprising-on-call) (Kiril Urbonas, DevOpsNess, 2025-03-07)

KU

Kiril Urbonas

Founder & Chief Editor

586 articles
View all articles by Kiril Urbonas

You might have missed

Evergreen posts worth revisiting.