Terraform State Management Strategies

We have ~40 services across 12 AWS accounts (dev, staging, prod, plus per-team accounts). Terraform manages most of the infrastructure. State management is the part of Terraform that has caused us the most operational pain — more than any specific resource type or provider quirk. This is what we've landed on after a few migrations and a couple of near-misses.

The state file is the database#

A Terraform state file is the source of truth for "what does Terraform think exists." If it disagrees with reality (because someone clicked something in the AWS console, or because the file got corrupted, or because two terraform apply runs happened concurrently), the next plan is wrong and the next apply will do the wrong thing.

Treat state files like databases:

• They need backups
• They need access controls
• They need locking
• They need migration plans when you split or merge them

Most Terraform pain comes from skipping at least one of those four.

Backend choice: S3 + DynamoDB#

We use the S3 backend with DynamoDB for locking. Setup:

terraform {
  backend "s3" {
    bucket         = "company-tf-state-prod"
    key            = "services/payments/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "tf-state-locks"
    encrypt        = true
  }
}

The S3 bucket has:

• Versioning enabled (state file recovery)
• Server-side encryption (KMS, with a CMK we own)
• Public access block (blanket deny)
• Lifecycle rule: keep all versions for 90 days, then move old versions to Glacier

DynamoDB table is tf-state-locks with primary key LockID. Locking is per-state-file, not global. Two engineers can run apply against different state files concurrently; same state file, the second blocks.

We tried Terraform Cloud briefly. It works fine but the per-seat pricing didn't justify the additional features for our team. S3 + DynamoDB is free if you already have an AWS account.

State partitioning: per-service, per-environment#

The hardest decision is how to split state files. Too few = one apply touches half your infrastructure, blast radius is huge. Too many = endless dependencies and terraform_remote_state lookups.

Our partitioning rules:

1. Per environment: each environment (dev/staging/prod) has its own state files. No cross-environment state.
2. Per service: each service has its own state file (per env). So services/payments/dev, services/payments/staging, services/payments/prod are three separate state files.
3. Shared infrastructure (VPCs, IAM baselines, shared databases) is a separate state per env: shared/networking/prod, shared/iam/prod.

Cross-state references use terraform_remote_state:

data "terraform_remote_state" "networking" {
  backend = "s3"
  config = {
    bucket = "company-tf-state-prod"
    key    = "shared/networking/terraform.tfstate"
    region = "us-east-1"
  }
}

resource "aws_security_group" "app" {
  vpc_id = data.terraform_remote_state.networking.outputs.vpc_id
}

The networking state's outputs become the contract. Changing an output is a breaking change for every consumer.

What goes in shared vs per-service#

We've moved this line a few times. Current rules:

Shared state (one per env):

• VPC, subnets, NAT gateways
• Route 53 hosted zones
• IAM baseline (cross-account roles, SSO config)
• Centralized logging (CloudWatch logs, log destinations)

Per-service state:

• ECS services / Lambda functions / EC2 instances
• Service-specific RDS or DynamoDB tables
• Service-specific IAM roles
• ALBs, target groups, security groups for that service
• Route 53 records pointing to that service

The principle: shared state contains things that change rarely (months apart) and have many consumers. Per-service state contains things that change frequently (deploys) and only the service team owns.

Locking: enabled, always#

DynamoDB locking is non-optional. We've twice had incidents from concurrent applies (before we standardized):

• Two engineers ran apply on the same state at the same time. Each saw a partial view of the changes. The state file ended up with one engineer's resources tracked but pointing at IDs the other engineer's resources actually had. We had to manually edit the state file to fix it.
• A CI job and a human ran apply concurrently. Same outcome. We added a "no manual apply against prod state" rule and CI is the only way; locking now protects against the rare exceptions.

Lock timeout is the default (no auto-release). If a lock is stuck (CI killed mid-apply), we manually delete the lock entry from DynamoDB. We documented this and it happens maybe once a quarter.

State file size: keep it reasonable#

A state file with 5,000 resources is workable. A state file with 50,000 resources is painful — every plan/apply is slow, refresh takes minutes, errors are hard to find.

We try to keep state files under ~500 resources. If a service grows beyond that, we split it (e.g., compute and storage into separate states). Splitting is the painful operation; we plan for it before crossing the threshold rather than after.

Splitting state files: the procedure that works#

Splitting state is the hardest Terraform operation. We've done it a dozen times; here's the procedure:

1. Plan the split on paper. Which resources move to the new state file? Which references break? What new outputs are needed?
2. Create the new state file location (S3 key + Terraform config) but don't apply yet.
3. Use terraform state mv with the -state-out flag to move resources from old state to new state, one resource at a time:
   bash.bash

   terraform state mv -state=old.tfstate -state-out=new.tfstate \
     aws_s3_bucket.data aws_s3_bucket.data
   

4. Run terraform plan against both states. Expect: zero changes in both. If either shows changes, the split is wrong; revert.
5. Update any consumers of the moved resources to use terraform_remote_state against the new state.
6. Apply (which should be a no-op).

The "expect zero changes" check is critical. We once split a state and the new plan showed it would destroy and recreate a database — because we'd also accidentally changed an attribute during the split. Caught it in plan; never applied. Always run plan first.

What we got wrong: the great account migration#

We migrated from a single AWS account to multi-account about two years ago. The Terraform state migration was painful.

The plan: re-apply each module against the new account, which would create new resources. Then destroy the resources in the old account.

What actually happened: the apply against the new account, against an existing state file that referenced old-account resource IDs, did weird things. Some resources tried to update in place (changing region or account, which doesn't actually work — Terraform tried to update the ARN field and failed). Others tried to create new and got conflicts.

What we should have done:

1. Made fresh state files for the new account from scratch.
2. Imported existing resources from the old account into the new state files (terraform import).
3. Done the actual data migration separately (RDS replication, S3 sync, etc.).
4. Destroyed the old state and old resources at the end.

We did roughly the right thing, just messy and out of order. Learned: state files are tied to a backend location and a set of resource IDs. Changing both at once is a multi-step operation; never try to do it as one big apply.

Refreshing state vs trusting state#

terraform refresh (or terraform plan -refresh-only) re-reads actual resource state from AWS and updates the state file. By default, plan does a refresh first.

We disable refresh on plans in CI for performance (-refresh=false), then run a periodic full refresh on a schedule. This makes plan-on-PR fast (~30s vs 5min for big states) but means drift between TF and reality might lag.

The schedule: nightly job runs terraform plan -refresh-only against every state file and posts diffs to Slack. Drift surfaces within 24 hours.

Drift detection: the early warning#

Beyond refresh, we run driftctl against accounts to detect resources that exist in AWS but not in Terraform. Most drift comes from:

• Console clicks during incident response (someone changed a security group manually to fix something at 2 AM)
• AWS auto-creating resources (default VPCs, default security groups)
• Pre-Terraform legacy resources

The driftctl report goes into a weekly review. Each item is either: import to TF, accept and ignore, or delete.

State file recovery#

S3 versioning has saved us twice. Both times, a terraform state rm was run that shouldn't have been (the engineer thought they were removing one resource; they removed a module path that contained 30). We rolled back via:

1. List versions: aws s3api list-object-versions --bucket tf-state --prefix path/to/state
2. Identify the version-id of the pre-corruption state
3. aws s3api copy-object --copy-source bucket/path/to/state?versionId=XXX --bucket bucket --key path/to/state
4. Re-run plan to verify.

Without versioning, both incidents would have required hours of manual state reconstruction. With versioning, ~5 minutes each.

Sensitive data in state#

Terraform state contains sensitive values: database passwords, API keys, etc. Anyone with read access to the state file can see them.

Our rules:

• State files live in S3 with encryption (KMS CMK).
• Read access to state buckets is granted only to engineers who need it; everyone else uses CI's read-only outputs.
• We don't store the state files locally outside of CI.
• We rotate sensitive values periodically (quarterly), and have automated detection for state files containing AWS access keys (we have zero, by policy).

What we still don't have#

Multi-region state replication. Our state buckets are in us-east-1. If that region has a long outage, we can't run Terraform until it comes back. Acceptable risk for now; mitigation would be cross-region replication, which we haven't bothered with.

State file diff visualization. When a state file changes, we don't have a diff in the PR. We rely on terraform plan output. A nice-to-have would be a "what changed in state since last apply" view; haven't built it.

Per-service Terraform versions. All states are pinned to the same Terraform version. Upgrading TF means coordinating across all services. We've stayed disciplined about this; some teams haven't and it bit them.

What I'd tell a team starting#

Pick S3 + DynamoDB if you're on AWS. Don't fight this. It's free, it works, and it's the documented path.

Partition by service-and-environment from day one. Splitting later is painful. The instinct to "just put it all in one state for now" creates technical debt that compounds.

Enable versioning on the state bucket before you write any state to it. State recovery has saved us multiple times; it's not optional.

Always run terraform plan after a state move and expect zero changes. If plan shows changes, the move is wrong. This rule has caught dozens of mistakes.

Document the split-state procedure and run it once on staging before doing it in prod. Splitting state is a multi-step process with sharp edges. Practice helps.

State management is the unsexy part of Terraform that determines whether your team scales smoothly or constantly fights itself. Get the partitioning right and the rest gets a lot easier.