What We Learned Running Weekly Game Days on Our CI/CD Pipeline

What We Learned Running Weekly Game Days on Our CI/CD Pipeline#

Most teams say they have a CI/CD pipeline; fewer can explain what happens when a deploy half-fails on a Friday night.

Game Day Scenario: Rollback That Never Rolls Back#

We simulated a bad deploy by merging a PR that intentionally broke a health check.

Observed:

• The canary failed, alerts fired, but the pipeline stopped for manual intervention.
• On-call engineers had different mental models of how rollback should work.

Fixes:

1. We made rollback a first-class job in the pipeline:

```yaml jobs: deploy_prod: steps: - run: ./scripts/deploy.sh rollback_prod: if: failure() steps: - run: ./scripts/rollback.sh ```

2. We documented one canonical rollback path per service.

Game Day Scenario: Missing Permissions#

In another exercise, we revoked a service account permission in staging.

• The deploy failed halfway through, leaving stale pods.
• Logs showed a generic “Forbidden” error; the pipeline reported only “step failed”.

Changes:

• We added structured logging around each infra call.
• We taught the pipeline to surface the exact failing command and principal.

Takeaways#

• Run game days regularly; don’t wait for production to teach you.
• Practice rollback as much as you practice deploy.
• Make pipeline failures boring and obvious, not puzzles.