What We Learned Running Weekly Game Days on Our CI/CD Pipeline
Practical game day scenarios for CI/CD: broken rollbacks, permission issues, and slow feedback loops—and how we fixed them.
Key takeaways
Practical game day scenarios for CI/CD: broken rollbacks, permission issues, and slow feedback loops—and how we fixed them.
On this page
What We Learned Running Weekly Game Days on Our CI/CD Pipeline#
Most teams say they have a CI/CD pipeline; fewer can explain what happens when a deploy half-fails on a Friday night.
Game Day Scenario: Rollback That Never Rolls Back#
We simulated a bad deploy by merging a PR that intentionally broke a health check.
Observed:
- The canary failed, alerts fired, but the pipeline stopped for manual intervention.
- On-call engineers had different mental models of how rollback should work.
Fixes:
- We made rollback a first-class job in the pipeline:
```yaml jobs: deploy_prod: steps: - run: ./scripts/deploy.sh rollback_prod: if: failure() steps: - run: ./scripts/rollback.sh ```
- We documented one canonical rollback path per service.
Game Day Scenario: Missing Permissions#
In another exercise, we revoked a service account permission in staging.
- The deploy failed halfway through, leaving stale pods.
- Logs showed a generic “Forbidden” error; the pipeline reported only “step failed”.
Changes:
- We added structured logging around each infra call.
- We taught the pipeline to surface the exact failing command and principal.
Takeaways#
- Run game days regularly; don’t wait for production to teach you.
- Practice rollback as much as you practice deploy.
- Make pipeline failures boring and obvious, not puzzles.
Get the DevOps Troubleshooting Cheat Sheet
Subscribe and get our free one-page reference for the errors that eat an afternoon — CrashLoopBackOff, OOMKilled, Terraform state locks, and more — plus new guides as we publish them.
Real-World RAG Incidents: Lessons from a Production Rollout
A field report from rolling out retrieval-augmented generation in production, including cache bugs, bad embeddings, and how we fixed them.
A Pragmatic Multi-Region Strategy for Small Teams
How a small team moved from single-region risk to a simple active/passive multi-region setup without doubling complexity.
More from DevOps
Explore more articles in this category
GitLab's New Rate Limits: What to Fix Before Oct 19
GitLab is capping unauthenticated API calls at 60 an hour starting October 19, and the preview windows land before most teams will have noticed.
Storm-3068: A CI/CD Pipeline Is a Kubeconfig Exfiltration Machine
Microsoft's Storm-3068 report used zero malware to steal Kubernetes credentials, just a password reset, a pipeline edit, and permissions nobody had scoped down.
Docker Cloud Sandboxes: Why Agents Need MicroVMs, Not Containers
Docker put AI coding agents in hosted microVMs instead of containers, because a container was never the isolation boundary this job needed.
You might have missed
Evergreen posts worth revisiting.
