What We Learned Running Weekly Game Days on Our CI/CD Pipeline
Practical game day scenarios for CI/CD: broken rollbacks, permission issues, and slow feedback loops—and how we fixed them.
On this page
Most teams say they have a CI/CD pipeline; fewer can explain what happens when a deploy half-fails on a Friday night.
Game Day Scenario: Rollback That Never Rolls Back
We simulated a bad deploy by merging a PR that intentionally broke a health check.
Observed:
- The canary failed, alerts fired, but the pipeline stopped for manual intervention.
- On-call engineers had different mental models of how rollback should work.
Fixes:
- We made rollback a first-class job in the pipeline:
```yaml jobs: deploy_prod: steps: - run: ./scripts/deploy.sh rollback_prod: if: failure() steps: - run: ./scripts/rollback.sh ```
- We documented one canonical rollback path per service.
Game Day Scenario: Missing Permissions
In another exercise, we revoked a service account permission in staging.
- The deploy failed halfway through, leaving stale pods.
- Logs showed a generic “Forbidden” error; the pipeline reported only “step failed”.
Changes:
- We added structured logging around each infra call.
- We taught the pipeline to surface the exact failing command and principal.
Takeaways
- Run game days regularly; don’t wait for production to teach you.
- Practice rollback as much as you practice deploy.
- Make pipeline failures boring and obvious, not puzzles.
Real-World RAG Incidents: Lessons from a Production Rollout
A field report from rolling out retrieval-augmented generation in production, including cache bugs, bad embeddings, and how we fixed them.
A Pragmatic Multi-Region Strategy for Small Teams
How a small team moved from single-region risk to a simple active/passive multi-region setup without doubling complexity.
More from DevOps
Explore more articles in this category
PHP2Shell (CVE-2026-109618, CVE-2026-109619): Patch Before the Bots Do
Two unauthenticated RCEs in PHP core, leaked a day before the patch, with exploit attempts already logged. Here is how to tell if you are exposed and what to do tonight.
GitHub Copilot CLI Sandboxing Is GA: What Actually Changed
GitHub made local sandboxing generally available for Copilot CLI on October 7, so the default now restricts the agent instead of trusting it.
The Tensorlake npm Compromise Targeted Your AI Tool Configs
A compromised npm release went after cloud credentials as expected, and local AI coding-assistant config files, which most teams never treated as secrets.
You might have missed
Evergreen posts worth revisiting.
