Practical game day scenarios for CI/CD: broken rollbacks, permission issues, and slow feedback loops—and how we fixed them.
Most teams say they have a CI/CD pipeline; fewer can explain what happens when a deploy half-fails on a Friday night.
We simulated a bad deploy by merging a PR that intentionally broke a health check.
Observed:
Fixes:
```yaml jobs: deploy_prod: steps: - run: ./scripts/deploy.sh rollback_prod: if: failure() steps: - run: ./scripts/rollback.sh ```
In another exercise, we revoked a service account permission in staging.
Changes:
Get the latest tutorials, guides, and insights on AI, DevOps, Cloud, and Infrastructure delivered directly to your inbox.
A field report from rolling out retrieval-augmented generation in production, including cache bugs, bad embeddings, and how we fixed them.
How a small team moved from single-region risk to a simple active/passive multi-region setup without doubling complexity.
Explore more articles in this category
A practical guide to AIOps and self-healing infrastructure: the maturity ladder from alert to autonomous remediation, observability as the foundation, anomaly detection, Kubernetes-native self-healing, an operator-driven remediation loop, agentic SRE, and the guardrails that keep automation safe.
Run only the CI jobs a change actually affects using if conditionals, trigger path filters, and per-job path detection in a monorepo.
A prioritized toolkit for cutting CI time: measure the critical path first, then cache, parallelize, run only what changed, and shrink the work itself.
Evergreen posts worth revisiting.