# What We Learned Running Weekly Game Days on Our CI/CD Pipeline

URL: https://www.devopsness.com/blog/what-we-learned-running-weekly-game-days-on-our-ci-cd-pipeline
Author: Kiril Urbonas
Published: 2025-03-03
Updated: 2025-03-03
Categories: DevOps
Tags: Kubernetes, CI/CD, GitHub Actions

Summary: Practical game day scenarios for CI/CD: broken rollbacks, permission issues, and slow feedback loops—and how we fixed them.

---
# What We Learned Running Weekly Game Days on Our CI/CD Pipeline

Most teams say they have a CI/CD pipeline; fewer can explain what happens when a deploy half-fails on a Friday night.

## Game Day Scenario: Rollback That Never Rolls Back

We simulated a bad deploy by merging a PR that intentionally broke a health check.

Observed:

- The canary failed, alerts fired, but the pipeline **stopped for manual intervention**.
- On-call engineers had different mental models of how rollback should work.

Fixes:

1. We made rollback a **first-class job** in the pipeline:

\`\`\`yaml
jobs:
  deploy_prod:
    steps:
      - run: ./scripts/deploy.sh
  rollback_prod:
    if: failure()
    steps:
      - run: ./scripts/rollback.sh
\`\`\`

2. We documented one canonical rollback path per service.

## Game Day Scenario: Missing Permissions

In another exercise, we revoked a service account permission in staging.

- The deploy failed halfway through, leaving stale pods.
- Logs showed a generic “Forbidden” error; the pipeline reported only “step failed”.

Changes:

- We added **structured logging** around each infra call.
- We taught the pipeline to surface the **exact failing command** and principal.

## Takeaways

- Run game days regularly; don’t wait for production to teach you.
- Practice rollback as much as you practice deploy.
- Make pipeline failures **boring and obvious**, not puzzles.