# Chaos Engineering: What We Actually Run as Game Days URL: https://www.devopsness.com/blog/chaos-engineering-game-days-platform-teams Author: Kiril Urbonas Published: 2026-05-01 Updated: 2026-10-31 Categories: DevOps Tags: Chaos Engineering, Reliability, Game Days, DevOps, SRE Summary: How we'd run a quarterly chaos game day. The scenarios that tend to surface real problems, the ones that rarely do, and the operational discipline that makes the practice pay back. --- # Chaos Engineering: What We Actually Run as Game Days A chaos game day is a scheduled exercise where a team injects one controlled failure into production, responds as if it were real, then fixes what broke. Our format: one scenario per quarter, a few hours on a weekday, a written blast-radius cap, and an abort condition wired to an alarm. The fixes are the payoff, not the drill. > [!VERIFIED] > AWS Fault Injection Service (FIS), Chaos Mesh 2.8.7, against the AWS FIS user guide (actions reference, stop conditions, template examples), the chaos-mesh.org docs and principlesofchaos.org. Checked October 2026. ## What a game day is, in our flavor Five to ten people on a call, one scenario picked in advance from a backlog of "things we should test", and an explicit blast-radius limit. Someone injects the failure; on-call responds with the real runbooks and real dashboards. Afterward, a debrief: what worked, what didn't, what changes, and who owns each change. It is neither continuous chaos (Netflix's Chaos Monkey model) nor a full disaster-recovery drill. It follows the loop from the Principles of Chaos Engineering: define steady state as a measurable output (error rate, latency, throughput), predict it will hold, inject a realistic fault, and try to disprove the prediction. ## The scenarios that tend to find real problems These are the five we'd put at the top of any backlog, because each exposes a class of problem that dev and staging hide. **Fail over the primary database.** FIS has `aws:rds:reboot-db-instances` with `forceFailover: true` for Multi-AZ instances, and `aws:rds:failover-db-cluster` for Aurora. The database usually recovers on schedule. What typically fails is the application: connection pools holding dead connections, no reconnect logic, a DNS TTL cached too long. Measure time to recovery from the app's side, not from the RDS event. **Lose one Availability Zone's worth of nodes.** Stateless multi-AZ deployments shrug this off. What it exposes is the single-replica thing nobody remembers: an in-cluster database chart without HA, a single-AZ NAT gateway, a PodDisruptionBudget that blocks rescheduling. **Saturate outbound network from one workload.** Shared egress (NAT gateways, proxies) is a shared fate. One noisy pod can starve unrelated services of outbound calls. **Add latency to a dependency.** Use enough to matter: 100 ms often hides inside normal jitter, while 500 ms to 1 s exposes oversized timeouts, missing circuit breakers and request handlers piling up until a pod falls over. **Kill the telemetry pipeline.** Stop the OpenTelemetry collectors and see whether traces buffer or vanish, and whether anyone notices. Losing visibility during an incident is its own incident. ## The scenarios that rarely find much Negative results are still information, so log them: - **Killing one app pod.** Kubernetes replaces it. Good as a first drill to calibrate the team, not as a finding. - **Small latency on one downstream.** Within jitter. Escalate the number or drop the scenario. - **Rehearsed DNS-based region failover.** If it already works twice in a row, keep it as practice, not discovery. - **Killing a cache primary with working Sentinel or replication.** A few seconds of failed reads, then recovery. Check it once, then move on. ## The tooling, with configs you can adapt **AWS FIS** for cloud-level faults. Stop conditions are CloudWatch alarms; when one fires, FIS stops the experiment and it cannot be resumed. This template, adapted from the AWS example, stops tagged worker instances in one AZ and restarts them after ten minutes: ```json # fis/az-a-node-loss.json { "description": "Game day: stop prod workers in us-east-1a, restart after 10m", "targets": { "azWorkers": { "resourceType": "aws:ec2:instance", "resourceTags": { "role": "k8s-worker", "env": "prod" }, "filters": [ { "path": "Placement.AvailabilityZone", "values": ["us-east-1a"] }, { "path": "State.Name", "values": ["running"] } ], "selectionMode": "PERCENT(50)" } }, "actions": { "stopWorkers": { "actionId": "aws:ec2:stop-instances", "parameters": { "startInstancesAfterDuration": "PT10M" }, "targets": { "Instances": "azWorkers" } } }, "stopConditions": [ { "source": "aws:cloudwatch:alarm", "value": "arn:aws:cloudwatch:us-east-1:111122223333:alarm:api-5xx-over-1pct" } ], "roleArn": "arn:aws:iam::111122223333:role/fis-gameday" } ``` (Strip the first comment line before passing it to the CLI; JSON has no comments.) For EKS managed node groups, `aws:eks:terminate-nodegroup-instances` takes `instanceTerminationPercentage` (1-100) instead. ``` $ aws fis create-experiment-template --cli-input-json file://fis/az-a-node-loss.json $ aws fis start-experiment --experiment-template-id EXT1a2b3c4d5e6f7 $ aws fis stop-experiment --id EXP1a2b3c4d5e6f7 # manual abort ``` **Chaos Mesh** (a CNCF incubating project) for pod-level faults: kill, network delay or partition, IO faults, stress. The delay experiment, with `mode` capping the blast radius: ```yaml # chaos/payments-db-latency.yaml apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: payments-db-latency namespace: payments spec: action: delay mode: fixed-percent value: "30" selector: namespaces: [payments] labelSelectors: app: payments-api delay: latency: "500ms" jitter: "50ms" correlation: "100" direction: to duration: "10m" ``` ``` $ kubectl apply -f chaos/payments-db-latency.yaml $ kubectl delete -f chaos/payments-db-latency.yaml # abort, removes the fault ``` Modes are `one`, `all`, `fixed`, `fixed-percent` and `random-max-percent`. Never use `all` on a first run. ## Safety discipline **A one-page plan per drill.** What we'll do, when, expected impact, abort criteria, comms, roles. Signed off by the platform lead. The value is the conversation it forces. **Comms 24 hours ahead.** Post in the channels of every team whose services might be affected. They don't act; they just don't panic at odd graphs. **Blast-radius caps in config, not in intent.** `PERCENT(50)` in FIS, `fixed-percent` in Chaos Mesh. The cap bounds the worst case when the drill goes wrong. **Abort criteria written in advance.** "API 5xx rate above 1% for 60 seconds" becomes the CloudWatch alarm FIS watches. Without a number, abort becomes an argument mid-incident. **Follow-ups tracked to closure.** Each finding gets an owner and a ticket, reviewed at the next drill. Unfixed findings teach the team that drills don't matter. Run the debrief like any [post-mortem that drives change](/blog/incident-post-mortems-that-drive-change). ## Failure modes when running game days - **The experiment goes to `stopped` seconds after `start-experiment`.** A stop condition triggered: the alarm was already in `ALARM` state, or it watches a metric that is noisy at baseline. Fix the alarm before the drill. - **FIS action fails with an IAM access denied error.** The experiment role is missing the action's permissions, e.g. `ec2:StopInstances` or `rds:RebootDBInstance`; each action in the reference lists what it needs. - **`aws:eks:terminate-nodegroup-instances` targets nothing.** It supports only EKS managed node groups, not self-managed ones. - **Chaos Mesh delay has no effect.** The selector matched no pods, or the network action is blocked by the pod's security policy. `kubectl describe networkchaos ` shows the targeted pods and injection status. - **The fault outlives the drill.** Always set `duration` (Chaos Mesh) or a restart parameter (FIS), and rehearse the abort command before you start. ## What we don't do **Continuous chaos in production.** At our size, a 24/7 random killer trains people to dismiss alerts as drill noise. **Chaos only in staging.** Staging builds confidence; production-shaped traffic finds the real issues. **3 AM drills.** Nobody can respond at 3 AM either, which defeats the point. Weekday, mid-morning to early afternoon, full staffing. ## Where this breaks Skip game days until three things exist: observability you trust, an incident process the team already uses, and leadership that will fund the fixes. Without them a drill adds stress and produces findings nobody acts on. Regulated workloads may also forbid deliberate production faults; there, run the same scenarios against a production-scale replica and say so plainly in the report. ## Questions people ask ### What is a game day in chaos engineering? A game day is a planned session where a team deliberately injects a failure, such as a database failover or losing an Availability Zone, into a real system and responds as if it were an incident. It has a written plan, a blast-radius limit, abort criteria and a debrief. The goal is to find weaknesses in systems and in the human response before a real outage does. ### How often should you run chaos game days? Quarterly is enough for most platform teams. Each drill needs planning time beforehand and engineering time afterward to fix what it found, and running more often than you can fix just piles up findings. Start with low-risk scenarios like killing a single pod, then a node, then an AZ, and only automate continuous experiments once the quarterly findings dry up. ### Is it safe to run chaos experiments in production? It is safe enough when the blast radius is capped in configuration, an alarm-backed stop condition can halt the experiment automatically, and someone has rehearsed the manual abort. AWS FIS stops an experiment when a CloudWatch alarm fires, and Chaos Mesh limits targets with modes like `fixed-percent`. Without working observability and an incident process, it is not safe. ## The call we'd make Run one production game day per quarter, during business hours, starting with the database failover and AZ-loss scenarios, because those expose the application-side gaps no staging environment will. Cap the blast radius in config, wire the abort to an alarm, and treat the follow-up tickets as the actual deliverable. If you can't commit to fixing what the drill finds, don't run it yet; spend that quarter on the [error budget process](/blog/error-budgets-to-roadmap-prioritization) that will justify the fixes. ## Sources - [AWS FIS User Guide: Actions reference](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html) - [AWS FIS User Guide: Stop conditions](https://docs.aws.amazon.com/fis/latest/userguide/stop-conditions.html) - [Chaos Mesh docs: Simulate network faults on Kubernetes](https://chaos-mesh.org/docs/simulate-network-chaos-on-kubernetes/) - [Principles of Chaos Engineering](https://principlesofchaos.org/)