Chaos Engineering: What We Actually Run as Game Days

A chaos game day is a scheduled exercise where a team injects one controlled failure into production, responds as if it were real, then fixes what broke. Our format: one scenario per quarter, a few hours on a weekday, a written blast-radius cap, and an abort condition wired to an alarm. The fixes are the payoff, not the drill.

What a game day is, in our flavor

Five to ten people on a call, one scenario picked in advance from a backlog of "things we should test", and an explicit blast-radius limit. Someone injects the failure; on-call responds with the real runbooks and real dashboards. Afterward, a debrief: what worked, what didn't, what changes, and who owns each change.

It is neither continuous chaos (Netflix's Chaos Monkey model) nor a full disaster-recovery drill. It follows the loop from the Principles of Chaos Engineering: define steady state as a measurable output (error rate, latency, throughput), predict it will hold, inject a realistic fault, and try to disprove the prediction.

The scenarios that tend to find real problems

These are the five we'd put at the top of any backlog, because each exposes a class of problem that dev and staging hide.

Fail over the primary database. FIS has aws:rds:reboot-db-instances with forceFailover: true for Multi-AZ instances, and aws:rds:failover-db-cluster for Aurora. The database usually recovers on schedule. What typically fails is the application: connection pools holding dead connections, no reconnect logic, a DNS TTL cached too long. Measure time to recovery from the app's side, not from the RDS event.

Lose one Availability Zone's worth of nodes. Stateless multi-AZ deployments shrug this off. What it exposes is the single-replica thing nobody remembers: an in-cluster database chart without HA, a single-AZ NAT gateway, a PodDisruptionBudget that blocks rescheduling.

Saturate outbound network from one workload. Shared egress (NAT gateways, proxies) is a shared fate. One noisy pod can starve unrelated services of outbound calls.

Add latency to a dependency. Use enough to matter: 100 ms often hides inside normal jitter, while 500 ms to 1 s exposes oversized timeouts, missing circuit breakers and request handlers piling up until a pod falls over.

Kill the telemetry pipeline. Stop the OpenTelemetry collectors and see whether traces buffer or vanish, and whether anyone notices. Losing visibility during an incident is its own incident.

The scenarios that rarely find much

Negative results are still information, so log them:

• Killing one app pod. Kubernetes replaces it. Good as a first drill to calibrate the team, not as a finding.
• Small latency on one downstream. Within jitter. Escalate the number or drop the scenario.
• Rehearsed DNS-based region failover. If it already works twice in a row, keep it as practice, not discovery.
• Killing a cache primary with working Sentinel or replication. A few seconds of failed reads, then recovery. Check it once, then move on.

The tooling, with configs you can adapt

AWS FIS for cloud-level faults. Stop conditions are CloudWatch alarms; when one fires, FIS stops the experiment and it cannot be resumed. This template, adapted from the AWS example, stops tagged worker instances in one AZ and restarts them after ten minutes:

{
  "description": "Game day: stop prod workers in us-east-1a, restart after 10m",
  "targets": {
    "azWorkers": {
      "resourceType": "aws:ec2:instance",
      "resourceTags": { "role": "k8s-worker", "env": "prod" },
      "filters": [
        { "path": "Placement.AvailabilityZone", "values": ["us-east-1a"] },
        { "path": "State.Name", "values": ["running"] }
      ],
      "selectionMode": "PERCENT(50)"
    }
  },
  "actions": {
    "stopWorkers": {
      "actionId": "aws:ec2:stop-instances",
      "parameters": { "startInstancesAfterDuration": "PT10M" },
      "targets": { "Instances": "azWorkers" }
    }
  },
  "stopConditions": [
    { "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:us-east-1:111122223333:alarm:api-5xx-over-1pct" }
  ],
  "roleArn": "arn:aws:iam::111122223333:role/fis-gameday"
}

(Strip the first comment line before passing it to the CLI; JSON has no comments.) For EKS managed node groups, aws:eks:terminate-nodegroup-instances takes instanceTerminationPercentage (1-100) instead.

aws fis create-experiment-template --cli-input-json file://fis/az-a-node-loss.json
aws fis start-experiment --experiment-template-id EXT1a2b3c4d5e6f7
aws fis stop-experiment --id EXP1a2b3c4d5e6f7   # manual abort

Chaos Mesh (a CNCF incubating project) for pod-level faults: kill, network delay or partition, IO faults, stress. The delay experiment, with mode capping the blast radius:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payments-db-latency
  namespace: payments
spec:
  action: delay
  mode: fixed-percent
  value: "30"
  selector:
    namespaces: [payments]
    labelSelectors:
      app: payments-api
  delay:
    latency: "500ms"
    jitter: "50ms"
    correlation: "100"
  direction: to
  duration: "10m"

kubectl apply -f chaos/payments-db-latency.yaml
kubectl delete -f chaos/payments-db-latency.yaml   # abort, removes the fault

Modes are one, all, fixed, fixed-percent and random-max-percent. Never use all on a first run.

Safety discipline

A one-page plan per drill. What we'll do, when, expected impact, abort criteria, comms, roles. Signed off by the platform lead. The value is the conversation it forces.

Comms 24 hours ahead. Post in the channels of every team whose services might be affected. They don't act; they just don't panic at odd graphs.

Blast-radius caps in config, not in intent. PERCENT(50) in FIS, fixed-percent in Chaos Mesh. The cap bounds the worst case when the drill goes wrong.

Abort criteria written in advance. "API 5xx rate above 1% for 60 seconds" becomes the CloudWatch alarm FIS watches. Without a number, abort becomes an argument mid-incident.

Follow-ups tracked to closure. Each finding gets an owner and a ticket, reviewed at the next drill. Unfixed findings teach the team that drills don't matter. Run the debrief like any post-mortem that drives change.

Failure modes when running game days

• The experiment goes to stopped seconds after start-experiment. A stop condition triggered: the alarm was already in ALARM state, or it watches a metric that is noisy at baseline. Fix the alarm before the drill.
• FIS action fails with an IAM access denied error. The experiment role is missing the action's permissions, e.g. ec2:StopInstances or rds:RebootDBInstance; each action in the reference lists what it needs.
• aws:eks:terminate-nodegroup-instances targets nothing. It supports only EKS managed node groups, not self-managed ones.
• Chaos Mesh delay has no effect. The selector matched no pods, or the network action is blocked by the pod's security policy. kubectl describe networkchaos <name> shows the targeted pods and injection status.
• The fault outlives the drill. Always set duration (Chaos Mesh) or a restart parameter (FIS), and rehearse the abort command before you start.

What we don't do

Continuous chaos in production. At our size, a 24/7 random killer trains people to dismiss alerts as drill noise.

Chaos only in staging. Staging builds confidence; production-shaped traffic finds the real issues.

3 AM drills. Nobody can respond at 3 AM either, which defeats the point. Weekday, mid-morning to early afternoon, full staffing.

Where this breaks

Skip game days until three things exist: observability you trust, an incident process the team already uses, and leadership that will fund the fixes. Without them a drill adds stress and produces findings nobody acts on. Regulated workloads may also forbid deliberate production faults; there, run the same scenarios against a production-scale replica and say so plainly in the report.

Questions people ask

What is a game day in chaos engineering?

A game day is a planned session where a team deliberately injects a failure, such as a database failover or losing an Availability Zone, into a real system and responds as if it were an incident. It has a written plan, a blast-radius limit, abort criteria and a debrief. The goal is to find weaknesses in systems and in the human response before a real outage does.

How often should you run chaos game days?

Quarterly is enough for most platform teams. Each drill needs planning time beforehand and engineering time afterward to fix what it found, and running more often than you can fix just piles up findings. Start with low-risk scenarios like killing a single pod, then a node, then an AZ, and only automate continuous experiments once the quarterly findings dry up.

Is it safe to run chaos experiments in production?

It is safe enough when the blast radius is capped in configuration, an alarm-backed stop condition can halt the experiment automatically, and someone has rehearsed the manual abort. AWS FIS stops an experiment when a CloudWatch alarm fires, and Chaos Mesh limits targets with modes like fixed-percent. Without working observability and an incident process, it is not safe.

The call we'd make

Run one production game day per quarter, during business hours, starting with the database failover and AZ-loss scenarios, because those expose the application-side gaps no staging environment will. Cap the blast radius in config, wire the abort to an alarm, and treat the follow-up tickets as the actual deliverable. If you can't commit to fixing what the drill finds, don't run it yet; spend that quarter on the error budget process that will justify the fixes.

Sources

• AWS FIS User Guide: Actions reference
• AWS FIS User Guide: Stop conditions
• Chaos Mesh docs: Simulate network faults on Kubernetes
• Principles of Chaos Engineering