Running Production on Spot Instances Safely

Spot instances are spare EC2 capacity that AWS rents out at a steep discount, usually 60-90% off on-demand. The catch is the whole deal: AWS can reclaim the machine whenever it needs the capacity back, and you get exactly two minutes of warning before it's gone. For years that reputation kept spot pinned to "batch jobs only." We run a good chunk of our production fleet on it now, and the fleet is quieter than the on-demand version was. The difference is entirely in how you set it up.

What actually belongs on spot#

The question isn't "is this workload important." Everything in production is important. The question is "what happens when this specific instance disappears with two minutes notice."

Safe on spot:

• Stateless services. Web tiers, API servers, anything behind a load balancer that can lose a node and reschedule the pod elsewhere. If a replacement instance serves the same request, spot is fine.
• Batch and async work. ETL, video encoding, report generation. Worst case a job restarts. These are the textbook fit.
• CI runners. Build agents are ephemeral by design. Losing one mid-build costs you a retry, not data.
• Kubernetes worker nodes. Pods are already designed to move. This is where spot pays off hardest.

Not safe on spot:

• Stateful primaries. Your database writer, the leader of a consensus group, a single-replica cache holding warm state. Anything where losing the node means losing data or triggering a failover you didn't schedule. Keep these on on-demand or reserved capacity.
• Anything with a long, uninterruptible startup. If a node takes eight minutes to become useful, a two-minute eviction notice is a bad trade.

The honest rule we use: if you can't answer "the workload reschedules cleanly" without a pause, it doesn't go on spot yet.

Handling the two-minute notice#

The two-minute interruption notice is the entire game. AWS publishes it two ways: the instance metadata endpoint exposes a spot/instance-action field, and there's a CloudWatch/EventBridge EC2 Spot Instance Interruption Warning event. Something in your stack has to watch for it and react.

React means three things, in order:

1. Stop taking new work. Deregister the node from the load balancer, cordon it in Kubernetes, pull it out of the queue consumer group.
2. Drain in-flight work. Let existing connections finish. Connection draining on the load balancer plus a sane pod termination grace period covers most of this.
3. Checkpoint if you have to. For longer batch jobs, write progress somewhere durable so the restart picks up instead of starting over.

Two minutes is enough time for a graceful shutdown and not much else. Design for it and it's a non-event. Ignore it and every eviction becomes a spike of 502s.

Diversify or suffer#

The single biggest lever on interruption frequency is diversification, and most teams get it wrong by asking for one instance type in one AZ. That's the worst possible spot request: you're competing for the narrowest possible pool, and when that pool tightens, everything you have goes at once.

Spread across many instance types and every AZ in the region. m6i.large, m6a.large, m5.large, m5a.large, m5n.large are near-interchangeable for a stateless service. Each is a separate capacity pool. If you'll accept fifteen instance types across three AZs, AWS is drawing from forty-five pools, and a shortage in one barely registers.

Then set the allocation strategy to capacity-optimized (or price-capacity-optimized, which we prefer). Do not use the old lowest-price strategy for anything you care about. Lowest-price chases the cheapest pool, which is cheap precisely because it's about to run dry, so it hands you the highest interruption rate. Capacity-optimized places instances in the deepest pools, trading a few cents for far fewer evictions. The few cents are worth it every time.

Kubernetes on spot#

This is where we run most of our spot capacity, because Kubernetes already assumes nodes come and go. A few pieces have to be in place.

Node provisioning. We use Karpenter. It watches for unschedulable pods and launches the cheapest node that fits, and it's genuinely good at spot diversification if you give it room. Cluster Autoscaler works too, but Karpenter's flexibility across instance types maps naturally onto how spot wants to be used.

The termination handler. You need something that catches the interruption notice and cordons plus drains the node before AWS pulls it. Karpenter handles interruptions natively when you wire up the SQS/EventBridge queue; on Cluster Autoscaler setups, run the AWS Node Termination Handler as a DaemonSet.

PodDisruptionBudgets. These stop a drain from taking down too many replicas of one service at once. Without a PDB, a wave of evictions plus a rolling deploy can briefly leave you with zero healthy pods.

A Karpenter NodePool that captures the shape:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-general
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["m", "c", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
  limits:
    cpu: "1000"

And a PodDisruptionBudget so drains stay polite:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 75%
  selector:
    matchLabels:
      app: api

The wide instance-category and Gt: "5" generation filter are doing the diversification work. Don't pin instance types by hand here; let Karpenter draw from the whole menu.

Mixed on-demand and spot#

Pure spot is a bad bet for the baseline you can never drop below. Run a floor of on-demand or reserved capacity that keeps the service alive if spot dries up entirely, and put the elastic layer on top of it on spot. In Karpenter, a second NodePool for on-demand with a higher weight, plus pod topology spread and a PDB, gets you there. We land around 70-80% spot on stateless tiers and sleep fine.

This pairs with the rest of our cloud cost optimization work; spot is the highest-leverage single move on the compute line, but only after the diversification and draining are real.

The call we'd make#

Put every stateless service and every Kubernetes worker on spot, diversified across a dozen-plus instance types with price-capacity-optimized, behind a proper termination handler and PDBs, on an on-demand floor you can't fall through. Keep stateful primaries off it. Do that and spot stops being a gamble and starts being the default, with the on-demand bill as the thing you have to justify.