Karpenter: Node Provisioning Patterns at Scale

Use three shared NodePools (system, web, batch), allow spot and on-demand in the same pool so Karpenter falls back on its own, and give spot workloads PodDisruptionBudgets plus longer grace periods. That layout replaced our Cluster Autoscaler node groups about two years ago and survived several rounds of iteration. Per-team pools did not.

What Karpenter actually does

Karpenter watches for unschedulable pods, decides what kind of node would fit them, and provisions one (or several) directly through the cloud provider's API. It also handles the reverse: when nodes are underutilized, it consolidates pods onto fewer nodes and terminates the surplus.

Compared to Cluster Autoscaler, Karpenter is faster (no fixed node groups, decisions in seconds), more flexible (the right instance type per pod), and simpler to operate (one controller, not a maze of ASGs).

The tradeoff is more responsibility on the team. Karpenter's defaults work for simple workloads; production usually needs tuning.

Our NodePool layout

Three pools. Each is purpose-built; we tried fewer and more and landed on three.

system: for cluster add-ons (Argo CD, monitoring agents, CoreDNS and so on). On-demand nodes only, no spot. Stable resource shapes, small types (m6i.large-ish). Taints keep application pods off it.

web: for stateless application pods. Mixed instance types (Karpenter picks the cheapest fit from a wide list), mostly spot. Application pods tolerate this pool by default.

batch: for batch jobs, ML preprocessing, anything that can survive interruption. Spot only, wide instance variety. Different tolerations so only batch workloads land here.

We tried one NodePool per service early on, with every team owning its own. It was a maintenance disaster. Three shared pools with clear differentiators (interruption tolerance, workload type) is the right granularity.

Configuring web for spot plus stability

The pool that took the most iteration:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: web
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: node.kubernetes.io/instance-type
          operator: NotIn
          values: ["t3.nano", "t3.micro"]   # too small to be useful
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  limits:
    cpu: "2000"
    memory: 8000Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    budgets:
      - nodes: "10%"

Key choices:

• Spot and on-demand both allowed. Karpenter's docs say it prioritizes reserved capacity, then spot, then on-demand, and only falls back to types the pool allows. The fallback is what makes spot reliable at scale: when spot capacity is unavailable in an instance family or AZ, on-demand fills the gap automatically.
• AMD64 and ARM64 both allowed. Karpenter picks the cheapest fit, and Graviton instances are usually cheaper per unit of work. Apps without ARM compatibility issues benefit automatically; apps with them are pinned via affinity. Our Graviton migration notes cover what broke along the way.
• consolidateAfter: 1m. A node must go a full minute without a pod being added or removed before Karpenter considers it for consolidation (the default is 0s). Aggressive enough to save money, not so aggressive it thrashes during normal churn.
• budgets: 10% nodes. At most 10% of nodes in this pool can be disrupted at once, which is also Karpenter's default when no budget is set. We write it explicitly anyway.

Spot interruption handling

AWS gives a two-minute warning before reclaiming a spot instance. Karpenter receives it through an SQS queue fed by EventBridge rules (set with --interruption-queue), taints and drains the node, and provisions a replacement in parallel. By the time the reclaim happens, the pods have ideally rescheduled elsewhere.

Two minutes is not always enough. Things we changed:

• Pod terminationGracePeriodSeconds. Many of our pods had the 30-second default. For pods doing real work (database connections, in-flight requests), we raised it to 60-120 seconds, which gives a pod time to drain connections before it is killed.
• PodDisruptionBudgets for critical services. PDBs stop Karpenter from evicting too many pods of the same Deployment at once, which prevents the case where five replicas of one service all sit on the spot node that just got interrupted.

With these mitigations, spot interruptions on our busiest days rarely turn into customer-visible incidents. If you are still deciding how much spot to run at all, our spot instances guide covers the economics.

When not to use spot

Two cases we keep off spot:

• Stateful services without HA. A self-hosted Redis without sentinel or cluster mode: losing the spot node means losing the cache, which often cascades.
• Services with long-running connections. WebSocket servers, gRPC streaming services. A two-minute warning isn't enough to drain thousands of clients gracefully.

Both go on the system (on-demand) pool with explicit selectors.

EC2NodeClass: the AWS-specific part

The NodePool says "what kind of node"; the EC2NodeClass says "what does that node look like."

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiSelectorTerms:
    - alias: bottlerocket@v1.20.4   # example pin from the docs; @latest is not recommended in production
  role: KarpenterNodeRole
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "main-cluster"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "main-cluster"
  blockDeviceMappings:
    - deviceName: /dev/xvdb
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        encrypted: true

A few choices that matter:

• Bottlerocket via amiSelectorTerms. In the v1 API, amiSelectorTerms is required, and an alias sets the AMI family, so amiFamily can be dropped. Bottlerocket is AWS's minimal, container-optimized OS, with a smaller attack surface than full Amazon Linux. Pin the version: the docs warn against @latest in production.
• Subnet and security group discovery via tags. Karpenter picks them by tag, so we don't hardcode IDs. We tag everything with karpenter.sh/discovery: <cluster-name> in Terraform.
• EBS encryption and gp3. On Bottlerocket, /dev/xvdb is the data volume that holds images and logs, so that is the one we enlarge. Always encrypted.

Cost: what changed

Before Karpenter, we ran Cluster Autoscaler with fixed node groups: m5.4xlarge for general compute, m5.large for system pods. Lots of partial-fit nodes and low effective utilization.

After Karpenter, utilization rose noticeably because instance sizes track pod shapes, most compute moved to spot, and a meaningful share of the fleet moved to ARM. The compute bill dropped substantially for the same workload over the following year. We won't quote precise percentages, because they depend on your starting point.

What still bites us

Two ongoing pain points:

Pod-to-node-startup race. When Karpenter provisions a new node for a pending pod, the pod sometimes lands on an existing node that freed up in the meantime. The new node sits idle until consolidation reclaims it. Wasted spend, but small.

Mixed-arch in the same Deployment. A Deployment with replicas: 10 might end up with 6 pods on AMD64 nodes and 4 on ARM64. Both work, but it confuses perf analysis, since per-pod metrics across two CPU shapes are not directly comparable. For perf-sensitive services we pin to one arch with node selectors.

Migration from Cluster Autoscaler

Worth it but not trivial. The path that worked for us:

1. Install Karpenter alongside Cluster Autoscaler, both running.
2. Create one NodePool for low-stakes workloads (batch jobs, for example).
3. Migrate one workload to the new pool, observe.
4. Migrate more, expand NodePool variety.
5. Gradually drain and remove Cluster Autoscaler node groups.

It took about two months for our fleet. Dual-running is ugly but lets you migrate incrementally.

Questions people ask

How many NodePools should a Karpenter cluster have?

For most teams, three: an on-demand system pool for add-ons, a mixed web pool for stateless apps, and a spot-only batch pool for interruptible work. Split by interruption tolerance and workload type, not by team. Per-team pools multiply config to review and upgrade, and they fragment capacity so consolidation has fewer places to pack pods.

Does Karpenter fall back to on-demand when spot is unavailable?

Yes, if the NodePool allows it. Karpenter prioritizes reserved capacity, then spot, then on-demand, and only launches capacity types listed in the pool's karpenter.sh/capacity-type requirement. A pool that allows only spot will never launch on-demand, so put both values in one pool when you want automatic fallback.

What does consolidateAfter do in Karpenter?

It sets how long a node must stay stable, with no pods added or removed, before Karpenter considers it for consolidation. The timer resets on every pod change. The default is 0s; Never disables consolidation for that pool. A minute or two suits most web workloads, while churny batch pools often benefit from a longer value.

What I'd tell a team starting

Three NodePools is enough for most teams. System, web, batch. Resist the per-team-pool urge.

Spot plus on-demand fallback in the same pool. Don't make separate pools by capacity type.

Tune consolidateAfter to your tolerance. Faster consolidation means more savings and more churn. One minute works for us.

PDBs and termination grace periods matter on spot. Otherwise spot interruptions become incidents.

Watch interruption rates. The AWS Spot Instance Advisor publishes interruption frequency by instance type, and some families are far more interruption-prone than others. We exclude the worst offenders from our pools.

Karpenter is one operator; treat it like any other. Monitor it, alert on its health, upgrade it on the same cadence as other operators.

Karpenter is one of the highest-return Kubernetes operational moves we've made. The patterns above aren't exotic; they're just what survived a few iterations.

Sources

• Karpenter NodePools documentation
• Karpenter Disruption documentation
• Karpenter NodeClasses documentation
• AWS EC2 Spot Instance interruption notices