# Karpenter: Node Provisioning Patterns at Scale URL: https://www.devopsness.com/blog/karpenter-node-provisioning-patterns-at-scale Author: Kiril Urbonas Published: 2026-05-03 Updated: 2026-10-31 Categories: Cloud Tags: Karpenter, Kubernetes, AWS, EKS, Autoscaling Summary: After two years of running Karpenter on production EKS clusters, the NodePool patterns that survived, the ones we replaced, and the tuning that matters. --- # Karpenter: Node Provisioning Patterns at Scale Use three shared NodePools (`system`, `web`, `batch`), allow spot and on-demand in the same pool so Karpenter falls back on its own, and give spot workloads PodDisruptionBudgets plus longer grace periods. That layout replaced our Cluster Autoscaler node groups about two years ago and survived several rounds of iteration. Per-team pools did not. > [!VERIFIED] > Karpenter 1.15 (v1.15.1, released 2026-10-23), `karpenter.sh/v1` and `karpenter.k8s.aws/v1` APIs, against the karpenter.sh NodePool, NodeClass and Disruption docs and the AWS EC2 spot interruption notice docs. Checked October 2026. ## What Karpenter actually does Karpenter watches for unschedulable pods, decides what kind of node would fit them, and provisions one (or several) directly through the cloud provider's API. It also handles the reverse: when nodes are underutilized, it consolidates pods onto fewer nodes and terminates the surplus. Compared to Cluster Autoscaler, Karpenter is faster (no fixed node groups, decisions in seconds), more flexible (the right instance type per pod), and simpler to operate (one controller, not a maze of ASGs). The tradeoff is more responsibility on the team. Karpenter's defaults work for simple workloads; production usually needs tuning. ## Our NodePool layout Three pools. Each is purpose-built; we tried fewer and more and landed on three. **`system`:** for cluster add-ons (Argo CD, monitoring agents, CoreDNS and so on). On-demand nodes only, no spot. Stable resource shapes, small types (`m6i.large`-ish). Taints keep application pods off it. **`web`:** for stateless application pods. Mixed instance types (Karpenter picks the cheapest fit from a wide list), mostly spot. Application pods tolerate this pool by default. **`batch`:** for batch jobs, ML preprocessing, anything that can survive interruption. Spot only, wide instance variety. Different tolerations so only batch workloads land here. We tried one NodePool per service early on, with every team owning its own. It was a maintenance disaster. Three shared pools with clear differentiators (interruption tolerance, workload type) is the right granularity. ## Configuring `web` for spot plus stability The pool that took the most iteration: ```yaml # karpenter/nodepool-web.yaml apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: web spec: template: spec: requirements: - key: karpenter.sh/capacity-type operator: In values: ["spot", "on-demand"] - key: kubernetes.io/arch operator: In values: ["amd64", "arm64"] - key: node.kubernetes.io/instance-type operator: NotIn values: ["t3.nano", "t3.micro"] # too small to be useful nodeClassRef: group: karpenter.k8s.aws kind: EC2NodeClass name: default limits: cpu: "2000" memory: 8000Gi disruption: consolidationPolicy: WhenEmptyOrUnderutilized consolidateAfter: 1m budgets: - nodes: "10%" ``` Key choices: - **Spot and on-demand both allowed.** Karpenter's docs say it prioritizes reserved capacity, then spot, then on-demand, and only falls back to types the pool allows. The fallback is what makes spot reliable at scale: when spot capacity is unavailable in an instance family or AZ, on-demand fills the gap automatically. - **AMD64 and ARM64 both allowed.** Karpenter picks the cheapest fit, and Graviton instances are usually cheaper per unit of work. Apps without ARM compatibility issues benefit automatically; apps with them are pinned via affinity. Our [Graviton migration notes](/blog/aws-graviton-migration-what-broke-what-we-saved) cover what broke along the way. - **`consolidateAfter: 1m`.** A node must go a full minute without a pod being added or removed before Karpenter considers it for consolidation (the default is `0s`). Aggressive enough to save money, not so aggressive it thrashes during normal churn. - **`budgets: 10% nodes`.** At most 10% of nodes in this pool can be disrupted at once, which is also Karpenter's default when no budget is set. We write it explicitly anyway. ## Spot interruption handling AWS gives a two-minute warning before reclaiming a spot instance. Karpenter receives it through an SQS queue fed by EventBridge rules (set with `--interruption-queue`), taints and drains the node, and provisions a replacement in parallel. By the time the reclaim happens, the pods have ideally rescheduled elsewhere. Two minutes is not always enough. Things we changed: - **Pod `terminationGracePeriodSeconds`.** Many of our pods had the 30-second default. For pods doing real work (database connections, in-flight requests), we raised it to 60-120 seconds, which gives a pod time to drain connections before it is killed. - **PodDisruptionBudgets for critical services.** PDBs stop Karpenter from evicting too many pods of the same Deployment at once, which prevents the case where five replicas of one service all sit on the spot node that just got interrupted. With these mitigations, spot interruptions on our busiest days rarely turn into customer-visible incidents. If you are still deciding how much spot to run at all, our [spot instances guide](/blog/spot-instances-guide) covers the economics. ## When not to use spot Two cases we keep off spot: - **Stateful services without HA.** A self-hosted Redis without sentinel or cluster mode: losing the spot node means losing the cache, which often cascades. - **Services with long-running connections.** WebSocket servers, gRPC streaming services. A two-minute warning isn't enough to drain thousands of clients gracefully. Both go on the `system` (on-demand) pool with explicit selectors. ## EC2NodeClass: the AWS-specific part The NodePool says "what kind of node"; the EC2NodeClass says "what does that node look like." ```yaml # karpenter/ec2nodeclass-default.yaml apiVersion: karpenter.k8s.aws/v1 kind: EC2NodeClass metadata: name: default spec: amiSelectorTerms: - alias: bottlerocket@v1.20.4 # example pin from the docs; @latest is not recommended in production role: KarpenterNodeRole subnetSelectorTerms: - tags: karpenter.sh/discovery: "main-cluster" securityGroupSelectorTerms: - tags: karpenter.sh/discovery: "main-cluster" blockDeviceMappings: - deviceName: /dev/xvdb ebs: volumeSize: 100Gi volumeType: gp3 encrypted: true ``` A few choices that matter: - **Bottlerocket via `amiSelectorTerms`.** In the v1 API, `amiSelectorTerms` is required, and an `alias` sets the AMI family, so `amiFamily` can be dropped. Bottlerocket is AWS's minimal, container-optimized OS, with a smaller attack surface than full Amazon Linux. Pin the version: the docs warn against `@latest` in production. - **Subnet and security group discovery via tags.** Karpenter picks them by tag, so we don't hardcode IDs. We tag everything with `karpenter.sh/discovery: ` in Terraform. - **EBS encryption and gp3.** On Bottlerocket, `/dev/xvdb` is the data volume that holds images and logs, so that is the one we enlarge. Always encrypted. ## Cost: what changed Before Karpenter, we ran Cluster Autoscaler with fixed node groups: `m5.4xlarge` for general compute, `m5.large` for system pods. Lots of partial-fit nodes and low effective utilization. After Karpenter, utilization rose noticeably because instance sizes track pod shapes, most compute moved to spot, and a meaningful share of the fleet moved to ARM. The compute bill dropped substantially for the same workload over the following year. We won't quote precise percentages, because they depend on your starting point. ## What still bites us Two ongoing pain points: **Pod-to-node-startup race.** When Karpenter provisions a new node for a pending pod, the pod sometimes lands on an existing node that freed up in the meantime. The new node sits idle until consolidation reclaims it. Wasted spend, but small. **Mixed-arch in the same Deployment.** A Deployment with `replicas: 10` might end up with 6 pods on AMD64 nodes and 4 on ARM64. Both work, but it confuses perf analysis, since per-pod metrics across two CPU shapes are not directly comparable. For perf-sensitive services we pin to one arch with node selectors. ## Migration from Cluster Autoscaler Worth it but not trivial. The path that worked for us: 1. Install Karpenter alongside Cluster Autoscaler, both running. 2. Create one NodePool for low-stakes workloads (batch jobs, for example). 3. Migrate one workload to the new pool, observe. 4. Migrate more, expand NodePool variety. 5. Gradually drain and remove Cluster Autoscaler node groups. It took about two months for our fleet. Dual-running is ugly but lets you migrate incrementally. ## Questions people ask ### How many NodePools should a Karpenter cluster have? For most teams, three: an on-demand `system` pool for add-ons, a mixed `web` pool for stateless apps, and a spot-only `batch` pool for interruptible work. Split by interruption tolerance and workload type, not by team. Per-team pools multiply config to review and upgrade, and they fragment capacity so consolidation has fewer places to pack pods. ### Does Karpenter fall back to on-demand when spot is unavailable? Yes, if the NodePool allows it. Karpenter prioritizes reserved capacity, then spot, then on-demand, and only launches capacity types listed in the pool's `karpenter.sh/capacity-type` requirement. A pool that allows only `spot` will never launch on-demand, so put both values in one pool when you want automatic fallback. ### What does consolidateAfter do in Karpenter? It sets how long a node must stay stable, with no pods added or removed, before Karpenter considers it for consolidation. The timer resets on every pod change. The default is `0s`; `Never` disables consolidation for that pool. A minute or two suits most web workloads, while churny batch pools often benefit from a longer value. ## What I'd tell a team starting **Three NodePools is enough for most teams.** System, web, batch. Resist the per-team-pool urge. **Spot plus on-demand fallback in the same pool.** Don't make separate pools by capacity type. **Tune `consolidateAfter` to your tolerance.** Faster consolidation means more savings and more churn. One minute works for us. **PDBs and termination grace periods matter on spot.** Otherwise spot interruptions become incidents. **Watch interruption rates.** The AWS Spot Instance Advisor publishes interruption frequency by instance type, and some families are far more interruption-prone than others. We exclude the worst offenders from our pools. **Karpenter is one operator; treat it like any other.** Monitor it, alert on its health, upgrade it on the same cadence as other operators. Karpenter is one of the highest-return Kubernetes operational moves we've made. The patterns above aren't exotic; they're just what survived a few iterations. ## Sources - [Karpenter NodePools documentation](https://karpenter.sh/docs/concepts/nodepools/) - [Karpenter Disruption documentation](https://karpenter.sh/docs/concepts/disruption/) - [Karpenter NodeClasses documentation](https://karpenter.sh/docs/concepts/nodeclasses/) - [AWS EC2 Spot Instance interruption notices](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html)