# Kubernetes FailedScheduling — Taints, Affinity, and Resources URL: https://www.devopsness.com/blog/fix-failedscheduling Author: Kiril Urbonas Published: 2026-07-10 Updated: 2026-07-10 Categories: DevOps Tags: Kubernetes, DevOps, Scheduling, Debugging, Operations Summary: A pod sits Pending and the scheduler quietly logs why it rejected every node. Learn to read that one line and fix the real cause instead of guessing. --- # Kubernetes FailedScheduling — Taints, Affinity, and Resources A `FailedScheduling` event means the scheduler did its job and came up empty. It walked every node in the cluster, ran each one through a set of predicates, and not a single node passed all of them. The pod stays Pending. Nothing is broken; the cluster is telling you, in one dense line, that what you asked for and what you have don't line up. The mistake I see people make is scaling the node group the moment they see Pending. Sometimes that's right. Often it's not, and you've just paid for capacity that won't fix a taint or an affinity typo. Read the event first. ## Read the event, not the vibes ``` kubectl describe pod api-7f9c8-2xk4p ``` Scroll to the bottom: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 47s default-scheduler 0/5 nodes are available: 3 Insufficient cpu, 2 node(s) had untolerated taint {dedicated: gpu}. preemption: 0/5 nodes are available: 3 No preemption victims found, 2 Preemption is not helpful for scheduling. ``` That message is a manifest. `0/5 nodes are available` is the headline. The comma-separated list is a tally of *why each node failed*, and the counts add up to your node total. Here: three nodes had no CPU room, two carried a taint the pod didn't tolerate. Five nodes, zero fits. Learn to parse each phrase. Below are the ones you'll actually meet. ## Insufficient cpu / Insufficient memory ``` 0/5 nodes are available: 5 Insufficient cpu. ``` The pod's total CPU *requests* (not limits, requests) exceed what's free on every node. Free means allocatable minus what's already reserved by other pods, whether or not they're using it. A node can sit at 10% real utilization and still reject a pod because everything on it over-requested. Two fixes. If the request is honest, add capacity. If it's inflated — and a lot of `resources.requests` are copy-pasted guesses — lower it: ```yaml resources: requests: cpu: "250m" # was "2" because nobody measured memory: "256Mi" ``` Check what's actually free before you decide: ``` kubectl describe node ip-10-0-1-14 | grep -A5 "Allocated resources" ``` This is the same territory as a [pod Pending](/blog/fix-pod-pending) that never gets placed — resource math is the most common root cause. ## Untolerated taint ``` 2 node(s) had untolerated taint {dedicated: gpu: NoSchedule}. ``` Someone tainted those nodes to keep general workloads off them. Your pod isn't allowed there unless it carries a matching toleration. If this pod *should* run there: ```yaml tolerations: - key: "dedicated" operator: "Equal" value: "gpu" effect: "NoSchedule" ``` If it shouldn't, the event is doing its job — those nodes are reserved and the pod belongs elsewhere. A toleration only grants permission; it doesn't pull the pod onto the node. That's affinity's job. ## Node affinity / nodeSelector didn't match ``` 0/5 nodes are available: 5 node(s) didn't match Pod's node affinity/selector. ``` You asked for nodes with a label none of them carry. Ninety percent of the time it's a typo or a zone that doesn't exist. Print the labels: ``` kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type ``` Then reconcile your selector with reality: ```yaml affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node.kubernetes.io/instance-type operator: In values: ["m6i.large", "m6i.xlarge"] ``` Use `required` when the pod genuinely cannot run elsewhere. Reach for `preferredDuringScheduling` when it's a nudge, not a law — `required` plus a label typo is how you strand a pod forever. ## Pod affinity / anti-affinity ``` 0/5 nodes are available: 5 node(s) didn't satisfy existing pods anti-affinity rules. ``` Common with a StatefulSet that spreads replicas one-per-node using `requiredDuringScheduling` anti-affinity. Ask for four replicas on a three-node cluster and the fourth has nowhere legal to sit. Either add a node or loosen the rule to `preferred`: ```yaml affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app: cache topologyKey: kubernetes.io/hostname ``` `preferred` still spreads pods when it can, but it won't wedge the deploy when it can't. ## Volume zone conflict ``` 0/3 nodes are available: 3 node(s) had volume node affinity conflict. ``` This one trips everybody. An EBS volume lives in one availability zone. A pod bound to that PVC can only schedule onto nodes in the *same* zone. If your only free capacity is in `us-east-1b` and the volume is in `us-east-1a`, no dice. Check the PV: ``` kubectl get pv -o jsonpath='{.spec.nodeAffinity}' ``` The durable fix is a `WaitForFirstConsumer` StorageClass so the volume is provisioned *after* the scheduler picks a zone, not before: ```yaml volumeBindingMode: WaitForFirstConsumer ``` ## Unschedulable / cordoned nodes ``` 0/5 nodes are available: 2 node(s) were unschedulable, 3 Insufficient cpu. ``` `unschedulable` means someone ran `kubectl cordon` (or a drain in progress) and those nodes won't accept new pods. Sometimes that's a forgotten cordon from last week's maintenance. Confirm and uncordon: ``` kubectl get nodes | grep SchedulingDisabled kubectl uncordon ip-10-0-1-22 ``` ## Topology spread constraints ``` 0/6 nodes are available: 3 node(s) didn't match pod topology spread constraints. ``` You've asked the scheduler to balance replicas across zones or nodes within a `maxSkew`, and honoring it would break the limit. With `whenUnsatisfiable: DoNotSchedule`, the pod waits. If even spread matters less than getting the pod running, switch to `ScheduleAnyway`: ```yaml topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app: api ``` ## The call we'd make Read the tally before you touch anything. The count after each phrase tells you how many nodes failed for that reason, and that alone usually points at the fix: taints and affinity are almost always a manifest problem you solve in the YAML, while `Insufficient cpu` split evenly across every node is the one case where scaling is honest. Fix inflated requests before you add nodes — measured requests fix more Pending pods than a bigger cluster ever will. And keep `required` rules for things that are genuinely required; everything softer belongs in `preferred`, where a bad label can't strand a workload. For the wider map of failure modes, our [Kubernetes troubleshooting](/blog/kubernetes-error-troubleshooting-guide) guide connects the events that tend to show up together.