Kubernetes FailedScheduling — Taints, Affinity, and Resources

A FailedScheduling event means the scheduler did its job and came up empty. It walked every node in the cluster, ran each one through a set of predicates, and not a single node passed all of them. The pod stays Pending. Nothing is broken; the cluster is telling you, in one dense line, that what you asked for and what you have don't line up.

The mistake I see people make is scaling the node group the moment they see Pending. Sometimes that's right. Often it's not, and you've just paid for capacity that won't fix a taint or an affinity typo. Read the event first.

Read the event, not the vibes

kubectl describe pod api-7f9c8-2xk4p

Scroll to the bottom:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  47s   default-scheduler  0/5 nodes are available: 3 Insufficient cpu, 2 node(s) had untolerated taint {dedicated: gpu}. preemption: 0/5 nodes are available: 3 No preemption victims found, 2 Preemption is not helpful for scheduling.

That message is a manifest. 0/5 nodes are available is the headline. The comma-separated list is a tally of why each node failed, and the counts add up to your node total. Here: three nodes had no CPU room, two carried a taint the pod didn't tolerate. Five nodes, zero fits.

Learn to parse each phrase. Below are the ones you'll actually meet.

Insufficient cpu / Insufficient memory

0/5 nodes are available: 5 Insufficient cpu.

The pod's total CPU requests (not limits, requests) exceed what's free on every node. Free means allocatable minus what's already reserved by other pods, whether or not they're using it. A node can sit at 10% real utilization and still reject a pod because everything on it over-requested.

Two fixes. If the request is honest, add capacity. If it's inflated — and a lot of resources.requests are copy-pasted guesses — lower it:

resources:
  requests:
    cpu: "250m"      # was "2" because nobody measured
    memory: "256Mi"

Check what's actually free before you decide:

kubectl describe node ip-10-0-1-14 | grep -A5 "Allocated resources"

This is the same territory as a pod Pending that never gets placed — resource math is the most common root cause.

Untolerated taint

2 node(s) had untolerated taint {dedicated: gpu: NoSchedule}.

Someone tainted those nodes to keep general workloads off them. Your pod isn't allowed there unless it carries a matching toleration. If this pod should run there:

tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "gpu"
  effect: "NoSchedule"

If it shouldn't, the event is doing its job — those nodes are reserved and the pod belongs elsewhere. A toleration only grants permission; it doesn't pull the pod onto the node. That's affinity's job.

Node affinity / nodeSelector didn't match

0/5 nodes are available: 5 node(s) didn't match Pod's node affinity/selector.

You asked for nodes with a label none of them carry. Ninety percent of the time it's a typo or a zone that doesn't exist. Print the labels:

kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type

Then reconcile your selector with reality:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["m6i.large", "m6i.xlarge"]

Use required when the pod genuinely cannot run elsewhere. Reach for preferredDuringScheduling when it's a nudge, not a law — required plus a label typo is how you strand a pod forever.

Pod affinity / anti-affinity

0/5 nodes are available: 5 node(s) didn't satisfy existing pods anti-affinity rules.

Common with a StatefulSet that spreads replicas one-per-node using requiredDuringScheduling anti-affinity. Ask for four replicas on a three-node cluster and the fourth has nowhere legal to sit. Either add a node or loosen the rule to preferred:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchLabels:
            app: cache
        topologyKey: kubernetes.io/hostname

preferred still spreads pods when it can, but it won't wedge the deploy when it can't.

Volume zone conflict

0/3 nodes are available: 3 node(s) had volume node affinity conflict.

This one trips everybody. An EBS volume lives in one availability zone. A pod bound to that PVC can only schedule onto nodes in the same zone. If your only free capacity is in us-east-1b and the volume is in us-east-1a, no dice. Check the PV:

kubectl get pv <name> -o jsonpath='{.spec.nodeAffinity}'

The durable fix is a WaitForFirstConsumer StorageClass so the volume is provisioned after the scheduler picks a zone, not before:

volumeBindingMode: WaitForFirstConsumer

Unschedulable / cordoned nodes

0/5 nodes are available: 2 node(s) were unschedulable, 3 Insufficient cpu.

unschedulable means someone ran kubectl cordon (or a drain in progress) and those nodes won't accept new pods. Sometimes that's a forgotten cordon from last week's maintenance. Confirm and uncordon:

kubectl get nodes | grep SchedulingDisabled
kubectl uncordon ip-10-0-1-22

Topology spread constraints

0/6 nodes are available: 3 node(s) didn't match pod topology spread constraints.

You've asked the scheduler to balance replicas across zones or nodes within a maxSkew, and honoring it would break the limit. With whenUnsatisfiable: DoNotSchedule, the pod waits. If even spread matters less than getting the pod running, switch to ScheduleAnyway:

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: ScheduleAnyway
  labelSelector:
    matchLabels:
      app: api

The call we'd make

Read the tally before you touch anything. The count after each phrase tells you how many nodes failed for that reason, and that alone usually points at the fix: taints and affinity are almost always a manifest problem you solve in the YAML, while Insufficient cpu split evenly across every node is the one case where scaling is honest. Fix inflated requests before you add nodes — measured requests fix more Pending pods than a bigger cluster ever will. And keep required rules for things that are genuinely required; everything softer belongs in preferred, where a bad label can't strand a workload. For the wider map of failure modes, our Kubernetes troubleshooting guide connects the events that tend to show up together.