A Pending pod means the scheduler couldn't place it or the node couldn't start it. Here's how to read the events and clear the eight causes we hit most.
Why Your Kubernetes Pod Is Stuck in Pending (and How to Fix It)
A pod in Pending is a pod that Kubernetes has accepted but not run. That's the whole definition, and it splits into two very different situations. Either the scheduler looked at every node and refused to place the pod anywhere, or the scheduler already picked a node and the kubelet on that node hasn't been able to start the containers yet. Those are separate problems with separate fixes, and the first thing you do is figure out which one you're looking at.
The good news: the pod already told you why. People burn an hour guessing at cluster capacity when the answer is sitting in the pod's event log. So before anything else, describe it.
kubectl describe pod checkout-api-7d9f8b6c4-x2ktp
Scroll to the bottom, to Events. The scheduler writes a FailedScheduling event with a per-node breakdown, and that breakdown names the cause. Here's a real one from a cluster that ran out of room:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m14s default-scheduler 0/6 nodes are available: 3 Insufficient cpu, 3 Insufficient memory. preemption: 0/6 nodes are available: 6 No preemption victims found.
Read that literally. Six nodes, none usable, and it tells you the split: three couldn't fit the CPU request, three couldn't fit the memory request. If the Events section is empty and the pod already shows a node assignment (Node: ip-10-0-3-14), you're in the second situation. That's a kubelet problem, image pull, a volume that won't attach, and you go look at that node instead.
Below are the causes we actually hit, roughly in order of frequency.
The example above. Every node has less allocatable CPU or memory than your pod requests. You have three fixes and you should pick the right one, not the loud one.
Lower the request if it's inflated. Half the "we need bigger nodes" incidents I've seen were a pod asking for 4 CPU when it uses 400m. Check actual usage with kubectl top pod before you scale anything.
resources:
requests:
cpu: 500m
memory: 512Mi
If the request is honest and the cluster is genuinely full, add capacity. With Cluster Autoscaler or Karpenter installed, a Pending pod that can't fit is the trigger, a new node appears in a minute or two. Without an autoscaler, you're scaling the node group by hand. Our take: run an autoscaler. A pod stuck Pending because a human hasn't noticed the node group is full is an avoidable page.
Related but nastier: no node is big enough, full stop. A request of memory: 64Gi on a fleet of 32Gi nodes will never schedule, and no amount of autoscaling the count fixes it because scaling adds more of the same small node. The event looks identical to plain insufficiency. The tell is that scaling out doesn't help. Fix the request or add a node type that can hold it.
Nodes get tainted to repel pods, GPU nodes, spot nodes, control-plane nodes. If your pod has no matching toleration, the scheduler skips those nodes:
0/4 nodes are available: 4 node(s) had untolerated taint {dedicated: gpu}.
Either add the toleration or stop targeting tainted nodes:
tolerations:
- key: dedicated
operator: Equal
value: gpu
effect: NoSchedule
A nodeSelector or nodeAffinity rule that no node satisfies produces:
0/8 nodes are available: 8 node(s) didn't match Pod's node affinity/selector.
Usually a typo or a stale label. You asked for disktype: ssd and the nodes are labeled disk-type: ssd, or the label got dropped in a node-pool rebuild. Check kubectl get nodes --show-labels and reconcile.
Anti-affinity that requires one replica per node runs out of nodes once replicas equal nodes. The message reads node(s) didn't match pod anti-affinity rules. This one bites during scale-ups: you bump replicas past your node count and the extras sit Pending forever. Either add nodes or loosen requiredDuringScheduling to preferred.
A pod that mounts a PVC won't schedule until that claim binds. Describe the PVC, not just the pod:
Warning ProvisioningFailed storageclass.storage.k8s.io "fast-ssd" not found
Common variants: the StorageClass name is wrong, there's no default StorageClass and the PVC didn't name one, or the volume's zone is pinned and the schedulable nodes are in a different zone. That last one is sneaky on regional clusters. The volume lives in us-east-1a, every node with room is in 1b, and neither the pod nor the PVC event spells it out plainly.
If the namespace has a ResourceQuota and this pod would push it over, the pod is rejected or held. You'll see it referenced in the event or when you inspect the quota:
kubectl describe resourcequota -n payments
Resource Used Hard
-------- ---- ----
requests.cpu 19500m 20
requests.memory 38Gi 40Gi
The cluster has room, the namespace doesn't. Raise the quota or free it up by cleaning out dead workloads. This one confuses people because every node looks healthy, capacity is a namespace-level accounting limit here, not a hardware one.
Sometimes the humble answer is right: there are zero schedulable nodes. A node group scaled to zero, every node cordoned during maintenance, or the whole pool NotReady after a bad kubelet rollout. kubectl get nodes tells you in one line. If nodes are NotReady, the pod problem is a symptom and you go fix the nodes.
Describe first, always. The scheduler's FailedScheduling message is not vague, it enumerates every node and the exact reason each was rejected, and nine times out of ten the fix is written into that text. Guessing at capacity before reading the events is how a five-minute fix turns into an afternoon. Build the reflex: pod stuck, kubectl describe, read the last event, act on what it literally says. If Pending shows up a lot under load, that's your cue to run an autoscaler and set honest resource requests, not to keep hand-scaling node groups at 2am. For the wider picture, our Kubernetes troubleshooting guide covers the failure modes beyond scheduling.
