Kubernetes Node NotReady — Diagnosis and Fix

The page comes in at 3am: a node is NotReady, and half a dozen pods that were happily running on it are now Terminating or already gone. If you've never seen this cold, the instinct is to reboot the node and hope. Don't. Nine times out of ten the node is telling you exactly what's wrong, and rebooting throws that evidence away.

What NotReady actually means#

A node is Ready when its kubelet posts a healthy status to the API server on a heartbeat, roughly every 10 seconds by default. When those heartbeats stop, or the kubelet reports it can't run pods, the node-lifecycle controller marks the node NotReady.

That status is not just cosmetic. After the eviction timeout (--pod-eviction-timeout, historically 5 minutes, now driven by the node.kubernetes.io/not-ready taint and per-pod toleration seconds), the control plane starts evicting the pods off that node so they can reschedule elsewhere. So a NotReady node is a countdown: you have a few minutes before your workloads get shuffled. If the node comes back inside that window, nothing moves. If it doesn't, you get a rescheduling storm on top of whatever broke the node.

The kubelet is the thing reporting health. So most of your debugging happens on the node, not in the cluster.

Start from the control plane#

Before you SSH anywhere, get the cluster's view:

kubectl get nodes -o wide
kubectl describe node ip-10-0-3-14

describe is where the answer usually is. Read the Conditions block:

Type             Status    Reason
MemoryPressure   True      KubeletHasInsufficientMemory
DiskPressure     False     KubeletHasNoDiskPressure
PIDPressure      False     KubeletHasSufficientPID
Ready            False     KubeletNotReady

If a pressure condition is True, you have your lead already. Also check LastHeartbeatTime. If the heartbeat went stale while every condition still reads healthy, the kubelet or the network died mid-report rather than the node running out of a resource. Those are two different investigations.

Get onto the node#

ssh ip-10-0-3-14
systemctl status kubelet
journalctl -u kubelet -n 200 --no-pager

The kubelet logs are blunt. They'll say the config file failed to parse, the container runtime is unreachable, the bootstrap cert expired, or disk is full. Read them before you touch anything.

Then check the runtime, because a healthy kubelet talking to a dead runtime looks a lot like a dead kubelet:

systemctl status containerd
crictl info
crictl ps

If crictl hangs or errors, the runtime is your problem, not the kubelet.

The usual culprits, and what to do#

Kubelet crashed or misconfigured. systemctl status kubelet shows it flapping or dead. If it's a config change that broke it (someone edited /var/lib/kubelet/config.yaml or the systemd drop-in), the journal names the offending field. Fix the config, systemctl daemon-reload, systemctl restart kubelet. If it's OOM-killed, that points you back at memory pressure below.

Container runtime down. containerd wedged or crashed. systemctl restart containerd, then restart the kubelet so it reconnects. If containerd won't start, check its own logs (journalctl -u containerd) for a corrupt state dir or a bad config in /etc/containerd/config.toml.

Disk or memory pressure. DiskPressure=True usually means the image or log volume filled up. Check df -h, and remember the kubelet evicts pods to reclaim space, which can look like the node is "losing" pods for no reason. Clear it: crictl rmi --prune for dangling images, hunt down runaway container logs under /var/log/pods, and check for a wedged log file that never rotated. MemoryPressure means the same story for RAM. Find the hog with top or check for a pod with no memory limit eating the box.

CNI plugin failure. The node is up, the kubelet is up, but it can't reach the API server or set up pod networking. journalctl shows dial timeouts to the control plane. Check the CNI daemonset pod on that node (Calico, Cilium, whichever), look at /etc/cni/net.d, and confirm basic reachability with curl -k https://<apiserver>:6443/healthz. A missing or half-written CNI config file is a classic post-upgrade failure.

Certificate expiry. On self-managed clusters this bites about a year in. The kubelet client cert expired and the API server rejects its heartbeats. journalctl shows x509: certificate has expired. kubeadm certs check-expiration on the control plane, rotate, and make sure kubelet cert rotation (rotateCertificates: true) is actually on so you're not back here next year.

Cloud instance unhealthy. Sometimes the box itself is sick: failed EBS volume, hardware retirement, a hypervisor hiccup. Check the cloud console's instance health and system logs. If the instance is toast, don't nurse it. Drain and replace.

Clock skew. Rare but nasty. If NTP drifted far enough, TLS handshakes to the API server fail with time-based validation errors and certs look "not yet valid." timedatectl, confirm chronyd/systemd-timesyncd is running and synced.

For the broader map of what breaks and how to read it, our Kubernetes troubleshooting guide covers the adjacent failure modes.

Recovering safely#

If you fixed the root cause and the node came back inside the eviction window, you're done: pods stay put. If you need to do disruptive work, cordon first so nothing new lands there:

kubectl cordon ip-10-0-3-14

Then drain when you're ready to move workloads off, respecting PodDisruptionBudgets:

kubectl drain ip-10-0-3-14 --ignore-daemonsets --delete-emptydir-data

--ignore-daemonsets because you can't evict daemonset pods anyway, and --delete-emptydir-data only if you accept losing scratch data on those pods. When the node is healthy again:

kubectl uncordon ip-10-0-3-14

If the box is genuinely dead, skip the nursing and terminate the instance so the autoscaler or machine controller brings up a fresh one.

The call we'd make#

Read the node conditions and the kubelet journal before you do anything reactive. Reboot is the last move, not the first, because it destroys the logs that would have told you why. Cordon early, drain deliberately, and if the instance itself is unhealthy, replace it instead of babysitting it. Nodes are cattle. Treat this one like it.