Evicted pods are the kubelet telling you a node ran out of something. Here's how to read the signal, stop the bleeding, and keep it from happening again.
Fixing Evicted Pods and Node Pressure in Kubernetes
You open a namespace expecting the usual and instead find a column of pods in Evicted status, sometimes dozens of them. Nothing crashed in your code. No bad deploy. The pods just got shown the door.
Eviction is not a bug. It is the kubelet doing exactly what it was told: when a node runs low on a resource it cannot page out or reclaim fast enough, the kubelet starts killing pods to protect itself. If the node dies, everything on it dies. So the kubelet picks victims first.
The resources that trigger this are memory, ephemeral storage (disk used by container logs, emptyDir volumes, and the writable container layer), the imagefs / nodefs the container runtime lives on, and PIDs. When one of those crosses a threshold, the node flips a condition and the kubelet goes hunting.
Start with the pod itself.
kubectl describe pod payments-api-7c9f-abc12 -n prod
At the top you get Status: Failed and Reason: Evicted, and in the message a line like The node was low on resource: ephemeral-storage. Container app was using 2Gi, request is 0. That last clause is the whole story. The request was zero, so from the scheduler's point of view this pod promised to use nothing, which makes it the first thing thrown overboard.
Then look at what the node was actually screaming about.
kubectl get events -n prod --sort-by=.lastTimestamp | grep -i evict
kubectl describe node ip-10-2-4-19 | grep -A5 Conditions
Node conditions are the ground truth. You are looking for MemoryPressure, DiskPressure, or PIDPressure reading True. A node under DiskPressure will keep evicting pods, and freshly scheduled replacements land, fill up, and get evicted again. That loop is the classic symptom people mistake for a crash-looping app.
For a fast overview of who is hurting:
kubectl top nodes
kubectl top pods -n prod --sort-by=memory
Memory is the one people hit first. A node reports MemoryPressure=True, the kubelet reclaims what it can, and when that is not enough it evicts.
The kubelet ranks candidates by QoS class. Pods with no requests or limits are BestEffort and go first. Pods where usage exceeds their request are next. Pods in the Guaranteed class, where requests equal limits on every container, are evicted last, only when there is genuinely nothing else to give.
So the fix is not a bigger node. The fix is telling the scheduler the truth.
resources:
requests:
memory: 512Mi
cpu: 250m
limits:
memory: 512Mi
cpu: 500m
Set the memory request equal to the limit and that container is Guaranteed. It becomes the last thing evicted instead of the first. For workloads that genuinely spike, keep the request honest and let the limit sit higher, which puts you in Burstable and still protects you far better than the zero-request default. My rule: no production pod ships without a memory request. If you cannot say how much a service needs, you do not understand it well enough to run it.
This one is quieter and bites harder. Ephemeral storage fills up from three places, and it is almost always a mix.
Container logs are first. A chatty app writing to stdout with no rotation will pack the node disk over a few days. emptyDir volumes are second, because people treat them as free scratch space and forget they count against the node. The writable container layer is third, from anything an app writes outside a mounted volume.
Cap it so a single greedy pod cannot take the node down with it.
resources:
requests:
ephemeral-storage: 1Gi
limits:
ephemeral-storage: 2Gi
A pod that blows past its ephemeral-storage limit gets evicted on its own, before it touches the node's overall health. That is the outcome you want. Blast radius of one instead of the whole node.
Disk pressure also comes from image accumulation, which is not the app's fault at all. Nodes that stay up for weeks pull dozens of image versions and never garbage-collect aggressively enough. Check it:
kubectl describe node ip-10-2-4-19 | grep -i imagefs
The kubelet has image-gc-high-threshold and image-gc-low-threshold flags that control when it prunes unused images. If your nodes are long-lived, tune those down or set eviction thresholds explicitly in the kubelet config so cleanup starts before you are in the red:
--eviction-hard=memory.available<500Mi,nodefs.available<10%,imagefs.available<15%
Evicted pods do not clean themselves. They sit in Failed state as tombstones, cluttering kubectl get pods and, on older setups, counting toward quotas. Sweep them:
kubectl get pods -n prod --field-selector=status.phase=Failed \
-o name | xargs kubectl delete -n prod
That deletes the dead objects. It does not fix the cause, so do it after you have set requests, not instead.
If the pressure is real and your requests are already honest, the node is simply too small. Add capacity or spread the load. Cluster Autoscaler or Karpenter should be doing this for you, but a node stuck evicting can be one that autoscaling has not caught up to yet, so check that the scaler is not blocked on something.
For workloads you cannot afford to lose several replicas of at once, a PodDisruptionBudget limits voluntary disruptions during drains and upgrades. Note that a PDB does not protect against hard eviction under node pressure, since that is involuntary. It buys you safety during the planned events, which is where most self-inflicted outages actually come from. Pair it with proper requests and you have covered both the accident and the plan.
If you want the wider map of failure modes, our Kubernetes troubleshooting guide sits next to this one.
Set memory and ephemeral-storage requests on everything, no exceptions. Make the pods you cannot lose Guaranteed. Cap ephemeral storage so one bad actor evicts itself instead of the node. Turn on real image garbage collection on long-lived nodes. Then sweep the tombstones. Ninety percent of eviction incidents we have seen trace back to a zero request somewhere, and that is a config line, not a capacity problem.
