# Fixing Evicted Pods and Node Pressure in Kubernetes URL: https://www.devopsness.com/blog/fix-pod-evicted Author: Kiril Urbonas Published: 2026-07-10 Updated: 2026-07-10 Categories: DevOps Tags: Kubernetes, DevOps, Performance, Debugging, Operations Summary: Evicted pods are the kubelet telling you a node ran out of something. Here's how to read the signal, stop the bleeding, and keep it from happening again. --- # Fixing Evicted Pods and Node Pressure in Kubernetes You open a namespace expecting the usual and instead find a column of pods in `Evicted` status, sometimes dozens of them. Nothing crashed in your code. No bad deploy. The pods just got shown the door. Eviction is not a bug. It is the kubelet doing exactly what it was told: when a node runs low on a resource it cannot page out or reclaim fast enough, the kubelet starts killing pods to protect itself. If the node dies, everything on it dies. So the kubelet picks victims first. The resources that trigger this are memory, ephemeral storage (disk used by container logs, `emptyDir` volumes, and the writable container layer), the imagefs / nodefs the container runtime lives on, and PIDs. When one of those crosses a threshold, the node flips a condition and the kubelet goes hunting. ## Reading the signal Start with the pod itself. ``` kubectl describe pod payments-api-7c9f-abc12 -n prod ``` At the top you get `Status: Failed` and `Reason: Evicted`, and in the message a line like `The node was low on resource: ephemeral-storage. Container app was using 2Gi, request is 0.` That last clause is the whole story. The request was zero, so from the scheduler's point of view this pod promised to use nothing, which makes it the first thing thrown overboard. Then look at what the node was actually screaming about. ``` kubectl get events -n prod --sort-by=.lastTimestamp | grep -i evict kubectl describe node ip-10-2-4-19 | grep -A5 Conditions ``` Node conditions are the ground truth. You are looking for `MemoryPressure`, `DiskPressure`, or `PIDPressure` reading `True`. A node under `DiskPressure` will keep evicting pods, and freshly scheduled replacements land, fill up, and get evicted again. That loop is the classic symptom people mistake for a crash-looping app. For a fast overview of who is hurting: ``` kubectl top nodes kubectl top pods -n prod --sort-by=memory ``` ## Memory pressure Memory is the one people hit first. A node reports `MemoryPressure=True`, the kubelet reclaims what it can, and when that is not enough it evicts. The kubelet ranks candidates by QoS class. Pods with no requests or limits are `BestEffort` and go first. Pods where usage exceeds their request are next. Pods in the `Guaranteed` class, where requests equal limits on every container, are evicted last, only when there is genuinely nothing else to give. So the fix is not a bigger node. The fix is telling the scheduler the truth. ```yaml resources: requests: memory: 512Mi cpu: 250m limits: memory: 512Mi cpu: 500m ``` Set the memory request equal to the limit and that container is `Guaranteed`. It becomes the last thing evicted instead of the first. For workloads that genuinely spike, keep the request honest and let the limit sit higher, which puts you in `Burstable` and still protects you far better than the zero-request default. My rule: no production pod ships without a memory request. If you cannot say how much a service needs, you do not understand it well enough to run it. ## Ephemeral storage and disk pressure This one is quieter and bites harder. Ephemeral storage fills up from three places, and it is almost always a mix. Container logs are first. A chatty app writing to stdout with no rotation will pack the node disk over a few days. `emptyDir` volumes are second, because people treat them as free scratch space and forget they count against the node. The writable container layer is third, from anything an app writes outside a mounted volume. Cap it so a single greedy pod cannot take the node down with it. ```yaml resources: requests: ephemeral-storage: 1Gi limits: ephemeral-storage: 2Gi ``` A pod that blows past its ephemeral-storage limit gets evicted on its own, before it touches the node's overall health. That is the outcome you want. Blast radius of one instead of the whole node. Disk pressure also comes from image accumulation, which is not the app's fault at all. Nodes that stay up for weeks pull dozens of image versions and never garbage-collect aggressively enough. Check it: ``` kubectl describe node ip-10-2-4-19 | grep -i imagefs ``` The kubelet has `image-gc-high-threshold` and `image-gc-low-threshold` flags that control when it prunes unused images. If your nodes are long-lived, tune those down or set eviction thresholds explicitly in the kubelet config so cleanup starts before you are in the red: ``` --eviction-hard=memory.available<500Mi,nodefs.available<10%,imagefs.available<15% ``` ## Stopping the loop and cleaning up Evicted pods do not clean themselves. They sit in `Failed` state as tombstones, cluttering `kubectl get pods` and, on older setups, counting toward quotas. Sweep them: ``` kubectl get pods -n prod --field-selector=status.phase=Failed \ -o name | xargs kubectl delete -n prod ``` That deletes the dead objects. It does not fix the cause, so do it after you have set requests, not instead. If the pressure is real and your requests are already honest, the node is simply too small. Add capacity or spread the load. Cluster Autoscaler or Karpenter should be doing this for you, but a node stuck evicting can be one that autoscaling has not caught up to yet, so check that the scaler is not blocked on something. For workloads you cannot afford to lose several replicas of at once, a PodDisruptionBudget limits voluntary disruptions during drains and upgrades. Note that a PDB does not protect against hard eviction under node pressure, since that is involuntary. It buys you safety during the planned events, which is where most self-inflicted outages actually come from. Pair it with proper requests and you have covered both the accident and the plan. If you want the wider map of failure modes, our [Kubernetes troubleshooting](/blog/kubernetes-error-troubleshooting-guide) guide sits next to this one. ## The call we'd make Set memory and ephemeral-storage requests on everything, no exceptions. Make the pods you cannot lose `Guaranteed`. Cap ephemeral storage so one bad actor evicts itself instead of the node. Turn on real image garbage collection on long-lived nodes. Then sweep the tombstones. Ninety percent of eviction incidents we have seen trace back to a zero request somewhere, and that is a config line, not a capacity problem.