# Kubernetes Pod Disruption Budgets: Surviving Node Drains Without an Outage URL: https://www.devopsness.com/blog/kubernetes-pod-disruption-budgets-node-drains Author: Kiril Urbonas Published: 2026-06-25 Updated: 2026-10-31 Categories: DevOps Tags: Kubernetes, Reliability, Operations, Autoscaling, Production Summary: Node upgrades, autoscaler scale-downs, and spot reclaims all drain nodes. Without PDBs they can take all your replicas at once. The budgets, probes, and graceful-shutdown handling that keep voluntary disruptions invisible to users. --- # Kubernetes Pod Disruption Budgets: Surviving Node Drains Without an Outage A PodDisruptionBudget (PDB) tells the Kubernetes Eviction API how many of an app's pods may be down at once during voluntary disruptions such as `kubectl drain`. Set `maxUnavailable: 1` on most stateless services, keep at least one replica more than the budget requires, and pair it with real readiness probes and a graceful shutdown. > [!VERIFIED] > Kubernetes 1.37 (1.37.2, the current patch release), `policy/v1` PodDisruptionBudget, against the kubernetes.io disruptions concept page, the configure-PDB and safely-drain-node task pages, and the API-initiated eviction docs. Checked October 2026. The classic way to learn this: three replicas spread across three nodes look redundant, until a node-pool upgrade drains all three nodes in quick succession and every replica goes down at once. The Deployment says 3/3; reality says 0 serving. A PDB is how you tell Kubernetes "you may disrupt my pods, but not all of them at once." ## Voluntary vs involuntary disruption PDBs protect against **voluntary** disruptions, the ones Kubernetes initiates and can be asked to slow down: - `kubectl drain` during node upgrades or repair - Cluster Autoscaler or Karpenter removing a node to consolidate - Spot or preemptible node reclamation, when a handler drains the node on notice They do **not** protect against involuntary disruptions: a kernel panic, a hardware failure, a network partition, or the kubelet evicting pods because the node ran out of resources. Those just happen. PDBs are a contract with the Eviction API, and only evictions go through it. The docs are explicit that deleting a pod or a Deployment directly bypasses the budget, so a script that runs `kubectl delete pod` in a loop gets no protection at all. ## The budget ```yaml # k8s/web-pdb.yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: web-pdb spec: maxUnavailable: 1 unhealthyPodEvictionPolicy: AlwaysAllow selector: matchLabels: app: web ``` When something tries to evict a `web` pod, the API server checks the PDB. If the eviction would push the app past its budget, the API answers `429 Too Many Requests` and `kubectl drain` keeps retrying until a replacement pod is `Ready` elsewhere or its timeout expires. The drain proceeds one pod at a time, waiting for recovery between each, which is exactly the rolling behavior you wanted. You can watch the budget from the command line: ``` $ kubectl apply -f k8s/web-pdb.yaml $ kubectl get pdb web-pdb NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE web-pdb N/A 1 1 12s $ kubectl drain node-a --ignore-daemonsets --delete-emptydir-data --timeout=10m ``` `ALLOWED DISRUPTIONS` is the number to watch. If it reads `0` before you start a drain, the drain will hang. ## minAvailable vs maxUnavailable Two ways to express the budget; pick by what stays stable as you scale: ```yaml minAvailable: 2 # absolute floor, but that is 50% at 4 replicas and 20% at 10 # vs maxUnavailable: 1 # at most 1 down at a time, regardless of replica count ``` - Use **`maxUnavailable: 1`** for most stateless services. It scales naturally and clearly says "drain one at a time." - Use **`minAvailable`** as a percentage (`minAvailable: 80%`) when you need a capacity floor to handle load, not just availability. Kubernetes rounds percentages up, so `minAvailable: 50%` on 7 pods keeps 4. **Never set `minAvailable` equal to the replica count**, and never use `maxUnavailable: 0` on a multi-replica app. Both mean no pod can ever be voluntarily evicted: the drain blocks forever and your node upgrade hangs. The docs reserve `maxUnavailable: 0` for single-instance stateful apps where the owner wants to be consulted, deleting the PDB to allow a drain and recreating it after. ## PDBs are necessary, not sufficient A PDB controls *how many* pods go down at once. It does nothing about whether each individual shutdown is graceful. You also need: **Readiness probes that mean it.** The PDB counts a pod as healthy when it is `Ready`. If your readiness probe goes green before the app can serve, the PDB lets the next eviction proceed into a pod that isn't really ready. The budget is only as honest as the probe, a failure mode we covered in [readiness probes that lie during rolling updates](/blog/kubernetes-readiness-probes-lie-during-rolling-updates). **Graceful shutdown.** On eviction, Kubernetes runs any `preStop` hook, sends `SIGTERM`, waits up to `terminationGracePeriodSeconds`, then sends `SIGKILL`. The process must use that window: stop accepting new connections, finish in-flight requests, then exit. ```yaml # k8s/web-deployment.yaml (pod spec excerpt) terminationGracePeriodSeconds: 30 containers: - name: web lifecycle: preStop: exec: command: ["sh", "-c", "sleep 5"] # let endpoint removal propagate ``` The `preStop` sleep matters more than it looks: pod termination and Service endpoint removal happen concurrently, not in order. Without the brief sleep, the pod can receive `SIGTERM` and start shutting down while a load balancer still routes new requests to it, which shows up as refused connections and 502s. The sleep holds the pod alive long enough for endpoint removal to reach every proxy. ## The gotchas: PDBs can block legitimate operations **A degraded app plus a strict budget stalls drains.** If one pod is already crashlooping and the budget leaves no slack, the drain can't make progress. Before Kubernetes 1.31 this was a common hang. The `unhealthyPodEvictionPolicy` field is now stable: the default, `IfHealthyBudget`, only lets a not-yet-ready pod be evicted when the app is otherwise within budget, while `AlwaysAllow` lets drains evict pods that are running but not ready. The docs recommend `AlwaysAllow` so a broken pod never pins a node; we set it everywhere unless an app has a reason not to. **Leave headroom.** Run enough replicas that the PDB permits at least one eviction even during a partial outage. Our rule: `replicas >= minAvailable + 1`, always. **Rolling updates are not governed by the PDB.** Pods replaced during a Deployment rollout count against the budget, but the Deployment controller does not ask permission. Rollout safety lives in `maxSurge` and `maxUnavailable` on the Deployment itself. **Single-replica apps get no protection.** With one replica, either the PDB blocks every drain or it allows the only pod to go. Run two, or accept the blip. When a node does get stuck, `kubectl get pdb -A` and the drain output naming the blocking pod usually tell you which app needs attention; pods that disappear for other reasons are covered in [fixing evicted pods](/blog/fix-pod-evicted). ## Questions people ask ### Why is kubectl drain stuck evicting a pod? Almost always a PodDisruptionBudget with zero allowed disruptions. The Eviction API returns 429 and `kubectl drain` retries until its timeout. Run `kubectl get pdb -A` and look for `ALLOWED DISRUPTIONS` of 0. Common causes are `minAvailable` equal to the replica count, `maxUnavailable: 0`, or an unhealthy pod under the default `IfHealthyBudget` policy. Scale up, fix the unhealthy pod, or set `unhealthyPodEvictionPolicy: AlwaysAllow`. ### Should I use minAvailable or maxUnavailable? Use `maxUnavailable: 1` for most stateless services, because it keeps meaning "one at a time" as replica counts change. Use `minAvailable` as a percentage when the app needs a capacity floor to carry load during a drain. Kubernetes rounds percentages up in both cases. Avoid an absolute `minAvailable` equal to replicas, which blocks every voluntary eviction and stalls node upgrades indefinitely. ### Does a PodDisruptionBudget protect against node failures? No. PDBs only constrain voluntary disruptions that go through the Eviction API, such as `kubectl drain` and node autoscaler scale-downs. Hardware failure, kernel panics, network partitions and kubelet evictions under resource pressure ignore the budget, as do direct `kubectl delete pod` calls. For those you need replicas spread across nodes and zones with topology spread constraints or anti-affinity. ## The call we'd make Put a PDB on every multi-replica service: `maxUnavailable: 1` (or a percentage floor where capacity matters), `unhealthyPodEvictionPolicy: AlwaysAllow`, honest readiness probes, a short `preStop` delay, and always one more replica than the budget requires. Done that way, upgrades, autoscaler scale-downs and spot reclaims, which happen constantly in a healthy cluster, stay invisible to the people using your service. The caveat is single-instance stateful workloads, where a `maxUnavailable: 0` budget and a human in the loop beat any automation. ## Sources - [Kubernetes docs: Disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) - [Kubernetes docs: Specifying a Disruption Budget for your Application](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) - [Kubernetes docs: Safely Drain a Node](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/) - [Kubernetes docs: API-initiated Eviction](https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/)