Kubernetes Pod Disruption Budgets: Surviving Node Drains Without an Outage

A PodDisruptionBudget (PDB) tells the Kubernetes Eviction API how many of an app's pods may be down at once during voluntary disruptions such as kubectl drain. Set maxUnavailable: 1 on most stateless services, keep at least one replica more than the budget requires, and pair it with real readiness probes and a graceful shutdown.

The classic way to learn this: three replicas spread across three nodes look redundant, until a node-pool upgrade drains all three nodes in quick succession and every replica goes down at once. The Deployment says 3/3; reality says 0 serving. A PDB is how you tell Kubernetes "you may disrupt my pods, but not all of them at once."

Voluntary vs involuntary disruption

PDBs protect against voluntary disruptions, the ones Kubernetes initiates and can be asked to slow down:

• kubectl drain during node upgrades or repair
• Cluster Autoscaler or Karpenter removing a node to consolidate
• Spot or preemptible node reclamation, when a handler drains the node on notice

They do not protect against involuntary disruptions: a kernel panic, a hardware failure, a network partition, or the kubelet evicting pods because the node ran out of resources. Those just happen. PDBs are a contract with the Eviction API, and only evictions go through it. The docs are explicit that deleting a pod or a Deployment directly bypasses the budget, so a script that runs kubectl delete pod in a loop gets no protection at all.

The budget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app: web

When something tries to evict a web pod, the API server checks the PDB. If the eviction would push the app past its budget, the API answers 429 Too Many Requests and kubectl drain keeps retrying until a replacement pod is Ready elsewhere or its timeout expires. The drain proceeds one pod at a time, waiting for recovery between each, which is exactly the rolling behavior you wanted. You can watch the budget from the command line:

kubectl apply -f k8s/web-pdb.yaml
kubectl get pdb web-pdb
NAME      MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
web-pdb   N/A             1                 1                     12s
kubectl drain node-a --ignore-daemonsets --delete-emptydir-data --timeout=10m

ALLOWED DISRUPTIONS is the number to watch. If it reads 0 before you start a drain, the drain will hang.

minAvailable vs maxUnavailable

Two ways to express the budget; pick by what stays stable as you scale:

minAvailable: 2        # absolute floor, but that is 50% at 4 replicas and 20% at 10
# vs
maxUnavailable: 1      # at most 1 down at a time, regardless of replica count

• Use maxUnavailable: 1 for most stateless services. It scales naturally and clearly says "drain one at a time."
• Use minAvailable as a percentage (minAvailable: 80%) when you need a capacity floor to handle load, not just availability. Kubernetes rounds percentages up, so minAvailable: 50% on 7 pods keeps 4.

Never set minAvailable equal to the replica count, and never use maxUnavailable: 0 on a multi-replica app. Both mean no pod can ever be voluntarily evicted: the drain blocks forever and your node upgrade hangs. The docs reserve maxUnavailable: 0 for single-instance stateful apps where the owner wants to be consulted, deleting the PDB to allow a drain and recreating it after.

PDBs are necessary, not sufficient

A PDB controls how many pods go down at once. It does nothing about whether each individual shutdown is graceful. You also need:

Readiness probes that mean it. The PDB counts a pod as healthy when it is Ready. If your readiness probe goes green before the app can serve, the PDB lets the next eviction proceed into a pod that isn't really ready. The budget is only as honest as the probe, a failure mode we covered in readiness probes that lie during rolling updates.

Graceful shutdown. On eviction, Kubernetes runs any preStop hook, sends SIGTERM, waits up to terminationGracePeriodSeconds, then sends SIGKILL. The process must use that window: stop accepting new connections, finish in-flight requests, then exit.

# k8s/web-deployment.yaml (pod spec excerpt)
terminationGracePeriodSeconds: 30
containers:
  - name: web
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "sleep 5"]   # let endpoint removal propagate

The preStop sleep matters more than it looks: pod termination and Service endpoint removal happen concurrently, not in order. Without the brief sleep, the pod can receive SIGTERM and start shutting down while a load balancer still routes new requests to it, which shows up as refused connections and 502s. The sleep holds the pod alive long enough for endpoint removal to reach every proxy.

The gotchas: PDBs can block legitimate operations

A degraded app plus a strict budget stalls drains. If one pod is already crashlooping and the budget leaves no slack, the drain can't make progress. Before Kubernetes 1.31 this was a common hang. The unhealthyPodEvictionPolicy field is now stable: the default, IfHealthyBudget, only lets a not-yet-ready pod be evicted when the app is otherwise within budget, while AlwaysAllow lets drains evict pods that are running but not ready. The docs recommend AlwaysAllow so a broken pod never pins a node; we set it everywhere unless an app has a reason not to.

Leave headroom. Run enough replicas that the PDB permits at least one eviction even during a partial outage. Our rule: replicas >= minAvailable + 1, always.

Rolling updates are not governed by the PDB. Pods replaced during a Deployment rollout count against the budget, but the Deployment controller does not ask permission. Rollout safety lives in maxSurge and maxUnavailable on the Deployment itself.

Single-replica apps get no protection. With one replica, either the PDB blocks every drain or it allows the only pod to go. Run two, or accept the blip.

When a node does get stuck, kubectl get pdb -A and the drain output naming the blocking pod usually tell you which app needs attention; pods that disappear for other reasons are covered in fixing evicted pods.

Questions people ask

Why is kubectl drain stuck evicting a pod?

Almost always a PodDisruptionBudget with zero allowed disruptions. The Eviction API returns 429 and kubectl drain retries until its timeout. Run kubectl get pdb -A and look for ALLOWED DISRUPTIONS of 0. Common causes are minAvailable equal to the replica count, maxUnavailable: 0, or an unhealthy pod under the default IfHealthyBudget policy. Scale up, fix the unhealthy pod, or set unhealthyPodEvictionPolicy: AlwaysAllow.

Should I use minAvailable or maxUnavailable?

Use maxUnavailable: 1 for most stateless services, because it keeps meaning "one at a time" as replica counts change. Use minAvailable as a percentage when the app needs a capacity floor to carry load during a drain. Kubernetes rounds percentages up in both cases. Avoid an absolute minAvailable equal to replicas, which blocks every voluntary eviction and stalls node upgrades indefinitely.

Does a PodDisruptionBudget protect against node failures?

No. PDBs only constrain voluntary disruptions that go through the Eviction API, such as kubectl drain and node autoscaler scale-downs. Hardware failure, kernel panics, network partitions and kubelet evictions under resource pressure ignore the budget, as do direct kubectl delete pod calls. For those you need replicas spread across nodes and zones with topology spread constraints or anti-affinity.

The call we'd make

Put a PDB on every multi-replica service: maxUnavailable: 1 (or a percentage floor where capacity matters), unhealthyPodEvictionPolicy: AlwaysAllow, honest readiness probes, a short preStop delay, and always one more replica than the budget requires. Done that way, upgrades, autoscaler scale-downs and spot reclaims, which happen constantly in a healthy cluster, stay invisible to the people using your service. The caveat is single-instance stateful workloads, where a maxUnavailable: 0 budget and a human in the loop beat any automation.

Sources

• Kubernetes docs: Disruptions
• Kubernetes docs: Specifying a Disruption Budget for your Application
• Kubernetes docs: Safely Drain a Node
• Kubernetes docs: API-initiated Eviction