Kubernetes Readiness Probes Lie During Rolling Updates

A passing readiness probe only means one handler, port, or command answered within a second. Rollout 502s come from that gap and from shutdown, where traffic still reaches dying pods. Fix it with a startupProbe, a local-only readiness check, a 15 second preStop sleep, a grace period that covers it, and maxUnavailable: 0.

Most rollout errors we see come from those two places. Fix both and a rolling update stops being an incident generator.

A green probe is a claim about one endpoint

Readiness is a boolean the kubelet computes from your probe. The Service machinery trusts it completely. If /healthz returns 200 from a handler that never touches the database, the connection pool, or the cache, the pod is "ready" the instant the web server binds a port. Real requests then hit a JVM still compiling hot paths or a Node process still opening its first pool connections.

We write readiness endpoints that check what a request needs: a pool connection is checked out and returned, config is loaded, the caches we can't serve without are warm. We do not check every downstream dependency, because then one flaky upstream flips every pod to unready at once and you have built an outage amplifier.

Startup is a different question from readiness

If your app needs 60 seconds to boot, you have two bad options with only a readiness probe and a liveness probe: a loose liveness threshold that hides real hangs, or a tight one that kills the pod mid-boot and produces a CrashLoopBackOff that only appears under load. A startupProbe exists for this. Per the Kubernetes docs, if a startup probe is configured, Kubernetes does not run liveness or readiness probes until it succeeds, so you can allow failureThreshold * periodSeconds of boot time without loosening anything afterwards.

Defaults matter here (per the Pod API reference): probes run every periodSeconds: 10, time out after timeoutSeconds: 1, and flip state after failureThreshold: 3. A readiness handler that occasionally takes 1.2 seconds under GC pressure will flap pods in and out of the Service.

The shutdown race is the real culprit

When a pod is deleted, two things start at roughly the same time: the control plane removes the pod from the Service's EndpointSlices, and the kubelet begins shutdown, running the preStop hook and then sending SIGTERM. Nothing orders these. Endpoint removal then has to propagate to kube-proxy on every node, or to your ingress controller, or to a cloud load balancer that polls on its own schedule. That takes anywhere from a fraction of a second to many seconds.

Meanwhile your app got SIGTERM, closed its listener, and exited fast. Requests still routed to the old pod get connection refused. This is what a "zero-downtime" rollout with a perfect readiness probe still produces: a burst of 502s and 503s that lines up with pod termination. The probe was never in the loop, because a terminating pod's fate is decided by endpoint removal, not by probe results.

The fix is unglamorous: delay shutdown so propagation can finish. A preStop sleep of 10 to 15 seconds keeps the process serving while the endpoint removal spreads. Kubernetes has a native sleep action for preStop (the PodLifecycleSleepAction gate went beta in 1.30 and stable in 1.34), so distroless images no longer need a shell. On clusters older than 1.30, use exec with sleep 15.

Grace period and surge decide how bad it gets

The preStop time counts against terminationGracePeriodSeconds, which defaults to 30 seconds. A 15 second sleep plus a 20 second in-flight drain does not fit, and the kubelet sends SIGKILL at the deadline, cutting off the requests you were trying to protect. Size it as sleep plus your slowest legitimate request plus a margin, and make the app stop accepting new connections on SIGTERM while finishing the ones it has.

The Deployment defaults are maxSurge: 25% and maxUnavailable: 25%. On four replicas that lets one pod go away before its replacement is ready, so capacity dips during every deploy. For latency-sensitive services we set maxUnavailable: 0 and maxSurge: 1, which costs one extra pod briefly and guarantees the new pod passes readiness before an old one is touched. Add minReadySeconds so a pod that goes ready and immediately falls over doesn't count as progress. Voluntary node drains are a separate hazard covered in Pod Disruption Budgets.

Put it together, then measure it

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 4
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: checkout
          image: registry.example.com/checkout:1.8.2
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /healthz/startup
              port: 8080
            periodSeconds: 5
            failureThreshold: 24
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: 8080
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2
          lifecycle:
            preStop:
              sleep:
                seconds: 15

Do not trust this until you have watched it. Run a request loop from inside the cluster against the Service while a rollout happens, and count status codes:

kubectl run probe --rm -i --restart=Never --image=curlimages/curl -- \
    sh -c 'for i in $(seq 1 1200); do curl -s -o /dev/null -w "%{http_code}\n" --max-time 2 http://checkout/api/ping; sleep 0.1; done' \
    | sort | uniq -c
# in a second terminal, while the loop runs:
kubectl rollout restart deployment/checkout
kubectl rollout status deployment/checkout

Run it once with the preStop block removed and once with it. The difference in non-200 counts is the most convincing argument you will make to a team that thinks its probes are fine.

The decision, concretely

• Is /healthz returning 200 without touching the pool or warm-up state? Split it: a cheap liveness check, and a readiness check for local prerequisites only.
• Does boot take longer than about 10 seconds? Add a startupProbe and leave liveness strict.
• Seeing 502s only at pod termination? Add a preStop sleep of 10 to 15 seconds and raise terminationGracePeriodSeconds to cover it plus in-flight work.
• Latency-sensitive service on few replicas? Use maxUnavailable: 0, maxSurge: 1, and minReadySeconds.

Questions people ask

Why do I get 502 errors during a Kubernetes rolling update?

Usually because endpoint removal and container shutdown happen in parallel. The pod receives SIGTERM and exits while kube-proxy, the ingress controller, or a cloud load balancer still sends it traffic, so those requests fail. A preStop sleep of 10 to 15 seconds keeps the pod serving until removal propagates, and terminationGracePeriodSeconds must be long enough to cover that sleep plus in-flight requests.

Does a readiness probe protect a pod that is terminating?

No. Once a pod is deleted, the control plane removes it from the Service's EndpointSlices regardless of what the readiness probe reports, and the kubelet starts the preStop hook and SIGTERM. The probe gates traffic into a starting or degraded pod; it does nothing to delay traffic that is still in flight toward a terminating one. That is why rollouts with perfect probes still drop requests.

What is the difference between a startup probe and a readiness probe?

A startup probe answers "has the app finished booting?" and runs first; until it succeeds, Kubernetes holds off liveness and readiness checks. A readiness probe answers "should this pod receive traffic right now?" and keeps running for the pod's life. Use the startup probe to give slow boots room, such as failureThreshold: 24 at periodSeconds: 5, without loosening liveness afterwards.

The call we'd make

Default every production Deployment to a startup probe, a local-only readiness check, a 15 second preStop sleep, and maxUnavailable: 0, then prove it with the request loop before trusting it. The caveat is that the sleep is a guess about propagation time. If you sit behind a cloud load balancer with slow deregistration, measure it and go longer. We could not verify the September 2026 Cloud Native Now piece that prompted this, so everything here rests on the documented Kubernetes mechanics.

Sources

• Kubernetes Docs: Liveness, Readiness and Startup Probes
• Kubernetes Docs: Pod Lifecycle (termination of Pods)
• Kubernetes Docs: Deployments (rolling update strategy)
• Kubernetes API reference: Pod v1 (Probe defaults)