Kubernetes Readiness Probes Lie During Rolling Updates

A passing readiness probe does not mean your app can serve real traffic. It means one HTTP handler, TCP port, or command returned success within a second. Most rollout errors we see come from the space between that signal and reality, and from the shutdown side, where Kubernetes keeps routing to pods that are already dying. Fix both and a rolling update stops being an incident generator.

A green probe is a claim about one endpoint#

Readiness is a boolean the kubelet computes from your probe. The Service machinery trusts it completely. If /healthz returns 200 from a handler that never touches the database, the connection pool, or the cache, the pod is "ready" the instant the web server binds a port. Real requests then hit a JVM still compiling hot paths or a Node process still opening its first pool connections.

We write readiness endpoints that check what a request needs: a pool connection is checked out and returned, config is loaded, the caches we can't serve without are warm. We do not check every downstream dependency, because then one flaky upstream flips every pod to unready at once and you have built an outage amplifier.

Startup is a different question from readiness#

If your app needs 60 seconds to boot, you have two bad options with only a readiness probe and a liveness probe: a loose liveness threshold that hides real hangs, or a tight one that kills the pod mid-boot and produces a CrashLoopBackOff that only appears under load. A startupProbe exists for this. Per the Kubernetes docs, liveness and readiness probes are disabled until the startup probe succeeds, so you can allow failureThreshold * periodSeconds of boot time without loosening anything afterwards.

Defaults matter here: probes run every periodSeconds: 10, time out after timeoutSeconds: 1, and flip state after failureThreshold: 3. A readiness handler that occasionally takes 1.2 seconds under GC pressure will flap pods in and out of the Service.

The shutdown race is the real culprit#

When a pod is deleted, two things start at roughly the same time: the control plane removes the pod from the Service's EndpointSlices, and the kubelet begins shutdown, running the preStop hook and then sending SIGTERM. Nothing orders these. Endpoint removal then has to propagate to kube-proxy on every node, or to your ingress controller, or to a cloud load balancer that polls on its own schedule. That takes anywhere from a fraction of a second to many seconds.

Meanwhile your app got SIGTERM, closed its listener, and exited fast. Requests still routed to the old pod get connection refused. This is what a "zero-downtime" rollout with a perfect readiness probe still produces: a burst of 502s and 503s that lines up with pod termination. The probe was never in the loop, because a terminating pod's fate is decided by endpoint removal, not by probe results.

The fix is unglamorous: delay shutdown so propagation can finish. A preStop sleep of 10 to 15 seconds keeps the process serving while the endpoint removal spreads. Kubernetes has a native sleep action for preStop (beta and on by default since 1.30), so distroless images no longer need a shell. On older clusters use exec with sleep 15.

Grace period and surge decide how bad it gets#

The preStop time counts against terminationGracePeriodSeconds, which defaults to 30. A 15 second sleep plus a 20 second in-flight drain does not fit, and the kubelet sends SIGKILL at the deadline, cutting off the requests you were trying to protect. Size it as sleep plus your slowest legitimate request plus a margin, and make the app stop accepting new connections on SIGTERM while finishing the ones it has.

The rolling update defaults are maxSurge: 25% and maxUnavailable: 25%. On four replicas that lets one pod go away before its replacement is ready, so capacity dips during every deploy. For latency-sensitive services we set maxUnavailable: 0 and maxSurge: 1, which costs one extra pod briefly and guarantees the new pod passes readiness before an old one is touched. Add minReadySeconds so a pod that goes ready and immediately falls over doesn't count as progress. Voluntary node drains are a separate hazard covered in Pod Disruption Budgets.

Put it together, then measure it#

# k8s/checkout-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 4
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: checkout
          image: registry.example.com/checkout:1.8.2
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /healthz/startup
              port: 8080
            periodSeconds: 5
            failureThreshold: 24
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: 8080
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2
          lifecycle:
            preStop:
              sleep:
                seconds: 15

Do not trust this until you have watched it. Run a request loop from inside the cluster against the Service while a rollout happens, and count status codes:

$ kubectl run probe --rm -i --restart=Never --image=curlimages/curl -- \
    sh -c 'for i in $(seq 1 1200); do curl -s -o /dev/null -w "%{http_code}\n" --max-time 2 http://checkout/api/ping; sleep 0.1; done' \
    | sort | uniq -c
# in a second terminal, while the loop runs:
$ kubectl rollout restart deployment/checkout
$ kubectl rollout status deployment/checkout

Run it once with the preStop block removed and once with it. The difference in non-200 counts is the most convincing argument you will make to a team that thinks its probes are fine.

The decision, concretely#

• Is /healthz returning 200 without touching the pool or warm-up state? Split it: a cheap liveness check, and a readiness check for local prerequisites only.
• Does boot take longer than about 10 seconds? Add a startupProbe and leave liveness strict.
• Seeing 502s only at pod termination? Add a preStop sleep of 10 to 15 seconds and raise terminationGracePeriodSeconds to cover it plus in-flight work.
• Latency-sensitive service on few replicas? Use maxUnavailable: 0, maxSurge: 1, and minReadySeconds.

The call we'd make#

Default every production Deployment to a startup probe, a local-only readiness check, a 15 second preStop sleep, and maxUnavailable: 0, then prove it with the request loop before trusting it. The caveat is that the sleep is a guess about propagation time. If you sit behind a cloud load balancer with slow deregistration, measure it and go longer. We could not verify the September 2026 Cloud Native Now piece that prompted this, so everything here rests on the documented Kubernetes mechanics.