# Kubernetes Readiness Probes Lie During Rolling Updates

URL: https://www.devopsness.com/blog/kubernetes-readiness-probes-lie-during-rolling-updates
Author: Kiril Urbonas
Published: 2026-09-20
Updated: 2026-10-31
Categories: DevOps
Tags: Kubernetes, Deployment, Reliability, Production, Debugging

Summary: A green readiness probe means the probe endpoint answered, nothing more. The gaps that cause 502s during rollouts, and the Deployment settings that close them.

---
# Kubernetes Readiness Probes Lie During Rolling Updates

A passing readiness probe only means one handler, port, or command answered within a second. Rollout 502s come from that gap and from shutdown, where traffic still reaches dying pods. Fix it with a `startupProbe`, a local-only readiness check, a 15 second `preStop` sleep, a grace period that covers it, and `maxUnavailable: 0`.

> [!VERIFIED]
> Kubernetes 1.37 (1.37.2, released 2026-10-28), against the kubernetes.io probe, Pod lifecycle, Deployment, Pod API reference and feature-gate pages. Checked October 2026.

Most rollout errors we see come from those two places. Fix both and a rolling update stops being an incident generator.

## A green probe is a claim about one endpoint

Readiness is a boolean the kubelet computes from your probe. The Service machinery trusts it completely. If `/healthz` returns `200` from a handler that never touches the database, the connection pool, or the cache, the pod is "ready" the instant the web server binds a port. Real requests then hit a JVM still compiling hot paths or a Node process still opening its first pool connections.

We write readiness endpoints that check what a request needs: a pool connection is checked out and returned, config is loaded, the caches we can't serve without are warm. We do not check every downstream dependency, because then one flaky upstream flips every pod to unready at once and you have built an outage amplifier.

## Startup is a different question from readiness

If your app needs 60 seconds to boot, you have two bad options with only a readiness probe and a liveness probe: a loose liveness threshold that hides real hangs, or a tight one that kills the pod mid-boot and produces a [CrashLoopBackOff](/blog/fix-crashloopbackoff) that only appears under load. A `startupProbe` exists for this. Per the Kubernetes docs, if a startup probe is configured, Kubernetes does not run liveness or readiness probes until it succeeds, so you can allow `failureThreshold * periodSeconds` of boot time without loosening anything afterwards.

Defaults matter here (per the Pod API reference): probes run every `periodSeconds: 10`, time out after `timeoutSeconds: 1`, and flip state after `failureThreshold: 3`. A readiness handler that occasionally takes 1.2 seconds under GC pressure will flap pods in and out of the Service.

## The shutdown race is the real culprit

When a pod is deleted, two things start at roughly the same time: the control plane removes the pod from the Service's EndpointSlices, and the kubelet begins shutdown, running the `preStop` hook and then sending `SIGTERM`. Nothing orders these. Endpoint removal then has to propagate to kube-proxy on every node, or to your ingress controller, or to a cloud load balancer that polls on its own schedule. That takes anywhere from a fraction of a second to many seconds.

Meanwhile your app got `SIGTERM`, closed its listener, and exited fast. Requests still routed to the old pod get connection refused. This is what a "zero-downtime" rollout with a perfect readiness probe still produces: a burst of 502s and 503s that lines up with pod termination. The probe was never in the loop, because a terminating pod's fate is decided by endpoint removal, not by probe results.

The fix is unglamorous: delay shutdown so propagation can finish. A `preStop` sleep of 10 to 15 seconds keeps the process serving while the endpoint removal spreads. Kubernetes has a native `sleep` action for `preStop` (the `PodLifecycleSleepAction` gate went beta in 1.30 and stable in 1.34), so distroless images no longer need a shell. On clusters older than 1.30, use `exec` with `sleep 15`.

## Grace period and surge decide how bad it gets

The `preStop` time counts against `terminationGracePeriodSeconds`, which defaults to 30 seconds. A 15 second sleep plus a 20 second in-flight drain does not fit, and the kubelet sends `SIGKILL` at the deadline, cutting off the requests you were trying to protect. Size it as sleep plus your slowest legitimate request plus a margin, and make the app stop accepting new connections on `SIGTERM` while finishing the ones it has.

The Deployment defaults are `maxSurge: 25%` and `maxUnavailable: 25%`. On four replicas that lets one pod go away before its replacement is ready, so capacity dips during every deploy. For latency-sensitive services we set `maxUnavailable: 0` and `maxSurge: 1`, which costs one extra pod briefly and guarantees the new pod passes readiness before an old one is touched. Add `minReadySeconds` so a pod that goes ready and immediately falls over doesn't count as progress. Voluntary node drains are a separate hazard covered in [Pod Disruption Budgets](/blog/kubernetes-pod-disruption-budgets-node-drains).

## Put it together, then measure it

```yaml
# k8s/checkout-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 4
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: checkout
          image: registry.example.com/checkout:1.8.2
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /healthz/startup
              port: 8080
            periodSeconds: 5
            failureThreshold: 24
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: 8080
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2
          lifecycle:
            preStop:
              sleep:
                seconds: 15
```

Do not trust this until you have watched it. Run a request loop from inside the cluster against the Service while a rollout happens, and count status codes:

```shell
$ kubectl run probe --rm -i --restart=Never --image=curlimages/curl -- \
    sh -c 'for i in $(seq 1 1200); do curl -s -o /dev/null -w "%{http_code}\n" --max-time 2 http://checkout/api/ping; sleep 0.1; done' \
    | sort | uniq -c
# in a second terminal, while the loop runs:
$ kubectl rollout restart deployment/checkout
$ kubectl rollout status deployment/checkout
```

Run it once with the `preStop` block removed and once with it. The difference in non-`200` counts is the most convincing argument you will make to a team that thinks its probes are fine.

## The decision, concretely

- **Is `/healthz` returning 200 without touching the pool or warm-up state?** Split it: a cheap liveness check, and a readiness check for local prerequisites only.
- **Does boot take longer than about 10 seconds?** Add a `startupProbe` and leave liveness strict.
- **Seeing 502s only at pod termination?** Add a `preStop` sleep of 10 to 15 seconds and raise `terminationGracePeriodSeconds` to cover it plus in-flight work.
- **Latency-sensitive service on few replicas?** Use `maxUnavailable: 0`, `maxSurge: 1`, and `minReadySeconds`.

## Questions people ask

### Why do I get 502 errors during a Kubernetes rolling update?

Usually because endpoint removal and container shutdown happen in parallel. The pod receives `SIGTERM` and exits while kube-proxy, the ingress controller, or a cloud load balancer still sends it traffic, so those requests fail. A `preStop` sleep of 10 to 15 seconds keeps the pod serving until removal propagates, and `terminationGracePeriodSeconds` must be long enough to cover that sleep plus in-flight requests.

### Does a readiness probe protect a pod that is terminating?

No. Once a pod is deleted, the control plane removes it from the Service's EndpointSlices regardless of what the readiness probe reports, and the kubelet starts the `preStop` hook and `SIGTERM`. The probe gates traffic into a starting or degraded pod; it does nothing to delay traffic that is still in flight toward a terminating one. That is why rollouts with perfect probes still drop requests.

### What is the difference between a startup probe and a readiness probe?

A startup probe answers "has the app finished booting?" and runs first; until it succeeds, Kubernetes holds off liveness and readiness checks. A readiness probe answers "should this pod receive traffic right now?" and keeps running for the pod's life. Use the startup probe to give slow boots room, such as `failureThreshold: 24` at `periodSeconds: 5`, without loosening liveness afterwards.

## The call we'd make

Default every production Deployment to a startup probe, a local-only readiness check, a 15 second `preStop` sleep, and `maxUnavailable: 0`, then prove it with the request loop before trusting it. The caveat is that the sleep is a guess about propagation time. If you sit behind a cloud load balancer with slow deregistration, measure it and go longer. We could not verify the September 2026 Cloud Native Now piece that prompted this, so everything here rests on the documented Kubernetes mechanics.

## Sources

- [Kubernetes Docs: Liveness, Readiness and Startup Probes](https://kubernetes.io/docs/concepts/workloads/pods/probes/)
- [Kubernetes Docs: Pod Lifecycle (termination of Pods)](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/)
- [Kubernetes Docs: Deployments (rolling update strategy)](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)
- [Kubernetes API reference: Pod v1 (Probe defaults)](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/)