Fixing 502 and 504 Errors from Kubernetes Ingress

There's a moment every on-call engineer knows: the pager goes off, someone pastes a screenshot of a 502 page, and the first instinct is to blame the app. Nine times out of ten the app is fine and the ingress just has nobody healthy to talk to. The trick is knowing which error you're actually looking at, because 502 and 504 point at completely different problems and the fixes don't overlap.

What the two codes actually mean at the ingress

The ingress controller (Nginx, in most of the clusters I've touched) is a reverse proxy. It takes the request, picks an upstream pod, opens a connection, and waits for a response. Two things can go wrong there.

A 502 Bad Gateway means the proxy tried to reach an upstream and the connection failed. No healthy endpoint to route to, connection refused, connection reset mid-flight, or the backend spoke a protocol the proxy didn't expect. The proxy never got a usable response, so it gives up and returns 502.

A 504 Gateway Timeout means the proxy did connect, sent the request, and then sat there waiting until its own read timeout expired. The backend is alive but too slow. That's a different animal entirely.

Rule of thumb I use: 502 is a connectivity problem, 504 is a latency problem. Don't touch timeouts for a 502, and don't restart pods for a 504.

Diagnosis before you touch anything

Start at the ingress controller logs. That's the source of truth for what the proxy saw.

kubectl logs -n ingress-nginx deploy/ingress-nginx-controller | grep -i 502

A real 502 line looks like this:

2026/07/12 09:14:03 [error] 41#41: *88213 connect() failed (111: Connection refused)
while connecting to upstream, client: 10.0.3.11, server: api.example.com,
upstream: "10.244.2.7:8080", request: "GET /v1/orders HTTP/1.1"

Note the upstream: "10.244.2.7:8080". That's a specific pod IP and port the proxy tried. Now check whether that endpoint should even exist:

kubectl get endpoints my-api -o wide

If that command prints <none> under ENDPOINTS, you've found your 502 already. The Service has no ready pods behind it, so the ingress has nowhere to send traffic. From there it's a readiness question:

kubectl get pods -l app=my-api
kubectl describe pod my-api-xxxx | grep -A5 Readiness

For a 504, the log reads differently:

2026/07/12 09:20:41 [error] 41#41: *91002 upstream timed out
(110: Connection timed out) while reading response header from upstream,
upstream: "10.244.2.7:8080", request: "GET /v1/report HTTP/1.1"

upstream timed out ... while reading response header is the signature. The connection worked; the wait didn't.

Fixing the 502

Once you know it's a connectivity failure, walk the causes roughly in order of how often they bite.

No ready endpoints. The most common one. A deploy rolled out, the new pods failed their readiness probe, and the old ones already terminated. The Endpoints list is empty. Fix the probe or fix whatever the probe is correctly catching. Don't paper over it by deleting the readinessProbe, that just moves the 502 to your users instead of your dashboard.

Wrong service or targetPort. The Service targetPort doesn't match the port the container actually listens on. Everything looks green because pods are Ready, but the proxy connects to a port nothing's bound to and gets a refusal. Check both sides:

kubectl get svc my-api -o jsonpath='{.spec.ports[0].targetPort}'

Then compare against the container's real listen port. This one hides well because the pods are healthy.

App not listening on the interface. Classic: the app binds to 127.0.0.1 instead of 0.0.0.0. Inside the pod it works, from another pod it refuses. Connection refused in the logs, every time.

Backend crashing under load. Connection reset (104) instead of refused (111) usually means the pod accepted the connection then died or got OOM-killed mid-request. Check kubectl get pods for restart counts and look for OOMKilled in the pod events.

TLS/protocol mismatch. If your backend serves HTTPS but the ingress is talking plain HTTP to it, you get a 502. Nginx needs to be told the backend is secure:

nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

Miss that annotation and the proxy sends cleartext into a TLS listener, which resets the connection.

Pods terminating without draining. During a rollout, a pod gets SIGTERM and is pulled from Endpoints, but in-flight connections still land on it for a second or two because of propagation lag. Add a preStop hook with a short sleep so the pod keeps serving while it's being removed from the endpoint list. Fifteen seconds of sleep has killed more rollout 502s for me than any other single change.

Fixing the 504

Every 504 comes down to the same thing: the backend took longer than the proxy was willing to wait. You either speed up the backend or extend the wait, and usually you need both.

The default Nginx proxy-read-timeout is 60 seconds. If your report endpoint sometimes takes 90, it gets cut off at 60 and the user sees a 504 even though the backend would have answered. Bump the timeouts on that ingress:

nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
nginx.ingress.kubernetes.io/proxy-send-timeout: "120"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "10"

Keep proxy-connect-timeout short. Connecting should be instant on a healthy network; a long connect timeout just delays the 502 you were going to get anyway.

If the slowness is a fat response streaming through, buffering settings matter too:

nginx.ingress.kubernetes.io/proxy-buffering: "on"
nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"

But be honest about root cause. Most 504s I've chased were a slow database query behind the endpoint, not the proxy. Raising the timeout to 120 seconds turns a fast failure into a slow one and hides the real problem: a missing index or a lock. Pull the backend's own latency metrics before you reach for the timeout annotation. If p99 is climbing, the ingress is the messenger, not the culprit.

For deeper pod-level and networking failures that these codes surface, our Kubernetes troubleshooting guide covers the layers underneath.

The call we'd make

Read the ingress log line first, always. If it says connect() failed or connection reset, it's a 502 and you go check Endpoints and readiness, not timeouts. If it says upstream timed out, it's a 504 and you look at backend latency before you touch a single annotation. The worst outages I've watched came from someone bumping proxy-read-timeout on a 502, or bouncing pods on a 504, because they never read the log line. Two minutes with kubectl logs and kubectl get endpoints beats an hour of guessing.