Debugging a Kubernetes Service That Won't Connect

Someone pings you: "the API is up but nothing can hit it." The pods are green. kubectl get svc shows the Service sitting there with a ClusterIP. And yet curl from the frontend just hangs until it times out. This is one of the most common Kubernetes support tickets there is, and almost every case falls into one of about eight buckets. The trick is checking them in the right order so you stop guessing.

Here's the path we walk, top to bottom. Don't skip steps. Half the time the answer shows up two commands in and you feel silly for suspecting NetworkPolicy.

Step 1: Is the pod actually Ready?#

Running is not the same as Ready. A pod can be Running with a failing readiness probe, and Kubernetes will keep it out of the Service rotation on purpose.

$ kubectl get pods -l app=orders
NAME                      READY   STATUS    RESTARTS   AGE
orders-7d9f8c6b5-2xk4p    0/1     Running   0          6m

0/1 is your smoking gun. The container is up but the readiness probe is red, so this pod is not a valid backend for anything. kubectl describe pod orders-7d9f8c6b5-2xk4p and read the Events at the bottom. Usually it's a probe hitting the wrong path or port. Fix the probe, watch it flip to 1/1, then move on.

Step 2: Does the Service selector match the pod labels?#

This is the single most common cause, so it's worth doing early. A Service finds its backends by matching labels. One typo and it matches nothing.

$ kubectl get svc orders -o jsonpath='{.spec.selector}'
{"app":"order-service"}

$ kubectl get pods --show-labels | grep orders
orders-7d9f8c6b5-2xk4p   1/1   Running   0   9m   app=orders,...

Look closely: the Service selects app=order-service, the pods carry app=orders. They will never meet. Someone renamed the deployment and forgot the Service. Align them and it starts working instantly.

Step 3: Does the Service have Endpoints?#

This one command tells you whether steps 1 and 2 are actually fine, because Endpoints are the ground truth of what a Service is routing to.

$ kubectl get endpoints orders
NAME     ENDPOINTS   AGE
orders   <none>      12m

<none> means zero pods are backing this Service. If you got here and step 1 and 2 looked okay, you missed something in one of them. Empty Endpoints is caused by exactly two things: no pod matches the selector, or no matching pod is Ready. There is no third option. On newer clusters run kubectl get endpointslices -l kubernetes.io/service-name=orders for the same answer.

When it's healthy you'll see real addresses:

$ kubectl get endpoints orders
NAME     ENDPOINTS                          AGE
orders   10.244.1.37:8080,10.244.2.14:8080  12m

Now you know traffic has somewhere to go. If Endpoints are populated and it still won't connect, the problem is below the Service, not the Service itself.

Step 4: Is targetPort right?#

The Service port is what clients hit. The targetPort is the port on the pod. People mix these up constantly.

$ kubectl get svc orders -o jsonpath='{.spec.ports[0]}'
{"port":80,"targetPort":8080,"protocol":"TCP"}

Now confirm the container really listens on 8080. Check the Deployment's containerPort, or just ask the pod:

$ kubectl exec orders-7d9f8c6b5-2xk4p -- netstat -tlnp
tcp   0   0   0.0.0.0:3000   0.0.0.0:*   LISTEN   1/node

The app is on 3000, the Service points at 8080. Traffic arrives at a closed port and hangs. Fix targetPort to 3000.

Step 5: Is the app bound to 0.0.0.0, not 127.0.0.1?#

That netstat line above is doing double duty. Look at the address, not just the port. 0.0.0.0:3000 means the app accepts connections on every interface. If instead you see 127.0.0.1:3000, the app only listens on loopback, and nothing outside the container, including kube-proxy, can reach it.

This bites people running frameworks with a default localhost bind. Node, Flask, Rails all do it. The fix is in the app config: bind to 0.0.0.0. No amount of Kubernetes YAML will route around a loopback bind.

Step 6: Can you curl it from another pod?#

Endpoints look good, ports line up, the app is on 0.0.0.0. Time to test the actual path. Get a shell inside the cluster and try it.

$ kubectl run tmp --rm -it --image=nicolaka/netshoot -- bash
tmp:~# curl -v http://orders.default.svc.cluster.local

Try three targets in order and note which one works:

• The pod IP directly (10.244.1.37:8080). If this fails, the problem is the pod or the app, not the Service.
• The ClusterIP. If the pod IP works but ClusterIP doesn't, suspect kube-proxy or a port mismatch.
• The DNS name. If ClusterIP works but the name doesn't, it's DNS.

This split is the whole game. It tells you which layer to blame instead of flailing across all of them.

Step 7: Is DNS resolving?#

From that same netshoot pod:

tmp:~# nslookup orders.default.svc.cluster.local
** server can't find orders.default.svc.cluster.local: NXDOMAIN

The format is service.namespace.svc.cluster.local. A frequent miss is calling a Service in another namespace by short name; orders only resolves inside the Service's own namespace. If everything NXDOMAINs, check CoreDNS itself: kubectl get pods -n kube-system -l k8s-app=kube-dns and look at its logs. A crashlooping CoreDNS takes the whole cluster's name resolution with it.

Step 8: Is a NetworkPolicy blocking you?#

If pod-IP curl works but only from some pods, you likely have a NetworkPolicy.

$ kubectl get networkpolicy
NAME            POD-SELECTOR   AGE
default-deny    <none>         30d

A default-deny policy with no matching ingress rule silently drops traffic. There's no error, no reset, just a hang that looks exactly like a routing problem. Read the policy and confirm your client's labels are allowed. This is the sneakiest item on the list because everything above it looks perfect.

Step 9: kube-proxy and the NodePort/LoadBalancer edge#

If ClusterIP works internally but an external NodePort or LoadBalancer doesn't, you've moved to a different layer. NodePort needs the node's firewall or cloud security group to allow that high port (30000 to 32767). LoadBalancer adds a cloud provisioning step that can quietly fail; kubectl describe svc shows <pending> on the external IP when the controller can't provision. And if ClusterIP is broken cluster-wide, check whether kube-proxy is healthy on the nodes: kubectl get pods -n kube-system -l k8s-app=kube-proxy. A dead kube-proxy means stale iptables and no ClusterIP routing at all.

The decision tree#

Run it in this order and stop at the first failure:

1. Pod 1/1 Ready? No -> fix the readiness probe.
2. Selector matches pod labels? No -> align them.
3. kubectl get endpoints populated? No -> go back to 1 or 2, you missed it.
4. targetPort matches the container's listening port? No -> fix it.
5. App on 0.0.0.0? No -> fix the app's bind address.
6. curl pod IP from another pod works? No -> app problem, not Service.
7. curl ClusterIP works? No -> kube-proxy / port.
8. DNS resolves? No -> namespace suffix or CoreDNS.
9. External access fails only? -> NodePort firewall or LoadBalancer provisioning.

For the wider set of cluster failures beyond connectivity, our Kubernetes troubleshooting guide covers the errors that put pods in a bad state to begin with.

The call we'd make#

Nine times out of ten, kubectl get endpoints ends the investigation before it starts. If it's empty, it's a selector or readiness issue and you never needed to think about DNS or kube-proxy. So that's our opinionated default: check Endpoints first, use it to decide whether the problem is above the Service or below it, and only then reach for the heavier tooling. Everything else on this list is just walking the layer the Endpoints result pointed you at.