Fixing DNS Resolution Failures in Kubernetes

The symptom almost never says "DNS." It says the app can't reach the database, or half the requests to an upstream time out at exactly five seconds, or a new deploy works fine in staging and falls over in prod. You go looking for a firewall rule or a bad service, and two hours later you find out a name never resolved. This one has bitten every cluster I've run, so here's how the resolution path actually works and where it breaks.

How cluster DNS actually works#

Every pod gets a generated /etc/resolv.conf. Look at one and you'll see three things that matter:

$ kubectl exec -it web-7d9f -- cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The nameserver is the cluster DNS service IP, backed by CoreDNS pods in kube-system. The search list is what turns postgres into a real name. And ndots:5 is the setting that quietly causes most of the slow-lookup pain.

Service names follow a fixed shape: service.namespace.svc.cluster.local. When your code asks for postgres, the resolver counts the dots. postgres has zero dots, which is fewer than five, so it treats the name as partial and walks the search list first: postgres.default.svc.cluster.local, then postgres.svc.cluster.local, then postgres.cluster.local, and only after those miss does it try postgres on its own. For an in-cluster service the first attempt hits, and you never notice. For an external name like api.stripe.com (three dots, still under five) you burn three failed cluster lookups before the real one succeeds. That's the ndots tax.

Diagnosis: prove it's DNS first#

Don't guess. Drop a debug pod in the same namespace as the broken workload and ask the resolver directly:

$ kubectl run dnsutils --rm -it --restart=Never \
    --image=registry.k8s.io/e2e-test-images/agnhost:2.45 -- bash

# inside the pod
$ nslookup postgres
$ nslookup postgres.default.svc.cluster.local
$ dig +search +short postgres.default
$ dig @10.96.0.10 kubernetes.default.svc.cluster.local

If the fully qualified name resolves but the short one doesn't, your problem is search domains or namespace, not CoreDNS. If nothing resolves, including kubernetes.default, look at CoreDNS itself:

$ kubectl -n kube-system get pods -l k8s-app=kube-dns
$ kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100
$ kubectl -n kube-system describe pod -l k8s-app=kube-dns | grep -A5 Events

Watch the log for SERVFAIL, plugin errors, or a restart count that keeps climbing. That last one is the tell for OOM.

The causes, roughly in the order they get you#

CoreDNS is down, crashing, or underscaled. Two replicas is the default and it's often not enough. Under load or a memory limit that's too tight, CoreDNS gets OOMKilled, the pod restarts, and every lookup during that window fails. Check kubectl -n kube-system get pods for restart counts and confirm the limits:

$ kubectl -n kube-system get deploy coredns -o jsonpath='{.spec.template.spec.containers[0].resources}'

Fix is boring and effective: raise the memory limit and scale out. Give it headroom, then add replicas or attach a HorizontalPodAutoscaler so it tracks demand instead of falling over at peak.

$ kubectl -n kube-system scale deploy coredns --replicas=4

A NetworkPolicy is eating port 53. The day a team adopts default-deny egress, DNS is the first casualty, because DNS runs on UDP and TCP port 53 and nobody remembers to allow it. The workload can't even ask the question. You need an explicit egress rule to kube-system on 53:

egress:
- to:
  - namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: kube-system
  ports:
  - protocol: UDP
    port: 53
  - protocol: TCP
    port: 53

If lookups work from a debug pod in kube-system but not from your app's namespace, this is your answer.

ndots is making external lookups slow. Those five failed attempts before an external name resolves add real latency, and worse, they multiply CoreDNS query volume by four. Two fixes. For a single workload that talks to the outside a lot, lower ndots per pod:

dnsConfig:
  options:
  - name: ndots
    value: "2"

Or sidestep the search list entirely by writing external names with a trailing dot, which marks them absolute: api.stripe.com. resolves in one query. That trailing dot looks like a typo in code review and it isn't. It's the fix.

Wrong namespace assumption. Someone hardcodes redis in a config, the app runs in payments, and redis actually lives in cache. The short name only searches the pod's own namespace. Cross-namespace calls need the namespace in the name: redis.cache or the full redis.cache.svc.cluster.local. This one hides because it works in the namespace where you tested it.

NodeLocal DNSCache issues. If you run node-local DNS (and on a busy cluster you should), each node gets a caching agent at a link-local address and resolv.conf points there instead of the service IP. When it's healthy it kills a huge amount of conntrack pressure. When its DaemonSet is unhealthy on one node, only pods on that node break, which makes for a maddening intermittent pattern. Check it per node: kubectl -n kube-system get pods -l k8s-app=node-local-dns -o wide.

Upstream resolver problems. CoreDNS forwards anything it doesn't own to an upstream, by default the node's resolver via forward . /etc/resolv.conf. If that upstream is slow or flaky, external resolution suffers while cluster names stay fine. The CoreDNS log shows the upstream SERVFAILs. Point forward at something reliable (1.1.1.1, 8.8.8.8) and add cache 30 in the Corefile if it isn't there.

Conntrack races. The classic one. Concurrent UDP DNS queries from the same pod can race in the kernel's conntrack table and get silently dropped, showing up as those five-second timeouts that correspond to the resolver's retry. It's a known kernel behavior. The durable fix is running NodeLocal DNSCache, which forces TCP upstream and dodges the race. A cheaper mitigation is setting options single-request-reopen in the pod's dnsConfig.

The call we'd make#

Prove it's DNS before you touch anything else, because the symptom lies. Run the debug pod, compare short name against FQDN, and read the CoreDNS log. Nine times out of ten it lands in one of two buckets: CoreDNS is starved and needs more memory and replicas, or a NetworkPolicy quietly ate port 53. Fix those, deploy NodeLocal DNSCache so conntrack stops biting you, and write your external names with the trailing dot. If you want the broader map of cluster failure modes this connects to, our Kubernetes troubleshooting guide covers the neighbors. DNS is not hard. It's just invisible until it isn't.