DNS Troubleshooting on Linux: A Systematic Approach

"It's always DNS" is a running joke because DNS resolution on a modern Linux box passes through more layers than people expect: an application's own resolver library, nsswitch.conf, systemd-resolved's local stub, /etc/resolv.conf, search domains, and often a caching layer on top of all of it. A name that resolves correctly from one tool and fails from another usually means two of those layers disagree, not that DNS itself is broken. The fix is finding out which layer is lying.

Start with the tool that bypasses the most layers#

dig talks directly to a nameserver and skips the local resolver stack entirely, which makes it the right first tool: it tells you what the actual authoritative or upstream server thinks, independent of anything misconfigured locally:

$ dig +short api.example.com
203.0.113.42

$ dig api.example.com
; <<>> DiG 9.18.24 <<>> api.example.com
;; ANSWER SECTION:
api.example.com.       284     IN      A       203.0.113.42
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Aug 26 14:02:11 UTC 2026

That SERVER: 127.0.0.53 line matters. On most current distros that's systemd-resolved's local stub listener, not your actual upstream DNS server. To skip the stub and query an upstream directly:

$ dig @8.8.8.8 api.example.com +short
203.0.113.42

If the direct query to a public resolver returns a different answer than the local stub, the problem is in local resolution (caching or config), not in DNS itself.

Find out what's actually answering locally#

$ resolvectl status
Global
       Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         DNS Servers: 10.0.0.2 10.0.0.3
          DNS Domain: ~.

resolvectl status (on distros using systemd-resolved) shows the real per-interface DNS servers and search domains, often more accurate than reading /etc/resolv.conf directly, because that file may just point at the local 127.0.0.53 stub rather than showing the real upstream servers systemd-resolved is actually using.

If the box doesn't run systemd-resolved, /etc/resolv.conf is the direct source of truth:

$ cat /etc/resolv.conf
nameserver 10.0.0.2
nameserver 10.0.0.3
search prod.internal example.com
options ndots:5

The ndots and search-domain trap#

ndots:5 (a common default in Kubernetes pods, inherited from many base images) means: if a name has fewer than 5 dots, try it against every search domain before trying it as-is. A lookup for api in a pod with search prod.internal.svc.cluster.local example.com doesn't try api first — it tries api.prod.internal.svc.cluster.local, then api.svc.cluster.local, and so on, only falling back to the bare name last, and that's the classic cause of DNS lookups that are slow (multiple failed queries before the real one) or resolve to the wrong thing entirely when an unexpected search domain happens to have a matching record.

$ dig +search +showsearch api
;; SEARCH LIST: prod.internal.svc.cluster.local example.com

If a hostname resolves differently (or times out longer) than a fully-qualified version of the same name, this is the first thing to check:

$ time dig api                    # walks the search list
$ time dig api.example.com        # fully qualified, no search-list walk

Caching: three layers that can each be stale independently#

1. systemd-resolved's own cache: flush it and retest before assuming anything upstream changed:
   bash.bash

   $ sudo resolvectl flush-caches
   

2. nscd or unbound, if either is running on the box, cache independently of systemd-resolved. Check with systemctl status nscd unbound; a box can be running more than one caching layer without anyone remembering why.
3. Application-level caching. The JVM, for example, caches successful DNS lookups forever by default (networkaddress.cache.ttl) unless a security manager is active. This is a frequent cause of "the DNS record changed 20 minutes ago but the app is still hitting the old IP" even though the OS resolved correctly, the app just never asked again.

Reading DNS off the wire when nothing else explains it#

$ sudo tcpdump -i eth0 -n port 53
14:05:02.118331 IP 10.0.0.5.51422 > 10.0.0.2.53: 41231+ A? api.example.com. (33)
14:05:02.119820 IP 10.0.0.2.53 > 10.0.0.5.51422: 41231 1/0/0 A 203.0.113.42 (49)

If a query never shows up in the capture at all, the application isn't even reaching the network for DNS. It's serving from an in-process cache, a hosts-file entry, or NSS module (like nss-mdns or a container's /etc/hosts injection) further up the resolution chain than the network layer. getent hosts <name> shows you the result after the full NSS chain runs, which is the closest thing to "what will the application actually see":

$ getent hosts api.example.com
203.0.113.42    api.example.com

The fix in order#

1. dig @<upstream> <name>: bypass every local layer and confirm what DNS actually says at the source.
2. resolvectl status (or /etc/resolv.conf directly): confirm which servers and search domains the box is actually configured to use, not what you assume it's using.
3. If a bare or short name behaves differently than the FQDN, suspect ndots and the search-domain walk; check with dig +search +showsearch.
4. Flush systemd-resolved's cache and check for a second caching daemon (nscd, unbound) before assuming the record itself is wrong.
5. If the app-level behavior still doesn't match what dig/getent show, suspect application-level DNS caching (the JVM is the classic offender); that's a restart or a cache-TTL config, not a DNS problem at all.
6. tcpdump port 53 when nothing above explains it: if the query never leaves the box, something upstream of the network (hosts file, NSS module, in-process cache) is answering first.

This pairs with Linux network debugging with tcpdump, ss, and eBPF for the rest of the network-layer diagnostic toolkit, and the broader Linux troubleshooting guide.

The call we'd make#

Reach for dig @<upstream> before anything else. It's the one query that isolates "is this actually a DNS problem" from "is this a local resolver, cache, or search-domain problem," and every other step in this guide exists to answer that second question once the first one is ruled out.