Fixing "Too Many Open Files" in Kubernetes Containers

The first time this bit me, the app had been running clean for three weeks. Then a node cordoned, pods rescheduled, traffic doubled onto fewer replicas, and one of them started spitting accept: too many open files into the logs. Requests hung. The pod stayed Ready because its liveness probe was cheap enough to still answer. Classic.

"Too many open files" means a process tried to open a new file descriptor and the kernel said no, because that process already holds as many FDs as its RLIMIT_NOFILE allows. The word "files" is what throws people. On Linux almost everything is a file descriptor: open files, yes, but also every TCP socket, every pipe, every epoll instance, every inotify watch. A busy web service can sit at thousands of FDs without touching a single file on disk, because each keep-alive connection is one FD, each upstream connection is another.

So the error is real, but it does not tell you which of two very different things is wrong. Either you have a leak and the FD count climbs forever, or you have honest concurrency and the limit is simply set too low. Fixing the second is one line. "Fixing" the first with that same one line just moves the crash to next Tuesday.

Find the process and count its FDs

Get a shell in the container, or use kubectl debug if the image has no shell:

kubectl exec -it mypod -- sh
# or, for a distroless image:
kubectl debug -it mypod --image=busybox --target=app -- sh

Find the PID (usually 1 in a container) and count what it holds:

ls /proc/1/fd | wc -l

That number is the truth. Compare it to the ceiling:

cat /proc/1/limits | grep 'open files'
# Max open files            1024      1048576    files

The first column is the soft limit, the one that actually bites. If ls /proc/1/fd | wc -l is sitting near that soft limit, that is your wall. Also check what the shell sees, since it can differ from what the app inherited:

ulimit -n

Now watch it over a minute or two:

while true; do echo "$(date +%T) $(ls /proc/1/fd | wc -l)"; sleep 5; done

Flat under load is honest concurrency. A line that only ever climbs, even when traffic dips, is a leak. That single observation decides everything that follows.

If you want to know what the FDs are, lsof -p 1 groups them, or just eyeball the symlinks:

ls -l /proc/1/fd | awk '{print $NF}' | sort | uniq -c | sort -rn | head

Ten thousand entries pointing at socket:[...] to the same upstream IP is a connection leak. A pile of anon_inode:inotify is a different animal, covered below.

When it's a real leak

This is the common one and the boring answer nobody wants: the fix is in the application, not the manifest. Somewhere a code path opens a socket or a file and never closes it on the error branch. HTTP clients that get created per-request instead of reused. Database connections checked out and never returned. A file handle opened in a loop without a defer close or a with block.

Reproduce it in a loop, watch /proc/1/fd climb, find the code path that correlates, and close the handle. Reuse your HTTP client and connection pool instead of minting one per call. If you genuinely cannot fix the upstream library today, raising the limit buys you time to schedule a restart before the crash, but be honest that it is a countdown, not a fix.

When the limit is honestly too low

If the count is flat and just high, raise the ceiling. Node defaults vary wildly by distro and container runtime, and older setups still ship a 1024 soft limit that a modern service blows through instantly.

Set it per-pod with a sysctl in the pod spec:

spec:
  securityContext:
    sysctls:
      - name: fs.file-max
        value: "2097152"

For the per-process nofile limit specifically, the reliable lever is the runtime. On most clusters the kubelet or containerd default LimitNOFILE governs it. Set it in the containerd/runtime config on the node:

# /etc/systemd/system/containerd.service.d/override.conf
[Service]
LimitNOFILE=1048576

Then systemctl daemon-reload && systemctl restart containerd and roll the pods. Confirm inside the container with cat /proc/1/limits. Pick a number with headroom over your steady-state count, not an arbitrary million because a blog said so.

Inotify exhaustion, the sneaky one

Sometimes the app is fine and the error still fires, because inotify watches are FDs too and they are governed by a node-level kernel setting shared across every pod on that node. File-watching tools, config-reload libraries, and log shippers each grab watches. Enough noisy neighbors and the node runs dry, and a totally innocent pod gets the error.

Check and raise the node sysctls:

sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances

# /etc/sysctl.d/99-inotify.conf
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 512

Apply with sysctl -p /etc/sysctl.d/99-inotify.conf. These are per-node and per-uid, so a DaemonSet or node-init handles it cleanly. While you are on the node, glance at conntrack too, because a full connection-tracking table (nf_conntrack: table full) produces the same "connections mysteriously fail" symptom from a different ceiling: sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max.

For the broader playbook on reading these signals, our Kubernetes troubleshooting guide walks the same diagnostic loop across other failure modes.

The call we'd make

Count the FDs before you touch a single manifest. If the number climbs, it is a leak and raising nofile only reschedules the outage, so fix the code. If the number is flat and high, raise the limit at the runtime, give it real headroom, and move on. And check the node sysctls for inotify and conntrack before you blame the app, because a shared ceiling makes an innocent pod look guilty. The limit is a smoke detector. Ripping the battery out because it keeps beeping is not maintenance.