Fix "Too Many Open Files" on Linux (ulimit)

Fix "Too Many Open Files" on Linux (ulimit)#

Your app was fine at lunch. By 3pm the logs are full of accept: too many open files, new connections hang, and the health check is flapping. The kernel is returning EMFILE because a process tried to allocate a file descriptor beyond its per-process limit. Every socket, pipe, epoll instance, inotify watch, and open file counts against that number.

The error is almost always per-process, not system-wide. That distinction decides where you fix it.

What the error actually means#

A file descriptor (fd) is a small integer the kernel hands back for anything openable. The soft limit RLIMIT_NOFILE caps how many a single process may hold at once. Cross it and open(), socket(), or accept() fail with EMFILE. You will see it as Too many open files in strace, application logs, or a Java java.net.SocketException.

Two limits exist per process. The soft limit is the enforced value. The hard limit is the ceiling a process may raise its own soft limit up to without root. Root can raise the hard limit; unprivileged processes cannot exceed it.

See the current limits#

$ ulimit -Sn        # soft
1024
$ ulimit -Hn        # hard
524288

ulimit reports the limits of your current shell. It does not tell you what a running daemon inherited. The daemon may have been started by systemd, cron, or an init script with a completely different environment. Read its actual limits from procfs:

$ cat /proc/$(pgrep -o nginx)/limits | grep -i "open files"
Max open files            1024                 4096                 files

That first column is the soft limit the process is really running with. If it says 1024 while your shell says 524288, your shell config is irrelevant to that process.

Count how many fds a process is holding#

$ ls /proc/12345/fd | wc -l
1017

At 1017 against a soft limit of 1024 you are about to fall over. To see what those fds are, lsof is the readable view:

$ lsof -p 12345 | awk '{print $5}' | sort | uniq -c | sort -rn
    880 IPv4
     71 REG
     12 unix
      8 DIR
      5 CHR

880 open IPv4 sockets on a process that should hold a few dozen is a leak signature, not a limit that is too low. Hold that thought.

Raise the limit the right way, per context#

There is no single place to change this. Where you edit depends on how the process starts.

Interactive shell or a script you launch by hand: ulimit -n 65536 in that shell, before starting the program. It applies to that shell and its children only, and it cannot exceed the hard limit.

Login sessions (SSH, su, getty): edit /etc/security/limits.conf and rely on pam_limits. This path only affects processes started through PAM.

# /etc/security/limits.conf
appuser   soft   nofile   65536
appuser   hard   nofile   131072

Confirm pam_limits.so is present in the relevant PAM stack (/etc/pam.d/common-session or /etc/pam.d/sshd), then log out and back in. A common trap: editing limits.conf and expecting a systemd-managed service to pick it up. It will not. PAM is not in that path.

systemd services: set it in the unit, not in limits.conf. systemd ignores pam_limits for services it starts.

# /etc/systemd/system/myapp.service  (or a drop-in)
[Service]
LimitNOFILE=65536

Use a drop-in to avoid editing packaged units:

$ sudo systemctl edit myapp
# add the [Service] block above, then:
$ sudo systemctl daemon-reload
$ sudo systemctl restart myapp

daemon-reload alone does not change a running process. The service must restart to inherit the new limit.

The system-wide ceiling#

Per-process limits sit under two kernel-wide sysctls. fs.file-max is the total number of open file handles the whole system will allow. fs.nr_open is the maximum any single process limit may be set to, so a LimitNOFILE higher than fs.nr_open is silently clamped.

$ sysctl fs.file-max fs.nr_open
fs.file-max = 9223372036854775807
fs.nr_open = 1048576

On modern kernels fs.file-max is effectively unbounded, so you rarely touch it. If you genuinely need per-process limits above ~1M, raise fs.nr_open first in /etc/sysctl.d/, then set the higher LimitNOFILE.

Verify it actually applied#

Do not trust the config file. Trust procfs on the live PID after restart:

$ systemctl show myapp -p MainPID --value
12345
$ grep "open files" /proc/12345/limits
Max open files            65536                65536                files

If that still reads 1024, you edited the wrong place for how the process starts. That mismatch is the single most common reason this error "won't go away."

Root cause vs bandaid#

Raising the limit is the correct fix when the workload legitimately needs more fds: a busy reverse proxy, a database with many connections, a service with a large connection pool. It is a bandaid when the fd count climbs without bound. A leak looks like a number that only goes up and never plateaus. Common sources:

• Connection pools: a pool with no max, or code that opens a client per request and never closes it. The fix is in the pool config, not the kernel.
• Unclosed files or sockets: a missing close() in an error path. Watch /proc/<pid>/fd over time; steady growth under steady load means a leak.
• inotify watches: watchers count against fs.inotify.max_user_watches, a separate limit. A tool watching a huge tree throws ENOSPC ("no space left on device") rather than EMFILE, but it is the same class of exhaustion. Raise it with sysctl fs.inotify.max_user_watches=524288 if the watch count is real.

If you raise ulimit and the process refills the new headroom in an hour, you bought an hour. Sample the fd count over time before deciding.

This is one entry in the broader Linux troubleshooting guide; the same procfs-first instinct applies when you hit address already in use.

The fix in order#

1. Read /proc/<pid>/limits on the actual running process. Ignore your shell's ulimit.
2. Count fds with ls /proc/<pid>/fd | wc -l and break them down with lsof -p. Decide leak vs legitimate load.
3. If it is real load, set the limit where the process starts: LimitNOFILE= for systemd, limits.conf for PAM logins, ulimit -n for hand-run scripts.
4. Restart the process and re-check /proc/<pid>/limits. Config that is not reflected on the live PID did nothing.
5. If it is a leak, fix the pool or the missing close. The kernel limit is a fuse, not a solution.

The call we would make on a paging alert: read the live limits, count the fds, and only reach for the config file once lsof proves the process needs the headroom rather than losing it.