Fix "No Space Left on Device" When df and du Disagree

You SSH into a box, a service is down, and the logs are screaming No space left on device. You run df, the filesystem is at 100%, and you go hunting for the offender. Nine times out of ten it's boring: a runaway log, a forgotten tarball, a cache that ate the world. The other one time, df and du flatly contradict each other, and that's the case that eats an hour if you don't know the pattern.

Let's do the boring case first, then the three that actually trip people up.

The basic case: find the big stuff

Start at the root of the full filesystem and walk down one level at a time. The -x flag keeps du on a single filesystem so it doesn't wander into /proc, /sys, or other mounts and hand you garbage numbers.

# biggest directories directly under /, human-sorted
du -x --max-depth=1 / 2>/dev/null | sort -h

# then drill into whatever's fat
du -x --max-depth=1 /var 2>/dev/null | sort -h

If ncdu is installed (or you can install it), it's faster to navigate:

ncdu -x /var

Usual suspects live in /var: /var/log, /var/lib/docker, /var/cache, and package manager leftovers. Clear the obvious offender, confirm df drops, move on. If du finds the space, you're done. If it doesn't, keep reading.

When df and du disagree

Here's the scenario that sends people in circles. df insists the disk is full:

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1   50G   50G     0  100% /

$ du -xsh /
23G     /

df says 50G used. du says 23G. Twenty-seven gigabytes are unaccounted for. du walks the directory tree and adds up files it can see; df asks the filesystem how many blocks are actually allocated. When those two numbers diverge, the space is being held by something that has no visible directory entry. There are three common reasons.

Case 1: deleted-but-still-open files

This is the big one. A process opens a file, something deletes it (or log rotation rms it), but the process still holds the file descriptor. The directory entry is gone so du can't see it, but the kernel won't release the blocks until the process closes the handle or dies. The space stays gone until then.

Find them with lsof:

# files marked deleted but still held open
lsof +L1

# or the grep version if +L1 isn't behaving
lsof -nP | grep '(deleted)'

You'll get output like this, and the SIZE column tells you who's hoarding:

COMMAND   PID USER   FD   TYPE DEVICE   SIZE/OFF NLINK    NODE NAME
java     2417 app     4w  REG  259,1  18253611008     0 1179651 /var/log/app/app.log (deleted)

Eighteen gigabytes, zero links, still open. Two ways to fix it. The clean fix is to restart (or HUP) the process so it releases the descriptor. If you can't restart it right now, truncate the file through its /proc handle without killing anything:

# free the space immediately without restarting the process
: > "/proc/2417/fd/4"

That empties the file the descriptor points at, and the blocks come back instantly. Then go fix the actual cause, which is almost always an app logging to a file that logrotate deletes instead of copytruncate-ing.

Case 2: inode exhaustion

Sometimes there's plenty of space and you still can't write a byte. Every file consumes an inode, and a filesystem has a fixed number of them set at creation time. Fill them up with millions of tiny files and you get No space left on device while df -h shows free gigabytes.

Check the inode table, not the block table:

$ df -i /
Filesystem       Inodes   IUsed  IFree IUse% Mounted on
/dev/nvme0n1p1  3276800 3276794      6  100% /

Blocks fine, inodes gone. Now find which directory is spawning files. This counts entries per top-level directory:

# find the directory hoarding inodes
for d in /var/*; do printf '%8d %s\n' "$(find "$d" -xdev | wc -l)" "$d"; done | sort -n

Typical culprits: PHP session dirs (/var/lib/php/sessions), mail queues, npm/pip caches, or an app dumping one file per event. The fix is deleting the files, but rm * will choke on the argument list. Use find with -delete:

find /var/lib/php/sessions -type f -mtime +2 -delete

Then fix the process so it stops, or add a cron job that reaps them.

Case 3: space hidden under a mount point

If a directory had files in it before something got mounted on top, those files still exist on the underlying filesystem. They're occupying space but they're shadowed by the mount, so you can't see them and du on the mounted tree won't count them.

The trick is to bind-mount the parent somewhere else and look underneath:

mkdir /mnt/underlay
mount --bind / /mnt/underlay
du -xsh /mnt/underlay/var/log   # look at what's hiding under the real mount
umount /mnt/underlay

If you find a fat directory there that's empty on the live system, that's your ghost. This bites people when /var or /var/log gets moved to its own volume but the old contents were never cleared off the root partition first.

Two more culprits worth knowing

Reserved blocks: ext4 reserves 5% of the filesystem for root by default. On a 50G root that's 2.5G you'll never see as "available" but which df counts as used-ish. It's there so root can still log in and clean up when a runaway process fills the disk. On a big data volume that 5% is pure waste. Check and lower it:

tune2fs -m 1 /dev/nvme0n1p1   # drop reserve from 5% to 1%

Leave it at 5% on the root filesystem. Only trim it on dedicated data or backup volumes.

journald and docker logs: systemd's journal and Docker's JSON log driver both grow without asking. Cap the journal with SystemMaxUse=500M in /etc/systemd/journal/journald.conf, or vacuum on the spot with journalctl --vacuum-size=500M. For Docker, per-container JSON logs live under /var/lib/docker/containers/*/*-json.log and grow forever unless you set max-size and max-file in /etc/docker/daemon.json.

The call we'd make

When df and du agree, it's a cleanup job: find the big directory, delete, done. When they disagree, stop hunting for files and think about what has no directory entry. Our reflex order is lsof +L1 first because deleted-open files are by far the most common surprise, then df -i for inodes, then a bind-mount to check for shadowed data. Fix the space, but always trace it back to the process that caused it, otherwise you're back here next week. And put a monitor on both block and inode usage so the disk tells you before the app does.

For the broader playbook on diagnosing a sick Linux box, see our Linux troubleshooting guide.