# Fix "No Space Left on Device" When df and du Disagree URL: https://www.devopsness.com/blog/fix-disk-full-linux Author: Kiril Urbonas Published: 2026-07-19 Updated: 2026-07-19 Categories: Linux Tags: Linux, Storage, Troubleshooting, Sysadmin, Disk Summary: A field guide to reclaiming Linux disk space when df reports full but du can't find where the bytes actually went. --- You SSH into a box, a service is down, and the logs are screaming `No space left on device`. You run `df`, the filesystem is at 100%, and you go hunting for the offender. Nine times out of ten it's boring: a runaway log, a forgotten tarball, a cache that ate the world. The other one time, `df` and `du` flatly contradict each other, and that's the case that eats an hour if you don't know the pattern. Let's do the boring case first, then the three that actually trip people up. ## The basic case: find the big stuff Start at the root of the full filesystem and walk down one level at a time. The `-x` flag keeps `du` on a single filesystem so it doesn't wander into `/proc`, `/sys`, or other mounts and hand you garbage numbers. ```bash # biggest directories directly under /, human-sorted du -x --max-depth=1 / 2>/dev/null | sort -h # then drill into whatever's fat du -x --max-depth=1 /var 2>/dev/null | sort -h ``` If `ncdu` is installed (or you can install it), it's faster to navigate: ```bash ncdu -x /var ``` Usual suspects live in `/var`: `/var/log`, `/var/lib/docker`, `/var/cache`, and package manager leftovers. Clear the obvious offender, confirm `df` drops, move on. If `du` finds the space, you're done. If it doesn't, keep reading. ## When df and du disagree Here's the scenario that sends people in circles. `df` insists the disk is full: ```bash $ df -h / Filesystem Size Used Avail Use% Mounted on /dev/nvme0n1p1 50G 50G 0 100% / $ du -xsh / 23G / ``` `df` says 50G used. `du` says 23G. Twenty-seven gigabytes are unaccounted for. `du` walks the directory tree and adds up files it can see; `df` asks the filesystem how many blocks are actually allocated. When those two numbers diverge, the space is being held by something that has no visible directory entry. There are three common reasons. ### Case 1: deleted-but-still-open files This is the big one. A process opens a file, something deletes it (or log rotation `rm`s it), but the process still holds the file descriptor. The directory entry is gone so `du` can't see it, but the kernel won't release the blocks until the process closes the handle or dies. The space stays gone until then. Find them with `lsof`: ```bash # files marked deleted but still held open lsof +L1 # or the grep version if +L1 isn't behaving lsof -nP | grep '(deleted)' ``` You'll get output like this, and the SIZE column tells you who's hoarding: ``` COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME java 2417 app 4w REG 259,1 18253611008 0 1179651 /var/log/app/app.log (deleted) ``` Eighteen gigabytes, zero links, still open. Two ways to fix it. The clean fix is to restart (or `HUP`) the process so it releases the descriptor. If you can't restart it right now, truncate the file through its `/proc` handle without killing anything: ```bash # free the space immediately without restarting the process : > "/proc/2417/fd/4" ``` That empties the file the descriptor points at, and the blocks come back instantly. Then go fix the actual cause, which is almost always an app logging to a file that logrotate deletes instead of `copytruncate`-ing. ### Case 2: inode exhaustion Sometimes there's plenty of space and you still can't write a byte. Every file consumes an inode, and a filesystem has a fixed number of them set at creation time. Fill them up with millions of tiny files and you get `No space left on device` while `df -h` shows free gigabytes. Check the inode table, not the block table: ```bash $ df -i / Filesystem Inodes IUsed IFree IUse% Mounted on /dev/nvme0n1p1 3276800 3276794 6 100% / ``` Blocks fine, inodes gone. Now find which directory is spawning files. This counts entries per top-level directory: ```bash # find the directory hoarding inodes for d in /var/*; do printf '%8d %s\n' "$(find "$d" -xdev | wc -l)" "$d"; done | sort -n ``` Typical culprits: PHP session dirs (`/var/lib/php/sessions`), mail queues, npm/pip caches, or an app dumping one file per event. The fix is deleting the files, but `rm *` will choke on the argument list. Use `find` with `-delete`: ```bash find /var/lib/php/sessions -type f -mtime +2 -delete ``` Then fix the process so it stops, or add a cron job that reaps them. ### Case 3: space hidden under a mount point If a directory had files in it before something got mounted on top, those files still exist on the underlying filesystem. They're occupying space but they're shadowed by the mount, so you can't see them and `du` on the mounted tree won't count them. The trick is to bind-mount the parent somewhere else and look underneath: ```bash mkdir /mnt/underlay mount --bind / /mnt/underlay du -xsh /mnt/underlay/var/log # look at what's hiding under the real mount umount /mnt/underlay ``` If you find a fat directory there that's empty on the live system, that's your ghost. This bites people when `/var` or `/var/log` gets moved to its own volume but the old contents were never cleared off the root partition first. ## Two more culprits worth knowing **Reserved blocks:** ext4 reserves 5% of the filesystem for root by default. On a 50G root that's 2.5G you'll never see as "available" but which `df` counts as used-ish. It's there so root can still log in and clean up when a runaway process fills the disk. On a big data volume that 5% is pure waste. Check and lower it: ```bash tune2fs -m 1 /dev/nvme0n1p1 # drop reserve from 5% to 1% ``` Leave it at 5% on the root filesystem. Only trim it on dedicated data or backup volumes. **journald and docker logs:** systemd's journal and Docker's JSON log driver both grow without asking. Cap the journal with `SystemMaxUse=500M` in `/etc/systemd/journal/journald.conf`, or vacuum on the spot with `journalctl --vacuum-size=500M`. For Docker, per-container JSON logs live under `/var/lib/docker/containers/*/*-json.log` and grow forever unless you set `max-size` and `max-file` in `/etc/docker/daemon.json`. ## The call we'd make When `df` and `du` agree, it's a cleanup job: find the big directory, delete, done. When they disagree, stop hunting for files and think about what has no directory entry. Our reflex order is `lsof +L1` first because deleted-open files are by far the most common surprise, then `df -i` for inodes, then a bind-mount to check for shadowed data. Fix the space, but always trace it back to the process that caused it, otherwise you're back here next week. And put a monitor on both block and inode usage so the disk tells you before the app does. For the broader playbook on diagnosing a sick Linux box, see our [Linux troubleshooting guide](/blog/linux-troubleshooting-guide).