Fix Docker "No Space Left on Device"

A CI runner paged us at 2am. Builds were failing with write /var/lib/docker/tmp: no space left on device, and the host had been provisioned with a 100GB disk that nobody had touched in months. Nothing was leaking in the app. Docker had quietly eaten the whole volume with old images, dead build cache, and container logs that had never rotated once.

This error almost never means your data is too big. It means Docker's housekeeping was never set up. Let's find where the space went and get it back without nuking anything you still need.

Where Docker actually eats disk

Everything Docker stores lives under /var/lib/docker (the data-root). Five things fill it up, roughly in this order of how often they bite:

• Images and layers. Every docker pull and docker build stacks layers. Old tags and intermediate layers stick around forever.
• Build cache. Since BuildKit, this is a separate store from images and it grows fast on busy CI. It's usually the biggest surprise.
• Stopped containers. Each one keeps its writable layer until you remove it.
• Volumes. Named and anonymous volumes hold real data. This is the one you can lose by being careless.
• Container logs. With the default json-file driver and no rotation, a chatty container writes an unbounded log file.

Diagnose before you delete

Start with Docker's own accounting. docker system df gives you the summary:

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          42        6         28.3GB    22.1GB (78%)
Containers      11        3         1.2GB     980MB (81%)
Local Volumes   17        4         6.4GB     3.1GB (48%)
Build Cache     213       0         31.7GB    31.7GB (100%)

That RECLAIMABLE column is the whole story. Here, 31.7GB of build cache is doing nothing and images are two-thirds waste. Add -v to break it down per image, container, and cache record so you can see the individual offenders:

$ docker system df -v

Cross-check against the OS so you know Docker is the culprit and not something else on the box:

$ df -h /var/lib/docker
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1   98G   97G  0.5G  99% /

$ sudo du -sh /var/lib/docker/*
28G  /var/lib/docker/overlay2
32G  /var/lib/docker/buildkit
2.1G /var/lib/docker/containers
6.4G /var/lib/docker/volumes

overlay2 is your image and container layers. buildkit is the cache. If containers is huge, that's almost always log files.

The fixes, in the order I run them

1. Prune build cache first

This is the safest big win and the one most people forget. Build cache never contains data you care about, only reusable build steps, so clearing it costs you nothing but a slower next build:

$ docker builder prune -f
Total reclaimed space: 31.7GB

On that CI host, this single command took us from 99% to 67%. Run it before anything riskier. If you want to keep recent cache but drop old, docker builder prune --filter until=168h clears anything older than a week.

2. Remove dangling and unused images

Dangling images are the untagged <none> layers left behind by rebuilds:

$ docker image prune -f
$ docker images -f dangling=true    # to see them first

To go further and drop every image not used by a running or stopped container, docker image prune -a. Be aware this will re-pull anything you still use.

3. Clean up stopped containers

$ docker container prune -f
Deleted Containers:
3f9a...   7c21...
Total reclaimed space: 980MB

This only removes containers that are already stopped. Running ones are never touched.

4. Fix container logs (the fix that actually sticks)

If du showed a fat containers directory, hunt the log:

$ sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail -3
14G  /var/lib/docker/containers/a1b2.../a1b2...-json.log

Fourteen gigabytes of logs from one container. You can truncate a single one live with truncate -s 0 <file>, but that's a band-aid. The real fix is rotation. Set it per container:

docker run --log-opt max-size=10m --log-opt max-file=3 myimage

Or set it globally so every new container inherits it. Edit /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Then sudo systemctl restart docker. Existing containers keep their old settings until recreated, so redeploy them to pick up the change.

5. The blunt instrument, and where it bites

docker system prune sweeps stopped containers, dangling images, unused networks, and build cache in one shot:

$ docker system prune
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all dangling images
  - unused build cache

Add -a and it also removes every unused image. That's fine on a CI box.

The one to respect is --volumes. docker system prune -a --volumes will delete any volume not attached to a running container, and that is where your Postgres data, your uploads, and your Redis dumps live. I have watched someone wipe a staging database with that flag because the container happened to be stopped during a deploy. Never put --volumes in an automated job, and think twice before typing it by hand. If you want to clean volumes, list them first with docker volume ls -f dangling=true and remove them by name.

Inode exhaustion: full disk with free space

Occasionally df -h shows space free but writes still fail. Check inodes:

$ df -i /var/lib/docker
Filesystem      Inodes  IUsed IFree IUse% Mounted on
/dev/nvme0n1p1  6.1M    6.1M  0     100%  /

Millions of tiny files from image layers can exhaust inodes before bytes run out. The same image and cache pruning fixes it, since each layer is a pile of small files.

Move or expand the data-root

If the box genuinely needs more space, point Docker at a bigger disk instead of fighting for scraps. Stop Docker, move the directory, and set data-root in daemon.json:

{
  "data-root": "/mnt/bigdisk/docker"
}

$ sudo systemctl stop docker
$ sudo rsync -aP /var/lib/docker/ /mnt/bigdisk/docker/
$ sudo systemctl start docker

Verify with docker info | grep "Docker Root Dir" before you delete the old copy.

Automate cleanup safely

The goal is a nightly job that reclaims the cheap stuff and never touches data. This is the cron line I trust:

0 3 * * * docker builder prune -f --filter until=168h && docker container prune -f && docker image prune -f

Cache older than a week, stopped containers, dangling images. No -a, no --volumes, nothing that can delete a running service or a database. For a deeper walkthrough of the errors around this, our Docker troubleshooting guide covers the rest.

The call we'd make

Prune build cache first, every time, because it's the biggest and safest win. Set log rotation in daemon.json on every host so the problem stops coming back. Automate the safe prunes on a nightly cron. And keep --volumes out of anything automated, because the disk you save isn't worth the database you might lose.