No space left on device

1 Logs that grew without rotation (most common on servers)

An application logging errors in a loop can fill hundreds of gigabytes overnight. /var is where this almost always happens.

$ du -h --max-depth=1 /var 2>/dev/null | sort -hr | head

$ sudo journalctl --vacuum-size=200M

For plain files, truncate rather than delete a log a process still holds open: sudo truncate -s 0 /var/log/huge.log. Deleting it would not free the space until the process restarts.

2 Deleted files still held open by a process

The classic head-scratcher: you delete a huge file, du no longer sees it, and df still reports the disk full. The space is not released until the process holding it closes the file.

$ sudo lsof +L1 | sort -k7 -n | tail

$ sudo systemctl restart theservice

3 Out of inodes, not bytes

Every file consumes an inode. Millions of tiny files: session files, mail queues, cache: can exhaust them while df -h still shows free space.

$ df -i

$ find /the/directory -type f -delete

If IUse% is at 100% while Use% is not, this is your cause. Find the offender with: for d in /*; do echo "$(find $d -xdev 2>/dev/null | wc -l) $d"; done | sort -rn | head.

4 Docker images, containers and volumes

On development machines and CI runners this is the usual culprit. Docker accumulates layers and orphaned volumes silently.

$ docker system df

$ docker system prune -a --volumes

That last command deletes unused images AND volumes. Make sure you are not relying on any of them before running it.

5 A different filesystem than the one you checked

/home, /var or /boot are often separate mounts. /boot in particular is small and fills up with old kernels.

$ df -h /the/exact/path/you/were/writing/to

$ sudo apt autoremove --purge    # clears old kernels from /boot

6 Package manager and build caches

Downloaded packages and build artefacts accumulate quietly.

$ du -sh /var/cache/apt ~/.cache 2>/dev/null

$ sudo apt clean && rm -rf ~/.cache/*