Linux "No Space Left on Device" Caused by Unlinked Deleted Files Holding Disk Space

When a Linux disk reports full despite deleted files, the space is very often still held by a long-running service — like journald, a database, or a web server — that had the file open at the moment it was deleted, and continues holding that space allocated until the specific process either closes the file or is restarted.

The Problem

A disk shows as full or nearly full, but manually reviewing the visible filesystem doesn't reveal files large enough to account for the reported usage:

df -h /var
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        20G   19G  200M   99% /var

du -sh /var/* 2>/dev/null | sort -rh
# totals don't add up anywhere close to 19G

Why It Happens

On Linux, deleting a file only removes its directory entry — if a running process still has that file open at the time of deletion, the underlying disk blocks remain allocated and unavailable until every process holding it open closes it. This specifically and commonly happens with long-running background services:

• A log rotation tool deletes or renames an old log file, but the service still writing to it (which opened the file by its original name/handle before rotation happened) continues writing to what's now an invisible, unlinked file, with the space it consumes remaining fully allocated and hidden from normal directory listings
• A database or long-running application had a large temporary or working file open, and that file was deleted (by the database's own cleanup logic, or manually) while the process was still actively using it
• journald (the systemd logging service) or another logging daemon holding an open handle to a log file that was deleted by an external cleanup process not coordinated with the logging daemon's own retention/rotation logic

The Fix

Identify exactly which process is holding open, deleted files, and how much space they're collectively consuming:

sudo lsof +L1

This lists every file with a link count of less than 1 (meaning deleted but still open), along with the process holding it and the size being held — giving you a direct, actionable list rather than needing to guess.

For the very common journald-specific case, where logs configured with a size or time limit have grown beyond expectation, or where the log directory itself was manually cleared without coordinating with the running service, restart the journal service directly, which releases its hold on any deleted files and lets it resume with a clean state:

sudo systemctl restart systemd-journald

Alternatively, if the actual concern is journal size specifically (rather than a deleted-file issue), use journald's own built-in size management rather than manually deleting its underlying files at all, which is both safer and less likely to produce this exact deleted-but-open-file scenario in the first place:

sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=7d

For a different specific service identified as holding deleted files (a database, a web server, a custom application), the general fix is the same in principle: restart the specific service to release its hold on the deleted file's disk space:

sudo systemctl restart your-service-name

Before restarting a critical production service purely to reclaim disk space, weigh the disruption of a restart against the urgency of the disk space issue — if disk space isn't yet critical, it may be more appropriate to schedule this for a planned maintenance window rather than restarting an important service immediately and disruptively purely to reclaim space.

If you don't want to restart the service, but the specific held file is a large, actively-growing log file, you can immediately free the space by truncating the file directly through its open file descriptor, without needing a full service restart:

# Using the specific PID and file descriptor number from the lsof +L1 output
: > /proc/<pid>/fd/<fd_number>

This truncates the file to zero length while the process continues writing to the same, now-emptied underlying file — a useful technique specifically for actively-growing log files where a full service restart isn't immediately practical, but disk space needs to be reclaimed right away.

Still Not Working?

To prevent this specific pattern from recurring — external tools deleting log or data files that a running service still has open — ensure any log rotation or cleanup tooling you use is properly configured to signal the relevant service after rotation (commonly via a "post-rotate" script sending a signal like SIGHUP, which many services interpret as "reopen your log files"), rather than simply deleting or renaming files out from under a service that has no way of knowing this happened and continues writing to the now-invisible original file:

# Example logrotate configuration snippet
postrotate
    systemctl reload your-service
endscript

Configuring this correctly for every service with externally-managed log rotation prevents the underlying disk-space-holding pattern from recurring, rather than needing to repeatedly discover and manually address it after the fact each time it happens.