Docker Container Running Out of Disk Space Due to Unbounded Log Files

Docker's default logging driver doesn't rotate or limit log file size unless explicitly configured — a container that logs continuously (a common pattern for many long-running services) can accumulate an unbounded log file that gradually consumes all available disk space on the host, entirely independent of anything wrong with the application itself.

The Problem

A host running Docker gradually runs out of disk space, seemingly without an obvious large file or dataset explaining the usage:

df -h
/dev/sda1   50G   49G   1.0G   98%   /

Checking for the actual culprit reveals a container's own JSON log file has grown enormous:

du -sh /var/lib/docker/containers/*/*-json.log | sort -rh | head -5
15G   /var/lib/docker/containers/a1b2c3.../a1b2c3-json.log

Why It Happens

Docker's default json-file logging driver captures every line of stdout/stderr from a container and writes it, by default, to a single, continuously growing log file with no automatic rotation or size limit at all unless explicitly configured. This is especially likely to become a problem with:

• Applications that log verbosely by default (debug-level logging left enabled in production, or a chatty framework/library) generating a much higher volume of log output than initially anticipated
• Long-running containers that accumulate log volume over weeks or months of continuous operation without ever being restarted, giving the unbounded log file plenty of time to grow very large
• An application caught in an error loop, repeatedly logging the same error message at high frequency, rapidly inflating log file size far faster than normal operation would

The Fix

Configure log rotation limits at the Docker daemon level, applying a sensible default to every container unless individually overridden:

# /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

sudo systemctl restart docker

max-size caps each individual log file at the specified size before rotating, and max-file caps how many rotated files are retained before the oldest is discarded — together, these bound the maximum total log storage per container to max-size × max-file, preventing any single container's logs from growing unboundedly regardless of how much it actually logs.

Note that daemon-level configuration only applies to newly created containers going forward — existing, already-running containers keep their original (potentially unbounded) logging configuration until they're recreated, not merely restarted:

# Existing containers need to be recreated, not just restarted, to pick up new log settings
docker compose down
docker compose up -d

For per-container overrides, rather than relying purely on the daemon-wide default, specify logging options directly in Docker Compose:

services:
  app:
    image: your-app
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

For containers with genuinely very high, expected log volume where losing older logs via rotation isn't acceptable, consider shipping logs to an external, dedicated logging system (like a centralized log aggregator) rather than relying on local rotation as the sole retention strategy — this separates the concern of "don't let local disk fill up" from "retain logs for as long as I actually need them for," which local rotation alone can't fully satisfy if you genuinely need long-term log retention beyond what's practical to keep on the container host's own local disk.

To immediately reclaim disk space for an already-oversized existing log file without waiting to recreate the container, you can safely truncate the log file directly, which doesn't affect the running container (it continues writing to the same, now-emptied file):

sudo truncate -s 0 /var/lib/docker/containers/<container-id>/<container-id>-json.log

This is a genuinely safe, immediate way to free space in an emergency, though it should be followed up with proper log rotation configuration to prevent the same file from simply growing unboundedly again afterward.

Still Not Working?

If disk space issues persist even after configuring log rotation, confirm the disk space is genuinely being consumed by container logs and not by something else entirely — Docker's own layer cache, unused images, stopped containers, or dangling volumes can all separately consume significant disk space, and a broader cleanup addressing all of these together (not just log rotation) is often needed for a host that's accumulated disk usage across multiple different Docker-related sources over an extended period:

docker system df
docker system prune -a --volumes

Review docker system prune's output carefully before confirming, particularly the --volumes flag, since this can remove data you didn't intend to lose if used without understanding exactly what it will delete in your specific environment.