PostgreSQL

Postgres WAL Logs Filling Up Disk Space Rapidly

· by DebuggedIt

Quick answer

WAL files accumulating and rapidly filling disk space almost always means something is preventing Postgres from safely removing old WAL segments — most...

WAL files accumulating and rapidly filling disk space almost always means something is preventing Postgres from safely removing old WAL segments — most commonly a replication slot that's stopped being consumed, or a WAL archiving process that's failing, both of which force Postgres to retain WAL indefinitely rather than recycling it as it normally would.

The Problem

The directory holding Postgres's WAL files grows rapidly, consuming disk space far faster than normal database activity would suggest:

du -sh /var/lib/postgresql/*/main/pg_wal
15G   /var/lib/postgresql/16/main/pg_wal

Left unaddressed, this can eventually consume all available disk space, which for Postgres is a genuinely serious situation — a Postgres instance that runs out of disk space for WAL can become unable to accept further writes at all.

Why WAL can't be recycled New WAL written Stalled replica / failed archive_command never consumed -> can't be removed

Why It Happens

Postgres normally recycles or removes WAL segments once they're no longer needed — but several conditions can prevent this, forcing WAL to accumulate indefinitely:

  • A replication slot exists but its corresponding replica or consumer has stopped, crashed, or fallen far behind — replication slots specifically prevent WAL removal until the slot confirms it's no longer needed, and an abandoned slot with no active consumer holds WAL indefinitely, exactly as it's designed to do to prevent data loss for a slot that might resume
  • A configured archive_command for WAL archiving is failing (a full archive destination, a network issue reaching remote storage, incorrect permissions) — Postgres won't remove a WAL segment until it's successfully archived, so a persistently failing archive command causes WAL to accumulate indefinitely
  • A very high volume of write activity generating WAL faster than it can normally be processed and removed, even without any specific failure — genuinely possible under extreme write load, though less common as the sole cause compared to the two reasons above

The Fix

First, check for replication slots and their current state, since an inactive slot is one of the most common causes:

SELECT slot_name, active, restart_lsn, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots;

A slot showing active = false with a large retained_wal value is very likely your culprit — this indicates a replica or consumer that's stopped connecting, while Postgres continues faithfully retaining WAL on its behalf, exactly as the slot mechanism is designed to do.

If the slot's corresponding replica or consumer is genuinely no longer needed (permanently decommissioned, no longer part of your architecture), drop the slot to allow WAL removal to resume:

SELECT pg_drop_replication_slot('slot_name');

If the replica or consumer is still needed but has simply fallen behind or disconnected, restore its connection rather than dropping the slot, since dropping it would require that consumer to perform a full resync from scratch rather than simply catching up from where it left off.

If WAL archiving is configured and the issue is a failing archive_command, check Postgres's logs directly for the specific archiving failure reason:

grep "archive command failed" /var/log/postgresql/postgresql-*.log

Common underlying causes include the archive destination running out of space itself, network connectivity issues to a remote archive location, or permission problems — address the specific reported cause, which typically resolves the accumulation once archiving can successfully resume and catch up on the backlog.

As an immediate, emergency measure if disk space is critically low and you need to free space right now while investigating the root cause, you can identify and manually remove genuinely orphaned WAL files — but do this very carefully, since removing WAL that's still needed by an active replica or a not-yet-completed archive operation can cause genuine data loss or force a full resync:

# Extremely cautious - confirm exactly what's safe to remove first
SELECT pg_walfile_name(pg_current_wal_lsn());
# Compare against files in pg_wal, only touch files clearly older than any active slot's needs

Generally, addressing the root cause (dropping an unneeded slot, fixing archiving) and letting Postgres's own normal WAL management resume is far safer than manual file removal, which should be treated as a genuine last resort under active emergency conditions.

Still Not Working?

If no replication slots or archiving issues explain the accumulation, and disk space continues growing under seemingly normal operation, review your max_wal_size and related checkpoint configuration, since these settings govern how aggressively Postgres checkpoints and recycles WAL under normal conditions — a max_wal_size set very high relative to your actual disk capacity means Postgres will legitimately retain more WAL before triggering a checkpoint, which is normal, intentional behavior but may need adjusting to better match your actual available disk space if the current setting is inadvertently too generous for your specific environment's capacity.