Linux / Bash

Linux Process Killed Automatically by System (Out of Memory)

A process that just stops running with no crash log, no error message, and no obvious cause is often a strong sign of the Linux kernel's OOM (Out of Memory) Killer stepping in — it terminates processes with SIGKILL specifically to prevent the system from running completely out of memory, and it does this silently from the affected application's own point of view.

The Problem

A long-running process — a server, a background worker — disappears without warning:

Killed

That single word, "Killed," printed to the terminal (if it was running in the foreground) is often the only visible sign anything happened at all, with no stack trace or application-level error message accompanying it.

Why It Happens

When available system memory (including swap) gets critically low, the Linux kernel's OOM Killer selects a process to terminate, based on an internal scoring system, in order to free up memory and keep the overall system responsive rather than let it grind to a complete halt or crash entirely. Common underlying causes:

  • A genuine memory leak in the application, where memory usage grows unbounded over time until it eventually exhausts what's available on the system
  • The system genuinely doesn't have enough total memory for the combined workload running on it, even without any single process behaving badly — several processes each using a reasonable amount can still collectively exceed what's available
  • A sudden spike in memory usage from processing an unusually large request or dataset, exceeding available memory even if typical usage is normally fine
  • No swap space configured at all, removing a buffer that would otherwise give the system (and the OOM Killer's decision-making) more room before an outright kill becomes necessary

The Fix

First, confirm the OOM Killer is actually responsible, rather than assuming, by checking the kernel log directly:

dmesg | grep -i "killed process"
sudo journalctl -k | grep -i "out of memory"

This typically shows a detailed log entry naming the exact process killed and the memory conditions at the time, confirming definitively whether this specific incident was OOM-related or something else entirely producing a similar symptom.

Once confirmed, identify what was actually consuming memory at the time using historical monitoring data if available, or reproduce and monitor in real time if the issue is recurring:

free -h
ps aux --sort=-%mem | head -10

If a specific process has a genuine memory leak, that's the underlying bug to actually fix in the application code — no amount of system-level tuning permanently solves a leak, it only delays how long the process survives before eventually being killed regardless.

If the system genuinely lacks sufficient memory for its total workload, either add more physical memory, reduce the number of concurrent processes/services running on the same machine, or add swap space as a buffer (though swap is a mitigation, not a substitute for adequate memory for CPU-bound or performance-sensitive workloads, since heavy swap usage significantly degrades performance):

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

For a specific process you want to protect from being an easy OOM Killer target relative to others on the system, adjust its OOM score, making the kernel less likely to select it when choosing what to kill under memory pressure:

echo -500 > /proc/<pid>/oom_score_adj

Lower values (down to -1000) make a process less likely to be selected; this doesn't prevent the OOM Killer from ever acting, but shifts relative priority away from a specific critical process in favor of others when a kill genuinely becomes necessary.

For containerized workloads specifically, set explicit, appropriate memory limits per container, so a single misbehaving container is contained to killing only itself rather than potentially triggering a host-level OOM event affecting unrelated processes:

docker run --memory=512m your-image

Still Not Working?

If the exact cause is hard to pin down because the kill happens infrequently and unpredictably, set up ongoing memory monitoring with alerting before memory pressure becomes critical, rather than only investigating after the fact through kernel logs. Tools like a basic cron-scheduled script checking free memory, or a proper monitoring stack, catch the trend of rising memory pressure in the lead-up to an OOM event, giving you the chance to investigate and intervene before the kernel makes the decision for you:

*/5 * * * * free -m | awk 'NR==2{print $3"/"$2}' >> /var/log/memory_usage.log

You might also like