The kernel killed your process to save the box, and the log looks like noise until you know exactly which fields to read.
The Linux OOM Killer — Why It Fired and How to Prevent It
You get paged because a service vanished. No panic in its own logs, no clean shutdown, just gone. Then you run dmesg and find the kernel talking about killing a process to reclaim memory. That is the OOM (out of memory) killer, and it is not a bug. It is the kernel deciding that the machine would fall over entirely unless something died right now.
Linux overcommits memory on purpose. Processes ask for far more than they touch, so the kernel hands out virtual mappings freely and only backs them with physical pages on first write. Most of the time this is fine. The trouble starts when actual usage climbs and the kernel needs a free page it does not have.
Before it kills anything, the kernel tries to reclaim: flush dirty page cache to disk, drop clean cache, swap anonymous pages out. The OOM killer fires only when all of that fails. Memory is exhausted, there are no reclaimable pages left, and there is no swap headroom. At that point the kernel picks a victim and sends it SIGKILL. Nothing else frees memory fast enough.
The record lands in the kernel ring buffer. Start there.
# Human-readable timestamps, filter for the OOM events
dmesg -T | grep -i -E 'oom|out of memory|killed process'
# Same story from the journal, kernel messages only
journalctl -k --since "1 hour ago" | grep -i oom
A real global OOM event looks like this:
[Tue Jul 21 03:14:22 2026] node invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[Tue Jul 21 03:14:22 2026] Mem-Info:
[Tue Jul 21 03:14:22 2026] Node 0 active_anon:3812044kB inactive_anon:12kB ... free:20964kB
[Tue Jul 21 03:14:22 2026] Tasks state (memory values in pages):
[Tue Jul 21 03:14:22 2026] [ pid ] uid tgid total_vm rss ... oom_score_adj name
[Tue Jul 21 03:14:22 2026] [ 1187] 1000 1187 1284410 961242 ... 0 node
[Tue Jul 21 03:14:22 2026] [ 902] 0 902 214880 4102 ... 0 sshd
[Tue Jul 21 03:14:22 2026] Out of memory: Killed process 1187 (node) total-vm:5137640kB, anon-rss:3844968kB, file-rss:0kB, shmem-rss:0kB
Read it top to bottom. The first line names who triggered the reclaim (node invoked oom-killer). Mem-Info shows how little was free. The Tasks state table is the important part: every process with its rss (resident set size, the real physical pages it holds) and its oom_score_adj. The final Out of memory: Killed process line names the victim and how much it was holding when it died. In this case node was sitting on roughly 3.8 GB of anonymous memory, so it was both the biggest user and the obvious target.
Sort that RSS column and you usually have your culprit without any other tooling. When you need to go the other way and catch the growth before the kill, see find what's using your memory.
The kernel does not kill randomly and it does not always kill the thing that asked for the page. It scores every eligible task and kills the one with the highest badness. The score is driven mostly by memory footprint: a process using half of RAM is roughly twice as likely to be picked as one using a quarter. Recent memory is what matters, not virtual size, which is why RSS is the column that counts.
You bias that decision with oom_score_adj, an integer from -1000 to +1000 per process. Add points to make a process a preferred victim, subtract to protect it. The live values are in /proc:
# Current badness score and the tunable knob for a running pid
cat /proc/1187/oom_score
cat /proc/1187/oom_score_adj
# Protect a critical daemon from selection (near-immune at -1000)
echo -900 > /proc/1187/oom_score_adj
Setting oom_score_adj to -1000 makes a task effectively unkillable, which is a foot-gun. Protect a database and you may hand the kill to sshd instead, locking yourself out of a box that is thrashing. Bias, do not exempt.
This is the distinction that trips people up. There are two different out-of-memory conditions and they look almost identical in the log.
Global OOM: the whole machine is out of memory. The kernel scans every process on the box and kills to save the system. This is the case above.
Cgroup OOM: a single control group hit its own memory.max limit while the host still has plenty of RAM free. Under cgroup v2 each container or systemd unit lives in a memory cgroup with a hard cap. Cross it and only tasks inside that cgroup are candidates. The host never noticed. The dmesg line calls this out:
[Tue Jul 21 09:41:08 2026] Memory cgroup out of memory: Killed process 44120 (python) ... oom_score_adj:0
[Tue Jul 21 09:41:08 2026] memory: usage 524288kB, limit 524288kB, failcnt 219
Memory cgroup out of memory plus a usage == limit line means the container ate its allowance, not the host's. That is the same mechanism behind Kubernetes OOMKilled and a Docker exit 137: both are cgroup limits doing exactly what you told them to. The fix path is completely different from a global OOM, so read that first line before you touch host-level swap or overcommit.
For global pressure you have a few levers.
Overcommit policy controls how generously the kernel hands out mappings. vm.overcommit_memory=0 is the default heuristic. Setting it to 2 enforces a hard ceiling of swap + overcommit_ratio% of RAM, so allocations fail with ENOMEM instead of succeeding and triggering an OOM kill later:
sysctl vm.overcommit_memory vm.overcommit_ratio
# Enforce a real ceiling: allocations get honest failures, not surprise kills
sysctl -w vm.overcommit_memory=2
sysctl -w vm.overcommit_ratio=80
Adding swap gives the reclaim path somewhere to push cold anonymous pages, which buys headroom before the killer fires. It trades kills for latency, so size it deliberately rather than defaulting to none. For a per-service cap, set MemoryMax on a systemd unit so a runaway daemon gets its own cgroup OOM at a limit you chose instead of taking the whole host down.
Tuning buys time. Fixing the cause is the actual work.
Right-size limits from measured usage, not from a guess doubled for safety. A limit set far above real need means the process can balloon until it takes the host with it; a limit set too tight means routine spikes get killed. Watch steady-state RSS plus peak, then cap a bit above peak.
Fix the leak. A process whose RSS only ever climbs is not being unlucky, it is holding references it should drop. The OOM killer is a symptom; the leak is the disease, and no sysctl fixes it.
Set oom_score_adj for the daemons that must survive, and set it modestly. Keep sshd and your monitoring agent slightly protected so a bad night still leaves you a way in and a way to see. Bias the killer toward the workload that is safe to restart.
When a process disappears without a trace, check dmesg -T | grep -i oom before anything else. The first log line tells you the whole shape of the problem: Out of memory alone means the host is exhausted, so look at overcommit, swap, and the biggest RSS in the task table. Memory cgroup out of memory means a container hit its own cap and the host is fine, so go fix the limit or the leak inside that unit. Then right-size the cap from real numbers and give your critical daemons a small negative oom_score_adj so the next shortage kills something restartable instead of your way in. For the wider playbook, keep the Linux troubleshooting guide handy.
