# The Linux OOM Killer — Why It Fired and How to Prevent It URL: https://www.devopsness.com/blog/fix-out-of-memory-linux Author: Kiril Urbonas Published: 2026-07-21 Updated: 2026-07-21 Categories: Linux Tags: Linux, Troubleshooting, Sysadmin, Memory, Oom Summary: The kernel killed your process to save the box, and the log looks like noise until you know exactly which fields to read. --- # The Linux OOM Killer — Why It Fired and How to Prevent It You get paged because a service vanished. No panic in its own logs, no clean shutdown, just gone. Then you run `dmesg` and find the kernel talking about killing a process to reclaim memory. That is the OOM (out of memory) killer, and it is not a bug. It is the kernel deciding that the machine would fall over entirely unless something died right now. ## When the kernel invokes it Linux overcommits memory on purpose. Processes ask for far more than they touch, so the kernel hands out virtual mappings freely and only backs them with physical pages on first write. Most of the time this is fine. The trouble starts when actual usage climbs and the kernel needs a free page it does not have. Before it kills anything, the kernel tries to reclaim: flush dirty page cache to disk, drop clean cache, swap anonymous pages out. The OOM killer fires only when all of that fails. Memory is exhausted, there are no reclaimable pages left, and there is no swap headroom. At that point the kernel picks a victim and sends it SIGKILL. Nothing else frees memory fast enough. ## Reading the evidence The record lands in the kernel ring buffer. Start there. ```bash # Human-readable timestamps, filter for the OOM events dmesg -T | grep -i -E 'oom|out of memory|killed process' # Same story from the journal, kernel messages only journalctl -k --since "1 hour ago" | grep -i oom ``` A real global OOM event looks like this: ``` [Tue Jul 21 03:14:22 2026] node invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [Tue Jul 21 03:14:22 2026] Mem-Info: [Tue Jul 21 03:14:22 2026] Node 0 active_anon:3812044kB inactive_anon:12kB ... free:20964kB [Tue Jul 21 03:14:22 2026] Tasks state (memory values in pages): [Tue Jul 21 03:14:22 2026] [ pid ] uid tgid total_vm rss ... oom_score_adj name [Tue Jul 21 03:14:22 2026] [ 1187] 1000 1187 1284410 961242 ... 0 node [Tue Jul 21 03:14:22 2026] [ 902] 0 902 214880 4102 ... 0 sshd [Tue Jul 21 03:14:22 2026] Out of memory: Killed process 1187 (node) total-vm:5137640kB, anon-rss:3844968kB, file-rss:0kB, shmem-rss:0kB ``` Read it top to bottom. The first line names who triggered the reclaim (`node invoked oom-killer`). `Mem-Info` shows how little was free. The `Tasks state` table is the important part: every process with its `rss` (resident set size, the real physical pages it holds) and its `oom_score_adj`. The final `Out of memory: Killed process` line names the victim and how much it was holding when it died. In this case `node` was sitting on roughly 3.8 GB of anonymous memory, so it was both the biggest user and the obvious target. Sort that RSS column and you usually have your culprit without any other tooling. When you need to go the other way and catch the growth before the kill, see [find what's using your memory](/blog/find-what-is-using-memory). ## How the victim is chosen The kernel does not kill randomly and it does not always kill the thing that asked for the page. It scores every eligible task and kills the one with the highest badness. The score is driven mostly by memory footprint: a process using half of RAM is roughly twice as likely to be picked as one using a quarter. Recent memory is what matters, not virtual size, which is why RSS is the column that counts. You bias that decision with `oom_score_adj`, an integer from `-1000` to `+1000` per process. Add points to make a process a preferred victim, subtract to protect it. The live values are in `/proc`: ```bash # Current badness score and the tunable knob for a running pid cat /proc/1187/oom_score cat /proc/1187/oom_score_adj # Protect a critical daemon from selection (near-immune at -1000) echo -900 > /proc/1187/oom_score_adj ``` Setting `oom_score_adj` to `-1000` makes a task effectively unkillable, which is a foot-gun. Protect a database and you may hand the kill to sshd instead, locking yourself out of a box that is thrashing. Bias, do not exempt. ## Global OOM versus cgroup OOM This is the distinction that trips people up. There are two different out-of-memory conditions and they look almost identical in the log. **Global OOM:** the whole machine is out of memory. The kernel scans every process on the box and kills to save the system. This is the case above. **Cgroup OOM:** a single control group hit its own `memory.max` limit while the host still has plenty of RAM free. Under cgroup v2 each container or systemd unit lives in a memory cgroup with a hard cap. Cross it and only tasks inside that cgroup are candidates. The host never noticed. The dmesg line calls this out: ``` [Tue Jul 21 09:41:08 2026] Memory cgroup out of memory: Killed process 44120 (python) ... oom_score_adj:0 [Tue Jul 21 09:41:08 2026] memory: usage 524288kB, limit 524288kB, failcnt 219 ``` `Memory cgroup out of memory` plus a `usage == limit` line means the container ate its allowance, not the host's. That is the same mechanism behind [Kubernetes OOMKilled](/blog/fix-oomkilled) and a [Docker exit 137](/blog/fix-docker-oom-137): both are cgroup limits doing exactly what you told them to. The fix path is completely different from a global OOM, so read that first line before you touch host-level swap or overcommit. ## Tuning knobs For global pressure you have a few levers. Overcommit policy controls how generously the kernel hands out mappings. `vm.overcommit_memory=0` is the default heuristic. Setting it to `2` enforces a hard ceiling of `swap + overcommit_ratio%` of RAM, so allocations fail with ENOMEM instead of succeeding and triggering an OOM kill later: ```bash sysctl vm.overcommit_memory vm.overcommit_ratio # Enforce a real ceiling: allocations get honest failures, not surprise kills sysctl -w vm.overcommit_memory=2 sysctl -w vm.overcommit_ratio=80 ``` Adding swap gives the reclaim path somewhere to push cold anonymous pages, which buys headroom before the killer fires. It trades kills for latency, so size it deliberately rather than defaulting to none. For a per-service cap, set `MemoryMax` on a systemd unit so a runaway daemon gets its own cgroup OOM at a limit you chose instead of taking the whole host down. ## Prevention Tuning buys time. Fixing the cause is the actual work. Right-size limits from measured usage, not from a guess doubled for safety. A limit set far above real need means the process can balloon until it takes the host with it; a limit set too tight means routine spikes get killed. Watch steady-state RSS plus peak, then cap a bit above peak. Fix the leak. A process whose RSS only ever climbs is not being unlucky, it is holding references it should drop. The OOM killer is a symptom; the leak is the disease, and no sysctl fixes it. Set `oom_score_adj` for the daemons that must survive, and set it modestly. Keep sshd and your monitoring agent slightly protected so a bad night still leaves you a way in and a way to see. Bias the killer toward the workload that is safe to restart. ## The call we'd make When a process disappears without a trace, check `dmesg -T | grep -i oom` before anything else. The first log line tells you the whole shape of the problem: `Out of memory` alone means the host is exhausted, so look at overcommit, swap, and the biggest RSS in the task table. `Memory cgroup out of memory` means a container hit its own cap and the host is fine, so go fix the limit or the leak inside that unit. Then right-size the cap from real numbers and give your critical daemons a small negative `oom_score_adj` so the next shortage kills something restartable instead of your way in. For the wider playbook, keep the [Linux troubleshooting guide](/blog/linux-troubleshooting-guide) handy.