# How to Fix OOMKilled (Exit Code 137) in Kubernetes

URL: https://www.devopsness.com/blog/fix-oomkilled
Author: Kiril Urbonas
Published: 2026-07-09
Updated: 2026-07-09
Categories: DevOps
Tags: Kubernetes, DevOps, Performance, Debugging, Operations

Summary: A pod dies, restarts, dies again, and the events say OOMKilled. Here is what exit code 137 actually means and how to stop the loop for good.

---
# How to Fix OOMKilled (Exit Code 137) in Kubernetes

The first time I chased this one, a payments worker kept restarting every four minutes. Logs cut off mid-request with nothing useful, CPU looked fine, and the only clue was a status field that read `OOMKilled`. No stack trace, no panic, just gone. That is the signature of exit code 137, and once you know what it is saying, the fix is usually mechanical.

## What OOMKilled and 137 actually mean

A container has a memory limit. When the process inside tries to use more than that limit, the Linux kernel's OOM (out of memory) killer steps in and sends the process a SIGKILL. SIGKILL is signal 9. When a process is terminated by a signal, its exit code is `128 + signal number`, so `128 + 9 = 137`. That is the whole story behind the number.

Kubernetes reads the exit code and the cgroup memory event, then labels the container `OOMKilled`. The container did not crash on its own; the kernel shot it because it crossed the line you drew in the pod spec. SIGKILL cannot be caught or handled, which is why you get no graceful shutdown and no last log line.

There are two flavors worth separating early. A **pod-level OOM** means one container blew past its own limit, and only that container dies. A **node-level OOM** means the whole node ran out of physical memory and the kernel starts killing processes to survive, which can take down pods that were behaving fine. They look similar in `kubectl` but the fix is different.

## Diagnosis: prove it before you touch anything

Start with describe. The `Last State` block is where the truth lives:

```bash
kubectl describe pod payments-worker-7f9c8 | grep -A5 "Last State"
```

```
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Thu, 09 Jul 2026 09:14:02
  Finished:     Thu, 09 Jul 2026 09:18:31
```

Now check actual usage against the limit. Metrics Server has to be installed for this:

```bash
kubectl top pod payments-worker-7f9c8 --containers
```

If the worker sits at 480Mi against a 512Mi limit all day, you have your answer without any deep forensics. For a leak, watch it climb over time instead of reading a single snapshot. A quick loop or a Grafana panel on `container_memory_working_set_bytes` tells you whether memory grows and never comes back down, or whether it just spikes under load.

To tell pod OOM from node OOM, look at the node:

```bash
kubectl describe node ip-10-0-3-14 | grep -A3 "MemoryPressure"
dmesg -T | grep -i "oom-kill"
```

If the node reports `MemoryPressure: True` and `dmesg` shows the kernel killing processes across several pods, you are dealing with a node that is oversubscribed, not a single greedy container.

## Requests, limits, and why QoS matters here

This is the part people skip, and it is the part that decides who dies first. A **request** is what the scheduler reserves for your pod when placing it on a node. A **limit** is the hard ceiling the kernel enforces. OOMKilled is always about the limit.

Those two numbers also set your pod's QoS class, which is the eviction pecking order:

- **Guaranteed**: requests equal limits for every container. Last to be evicted under node pressure.
- **Burstable**: requests set, limits higher or absent. Killed after Guaranteed pods survive.
- **BestEffort**: no requests or limits at all. First against the wall when a node runs out.

If you run BestEffort pods next to important workloads, one noisy neighbor with a memory spike can trigger a node OOM that evicts your good pods. That alone is a reason to always set requests.

## Causes and the fix for each

**Limit set too low.** The most common case. Someone copied a 256Mi limit from a template and the app legitimately needs 700Mi. Right-size it from real usage, not a guess. Look at the peak working set over a representative window and set the limit above the peak with headroom:

```yaml
resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "768Mi"
```

**A real leak.** If memory only ever climbs and a restart resets it, no limit will save you; you are just buying time between kills. Grab a heap profile (pprof for Go, a heap dump for the JVM, `--inspect` for Node) and find what is holding references. Raising the limit here is treating a fever with a bigger blanket.

**Runtime heap unaware of the cgroup.** The JVM and Node classically read the host's total memory, not the container limit, then size their heap for a machine that does not exist. A JVM on a 512Mi limit might try to claim a 4GB heap and get killed instantly. Fix it with runtime flags that respect the cgroup:

```
# JVM (Java 11+)
-XX:MaxRAMPercentage=75.0

# Node
--max-old-space-size=384   # roughly 75% of a 512Mi limit
```

Leave headroom between the heap and the container limit for off-heap allocations, thread stacks, and metaspace. A heap sized at the full limit will still get OOMKilled.

**Traffic spike.** Memory tracks concurrency, and a burst pushes you over a limit that was fine at steady state. Cap in-flight work, add backpressure, and set limits against your realistic peak, not your average.

**No limits at all.** Feels safe until one pod eats the node and the kernel starts culling neighbors. Always set requests so the scheduler plans correctly, and set limits so a single container cannot take the whole node hostage.

For workloads whose appetite genuinely moves around, the Vertical Pod Autoscaler can recommend and set requests and limits from observed usage instead of leaving it to a stale template. Run it in `recommendation` mode first and read what it suggests before you let it write values automatically. It is far better than the number someone guessed eighteen months ago.

If you want the broader map of pod failure states, our [Kubernetes troubleshooting](/blog/kubernetes-error-troubleshooting-guide) guide covers the neighbors of 137: CrashLoopBackOff, ImagePullBackOff, and the rest.

## The call we'd make

Confirm it is really OOMKilled with `describe`, then look at `kubectl top` before you change a single value. If usage sits flat near the limit, right-size the limit with headroom and move on. If it climbs forever, stop raising the limit and go find the leak, because a bigger ceiling only changes how often the pager fires. Set requests on everything so QoS works in your favor, teach the JVM and Node their real limit, and reach for VPA when the workload's appetite refuses to hold still.