Linux Memory Pressure: Reading PSI Before the OOM Killer Reads You

Pressure Stall Information (PSI) tells you how much time tasks spend stalled waiting for memory, which is the signal free memory and load average miss. Read /proc/pressure/memory: a rising some average means contention is starting, and a sustained non-zero full means everything is stalled and an OOM kill is close. Alert on both.

The OOM killer is the worst way to find out you're low on memory: it picks a process, kills it, and leaves you reconstructing what happened from a terse kernel log line. By then the incident already happened. The classic metrics don't warn you, because "free memory" looks fine right up until it doesn't; Linux uses spare memory for page cache. PSI measures the system struggling for memory, with enough lead time to act.

Why "free memory" misleads

On a healthy Linux box, free memory is usually near zero, and that's correct. The kernel fills unused RAM with page cache (recently-read files) because idle RAM is wasted RAM. That cache is reclaimable on demand, so "low free memory" is normal and says nothing about pressure.

The number that matters isn't how much memory is free; it's how much time the system spends stalled waiting for memory: blocked on reclaim, thrashing the page cache, swapping. That's exactly what PSI measures.

Reading /proc/pressure/memory

cat /proc/pressure/memory
some avg10=4.32 avg60=2.10 avg300=0.95 total=128934821
full avg10=1.20 avg60=0.55 avg300=0.20 total=43219003

The avg fields are percentages of wall-clock time over 10, 60 and 300-second windows. total is the cumulative stall time in microseconds, which is what you graph as a rate. PSI landed in kernel 4.20; if the file is missing, the kernel was built without CONFIG_PSI, and kernels built with CONFIG_PSI_DEFAULT_DISABLED need psi=1 on the kernel command line.

• some: the fraction of time at least one task was stalled waiting for memory. A leading indicator: the system is starting to work for memory.
• full: the fraction of time all non-idle tasks were stalled simultaneously. The kernel docs treat time in this state as thrashing: nothing makes progress because everyone is waiting on memory.

some avg10=4.32 means: over the last 10 seconds, 4.3% of the time something was stalled on memory. Rising some is your early warning. Non-trivial full means you're already in pain and an OOM kill may be near.

Why this beats load average and free memory

Load average lumps everything together and includes CPU-bound and I/O-bound tasks; it can't tell you why tasks are runnable-but-waiting. Free memory tells you about page cache, not pressure. PSI gives you a direct, attributable, time-based measure of memory contention specifically. It moves before the OOM killer fires, which is the entire point.

Per-cgroup pressure (the container angle)

Under cgroup v2, each cgroup exposes its own memory.pressure, cpu.pressure and io.pressure files in the same format, so you can attribute memory pressure to a specific container or service instead of the whole node:

cat /sys/fs/cgroup/system.slice/myapp.service/memory.pressure
some avg10=22.1 avg60=15.4 avg300=8.9 total=...

This is how you find which workload is causing node-level pressure. The node's /proc/pressure/memory says "something's struggling"; the per-cgroup file says "it's myapp, and it's stalled 22% of the last 10 seconds." On Kubernetes nodes this separates a noisy-neighbor problem from a genuinely under-provisioned node. It only works on cgroup v2 hosts, which is one more reason to finish the move covered in the kubelet cgroup v1 deprecation, and what cgroup v2 limits actually constrain explains how memory.high throttling shows up here as pressure.

Wiring it into alerts

We alert on sustained some and any meaningful full. The node_exporter pressure collector (on by default, kernel 4.20+) exposes the total counters as seconds, so rate() gives you the fraction of time stalled:

groups:
  - name: memory-pressure
    rules:
      - alert: MemoryPressureRising
        expr: rate(node_pressure_memory_waiting_seconds_total[5m]) > 0.10
        for: 5m
        labels: {severity: warning}
      - alert: MemoryThrashing
        expr: rate(node_pressure_memory_stalled_seconds_total[1m]) > 0.05
        for: 2m
        labels: {severity: critical}

memory_waiting maps to some and memory_stalled to full. Our starting thresholds are 10% some over five minutes for a warning and 5% full for a page. They are workload-specific: batch jobs tolerate more pressure than latency-sensitive services. The shape holds anyway: warn on rising some (you have time), page on full (you don't).

For sub-second reaction without a scrape interval, the kernel offers triggers: write some 150000 1000000 (150ms of stall in a 1s window) to /proc/pressure/memory, then poll() the descriptor. Windows run from 500ms to 10s, and unprivileged users must use multiples of 2s. This is how userspace OOM handlers act before the kernel does.

Acting on the signal

PSI is only useful if it drives an action with enough lead time:

• Autoscaling trigger: scale out or add a node when some avg60 crosses a threshold, rather than waiting for OOM kills to signal capacity shortage.
• Backpressure: a memory-pressure-aware service can shed load or pause intake when its own cgroup pressure rises, instead of growing until it's killed.
• Investigation: rising pressure on one cgroup points you straight at the leaking or under-limited workload.

Questions people ask

What is a good PSI memory threshold to alert on?

There is no universal number, because batch jobs tolerate stalls that would hurt a latency-sensitive API. Our starting point: warn when some stays above 10% for five minutes, page when full exceeds 5%. Watch a week of your own baseline first, then tighten. Any sustained full on a web tier deserves attention, since it means every runnable task was waiting on memory.

What is the difference between PSI some and full?

some is the share of time at least one task was stalled waiting for memory while others may still run. full is the share of time all non-idle tasks were stalled at once, which the kernel docs describe as thrashing. some is the early warning; full means throughput is already lost. At system level CPU has no meaningful full, but memory and I/O do.

Why is /proc/pressure/memory missing?

The kernel predates 4.20, was built without CONFIG_PSI, or was built with CONFIG_PSI_DEFAULT_DISABLED, in which case adding psi=1 to the kernel command line and rebooting turns it on. Check with ls /proc/pressure. Per-cgroup memory.pressure files additionally need cgroup v2 mounted, so a host still on cgroup v1 shows only the system-wide file.

The mental shift

Stop asking "how much memory is free?" It's nearly always near zero and tells you nothing. Start asking "how much time are we stalled waiting for memory?" PSI answers that directly, per-cgroup, with a leading some signal and a critical full signal. When the OOM killer does fire anyway, fixing out-of-memory kills on Linux covers the post-mortem. Wire PSI into your alerts and autoscaling and you trade the OOM killer's after-the-fact verdict for a warning you can act on while there's still time.

Sources

• Linux kernel docs: PSI, Pressure Stall Information
• Linux kernel docs: Control Group v2
• Prometheus node_exporter (pressure collector)