Find What's Eating Your RAM on Linux

Someone pages you: "the box is out of memory." You run free -h, see free sitting at 200 MB, and panic. Nine times out of ten nothing is wrong. The other time, you need to know exactly which process or subsystem is holding the pages. This is how to tell the two apart without guessing.

Start with free -h and read the right column

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            31Gi        8.2Gi       412Mi       1.1Gi        22Gi        21Gi
Swap:          8.0Gi        128Mi       7.9Gi

The trap is the free column. It shows 412 MiB and people assume the machine is starving. It is not. Linux deliberately fills unused RAM with page cache (recently read files, filesystem metadata) because idle RAM is wasted RAM. That cache is under buff/cache, and most of it is reclaimable the instant a process needs it.

The column that matters is available. It is the kernel's own estimate of how much memory a new workload can grab without swapping, and it already accounts for reclaimable cache. Here 21 GiB is available, so the box is healthy despite free looking tiny.

Rule of thumb: judge memory pressure by available, not free. If available is a small fraction of total and swap is climbing, then you have a real problem worth chasing.

Per-process the honest way

Sort processes by memory and look at the top of the list:

$ ps aux --sort=-%mem | head -6
USER       PID %CPU %MEM    VSZ    RSS TTY   STAT START   TIME COMMAND
postgres  2417  1.2 18.4 9420112 5900312 ?   Ss   Jul18  42:11 postgres: writer
java      3810  6.0 12.1 8123400 3880220 ?   Sl   Jul19  88:02 java -Xmx4g -jar app.jar
redis     1902  0.4  3.2 1120044 1030880 ?   Ssl  Jul17   9:44 redis-server
prom      2201  2.1  2.8 2455010  905122 ?   Ssl  Jul18  30:15 prometheus

Two columns describe each process:

VSZ (virtual size): everything the process has mapped, including memory-mapped files it never touched, shared libraries, and reserved-but-unused heap. It is almost always huge and almost always misleading. Ignore VSZ for "who is using RAM."

RSS (resident set size): the pages actually in physical RAM right now. Closer to the truth, but it has one big flaw: RSS counts shared pages in full for every process that maps them. Twenty PHP workers sharing 200 MB of the interpreter each report that 200 MB in their RSS. Sum the column and you will "find" more memory used than the machine has.

PSS: stop double-counting

Proportional set size (PSS) splits shared pages fairly. If a 200 MB library is shared by 20 workers, each is charged 10 MB. The per-process PSS values sum to something that actually reflects physical usage. The kernel exposes it per process:

$ grep -E '^(Rss|Pss|Shared|Private)' /proc/3810/smaps_rollup
Rss:             3880220 kB
Pss:             3120044 kB
Shared_Clean:     540210 kB
Shared_Dirty:     210880 kB
Private_Clean:     98110 kB
Private_Dirty:   3030120 kB

Here RSS is 3.7 GiB but PSS is 3.0 GiB; the difference is shared libraries counted against everyone. Private_Dirty (3.0 GiB) is the interesting number: pages this process alone owns and dirtied. Kill the process and that is what you get back. For a fleet-wide view, smem -tk -c "name pss rss" (from the smem package) reads smaps for every process and gives you a PSS-sorted table without the arithmetic.

What top's columns actually mean

In top (press M to sort by memory) the relevant fields are RES, SHR, and %MEM:

• RES is the same resident set as ps RSS, shared pages included.
• SHR is the portion of RES that is shareable (libraries, mapped files, shared memory segments).
• RES - SHR roughly approximates the private footprint, which is a decent quick proxy for PSS when you do not want to read smaps.

If a process shows RES 4 GiB and SHR 3.5 GiB, it is mostly touching shared pages and is not your leak. A process with RES 4 GiB and SHR 40 MB is holding 4 GiB of private memory and deserves your attention.

Containers live in cgroups, not free

Inside a container, free reports the host's totals, which is useless. The container's real limit and usage live in its cgroup. On cgroup v2:

$ cat /sys/fs/cgroup/memory.current
4187553792
$ cat /sys/fs/cgroup/memory.max
4294967296
$ grep -E '^(anon|file|slab|sock)' /sys/fs/cgroup/memory.stat
anon 3980210176
file 178954240
slab 24117248
sock 2097152

memory.current (about 3.9 GiB) against memory.max (4 GiB) tells you the container is near its limit and one bad allocation from the OOM killer. memory.stat breaks it down: anon is process heap and stacks, file is page cache the cgroup is charged for. A container getting OOM-killed with high anon is a genuine leak or an undersized limit. If yours is dying at the limit, see the Linux OOM killer for how that decision gets made.

When the culprit is the kernel

Sometimes ps and top add up to far less than used, and no cgroup explains it. The kernel itself is holding memory in slab caches (dentries, inodes, network buffers). slabtop shows it:

$ sudo slabtop --once --sort=c | head -8
 Active / Total Size (% used)  : 3120.44M / 3350.10M (93.1%)
  OBJS ACTIVE  USE OBJ SIZE  SLABS   NAME
 8912340 8900112 99%  0.19K  424397  dentry
 1204880 1201002 99%  1.05K   80325  ext4_inode_cache
  980220  975003 99%  0.10K   25130  buffer_head

Millions of dentry objects usually means something is stat-ing or opening enormous directory trees (a backup job, a misbehaving find, a container churning files). This memory is reclaimable under pressure, but a runaway can still squeeze real workloads. cat /proc/meminfo and check Slab, SReclaimable, and SUnreclaim for the split that matters.

Swap thrashing: the smoking gun

A machine can have "used" swap and be perfectly fine. What hurts is active swapping: pages moving in and out constantly because the working set no longer fits.

$ vmstat 2 4
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  2 6820112 210044  40122 880210 4820 6110  5200  6300 8100 12200 14 9 20 57 0
 2  3 6910220 198100  39980 861200 5100 6402  4900  6800 8400 12900 12 8 18 62 0

Watch si and so (swap in / swap out per second). Sustained non-zero values in both, combined with high wa (I/O wait, 57-62% here), is thrashing. The CPU is idle because everything is blocked waiting on disk to page memory back and forth. That is when latency falls off a cliff. A one-time so spike during a backup is normal; continuous churn is not.

A method that actually attributes memory

1. free -h: is available genuinely low, or does free just look scary? If available is healthy, stop, nothing is wrong.
2. If available is low, check swap with vmstat 2. Sustained si/so plus high wa confirms real pressure.
3. ps aux --sort=-%mem | head to find candidates. Treat RSS as a hint, not a verdict.
4. Confirm the top suspect with /proc/<pid>/smaps_rollup (PSS and Private_Dirty) or smem. Private_Dirty is the memory you reclaim by killing it.
5. If processes do not add up to used, check slabtop and /proc/meminfo for kernel/slab growth.
6. In containers, ignore free entirely and read memory.current / memory.max / memory.stat.

The call we'd make

Trust available over free, and trust PSS over RSS. Most "out of memory" pages are page cache doing its job, and the fix is to close the ticket. When it is real, the honest number is Private_Dirty (or per-cgroup anon), because that is the memory that actually disappears when you kill the offender. Everything else is either shared, reclaimable, or a rounding error. For the broader playbook, see the Linux troubleshooting guide.