A practical method for reading free, top, and ps correctly so you attribute Linux memory to a real cause instead of guessing.
Someone pages you: "the box is out of memory." You run free -h, see free sitting at 200 MB, and panic. Nine times out of ten nothing is wrong. The other time, you need to know exactly which process or subsystem is holding the pages. This is how to tell the two apart without guessing.
$ free -h
total used free shared buff/cache available
Mem: 31Gi 8.2Gi 412Mi 1.1Gi 22Gi 21Gi
Swap: 8.0Gi 128Mi 7.9Gi
The trap is the free column. It shows 412 MiB and people assume the machine is starving. It is not. Linux deliberately fills unused RAM with page cache (recently read files, filesystem metadata) because idle RAM is wasted RAM. That cache is under buff/cache, and most of it is reclaimable the instant a process needs it.
The column that matters is available. It is the kernel's own estimate of how much memory a new workload can grab without swapping, and it already accounts for reclaimable cache. Here 21 GiB is available, so the box is healthy despite free looking tiny.
Rule of thumb: judge memory pressure by available, not free. If available is a small fraction of total and swap is climbing, then you have a real problem worth chasing.
Sort processes by memory and look at the top of the list:
$ ps aux --sort=-%mem | head -6
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
postgres 2417 1.2 18.4 9420112 5900312 ? Ss Jul18 42:11 postgres: writer
java 3810 6.0 12.1 8123400 3880220 ? Sl Jul19 88:02 java -Xmx4g -jar app.jar
redis 1902 0.4 3.2 1120044 1030880 ? Ssl Jul17 9:44 redis-server
prom 2201 2.1 2.8 2455010 905122 ? Ssl Jul18 30:15 prometheus
Two columns describe each process:
VSZ (virtual size): everything the process has mapped, including memory-mapped files it never touched, shared libraries, and reserved-but-unused heap. It is almost always huge and almost always misleading. Ignore VSZ for "who is using RAM."
RSS (resident set size): the pages actually in physical RAM right now. Closer to the truth, but it has one big flaw: RSS counts shared pages in full for every process that maps them. Twenty PHP workers sharing 200 MB of the interpreter each report that 200 MB in their RSS. Sum the column and you will "find" more memory used than the machine has.
Proportional set size (PSS) splits shared pages fairly. If a 200 MB library is shared by 20 workers, each is charged 10 MB. The per-process PSS values sum to something that actually reflects physical usage. The kernel exposes it per process:
$ grep -E '^(Rss|Pss|Shared|Private)' /proc/3810/smaps_rollup
Rss: 3880220 kB
Pss: 3120044 kB
Shared_Clean: 540210 kB
Shared_Dirty: 210880 kB
Private_Clean: 98110 kB
Private_Dirty: 3030120 kB
Here RSS is 3.7 GiB but PSS is 3.0 GiB; the difference is shared libraries counted against everyone. Private_Dirty (3.0 GiB) is the interesting number: pages this process alone owns and dirtied. Kill the process and that is what you get back. For a fleet-wide view, smem -tk -c "name pss rss" (from the smem package) reads smaps for every process and gives you a PSS-sorted table without the arithmetic.
In top (press M to sort by memory) the relevant fields are RES, SHR, and %MEM:
- RES is the same resident set as
ps RSS, shared pages included.
- SHR is the portion of RES that is shareable (libraries, mapped files, shared memory segments).
RES - SHR roughly approximates the private footprint, which is a decent quick proxy for PSS when you do not want to read smaps.
If a process shows RES 4 GiB and SHR 3.5 GiB, it is mostly touching shared pages and is not your leak. A process with RES 4 GiB and SHR 40 MB is holding 4 GiB of private memory and deserves your attention.
Inside a container, free reports the host's totals, which is useless. The container's real limit and usage live in its cgroup. On cgroup v2:
$ cat /sys/fs/cgroup/memory.current
4187553792
$ cat /sys/fs/cgroup/memory.max
4294967296
$ grep -E '^(anon|file|slab|sock)' /sys/fs/cgroup/memory.stat
anon 3980210176
file 178954240
slab 24117248
sock 2097152
memory.current (about 3.9 GiB) against memory.max (4 GiB) tells you the container is near its limit and one bad allocation from the OOM killer. memory.stat breaks it down: anon is process heap and stacks, file is page cache the cgroup is charged for. A container getting OOM-killed with high anon is a genuine leak or an undersized limit. If yours is dying at the limit, see the Linux OOM killer for how that decision gets made.
Sometimes ps and top add up to far less than used, and no cgroup explains it. The kernel itself is holding memory in slab caches (dentries, inodes, network buffers). slabtop shows it:
$ sudo slabtop --once --sort=c | head -8
Active / Total Size (% used) : 3120.44M / 3350.10M (93.1%)
OBJS ACTIVE USE OBJ SIZE SLABS NAME
8912340 8900112 99% 0.19K 424397 dentry
1204880 1201002 99% 1.05K 80325 ext4_inode_cache
980220 975003 99% 0.10K 25130 buffer_head
Millions of dentry objects usually means something is stat-ing or opening enormous directory trees (a backup job, a misbehaving find, a container churning files). This memory is reclaimable under pressure, but a runaway can still squeeze real workloads. cat /proc/meminfo and check Slab, SReclaimable, and SUnreclaim for the split that matters.
A machine can have "used" swap and be perfectly fine. What hurts is active swapping: pages moving in and out constantly because the working set no longer fits.
$ vmstat 2 4
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 2 6820112 210044 40122 880210 4820 6110 5200 6300 8100 12200 14 9 20 57 0
2 3 6910220 198100 39980 861200 5100 6402 4900 6800 8400 12900 12 8 18 62 0
Watch si and so (swap in / swap out per second). Sustained non-zero values in both, combined with high wa (I/O wait, 57-62% here), is thrashing. The CPU is idle because everything is blocked waiting on disk to page memory back and forth. That is when latency falls off a cliff. A one-time so spike during a backup is normal; continuous churn is not.
free -h: is available genuinely low, or does free just look scary? If available is healthy, stop, nothing is wrong.- If available is low, check swap with
vmstat 2. Sustained si/so plus high wa confirms real pressure.
ps aux --sort=-%mem | head to find candidates. Treat RSS as a hint, not a verdict.- Confirm the top suspect with
/proc/<pid>/smaps_rollup (PSS and Private_Dirty) or smem. Private_Dirty is the memory you reclaim by killing it.
- If processes do not add up to
used, check slabtop and /proc/meminfo for kernel/slab growth.
- In containers, ignore
free entirely and read memory.current / memory.max / memory.stat.
Trust available over free, and trust PSS over RSS. Most "out of memory" pages are page cache doing its job, and the fix is to close the ticket. When it is real, the honest number is Private_Dirty (or per-cgroup anon), because that is the memory that actually disappears when you kill the offender. Everything else is either shared, reclaimable, or a rounding error. For the broader playbook, see the Linux troubleshooting guide.
