Diagnosing High Load Average on Linux

You get paged. A box is showing a load average of 40 and the first instinct is to assume the CPUs are pinned. You SSH in, run top, and the CPU is 85% idle. Now what? This is the moment where a lot of people stall, because they were taught that load equals CPU. On Linux it does not.

What load average actually measures

On most Unix systems, load average tracks tasks that want the CPU. On Linux it counts two things: tasks that are runnable (in the run queue or currently running) and tasks stuck in uninterruptible sleep, the D state. That second group is usually processes blocked waiting on I/O: a slow disk, a hung NFS mount, a saturated storage controller.

That design decision is why Linux load is a measure of demand for resources, not just CPU. A machine with idle processors can carry a load of 30 because 30 processes are all sitting in D waiting for a NAS to answer.

The three numbers from uptime are exponentially damped moving averages over 1, 5, and 15 minutes:

$ uptime
 14:22:07 up 61 days,  3:19,  2 users,  load average: 38.42, 24.17, 11.90

Read the trend, not the snapshot. Here the 1-minute figure is far above the 15-minute one, so load is climbing fast. If the 15-minute number were highest, the spike would already be receding and you might be chasing a ghost.

Load relative to core count

A load of 8 is a fire on a 4-core box and a nap on a 64-core one. Always divide by the number of logical CPUs:

$ nproc
16

Rough rule: load equal to core count means the machine is fully committed with nothing queued. Load well above core count means work is backing up. With nproc at 16, that 38.42 is roughly 2.4x oversubscribed, which is worth investigating but not yet catastrophic if it is I/O wait rather than CPU starvation.

The diagnostic path

Step 1: confirm the trend with top

top gives you the load line plus the CPU state breakdown. The field that matters most here is wa, the percentage of time CPUs sat idle waiting on I/O.

top - 14:23:11 up 61 days,  3:20,  2 users,  load average: 39.10, 26.44, 13.02
Tasks: 412 total,   2 running, 118 sleeping,   0 stopped,   0 zombie
%Cpu(s):  6.1 us,  3.4 sy,  0.0 ni, 12.5 id, 77.4 wa,  0.2 hi,  0.4 si,  0.0 st

us + sy is around 10%, id is 12%, and wa is a screaming 77%. That is the signature of an I/O-bound machine. If instead us were near 90% and wa near zero, you would be CPU-bound and reading a different story.

Step 2: split runnable vs blocked with vmstat

vmstat 1 samples every second. Watch two columns: r (processes runnable, wanting CPU) and b (processes in uninterruptible sleep, blocked on I/O).

$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2 36      0 481232  10240 2934112    0    0 40448     8 5120 8241  6  3 13 78  0
 1 34      0 479008  10240 2935200    0    0 39120    16 4998 8033  5  4 12 79  0
 3 38      0 477640  10240 2936880    0    0 41232    24 5310 8477  7  3 11 79  0

A high b column with a modest r column is the confirmation: this load is I/O-bound. Notice bi (blocks in) is huge while bo (blocks out) is tiny, so the pain is on the read side. If r were the tall column and b near zero, the story flips to CPU-bound.

Step 3: per-core view and iowait with mpstat

Aggregate CPU numbers hide single-threaded bottlenecks. mpstat -P ALL shows each core:

$ mpstat -P ALL 1 1
Linux 5.15.0-101-generic  07/20/2026  _x86_64_  (16 CPU)

14:24:02  CPU  %usr %nice  %sys %iowait  %irq  %soft %idle
14:24:02  all   6.0  0.0   3.3    77.9    0.2   0.4   12.2
14:24:02    0    5.9  0.0   3.0    79.1    0.0   0.3   11.7
14:24:02    1    6.2  0.0   3.4    78.0    0.2   0.5   11.7
14:24:02    3   90.1  0.0   8.0     0.0    0.0   1.9    0.0

Most cores are buried in %iowait, but core 3 is pinned at 90% user. That is a hint that at least one CPU-bound process is riding along with the I/O storm. In a pure CPU-bound incident you would see several cores near 100% %usr with %iowait flat at zero.

Step 4: name the D-state processes

Now find the processes actually stuck. Anything in D is contributing to load while doing no useful CPU work:

$ ps -eo state,pid,ppid,cmd | grep '^D'
D  8842  8801 /usr/bin/rsync -a /mnt/nfs/backup/ /data/restore/
D  8843  8801 /usr/bin/rsync -a /mnt/nfs/backup/ /data/restore/
D 11207     1 [nfsiod]
D 12934 12900 postgres: checkpointer

There is the culprit: parallel rsync jobs hammering an NFS mount, dragging Postgres' checkpointer into the wait along with them. Pair this with pidstat -d 1 to see per-process I/O throughput:

$ pidstat -d 1 3
14:25:10   UID    PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
14:25:10  1000   8842  38210.0      12.0      0.0    rsync
14:25:10  1000   8843    120.0    9840.0      0.0    rsync
14:25:10   999  12934      0.0    1024.0      0.0    postgres

Splitting the cause and what to do

I/O-bound (high b, high wa, processes in D). This is our case. Trace the storage backend. For NFS, check nfsstat -c and mount latency; a hung server will park every client process in D. For local disk, iostat -x 1 and look at %util near 100% with rising await. Next steps: throttle the offending job (ionice -c3, or --bwlimit for rsync), fix or fail over the slow backend, or move hot data off a saturated volume. You cannot fix I/O wait by adding CPUs.

CPU-bound (high r, wa near zero, cores near 100% %usr). The run queue is longer than the number of cores. Use pidstat 1 or top to find the busy PIDs, then profile with perf top to see where cycles go. Next steps: scale out, optimize the hot code path, or cap the process with cpulimit/cgroups. See debugging high CPU on Linux for the deeper hunt.

Too many threads (high r, moderate CPU, huge cs in vmstat). Sometimes load is high not from real work but from thousands of threads thrashing the scheduler. A runaway thread pool or a fork bomb shows a large r and a context-switch (cs) count in the tens of thousands. Check ps -eLf | wc -l for thread count and inspect the offending service's pool config. Next step: cap the pool size rather than adding hardware.

The call we'd make

High load with idle CPU almost always means I/O. Do not reboot and do not add cores on reflex. Run uptime for the trend, vmstat 1 to see whether r or b is tall, mpstat -P ALL to catch %iowait and any single hot core, then ps -eo state,... | grep '^D' to name the blocked processes. In this incident the fix was --bwlimit on the rsync jobs and a hard remount of the flaky NFS export; load fell back under nproc within two minutes. When you need the broader map of tools and symptoms, keep the Linux troubleshooting guide open in the next tab.