# Diagnosing High Load Average on Linux URL: https://www.devopsness.com/blog/fix-high-load-average Author: Kiril Urbonas Published: 2026-07-20 Updated: 2026-07-20 Categories: Linux Tags: Linux, Performance, Troubleshooting, Sysadmin, Load Average Summary: Load average counts more than CPU demand, so a high number needs a diagnostic path before you reach for a fix. --- You get paged. A box is showing a load average of 40 and the first instinct is to assume the CPUs are pinned. You SSH in, run `top`, and the CPU is 85% idle. Now what? This is the moment where a lot of people stall, because they were taught that load equals CPU. On Linux it does not. ## What load average actually measures On most Unix systems, load average tracks tasks that want the CPU. On Linux it counts two things: tasks that are runnable (in the run queue or currently running) **and** tasks stuck in uninterruptible sleep, the `D` state. That second group is usually processes blocked waiting on I/O: a slow disk, a hung NFS mount, a saturated storage controller. That design decision is why Linux load is a measure of demand for resources, not just CPU. A machine with idle processors can carry a load of 30 because 30 processes are all sitting in `D` waiting for a NAS to answer. The three numbers from `uptime` are exponentially damped moving averages over 1, 5, and 15 minutes: ```bash $ uptime 14:22:07 up 61 days, 3:19, 2 users, load average: 38.42, 24.17, 11.90 ``` Read the trend, not the snapshot. Here the 1-minute figure is far above the 15-minute one, so load is climbing fast. If the 15-minute number were highest, the spike would already be receding and you might be chasing a ghost. ## Load relative to core count A load of 8 is a fire on a 4-core box and a nap on a 64-core one. Always divide by the number of logical CPUs: ```bash $ nproc 16 ``` Rough rule: load equal to core count means the machine is fully committed with nothing queued. Load well above core count means work is backing up. With `nproc` at 16, that 38.42 is roughly 2.4x oversubscribed, which is worth investigating but not yet catastrophic if it is I/O wait rather than CPU starvation. ## The diagnostic path ### Step 1: confirm the trend with top `top` gives you the load line plus the CPU state breakdown. The field that matters most here is `wa`, the percentage of time CPUs sat idle waiting on I/O. ``` top - 14:23:11 up 61 days, 3:20, 2 users, load average: 39.10, 26.44, 13.02 Tasks: 412 total, 2 running, 118 sleeping, 0 stopped, 0 zombie %Cpu(s): 6.1 us, 3.4 sy, 0.0 ni, 12.5 id, 77.4 wa, 0.2 hi, 0.4 si, 0.0 st ``` `us` + `sy` is around 10%, `id` is 12%, and `wa` is a screaming 77%. That is the signature of an I/O-bound machine. If instead `us` were near 90% and `wa` near zero, you would be CPU-bound and reading a different story. ### Step 2: split runnable vs blocked with vmstat `vmstat 1` samples every second. Watch two columns: `r` (processes runnable, wanting CPU) and `b` (processes in uninterruptible sleep, blocked on I/O). ```bash $ vmstat 1 5 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 36 0 481232 10240 2934112 0 0 40448 8 5120 8241 6 3 13 78 0 1 34 0 479008 10240 2935200 0 0 39120 16 4998 8033 5 4 12 79 0 3 38 0 477640 10240 2936880 0 0 41232 24 5310 8477 7 3 11 79 0 ``` A high `b` column with a modest `r` column is the confirmation: this load is I/O-bound. Notice `bi` (blocks in) is huge while `bo` (blocks out) is tiny, so the pain is on the read side. If `r` were the tall column and `b` near zero, the story flips to CPU-bound. ### Step 3: per-core view and iowait with mpstat Aggregate CPU numbers hide single-threaded bottlenecks. `mpstat -P ALL` shows each core: ```bash $ mpstat -P ALL 1 1 Linux 5.15.0-101-generic 07/20/2026 _x86_64_ (16 CPU) 14:24:02 CPU %usr %nice %sys %iowait %irq %soft %idle 14:24:02 all 6.0 0.0 3.3 77.9 0.2 0.4 12.2 14:24:02 0 5.9 0.0 3.0 79.1 0.0 0.3 11.7 14:24:02 1 6.2 0.0 3.4 78.0 0.2 0.5 11.7 14:24:02 3 90.1 0.0 8.0 0.0 0.0 1.9 0.0 ``` Most cores are buried in `%iowait`, but core 3 is pinned at 90% user. That is a hint that at least one CPU-bound process is riding along with the I/O storm. In a pure CPU-bound incident you would see several cores near 100% `%usr` with `%iowait` flat at zero. ### Step 4: name the D-state processes Now find the processes actually stuck. Anything in `D` is contributing to load while doing no useful CPU work: ```bash $ ps -eo state,pid,ppid,cmd | grep '^D' D 8842 8801 /usr/bin/rsync -a /mnt/nfs/backup/ /data/restore/ D 8843 8801 /usr/bin/rsync -a /mnt/nfs/backup/ /data/restore/ D 11207 1 [nfsiod] D 12934 12900 postgres: checkpointer ``` There is the culprit: parallel `rsync` jobs hammering an NFS mount, dragging Postgres' checkpointer into the wait along with them. Pair this with `pidstat -d 1` to see per-process I/O throughput: ```bash $ pidstat -d 1 3 14:25:10 UID PID kB_rd/s kB_wr/s kB_ccwr/s Command 14:25:10 1000 8842 38210.0 12.0 0.0 rsync 14:25:10 1000 8843 120.0 9840.0 0.0 rsync 14:25:10 999 12934 0.0 1024.0 0.0 postgres ``` ## Splitting the cause and what to do **I/O-bound (high `b`, high `wa`, processes in `D`).** This is our case. Trace the storage backend. For NFS, check `nfsstat -c` and mount latency; a hung server will park every client process in `D`. For local disk, `iostat -x 1` and look at `%util` near 100% with rising `await`. Next steps: throttle the offending job (`ionice -c3`, or `--bwlimit` for rsync), fix or fail over the slow backend, or move hot data off a saturated volume. You cannot fix I/O wait by adding CPUs. **CPU-bound (high `r`, `wa` near zero, cores near 100% `%usr`).** The run queue is longer than the number of cores. Use `pidstat 1` or `top` to find the busy PIDs, then profile with `perf top` to see where cycles go. Next steps: scale out, optimize the hot code path, or cap the process with `cpulimit`/cgroups. See [debugging high CPU on Linux](/blog/debug-high-cpu-linux) for the deeper hunt. **Too many threads (high `r`, moderate CPU, huge `cs` in vmstat).** Sometimes load is high not from real work but from thousands of threads thrashing the scheduler. A runaway thread pool or a fork bomb shows a large `r` and a context-switch (`cs`) count in the tens of thousands. Check `ps -eLf | wc -l` for thread count and inspect the offending service's pool config. Next step: cap the pool size rather than adding hardware. ## The call we'd make High load with idle CPU almost always means I/O. Do not reboot and do not add cores on reflex. Run `uptime` for the trend, `vmstat 1` to see whether `r` or `b` is tall, `mpstat -P ALL` to catch `%iowait` and any single hot core, then `ps -eo state,... | grep '^D'` to name the blocked processes. In this incident the fix was `--bwlimit` on the rsync jobs and a hard remount of the flaky NFS export; load fell back under `nproc` within two minutes. When you need the broader map of tools and symptoms, keep the [Linux troubleshooting guide](/blog/linux-troubleshooting-guide) open in the next tab.