# Tracking Down High CPU on Linux URL: https://www.devopsness.com/blog/debug-high-cpu-linux Author: Kiril Urbonas Published: 2026-07-21 Updated: 2026-07-21 Categories: Linux Tags: Linux, Performance, Troubleshooting, CPU, Sysadmin Summary: A field-tested workflow for finding which process burns your CPU and whether the time is user, system, or iowait. --- A box pages at 3am with CPU pinned. Before you reach for the reboot button, the goal is narrow and answerable: which process, and what *kind* of CPU time is it burning? Those two facts point at completely different fixes. This is the workflow I run every time. For the wider picture (load average, memory pressure, disk saturation), see the [Linux troubleshooting guide](/blog/linux-troubleshooting-guide). Here we stay on CPU. ## Step 1: find the process Start where everyone starts, but read it properly. `top` sorted by CPU tells you the offender in seconds. ```bash $ top -o %CPU top - 03:14:22 up 41 days, 2:07, 1 user, load average: 7.81, 6.02, 4.10 Tasks: 214 total, 3 running, 211 sleeping, 0 stopped, 0 zombie %Cpu(s): 18.3 us, 71.2 sy, 0.0 ni, 6.1 id, 0.0 wa, 0.0 hi, 4.4 si, 0.0 st MiB Mem : 15869.4 total, 4021.1 free, 8912.3 used, 2936.0 buff/cache PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28471 app 20 0 4821140 612204 18932 R 388.4 3.8 91:22.14 api-server 1042 root 20 0 912304 44120 12008 S 12.6 0.3 14:02.90 node_exporter ``` `api-server` at 388% (nearly four full cores) is your target. Note the summary line already tells a story: **71.2 sy** means most of the machine's time is in the kernel, not application code. Hold that thought. If `top` is too noisy, `pidstat` gives you a clean per-process rate over a fixed interval, which is easier to trust than an instantaneous snapshot. ```bash $ pidstat 1 3 Linux 6.8.0-45-generic (web-03) 07/21/2026 _x86_64_ (4 CPU) 03:15:01 UID PID %usr %system %guest %CPU CPU Command 03:15:02 1001 28471 19.0 72.0 0.00 91.0 2 api-server 03:15:03 1001 28471 18.0 73.0 0.00 91.0 1 api-server 03:15:04 1001 28471 20.0 70.0 0.00 90.0 3 api-server ``` `%usr` vs `%system` split per process is the payoff here. `htop` (with `F5` tree view and `t` for thread display) does the same job interactively if you prefer it. ## Step 2: read the CPU breakdown The `%Cpu(s)` line is the most under-used diagnostic on the box. Each field means something specific: - **us (user):** time in application code. High user time is normal for a busy app. The fix lives in the code or its config. - **sy (system):** time in the kernel on behalf of the process, syscalls, context switches, network stack. High sys usually means the app is hammering the kernel: excessive `read`/`write`, tiny buffers, lock contention, or a context-switch storm. - **wa (iowait):** the CPU is idle *waiting* for disk or network storage. High iowait is not a CPU problem, it is a storage problem wearing a CPU costume. Adding cores will not help. - **si (softirq):** soft interrupt handling, mostly network packet processing. High softirq points at network load or a single-queue NIC. - **st (steal):** on a VM, time the hypervisor gave to *other* tenants while your vCPU was runnable. High steal means a noisy neighbour. You cannot fix it from inside the guest; resize or move the instance. So the box above (18 us / 71 sy / 4 si) is not a hot application loop. It is a syscall or context-switch problem. If instead you saw 40 wa, you would stop looking at CPU entirely and go chase the disk. If you saw 30 st, you would open a ticket with your cloud provider. Symptoms that also show up as load average are worth cross-checking against [diagnosing high load average](/blog/fix-high-load-average). ## Step 3: find the hot thread A process at 388% is really several threads. You want the one actually burning cycles, because a 40-thread app rarely melts down evenly. ```bash $ top -H -p 28471 PID USER PR NI VIRT RES S %CPU %MEM TIME+ COMMAND 28493 app 20 0 4821140 612204 R 98.7 3.8 22:14.02 worker-3 28494 app 20 0 4821140 612204 R 97.9 3.8 21:58.71 worker-4 28471 app 20 0 4821140 612204 S 1.1 3.8 0:44.10 api-server ``` Two `worker` threads are doing all the damage. Grab their thread IDs (TIDs) for the profiler. `ps -T -p 28471` gives the same mapping non-interactively: ```bash $ ps -T -p 28471 -o pid,tid,pcpu,comm --sort=-pcpu | head PID TID %CPU COMMAND 28471 28493 98.7 worker-3 28471 28494 97.9 worker-4 28471 28471 1.1 api-server ``` ## Step 4: profile what those threads are doing For high **user** time, sample the call stacks with `perf top`. It shows you the hottest functions live, no restart needed. ```bash $ sudo perf top -p 28471 Samples: 214K of event 'cpu-clock', 4000 Hz Overhead Shared Object Symbol 41.20% api-server [.] json_tokenize 18.60% libc-2.39.so [.] __memmove_avx_unaligned 11.04% api-server [.] regex_match_backtrack 6.71% api-server [.] gc_mark_sweep ``` That is a smoking gun: a JSON tokenizer and a backtracking regex are eating 50%+ of the thread. For a persistent record you can turn into a flame graph, record and report: ```bash $ sudo perf record -F 999 -g -p 28471 -- sleep 20 [ perf record: Captured and wrote 4.8 MB perf.data (19842 samples) ] $ sudo perf report --stdio | head -20 # Overhead Command Symbol 41.2% worker-3 [.] json_tokenize 11.0% worker-3 [.] regex_match_backtrack ``` Feed that `perf.data` through Brendan Gregg's `stackcollapse-perf.pl` and `flamegraph.pl` and you get the visual that makes the hotspot obvious to anyone. For high **system** time, the profiler is `strace`. Count syscalls to see what the process asks the kernel to do: ```bash $ sudo strace -f -c -p 28471 strace: Process 28471 attached ^C % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 68.44 4.812301 6 782140 read 22.10 1.554002 4 381004 futex 8.90 0.625810 7 88201 write ``` 782k `read` calls and 381k `futex` calls in a few seconds is the answer to the 71% sys time: tiny unbuffered reads plus lock contention on a futex. The fix is buffering and reducing lock churn, not more CPU. When you suspect a context-switch storm, `vmstat` confirms it. Watch the `cs` (context switches) and `in` (interrupts) columns: ```bash $ vmstat 1 3 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 0 0 411204 92100 300288 0 0 8 40 980 412088 18 71 6 0 0 4 0 0 410880 92100 300288 0 0 0 0 944 408221 19 72 5 0 0 ``` 400k context switches per second is pathological. That maps straight to the futex churn strace found: threads fighting over a lock, waking and sleeping constantly. ## Common causes, quick reference - **Busy loop / spin:** 100% user on one thread, flat `perf top` on one function. Usually a missing sleep or a poll loop with no backoff. - **GC pressure:** `perf top` heavy in `gc_*` or your runtime's collector. Tune heap size or reduce allocation rate. - **Regex / JSON hotspots:** pathological backtracking or parsing giant payloads. Cap input size, precompile patterns, or swap the parser. - **Context-switch storm:** high sys, huge `cs` in vmstat, `futex` in strace. Lock contention or too many runnable threads. - **iowait masquerading:** high wa, low us/sy. Not CPU at all, go profile the disk. ## The call we'd make Read the `%Cpu(s)` line first. It decides everything. High **user** means profile the code with `perf` and fix the hot function. High **system** means count syscalls with `strace` and cut the kernel chatter, usually buffering or lock contention. High **iowait** means walk away from CPU and go chase storage. High **steal** means it is the hypervisor's fault, resize or migrate. Then drill from process to thread to function so your fix lands on the exact line that matters, not a guess. Nine times out of ten the summary line and one `perf top` tell you everything before the pager stops buzzing.