A field-tested workflow for finding which process burns your CPU and whether the time is user, system, or iowait.
A box pages at 3am with CPU pinned. Before you reach for the reboot button, the goal is narrow and answerable: which process, and what kind of CPU time is it burning? Those two facts point at completely different fixes. This is the workflow I run every time.
For the wider picture (load average, memory pressure, disk saturation), see the Linux troubleshooting guide. Here we stay on CPU.
Start where everyone starts, but read it properly. top sorted by CPU tells you the offender in seconds.
$ top -o %CPU
top - 03:14:22 up 41 days, 2:07, 1 user, load average: 7.81, 6.02, 4.10
Tasks: 214 total, 3 running, 211 sleeping, 0 stopped, 0 zombie
%Cpu(s): 18.3 us, 71.2 sy, 0.0 ni, 6.1 id, 0.0 wa, 0.0 hi, 4.4 si, 0.0 st
MiB Mem : 15869.4 total, 4021.1 free, 8912.3 used, 2936.0 buff/cache
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28471 app 20 0 4821140 612204 18932 R 388.4 3.8 91:22.14 api-server
1042 root 20 0 912304 44120 12008 S 12.6 0.3 14:02.90 node_exporter
api-server at 388% (nearly four full cores) is your target. Note the summary line already tells a story: 71.2 sy means most of the machine's time is in the kernel, not application code. Hold that thought.
If top is too noisy, pidstat gives you a clean per-process rate over a fixed interval, which is easier to trust than an instantaneous snapshot.
$ pidstat 1 3
Linux 6.8.0-45-generic (web-03) 07/21/2026 _x86_64_ (4 CPU)
03:15:01 UID PID %usr %system %guest %CPU CPU Command
03:15:02 1001 28471 19.0 72.0 0.00 91.0 2 api-server
03:15:03 1001 28471 18.0 73.0 0.00 91.0 1 api-server
03:15:04 1001 28471 20.0 70.0 0.00 90.0 3 api-server
%usr vs %system split per process is the payoff here. htop (with F5 tree view and t for thread display) does the same job interactively if you prefer it.
The %Cpu(s) line is the most under-used diagnostic on the box. Each field means something specific:
- us (user): time in application code. High user time is normal for a busy app. The fix lives in the code or its config.
- sy (system): time in the kernel on behalf of the process, syscalls, context switches, network stack. High sys usually means the app is hammering the kernel: excessive
read/write, tiny buffers, lock contention, or a context-switch storm.
- wa (iowait): the CPU is idle waiting for disk or network storage. High iowait is not a CPU problem, it is a storage problem wearing a CPU costume. Adding cores will not help.
- si (softirq): soft interrupt handling, mostly network packet processing. High softirq points at network load or a single-queue NIC.
- st (steal): on a VM, time the hypervisor gave to other tenants while your vCPU was runnable. High steal means a noisy neighbour. You cannot fix it from inside the guest; resize or move the instance.
So the box above (18 us / 71 sy / 4 si) is not a hot application loop. It is a syscall or context-switch problem. If instead you saw 40 wa, you would stop looking at CPU entirely and go chase the disk. If you saw 30 st, you would open a ticket with your cloud provider. Symptoms that also show up as load average are worth cross-checking against diagnosing high load average.
A process at 388% is really several threads. You want the one actually burning cycles, because a 40-thread app rarely melts down evenly.
$ top -H -p 28471
PID USER PR NI VIRT RES S %CPU %MEM TIME+ COMMAND
28493 app 20 0 4821140 612204 R 98.7 3.8 22:14.02 worker-3
28494 app 20 0 4821140 612204 R 97.9 3.8 21:58.71 worker-4
28471 app 20 0 4821140 612204 S 1.1 3.8 0:44.10 api-server
Two worker threads are doing all the damage. Grab their thread IDs (TIDs) for the profiler. ps -T -p 28471 gives the same mapping non-interactively:
$ ps -T -p 28471 -o pid,tid,pcpu,comm --sort=-pcpu | head
PID TID %CPU COMMAND
28471 28493 98.7 worker-3
28471 28494 97.9 worker-4
28471 28471 1.1 api-server
For high user time, sample the call stacks with perf top. It shows you the hottest functions live, no restart needed.
$ sudo perf top -p 28471
Samples: 214K of event 'cpu-clock', 4000 Hz
Overhead Shared Object Symbol
41.20% api-server [.] json_tokenize
18.60% libc-2.39.so [.] __memmove_avx_unaligned
11.04% api-server [.] regex_match_backtrack
6.71% api-server [.] gc_mark_sweep
That is a smoking gun: a JSON tokenizer and a backtracking regex are eating 50%+ of the thread. For a persistent record you can turn into a flame graph, record and report:
$ sudo perf record -F 999 -g -p 28471 -- sleep 20
[ perf record: Captured and wrote 4.8 MB perf.data (19842 samples) ]
$ sudo perf report --stdio | head -20
# Overhead Command Symbol
41.2% worker-3 [.] json_tokenize
11.0% worker-3 [.] regex_match_backtrack
Feed that perf.data through Brendan Gregg's stackcollapse-perf.pl and flamegraph.pl and you get the visual that makes the hotspot obvious to anyone.
For high system time, the profiler is strace. Count syscalls to see what the process asks the kernel to do:
$ sudo strace -f -c -p 28471
strace: Process 28471 attached
^C
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
68.44 4.812301 6 782140 read
22.10 1.554002 4 381004 futex
8.90 0.625810 7 88201 write
782k read calls and 381k futex calls in a few seconds is the answer to the 71% sys time: tiny unbuffered reads plus lock contention on a futex. The fix is buffering and reducing lock churn, not more CPU.
When you suspect a context-switch storm, vmstat confirms it. Watch the cs (context switches) and in (interrupts) columns:
$ vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 0 0 411204 92100 300288 0 0 8 40 980 412088 18 71 6 0 0
4 0 0 410880 92100 300288 0 0 0 0 944 408221 19 72 5 0 0
400k context switches per second is pathological. That maps straight to the futex churn strace found: threads fighting over a lock, waking and sleeping constantly.
- Busy loop / spin: 100% user on one thread, flat
perf top on one function. Usually a missing sleep or a poll loop with no backoff.
- GC pressure:
perf top heavy in gc_* or your runtime's collector. Tune heap size or reduce allocation rate.
- Regex / JSON hotspots: pathological backtracking or parsing giant payloads. Cap input size, precompile patterns, or swap the parser.
- Context-switch storm: high sys, huge
cs in vmstat, futex in strace. Lock contention or too many runnable threads.
- iowait masquerading: high wa, low us/sy. Not CPU at all, go profile the disk.
Read the %Cpu(s) line first. It decides everything. High user means profile the code with perf and fix the hot function. High system means count syscalls with strace and cut the kernel chatter, usually buffering or lock contention. High iowait means walk away from CPU and go chase storage. High steal means it is the hypervisor's fault, resize or migrate. Then drill from process to thread to function so your fix lands on the exact line that matters, not a guess. Nine times out of ten the summary line and one perf top tell you everything before the pager stops buzzing.
