Tracking Down High CPU on Linux

A box pages at 3am with CPU pinned. Before you reach for the reboot button, the goal is narrow and answerable: which process, and what kind of CPU time is it burning? Those two facts point at completely different fixes. This is the workflow I run every time.

For the wider picture (load average, memory pressure, disk saturation), see the Linux troubleshooting guide. Here we stay on CPU.

Step 1: find the process

Start where everyone starts, but read it properly. top sorted by CPU tells you the offender in seconds.

$ top -o %CPU
top - 03:14:22 up 41 days,  2:07,  1 user,  load average: 7.81, 6.02, 4.10
Tasks: 214 total,   3 running, 211 sleeping,   0 stopped,   0 zombie
%Cpu(s): 18.3 us, 71.2 sy,  0.0 ni,  6.1 id,  0.0 wa,  0.0 hi,  4.4 si,  0.0 st
MiB Mem :  15869.4 total,   4021.1 free,   8912.3 used,   2936.0 buff/cache

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  28471 app       20   0 4821140 612204  18932 R 388.4   3.8  91:22.14 api-server
   1042 root      20   0  912304  44120  12008 S  12.6   0.3  14:02.90 node_exporter

api-server at 388% (nearly four full cores) is your target. Note the summary line already tells a story: 71.2 sy means most of the machine's time is in the kernel, not application code. Hold that thought.

If top is too noisy, pidstat gives you a clean per-process rate over a fixed interval, which is easier to trust than an instantaneous snapshot.

$ pidstat 1 3
Linux 6.8.0-45-generic (web-03)   07/21/2026   _x86_64_   (4 CPU)

03:15:01      UID       PID    %usr %system  %guest    %CPU   CPU  Command
03:15:02     1001     28471   19.0    72.0    0.00   91.0     2   api-server
03:15:03     1001     28471   18.0    73.0    0.00   91.0     1   api-server
03:15:04     1001     28471   20.0    70.0    0.00   90.0     3   api-server

%usr vs %system split per process is the payoff here. htop (with F5 tree view and t for thread display) does the same job interactively if you prefer it.

Step 2: read the CPU breakdown

The %Cpu(s) line is the most under-used diagnostic on the box. Each field means something specific:

• us (user): time in application code. High user time is normal for a busy app. The fix lives in the code or its config.
• sy (system): time in the kernel on behalf of the process, syscalls, context switches, network stack. High sys usually means the app is hammering the kernel: excessive read/write, tiny buffers, lock contention, or a context-switch storm.
• wa (iowait): the CPU is idle waiting for disk or network storage. High iowait is not a CPU problem, it is a storage problem wearing a CPU costume. Adding cores will not help.
• si (softirq): soft interrupt handling, mostly network packet processing. High softirq points at network load or a single-queue NIC.
• st (steal): on a VM, time the hypervisor gave to other tenants while your vCPU was runnable. High steal means a noisy neighbour. You cannot fix it from inside the guest; resize or move the instance.

So the box above (18 us / 71 sy / 4 si) is not a hot application loop. It is a syscall or context-switch problem. If instead you saw 40 wa, you would stop looking at CPU entirely and go chase the disk. If you saw 30 st, you would open a ticket with your cloud provider. Symptoms that also show up as load average are worth cross-checking against diagnosing high load average.

Step 3: find the hot thread

A process at 388% is really several threads. You want the one actually burning cycles, because a 40-thread app rarely melts down evenly.

$ top -H -p 28471
    PID USER      PR  NI    VIRT    RES  S  %CPU  %MEM     TIME+ COMMAND
  28493 app       20   0 4821140 612204  R  98.7   3.8  22:14.02 worker-3
  28494 app       20   0 4821140 612204  R  97.9   3.8  21:58.71 worker-4
  28471 app       20   0 4821140 612204  S   1.1   3.8   0:44.10 api-server

Two worker threads are doing all the damage. Grab their thread IDs (TIDs) for the profiler. ps -T -p 28471 gives the same mapping non-interactively:

$ ps -T -p 28471 -o pid,tid,pcpu,comm --sort=-pcpu | head
    PID     TID %CPU COMMAND
  28471   28493 98.7 worker-3
  28471   28494 97.9 worker-4
  28471   28471  1.1 api-server

Step 4: profile what those threads are doing

For high user time, sample the call stacks with perf top. It shows you the hottest functions live, no restart needed.

$ sudo perf top -p 28471
Samples: 214K of event 'cpu-clock', 4000 Hz
Overhead  Shared Object        Symbol
  41.20%  api-server           [.] json_tokenize
  18.60%  libc-2.39.so         [.] __memmove_avx_unaligned
  11.04%  api-server           [.] regex_match_backtrack
   6.71%  api-server           [.] gc_mark_sweep

That is a smoking gun: a JSON tokenizer and a backtracking regex are eating 50%+ of the thread. For a persistent record you can turn into a flame graph, record and report:

$ sudo perf record -F 999 -g -p 28471 -- sleep 20
[ perf record: Captured and wrote 4.8 MB perf.data (19842 samples) ]
$ sudo perf report --stdio | head -20
# Overhead  Command      Symbol
    41.2%   worker-3     [.] json_tokenize
    11.0%   worker-3     [.] regex_match_backtrack

Feed that perf.data through Brendan Gregg's stackcollapse-perf.pl and flamegraph.pl and you get the visual that makes the hotspot obvious to anyone.

For high system time, the profiler is strace. Count syscalls to see what the process asks the kernel to do:

$ sudo strace -f -c -p 28471
strace: Process 28471 attached
^C
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 68.44    4.812301           6    782140           read
 22.10    1.554002           4    381004           futex
  8.90    0.625810           7     88201           write

782k read calls and 381k futex calls in a few seconds is the answer to the 71% sys time: tiny unbuffered reads plus lock contention on a futex. The fix is buffering and reducing lock churn, not more CPU.

When you suspect a context-switch storm, vmstat confirms it. Watch the cs (context switches) and in (interrupts) columns:

$ vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 411204  92100 300288    0    0     8    40  980 412088 18 71  6  0  0
 4  0      0 410880  92100 300288    0    0     0     0  944 408221 19 72  5  0  0

400k context switches per second is pathological. That maps straight to the futex churn strace found: threads fighting over a lock, waking and sleeping constantly.

Common causes, quick reference

• Busy loop / spin: 100% user on one thread, flat perf top on one function. Usually a missing sleep or a poll loop with no backoff.
• GC pressure: perf top heavy in gc_* or your runtime's collector. Tune heap size or reduce allocation rate.
• Regex / JSON hotspots: pathological backtracking or parsing giant payloads. Cap input size, precompile patterns, or swap the parser.
• Context-switch storm: high sys, huge cs in vmstat, futex in strace. Lock contention or too many runnable threads.
• iowait masquerading: high wa, low us/sy. Not CPU at all, go profile the disk.

The call we'd make

Read the %Cpu(s) line first. It decides everything. High user means profile the code with perf and fix the hot function. High system means count syscalls with strace and cut the kernel chatter, usually buffering or lock contention. High iowait means walk away from CPU and go chase storage. High steal means it is the hypervisor's fault, resize or migrate. Then drill from process to thread to function so your fix lands on the exact line that matters, not a guess. Nine times out of ten the summary line and one perf top tell you everything before the pager stops buzzing.