Debugging Latency with eBPF: bpftrace One-Liners That Find It
When a service is slow and every dashboard looks green, bpftrace lets you watch the kernel directly. These one-liners found our tail latency.
Key takeaways
- When application traces stop at time spent in the handler, use bpftrace to watch the kernel.
- Count syscalls to find the suspect, histogram paired enter and exit probes to expose a bimodal tail, then separate disk from scheduling delay with block and run-queue probes.
- Use bcc offcputime for blocked stacks.
On this page
To find hidden latency with eBPF, run four bpftrace probes in order: count syscalls for the slow process, histogram one syscall's enter-to-exit time to expose a bimodal tail, then compare block I/O latency against run-queue latency to tell a slow disk from CPU starvation. Finish with bcc's offcputime to see exactly what the process blocked on.
Verified against
bpftrace 0.27.1 (released 2026-10-24) and bcc 0.37.0, against the bpftrace one-liner tutorial and migration guide and the bcc offcputime examples. Checked October 2026.
A service was returning p99 responses near a second while p50 sat in the low milliseconds. Classic tail latency. The application traces showed the time was spent "in the handler," which is where traces go to be useless, because the handler was blocked on something the application-level instrumentation couldn't see. bpftrace could see it, because it attaches to the kernel and watches what the process is actually doing between the lines your APM records.
If you've never used it, bpftrace is a high-level front end to eBPF. You write a short probe, it compiles to bytecode, the kernel runs it safely. No recompiling the kernel, no rebooting. You need root and a kernel with BTF type information, which current distro kernels ship. Two syntax changes catch people: since bpftrace 0.19, tracepoint arguments are args.field rather than args->field, and since 0.25 the args builtin requires BTF. Older blog posts still show the arrow form.
Where is the time actually going
Start broad. This counts syscalls by ID for the target process so you know what it's spending its time asking the kernel to do.
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter /pid == 24519/ {
@[args.id] = count();
}'
If you don't know the syscall name-to-number mapping, use the syscalls tracepoints instead, which are named:
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_* /comm == "api-server"/ {
@[probe] = count();
}'
For our slow service the top entry was sys_enter_fdatasync, which was the tell. The handler was calling fsync somewhere we didn't expect, probably a logging library flushing to disk synchronously.
Timing a specific syscall
Counting tells you what's frequent, not what's slow. To get latency, pair the enter and exit tracepoints and take the delta. This histogram shows how long read calls take, in microseconds bucketed by power of two:
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_read /comm == "api-server"/ { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_read /comm == "api-server" && @start[tid]/ {
@us = hist((nsecs - @start[tid]) / 1000);
delete(@start[tid]);
}'
The output shape is the whole point. This is the pattern we saw, trimmed:
@us:
[4, 8) 18201 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8, 16) 2100 |@@@@ |
...
[16K, 32K) 47 | |
[32K, 64K) 31 | |
Most reads finish in single-digit microseconds. But there's a second cluster up at 16-64 milliseconds. That bimodal shape is the fingerprint of tail latency. The average would have hidden it completely; the histogram shows two populations. The && @start[tid] filter matters: without it, reads that began before you attached produce garbage deltas.
Was it disk, or was it scheduling
Two of the most common hidden causes of tail latency are the disk taking too long and the process not getting CPU time when it's ready to run. eBPF sees both.
Block I/O latency, which reveals a slow or saturated disk:
sudo bpftrace -e '
tracepoint:block:block_rq_issue { @start[args.dev, args.sector] = nsecs; }
tracepoint:block:block_rq_complete /@start[args.dev, args.sector]/ {
@ms = hist((nsecs - @start[args.dev, args.sector]) / 1000000);
delete(@start[args.dev, args.sector]);
}'
Run-queue latency, which is how long a task waited on the run queue after becoming runnable before the scheduler gave it a CPU:
sudo bpftrace -e '
tracepoint:sched:sched_wakeup { @qtime[args.pid] = nsecs; }
tracepoint:sched:sched_switch /@qtime[args.next_pid]/ {
@runq_us = hist((nsecs - @qtime[args.next_pid]) / 1000);
delete(@qtime[args.next_pid]);
}'
For us the block histogram was clean, everything under a millisecond because it was NVMe. The run-queue histogram had a fat tail past 20ms, which pointed at CPU contention. A noisy neighbor container had no CPU limit and was starving our process during its batch jobs. A cpu cgroup quota on the neighbor fixed the p99, and we never touched our own code. If your nodes run containers, our notes on Linux performance tuning for container and Kubernetes nodes cover the quota side.
The off-CPU question
The sharpest tool for "my process is blocked and I don't know on what" is off-CPU analysis: when a thread leaves the CPU, record where it was in the stack and for how long. You can hand-roll this in bpftrace on sched_switch, but getting the stack capture and accounting right is fiddly, so we reach for offcputime from bcc instead (packaged as offcputime-bpfcc on Debian and Ubuntu):
sudo offcputime-bpfcc -p 24519 10
That traces one PID for ten seconds and prints each blocked stack with the total time spent there, ready to fold into a flame graph. Ours was blocked in fdatasync, closing the loop on the very first syscall count. For more day-to-day probes beyond latency, see our everyday bpftrace patterns.
Questions people ask
How do I measure syscall latency with bpftrace?
Pair the enter and exit tracepoints for the syscall. On sys_enter_read, store nsecs in a map keyed by tid; on sys_exit_read, subtract it, feed the result to hist(), and delete the key. Filter by comm or pid so you only time the process you care about. The power-of-two histogram shows whether slow calls form a separate cluster, which an average hides.
Why does bpftrace say args-> is invalid?
You are running a newer bpftrace with an old example. Since bpftrace 0.19, tracepoint arguments use dot syntax, args.filename instead of args->filename. Since 0.25, the args builtin in tracepoints also needs BTF type information from the kernel; on systems without it you can point the BPFTRACE_BTF environment variable at a BTF file. Update the probe syntax first, then check BTF.
What is the difference between on-CPU and off-CPU latency?
On-CPU time is your code actually running; profilers and flame graphs show it well. Off-CPU time is the thread waiting: on disk, locks, network, or the run queue. Tail latency usually hides off-CPU, which is why a CPU profile looks normal while p99 is terrible. Use run-queue histograms for scheduling delay and offcputime for blocked stacks.
The call we'd make
When application traces bottom out at "time spent in the handler," stop staring at them and drop to eBPF. Start with a syscall count to find the suspect, then a paired enter/exit histogram to confirm it's bimodal, then decide between disk and scheduling with the block and run-queue probes. It's a fifteen-minute investigation that has replaced days of guessing for us. Keep bpftrace and bcc-tools installed on your hosts before the incident, because the worst time to learn the probe syntax is at 3am with a pager going off.
Sources
Cite this article
Kiril Urbonas (2026). Debugging Latency with eBPF: bpftrace One-Liners That Find It. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/debugging-latency-with-ebpf-bpftrace
[Debugging Latency with eBPF: bpftrace One-Liners That Find It](https://www.devopsness.com/blog/debugging-latency-with-ebpf-bpftrace) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)
More from Linux
Explore more articles in this category
SSH Hardening in 2026: Keys, Certificates, and Bastion Patterns
We had 140 engineers with 300 static public keys scattered across authorized_keys files nobody could audit. Moving to SSH certificates with short TTLs made access reviewable again.
The Linux OOM Killer: Why It Fired and How to Prevent It
The kernel killed your process to save the box, and the log looks like noise until you know exactly which fields to read.
Practical Guide: Linux Performance Baseline Methodology
When everything seems "slow," a baseline gives you something to measure against. The capture-and-compare workflow we use on every Linux host.
You might have missed
Evergreen posts worth revisiting.