# Debugging Latency with eBPF: bpftrace One-Liners That Find It

URL: https://www.devopsness.com/blog/debugging-latency-with-ebpf-bpftrace
Author: Kiril Urbonas
Published: 2026-07-01
Updated: 2026-10-31
Categories: Linux
Tags: Linux, eBPF, bpftrace, Performance, Debugging

Summary: When a service is slow and every dashboard looks green, bpftrace lets you watch the kernel directly. These one-liners found our tail latency.

---
# Debugging Latency with eBPF: bpftrace One-Liners That Find It

To find hidden latency with eBPF, run four bpftrace probes in order: count syscalls for the slow process, histogram one syscall's enter-to-exit time to expose a bimodal tail, then compare block I/O latency against run-queue latency to tell a slow disk from CPU starvation. Finish with bcc's `offcputime` to see exactly what the process blocked on.

> [!VERIFIED]
> bpftrace 0.27.1 (released 2026-10-24) and bcc 0.37.0, against the bpftrace one-liner tutorial and migration guide and the bcc `offcputime` examples. Checked October 2026.

A service was returning p99 responses near a second while p50 sat in the low milliseconds. Classic tail latency. The application traces showed the time was spent "in the handler," which is where traces go to be useless, because the handler was blocked on something the application-level instrumentation couldn't see. `bpftrace` could see it, because it attaches to the kernel and watches what the process is actually doing between the lines your APM records.

If you've never used it, `bpftrace` is a high-level front end to eBPF. You write a short probe, it compiles to bytecode, the kernel runs it safely. No recompiling the kernel, no rebooting. You need root and a kernel with BTF type information, which current distro kernels ship. Two syntax changes catch people: since bpftrace 0.19, tracepoint arguments are `args.field` rather than `args->field`, and since 0.25 the `args` builtin requires BTF. Older blog posts still show the arrow form.

## Where is the time actually going

Start broad. This counts syscalls by ID for the target process so you know what it's spending its time asking the kernel to do.

```bash
$ sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter /pid == 24519/ {
  @[args.id] = count();
}'
```

If you don't know the syscall name-to-number mapping, use the `syscalls` tracepoints instead, which are named:

```bash
$ sudo bpftrace -e 'tracepoint:syscalls:sys_enter_* /comm == "api-server"/ {
  @[probe] = count();
}'
```

For our slow service the top entry was `sys_enter_fdatasync`, which was the tell. The handler was calling fsync somewhere we didn't expect, probably a logging library flushing to disk synchronously.

## Timing a specific syscall

Counting tells you what's frequent, not what's slow. To get latency, pair the enter and exit tracepoints and take the delta. This histogram shows how long `read` calls take, in microseconds bucketed by power of two:

```bash
$ sudo bpftrace -e '
tracepoint:syscalls:sys_enter_read /comm == "api-server"/ { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_read /comm == "api-server" && @start[tid]/ {
  @us = hist((nsecs - @start[tid]) / 1000);
  delete(@start[tid]);
}'
```

The output shape is the whole point. This is the pattern we saw, trimmed:

```
@us:
[4, 8)        18201 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8, 16)        2100 |@@@@                                    |
...
[16K, 32K)       47 |                                        |
[32K, 64K)       31 |                                        |
```

Most reads finish in single-digit microseconds. But there's a second cluster up at 16-64 milliseconds. That bimodal shape is the fingerprint of tail latency. The average would have hidden it completely; the histogram shows two populations. The `&& @start[tid]` filter matters: without it, reads that began before you attached produce garbage deltas.

## Was it disk, or was it scheduling

Two of the most common hidden causes of tail latency are the disk taking too long and the process not getting CPU time when it's ready to run. eBPF sees both.

Block I/O latency, which reveals a slow or saturated disk:

```bash
$ sudo bpftrace -e '
tracepoint:block:block_rq_issue { @start[args.dev, args.sector] = nsecs; }
tracepoint:block:block_rq_complete /@start[args.dev, args.sector]/ {
  @ms = hist((nsecs - @start[args.dev, args.sector]) / 1000000);
  delete(@start[args.dev, args.sector]);
}'
```

Run-queue latency, which is how long a task waited on the run queue after becoming runnable before the scheduler gave it a CPU:

```bash
$ sudo bpftrace -e '
tracepoint:sched:sched_wakeup { @qtime[args.pid] = nsecs; }
tracepoint:sched:sched_switch /@qtime[args.next_pid]/ {
  @runq_us = hist((nsecs - @qtime[args.next_pid]) / 1000);
  delete(@qtime[args.next_pid]);
}'
```

For us the block histogram was clean, everything under a millisecond because it was NVMe. The run-queue histogram had a fat tail past 20ms, which pointed at CPU contention. A noisy neighbor container had no CPU limit and was starving our process during its batch jobs. A `cpu` cgroup quota on the neighbor fixed the p99, and we never touched our own code. If your nodes run containers, our notes on [Linux performance tuning for container and Kubernetes nodes](/blog/linux-performance-tuning-for-containers-and-kubernetes-nodes) cover the quota side.

## The off-CPU question

The sharpest tool for "my process is blocked and I don't know on what" is off-CPU analysis: when a thread leaves the CPU, record where it was in the stack and for how long. You can hand-roll this in bpftrace on `sched_switch`, but getting the stack capture and accounting right is fiddly, so we reach for `offcputime` from bcc instead (packaged as `offcputime-bpfcc` on Debian and Ubuntu):

```bash
$ sudo offcputime-bpfcc -p 24519 10
```

That traces one PID for ten seconds and prints each blocked stack with the total time spent there, ready to fold into a flame graph. Ours was blocked in `fdatasync`, closing the loop on the very first syscall count. For more day-to-day probes beyond latency, see our [everyday bpftrace patterns](/blog/ebpf-tools-for-everyday-ops-bpftrace-patterns).

## Questions people ask

### How do I measure syscall latency with bpftrace?

Pair the enter and exit tracepoints for the syscall. On `sys_enter_read`, store `nsecs` in a map keyed by `tid`; on `sys_exit_read`, subtract it, feed the result to `hist()`, and delete the key. Filter by `comm` or `pid` so you only time the process you care about. The power-of-two histogram shows whether slow calls form a separate cluster, which an average hides.

### Why does bpftrace say args-> is invalid?

You are running a newer bpftrace with an old example. Since bpftrace 0.19, tracepoint arguments use dot syntax, `args.filename` instead of `args->filename`. Since 0.25, the `args` builtin in tracepoints also needs BTF type information from the kernel; on systems without it you can point the `BPFTRACE_BTF` environment variable at a BTF file. Update the probe syntax first, then check BTF.

### What is the difference between on-CPU and off-CPU latency?

On-CPU time is your code actually running; profilers and flame graphs show it well. Off-CPU time is the thread waiting: on disk, locks, network, or the run queue. Tail latency usually hides off-CPU, which is why a CPU profile looks normal while p99 is terrible. Use run-queue histograms for scheduling delay and `offcputime` for blocked stacks.

## The call we'd make

When application traces bottom out at "time spent in the handler," stop staring at them and drop to eBPF. Start with a syscall count to find the suspect, then a paired enter/exit histogram to confirm it's bimodal, then decide between disk and scheduling with the block and run-queue probes. It's a fifteen-minute investigation that has replaced days of guessing for us. Keep bpftrace and bcc-tools installed on your hosts before the incident, because the worst time to learn the probe syntax is at 3am with a pager going off.

## Sources

- [bpftrace one-liner tutorial (GitHub)](https://github.com/bpftrace/bpftrace/blob/master/docs/tutorial_one_liners.md)
- [bpftrace migration guide (args syntax and BTF changes)](https://github.com/bpftrace/bpftrace/blob/master/docs/migration_guide.md)
- [bpftrace v0.27.1 release notes](https://github.com/bpftrace/bpftrace/releases/tag/v0.27.1)
- [bcc offcputime examples and usage](https://github.com/iovisor/bcc/blob/master/tools/offcputime_example.txt)

