# Linux Network Debugging: tcpdump, ss, and eBPF in Anger

URL: https://www.devopsness.com/blog/linux-network-debugging-tcpdump-ss-ebpf
Author: Kiril Urbonas
Published: 2026-06-11
Updated: 2026-10-31
Categories: Linux
Tags: Linux, Networking, Tcpdump, Ebpf, Debugging

Summary: When the service is slow and the network is suspect, these are the tools we reach for, in this order, with the exact flags that find the answer.

---
# Linux Network Debugging: tcpdump, ss, and eBPF in Anger

Debug a slow Linux service in this order: `ss -tin` for RTT and retransmits, `ss -ltn` for a full listen queue, a filtered `tcpdump -w` capture, then `mtr` for path loss. Use `bpftrace` only when packets and sockets disagree with the application. Skip `netstat`; `ss` is faster and reads the same kernel state directly.

> [!VERIFIED]
> tcpdump 4.99.7, bpftrace 0.27.1, iproute2 `ss`, against the tcpdump.org man page, the bpftrace 0.27 stdlib docs, the man7.org ss(8) page and the kernel.org ip-sysctl docs. Checked October 2026.

## Start with `ss`: the connections view

`ss` (socket statistics) replaces the old `netstat`. First questions on any latency issue: how many connections do we have, are any in a weird state, are queues backed up?

**Connection counts by state.**
```bash
$ ss -tan state established | wc -l
$ ss -tan state time-wait | wc -l
$ ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c
```

A huge number of `TIME-WAIT` (tens of thousands) usually points at short-lived connections; every closed connection sits in TIME-WAIT for about 60 seconds. Fix it with connection reuse on the client (keepalive, pooling). The kernel knob `net.ipv4.tcp_tw_reuse` defaults to `2` (loopback only) per the ip-sysctl docs; `1` enables it globally for outgoing connections.

A huge number of `CLOSE-WAIT` is worse. It means the local side hasn't closed sockets that the peer already closed, which is almost always an application bug (a socket that never gets closed).

**Listen queue depth.**
```bash
$ ss -ltn
```

For a listening socket, `Recv-Q` is the current backlog (connections waiting to be accepted) and `Send-Q` is the configured backlog (capped by `somaxconn`). If `Recv-Q` sits near `Send-Q`, your application isn't calling `accept()` fast enough, usually an event-loop stall or thread starvation.

**Per-socket details.**
```bash
$ ss -tin
```

The `-i` flag adds TCP internals: RTT, congestion window, retransmits, RTO. A connection with high `retrans` is seeing packet loss; one with high `rtt` is far away or on a slow path.

## Then `tcpdump`: the packet view

When `ss` says something is wrong but not what, `tcpdump` does. Capture, then analyze in Wireshark.

**Capture to a file (reading live is too noisy).**
```bash
$ sudo tcpdump -i any -w /tmp/cap.pcap -s 0 'host <peer-ip> and port <port>'
```

`-s 0` sets the snapshot length to the default of 262144 bytes, so you get full packets. `-w` writes binary, much faster than printing. Always filter by host and port; capturing everything makes the file unusable.

For TLS traffic you still see TCP-level detail (handshake, retransmits, timing), just not the payload.

**Reading live (for sanity checks).**
```bash
$ sudo tcpdump -i any -nn -A 'host <peer> and port 80'
```

`-n` skips address and port-name lookups (`-nn` is the old habit and still works). `-A` prints ASCII payload, useful for plain HTTP.

**Common scenarios we capture for:**

- **"The server isn't responding."** Capture on the server, filter by client IP. Is the SYN arriving? Is the SYN-ACK going out? Did we send a RST? No SYN-ACK means the listener isn't there (port not bound, firewall, listen queue full).
- **"Connection dies after 60 seconds."** Look for keepalives or the FIN/RST that closed it. Often a load balancer idle timeout.
- **"Some requests are slow."** Capture for a few minutes, sort by response time in Wireshark, and look at the packet pattern around the slow ones.

## When `tcpdump` is too coarse: `bpftrace`

`tcpdump` shows packets. Sometimes you need to know what the kernel did with them. `bpftrace` runs eBPF programs from one-liners; the [bpftrace patterns we use day to day](/blog/ebpf-tools-for-everyday-ops-bpftrace-patterns) go deeper.

**Find which processes are opening IPv4 connections.**
```bash
$ sudo bpftrace -e 'tracepoint:syscalls:sys_enter_connect /args->uaddr->sa_family == 2/ {
  printf("%s -> %d.%d.%d.%d\n", comm,
    args->uaddr->sa_data[2], args->uaddr->sa_data[3],
    args->uaddr->sa_data[4], args->uaddr->sa_data[5]);
}'
```

(Cleaner with `tcpconnect` from the bcc tools, packaged as `bpfcc-tools` on Debian and Ubuntu, but the inline version needs no extra package.)

**TCP retransmits by destination.**
```bash
$ sudo bpftrace -e 'tracepoint:tcp:tcp_retransmit_skb { @[ntop(args->daddr)] = count(); }'
```

This shows which peers you keep resending to, which is the eBPF answer to "this service has weird latency to one dependency." `ntop()` accepts the 4-byte address array the tracepoint provides.

**Drop tracing.**
```bash
$ sudo bpftrace -e 'tracepoint:skb:kfree_skb { @[kstack] = count(); }'
```

Counts freed packets by kernel stack. Run for 30 seconds and read the top stacks; they usually point at firewall rules, queue overflow, or netfilter. Expect noise, since not every freed skb is a real drop.

The bcc tools also include `tcptracer`, `tcpretrans`, `tcptop` and `tcpsubnet`. We start with these before writing custom bpftrace. When the slowness turns out to be scheduling rather than packets, [debugging latency with bpftrace](/blog/debugging-latency-with-ebpf-bpftrace) is the next step.

## Who's sending what: `iftop`

For a live view of bandwidth by connection:

```bash
$ sudo iftop -i eth0 -n
```

Interactive, sorted by current bandwidth. It answers "who is saturating my NIC?" The usual answer: egress on a streaming endpoint, an rsync someone started during business hours, or a runaway log shipper.

## Path issues: `mtr`

When the question is "us or them?", `mtr` shows the hop-by-hop path with loss and latency.

```bash
$ mtr -rwn -c 100 <target>
```

`-r` is report mode (run, then exit), `-w` wide output, `-c 100` sends 100 packets. A high-loss hop mid-path that persists to the destination is the network's fault; loss only at the destination is yours. Loss at a single middle hop that disappears downstream is usually just ICMP rate limiting on that router.

When escalating to a cloud provider, "your network is dropping at hop X" beats "the network is slow."

## The order we run them

For a "service is slow" report:

1. **`ss -tin`** on the source. Connections established? RTT high? Retransmits?
2. **`ss -ltn`** on the destination. Is the listen queue full?
3. **`tcpdump`** between source and destination. Capture for 30 seconds; look at the pattern.
4. **`mtr`** if cross-region. Is there path loss?
5. **`bpftrace`, `tcptop`, `tcptracer`** if application symptoms don't match what the packets show.
6. **Read application logs** in parallel through all of this. Half the time it's an app bug dressed up as a network issue.

## Things we wish we'd known earlier

**`tcpdump` on a busy server can drop packets.** Raise the capture buffer with `-B` (in KiB) and write to a fast disk. If it still drops, capture on one interface or filter harder. The symptom is a trace where the first packets look fine and later ones look broken; tcpdump prints "packets dropped by kernel" when it exits, so check that line.

**`tcpdump` with TLS reveals less than you think.** You see the handshake, sizes and timing, not payload. Capture on the unencrypted side (the load balancer's internal hop) or use a proxy like mitmproxy for content.

**Conntrack table fills up.** On busy gateways the netfilter connection-tracking table can fill and drop new connections. Compare `/proc/sys/net/netfilter/nf_conntrack_count` with `nf_conntrack_max`. If close, raise the max or skip conntrack for that traffic.

**Reverse path filter.** The kernel default for `rp_filter` is `0`, but many distributions turn it on in startup scripts, and strict mode (`1`) drops packets that arrive on an interface that wouldn't route back to the source. It bites in multi-NIC setups, where an asymmetric route silently drops traffic. Check `/proc/sys/net/ipv4/conf/all/rp_filter`.

**MTU mismatches.** When certain sizes vanish (large requests hang, small ones succeed), suspect MTU. A VPN or tunnel shrinking the path MTU is the classic cause. Probe with `ping -M do -s 1472 <target>`.

## Questions people ask

### What replaced netstat on Linux?

`ss` from iproute2. It reads socket state straight from the kernel through netlink, so it is faster on hosts with many connections and shows more TCP detail. `ss -tan` lists all TCP sockets numerically, `ss -ltn` shows listeners with their backlog, and `ss -tin` adds RTT, congestion window and retransmit counters per connection. The net-tools `netstat` still works where installed, but `ss` is the tool to learn.

### How do I see why packets are being dropped on Linux?

Start cheap: check `nf_conntrack_count` against `nf_conntrack_max`, the `rp_filter` setting, and MTU with `ping -M do`. If none explain it, trace the kernel with `bpftrace -e 'tracepoint:skb:kfree_skb { @[kstack] = count(); }'` for 30 seconds. The top stacks show where packets are freed, usually netfilter, a full queue, or routing. Not every freed packet is a drop, so read the stacks.

### Should I use tcpdump or bpftrace?

Use `tcpdump` when you need to see what is on the wire: handshakes, resets, retransmits, timing between peers. Use `bpftrace` when the packets look fine but behavior doesn't, and you need to know which process connected, which kernel path dropped a packet, or which peer keeps triggering retransmits. In practice tcpdump answers most incidents and eBPF covers the harder remainder.

## What to read next

- [eBPF tools for everyday ops: bpftrace patterns](/blog/ebpf-tools-for-everyday-ops-bpftrace-patterns), the eBPF side expanded
- [Linux io_uring: async I/O patterns we use](/blog/linux-io-uring-async-io-patterns), adjacent kernel-side performance
- [Linux performance tuning for containers and Kubernetes nodes](/blog/linux-performance-tuning-for-containers-and-kubernetes-nodes), broader system-performance knobs
- [Container resource limits: what they actually do](/blog/container-resource-limits-what-they-actually-do), kernel resource accounting under containers

The kernel is usually right about what it sees. The discipline is reading these tools' output without jumping to conclusions; most answers are in `ss` or `tcpdump`.

## Sources

- [tcpdump(1) man page, tcpdump.org](https://www.tcpdump.org/manpages/tcpdump.1.html)
- [ss(8) man page, man7.org](https://man7.org/linux/man-pages/man8/ss.8.html)
- [Linux kernel docs: IP sysctl (rp_filter, tcp_tw_reuse)](https://docs.kernel.org/networking/ip-sysctl.html)
- [bpftrace 0.27 standard library reference](https://bpftrace.org/docs/release_027/stdlib)