Linux Network Debugging: tcpdump, ss, and eBPF in Anger

Debug a slow Linux service in this order: ss -tin for RTT and retransmits, ss -ltn for a full listen queue, a filtered tcpdump -w capture, then mtr for path loss. Use bpftrace only when packets and sockets disagree with the application. Skip netstat; ss is faster and reads the same kernel state directly.

Start with ss: the connections view

ss (socket statistics) replaces the old netstat. First questions on any latency issue: how many connections do we have, are any in a weird state, are queues backed up?

Connection counts by state.

$ ss -tan state established | wc -l
$ ss -tan state time-wait | wc -l
$ ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c

A huge number of TIME-WAIT (tens of thousands) usually points at short-lived connections; every closed connection sits in TIME-WAIT for about 60 seconds. Fix it with connection reuse on the client (keepalive, pooling). The kernel knob net.ipv4.tcp_tw_reuse defaults to 2 (loopback only) per the ip-sysctl docs; 1 enables it globally for outgoing connections.

A huge number of CLOSE-WAIT is worse. It means the local side hasn't closed sockets that the peer already closed, which is almost always an application bug (a socket that never gets closed).

Listen queue depth.

$ ss -ltn

For a listening socket, Recv-Q is the current backlog (connections waiting to be accepted) and Send-Q is the configured backlog (capped by somaxconn). If Recv-Q sits near Send-Q, your application isn't calling accept() fast enough, usually an event-loop stall or thread starvation.

Per-socket details.

$ ss -tin

The -i flag adds TCP internals: RTT, congestion window, retransmits, RTO. A connection with high retrans is seeing packet loss; one with high rtt is far away or on a slow path.

Then tcpdump: the packet view

When ss says something is wrong but not what, tcpdump does. Capture, then analyze in Wireshark.

Capture to a file (reading live is too noisy).

$ sudo tcpdump -i any -w /tmp/cap.pcap -s 0 'host <peer-ip> and port <port>'

-s 0 sets the snapshot length to the default of 262144 bytes, so you get full packets. -w writes binary, much faster than printing. Always filter by host and port; capturing everything makes the file unusable.

For TLS traffic you still see TCP-level detail (handshake, retransmits, timing), just not the payload.

Reading live (for sanity checks).

$ sudo tcpdump -i any -nn -A 'host <peer> and port 80'

-n skips address and port-name lookups (-nn is the old habit and still works). -A prints ASCII payload, useful for plain HTTP.

Common scenarios we capture for:

• "The server isn't responding." Capture on the server, filter by client IP. Is the SYN arriving? Is the SYN-ACK going out? Did we send a RST? No SYN-ACK means the listener isn't there (port not bound, firewall, listen queue full).
• "Connection dies after 60 seconds." Look for keepalives or the FIN/RST that closed it. Often a load balancer idle timeout.
• "Some requests are slow." Capture for a few minutes, sort by response time in Wireshark, and look at the packet pattern around the slow ones.

When tcpdump is too coarse: bpftrace

tcpdump shows packets. Sometimes you need to know what the kernel did with them. bpftrace runs eBPF programs from one-liners; the bpftrace patterns we use day to day go deeper.

Find which processes are opening IPv4 connections.

$ sudo bpftrace -e 'tracepoint:syscalls:sys_enter_connect /args->uaddr->sa_family == 2/ {
  printf("%s -> %d.%d.%d.%d\n", comm,
    args->uaddr->sa_data[2], args->uaddr->sa_data[3],
    args->uaddr->sa_data[4], args->uaddr->sa_data[5]);
}'

(Cleaner with tcpconnect from the bcc tools, packaged as bpfcc-tools on Debian and Ubuntu, but the inline version needs no extra package.)

TCP retransmits by destination.

$ sudo bpftrace -e 'tracepoint:tcp:tcp_retransmit_skb { @[ntop(args->daddr)] = count(); }'

This shows which peers you keep resending to, which is the eBPF answer to "this service has weird latency to one dependency." ntop() accepts the 4-byte address array the tracepoint provides.

Drop tracing.

$ sudo bpftrace -e 'tracepoint:skb:kfree_skb { @[kstack] = count(); }'

Counts freed packets by kernel stack. Run for 30 seconds and read the top stacks; they usually point at firewall rules, queue overflow, or netfilter. Expect noise, since not every freed skb is a real drop.

The bcc tools also include tcptracer, tcpretrans, tcptop and tcpsubnet. We start with these before writing custom bpftrace. When the slowness turns out to be scheduling rather than packets, debugging latency with bpftrace is the next step.

Who's sending what: iftop

For a live view of bandwidth by connection:

$ sudo iftop -i eth0 -n

Interactive, sorted by current bandwidth. It answers "who is saturating my NIC?" The usual answer: egress on a streaming endpoint, an rsync someone started during business hours, or a runaway log shipper.

Path issues: mtr

When the question is "us or them?", mtr shows the hop-by-hop path with loss and latency.

$ mtr -rwn -c 100 <target>

-r is report mode (run, then exit), -w wide output, -c 100 sends 100 packets. A high-loss hop mid-path that persists to the destination is the network's fault; loss only at the destination is yours. Loss at a single middle hop that disappears downstream is usually just ICMP rate limiting on that router.

When escalating to a cloud provider, "your network is dropping at hop X" beats "the network is slow."

The order we run them

For a "service is slow" report:

1. ss -tin on the source. Connections established? RTT high? Retransmits?
2. ss -ltn on the destination. Is the listen queue full?
3. tcpdump between source and destination. Capture for 30 seconds; look at the pattern.
4. mtr if cross-region. Is there path loss?
5. bpftrace, tcptop, tcptracer if application symptoms don't match what the packets show.
6. Read application logs in parallel through all of this. Half the time it's an app bug dressed up as a network issue.

Things we wish we'd known earlier

tcpdump on a busy server can drop packets. Raise the capture buffer with -B (in KiB) and write to a fast disk. If it still drops, capture on one interface or filter harder. The symptom is a trace where the first packets look fine and later ones look broken; tcpdump prints "packets dropped by kernel" when it exits, so check that line.

tcpdump with TLS reveals less than you think. You see the handshake, sizes and timing, not payload. Capture on the unencrypted side (the load balancer's internal hop) or use a proxy like mitmproxy for content.

Conntrack table fills up. On busy gateways the netfilter connection-tracking table can fill and drop new connections. Compare /proc/sys/net/netfilter/nf_conntrack_count with nf_conntrack_max. If close, raise the max or skip conntrack for that traffic.

Reverse path filter. The kernel default for rp_filter is 0, but many distributions turn it on in startup scripts, and strict mode (1) drops packets that arrive on an interface that wouldn't route back to the source. It bites in multi-NIC setups, where an asymmetric route silently drops traffic. Check /proc/sys/net/ipv4/conf/all/rp_filter.

MTU mismatches. When certain sizes vanish (large requests hang, small ones succeed), suspect MTU. A VPN or tunnel shrinking the path MTU is the classic cause. Probe with ping -M do -s 1472 <target>.

Questions people ask

What replaced netstat on Linux?

ss from iproute2. It reads socket state straight from the kernel through netlink, so it is faster on hosts with many connections and shows more TCP detail. ss -tan lists all TCP sockets numerically, ss -ltn shows listeners with their backlog, and ss -tin adds RTT, congestion window and retransmit counters per connection. The net-tools netstat still works where installed, but ss is the tool to learn.

How do I see why packets are being dropped on Linux?

Start cheap: check nf_conntrack_count against nf_conntrack_max, the rp_filter setting, and MTU with ping -M do. If none explain it, trace the kernel with bpftrace -e 'tracepoint:skb:kfree_skb { @[kstack] = count(); }' for 30 seconds. The top stacks show where packets are freed, usually netfilter, a full queue, or routing. Not every freed packet is a drop, so read the stacks.

Should I use tcpdump or bpftrace?

Use tcpdump when you need to see what is on the wire: handshakes, resets, retransmits, timing between peers. Use bpftrace when the packets look fine but behavior doesn't, and you need to know which process connected, which kernel path dropped a packet, or which peer keeps triggering retransmits. In practice tcpdump answers most incidents and eBPF covers the harder remainder.

What to read next

• eBPF tools for everyday ops: bpftrace patterns, the eBPF side expanded
• Linux io_uring: async I/O patterns we use, adjacent kernel-side performance
• Linux performance tuning for containers and Kubernetes nodes, broader system-performance knobs
• Container resource limits: what they actually do, kernel resource accounting under containers

The kernel is usually right about what it sees. The discipline is reading these tools' output without jumping to conclusions; most answers are in ss or tcpdump.

Sources

• tcpdump(1) man page, tcpdump.org
• ss(8) man page, man7.org
• Linux kernel docs: IP sysctl (rp_filter, tcp_tw_reuse)
• bpftrace 0.27 standard library reference