# Linux TCP Tuning for High-Throughput Services

URL: https://www.devopsness.com/blog/linux-tcp-tuning-high-throughput-services
Author: Kiril Urbonas
Published: 2026-07-02
Updated: 2026-11-15
Categories: Linux
Tags: Networking, TCP, Linux, Performance, Linux Kernel, tcpdump

Summary: A proxy that stalls with the CPU half-idle is often hitting kernel TCP defaults, not the hardware. The sysctls to change and how to measure each one.

---
# Linux TCP Tuning for High-Throughput Services

To tune Linux TCP for high throughput, raise `net.core.somaxconn` and the application's `listen()` backlog together, widen `ip_local_port_range` for proxies, raise the `tcp_rmem`/`tcp_wmem` maximums, and switch congestion control to BBR with the `fq` qdisc. Persist the values in `/etc/sysctl.d/` and check `nstat` counters before and after every change.

> [!VERIFIED]
> Linux 7.3 (stable 7.3.1, longterm 6.18.60), against the docs.kernel.org ip-sysctl reference, the tcp(7) man page and the nginx `listen` directive docs. Checked October 2026.

A busy reverse proxy can stall at tens of thousands of concurrent connections with the CPU half idle. The hardware wasn't the limit. Kernel TCP defaults are chosen to be safe on small machines, and on a busy proxy they become the wall you keep hitting.

Before changing anything, measure. Blind tuning is how you turn one problem into three. `ss -s` gives a connection summary, and `netstat -s` (or `nstat`) shows the counters that tell you what's actually being dropped.

```bash
$ ss -s
$ nstat -az | grep -E 'TcpExtListenDrops|TcpExtListenOverflows|TcpExtTCPBacklogDrop'
```

`ListenOverflows` climbing meant our accept queue was full. That was the first real clue.

## The accept backlog

When connections arrive faster than your app calls `accept()`, they queue. Two limits govern this: the app's `listen()` backlog argument and the kernel's `net.core.somaxconn`. If `somaxconn` is smaller than what the app asks for, the kernel silently caps it. The kernel docs list the default as 4096 since Linux 5.4 (it was 128 before), which is still small for a proxy taking bursts of tens of thousands of connections.

```bash
$ sudo sysctl -w net.core.somaxconn=65535
$ sudo sysctl -w net.ipv4.tcp_max_syn_backlog=65535
```

Then confirm your application actually passes a large backlog to `listen()`. Nginx sets it with `listen ... backlog=65535;` and does not inherit `somaxconn`: its docs say the backlog defaults to 511 on Linux. We set the kernel value and saw no change for an hour until we realized nginx was still calling `listen(sockfd, 511)`. Both sides have to agree. Our [nginx reverse proxy guide](/blog/nginx-reverse-proxy-guide) covers the rest of that config.

## Ephemeral ports and TIME_WAIT

For a proxy making outbound connections to backends, you can run out of source ports. Each outbound connection needs a local port, and the default range is 32768 to 60999.

```bash
$ sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535"
```

That gives you roughly 64k ports per destination tuple. Reserve any ports your own services listen on with `net.ipv4.ip_local_reserved_ports`, or an outbound connection can grab one first and the service fails with [address already in use](/blog/fix-address-already-in-use). Connections in `TIME_WAIT` hold their port for 60 seconds after closing, and under high churn they pile up fast. The right fix is not the dangerous one people copy from old forum posts: `tcp_tw_recycle` existed only from Linux 2.4 to 4.11 and was removed because it broke clients behind NAT. Enable reuse instead. The default, `2`, only covers loopback; `1` enables it globally.

```bash
$ sudo sysctl -w net.ipv4.tcp_tw_reuse=1
```

`tcp_tw_reuse` lets the kernel reuse a `TIME_WAIT` socket for a new outbound connection when it's provably safe using timestamps. It only affects the connecting side, so it's the correct knob for a proxy dialing backends. Better still, use keepalive connection pools to your backends so you're not churning connections at all, but when you must, `tw_reuse` is the safe lever.

## Buffer sizes for throughput

On a high-bandwidth, higher-latency link, small socket buffers cap throughput because the sender can't keep enough data in flight to fill the pipe. This is the bandwidth-delay product. Let the kernel autotune within a larger ceiling.

```bash
$ sudo sysctl -w net.core.rmem_max=16777216
$ sudo sysctl -w net.core.wmem_max=16777216
$ sudo sysctl -w net.ipv4.tcp_rmem="4096 131072 16777216"
$ sudo sysctl -w net.ipv4.tcp_wmem="4096 16384 16777216"
```

The three numbers are min, default, and max. The kernel autotunes between default and max based on conditions, so you're raising the ceiling, not pinning a fixed size. The kernel docs put the default `tcp_wmem` max between 64K and 4MB and the `tcp_rmem` max between 128K and 32MB, depending on RAM, so check what your box already has with `sysctl net.ipv4.tcp_rmem` before raising anything. The win shows up on high-RTT paths, where the window finally grows large enough to keep the link full. One trap: an application that calls `setsockopt()` with `SO_RCVBUF` or `SO_SNDBUF` disables autotuning for that socket, and these ceilings no longer apply.

## Congestion control: switch to BBR

The usual distro default, CUBIC, backs off hard on packet loss, which tanks throughput on links with any loss. BBR models the bottleneck bandwidth and round-trip time instead and holds throughput up far better on lossy paths. It has shipped in mainline kernels for years; check `sysctl net.ipv4.tcp_available_congestion_control` to see whether your kernel has it loaded.

```bash
$ sudo sysctl -w net.core.default_qdisc=fq
$ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
```

We set the `fq` queueing discipline alongside BBR so packets are paced. On a lossy long-haul path the per-stream difference against CUBIC can be large, but measure it on your own link rather than trusting anyone's chart. Confirm it took:

```bash
$ sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = bbr
```

## Making it stick and verifying

`sysctl -w` is lost on reboot. Put the final set in a file under `/etc/sysctl.d/` and apply it with `sysctl --system`, which reads that directory (plain `sysctl -p` only reads `/etc/sysctl.conf`):

```ini
# /etc/sysctl.d/99-tcp-tuning.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 131072 16777216
net.ipv4.tcp_wmem = 4096 16384 16777216
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
```

Then verify the behavior changed, don't just trust that you wrote the file. When we suspected retransmits were still high, `tcpdump` on the wire showed us the actual retransmitted segments:

```bash
$ sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-syn != 0' -c 20
$ nstat -az | grep -i retrans
```

Watching retransmit counters before and after each change is the only way to know a knob helped rather than hurt. [Debugging with tcpdump, ss and eBPF](/blog/linux-network-debugging-tcpdump-ss-ebpf) goes deeper on reading those counters. Once the full set was in, the connection ceiling moved well past where it had stalled and the CPU became the limiting resource, which is where you want to be.

## Questions people ask

### What should net.core.somaxconn be set to?

For a busy proxy or API server, 65535 is a common ceiling, but the number only matters if the application asks for it. The kernel default is 4096 since Linux 5.4. Raise `somaxconn`, then raise the backlog your server passes to `listen()`, such as nginx's `backlog=` parameter, which defaults to 511 on Linux. Confirm with `nstat` that `TcpExtListenOverflows` stops climbing.

### Is tcp_tw_reuse safe to enable?

Yes, for the connecting side. It lets the kernel reuse a `TIME_WAIT` socket for a new outbound connection when TCP timestamps make it safe, so it suits proxies dialing backends. The default value `2` enables it for loopback only; `1` enables it globally. Do not confuse it with `tcp_tw_recycle`, which broke NAT clients and was removed after Linux 4.11.

### Should I switch from CUBIC to BBR?

On long or lossy paths, usually yes: BBR models bandwidth and round-trip time instead of treating loss as congestion, so it keeps throughput up where CUBIC backs off. Set `net.core.default_qdisc=fq` with it. On a clean, low-latency datacenter network the gain is smaller, so measure retransmits and throughput on your own traffic before rolling it fleet-wide.

## The call we'd make

Change one thing at a time and watch `nstat` counters between changes, because half these knobs interact and a blind copy-paste of someone's `sysctl.conf` will bite you. If you only touch two settings, make them `net.core.somaxconn` (raise it, and match your app's `listen()` backlog) and switching congestion control to BBR with `fq`. Those two fixed the connection ceiling and the throughput ceiling for us, and everything else was fine-tuning on top. Never try to enable `tcp_tw_recycle`; it's gone for good reason. Prefer connection pooling over any `TIME_WAIT` trickery when you can.

## Sources

- [Linux kernel documentation: IP Sysctl (net.core and net.ipv4 parameters)](https://docs.kernel.org/networking/ip-sysctl.html)
- [tcp(7) Linux manual page](https://man7.org/linux/man-pages/man7/tcp.7.html)
- [nginx documentation: ngx_http_core_module listen directive](https://nginx.org/en/docs/http/ngx_http_core_module.html)
- [The Linux Kernel Archives (current stable and longterm releases)](https://www.kernel.org/)

