Skip to main content
Linux TCP Tuning for High-Throughput Services

Linux TCP Tuning for High-Throughput Services

KU
· Founder & Chief Editor
• 7 min read•Updated •45 views

A proxy that stalls with the CPU half-idle is often hitting kernel TCP defaults, not the hardware. The sysctls to change and how to measure each one.

Key takeaways

  • Raise net.core.somaxconn and your app's listen() backlog together, widen ip_local_port_range for outbound-heavy proxies, raise the tcp_rmem and tcp_wmem ceilings for high-latency links, and switch to BBR with the fq qdisc.
  • Measure ListenOverflows and retransmits with nstat before and after each change, and never enable tcp_tw_recycle.

To tune Linux TCP for high throughput, raise net.core.somaxconn and the application's listen() backlog together, widen ip_local_port_range for proxies, raise the tcp_rmem/tcp_wmem maximums, and switch congestion control to BBR with the fq qdisc. Persist the values in /etc/sysctl.d/ and check nstat counters before and after every change.

Verified against

Linux 7.3 (stable 7.3.1, longterm 6.18.60), against the docs.kernel.org ip-sysctl reference, the tcp(7) man page and the nginx listen directive docs. Checked October 2026.

A busy reverse proxy can stall at tens of thousands of concurrent connections with the CPU half idle. The hardware wasn't the limit. Kernel TCP defaults are chosen to be safe on small machines, and on a busy proxy they become the wall you keep hitting.

Before changing anything, measure. Blind tuning is how you turn one problem into three. ss -s gives a connection summary, and netstat -s (or nstat) shows the counters that tell you what's actually being dropped.

Bash
ss -s
nstat -az | grep -E 'TcpExtListenDrops|TcpExtListenOverflows|TcpExtTCPBacklogDrop'

ListenOverflows climbing meant our accept queue was full. That was the first real clue.

The accept backlog

When connections arrive faster than your app calls accept(), they queue. Two limits govern this: the app's listen() backlog argument and the kernel's net.core.somaxconn. If somaxconn is smaller than what the app asks for, the kernel silently caps it. The kernel docs list the default as 4096 since Linux 5.4 (it was 128 before), which is still small for a proxy taking bursts of tens of thousands of connections.

Bash
sudo sysctl -w net.core.somaxconn=65535
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=65535

Then confirm your application actually passes a large backlog to listen(). Nginx sets it with listen ... backlog=65535; and does not inherit somaxconn: its docs say the backlog defaults to 511 on Linux. We set the kernel value and saw no change for an hour until we realized nginx was still calling listen(sockfd, 511). Both sides have to agree. Our nginx reverse proxy guide covers the rest of that config.

Ephemeral ports and TIME_WAIT

For a proxy making outbound connections to backends, you can run out of source ports. Each outbound connection needs a local port, and the default range is 32768 to 60999.

Bash
sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535"

That gives you roughly 64k ports per destination tuple. Reserve any ports your own services listen on with net.ipv4.ip_local_reserved_ports, or an outbound connection can grab one first and the service fails with address already in use. Connections in TIME_WAIT hold their port for 60 seconds after closing, and under high churn they pile up fast. The right fix is not the dangerous one people copy from old forum posts: tcp_tw_recycle existed only from Linux 2.4 to 4.11 and was removed because it broke clients behind NAT. Enable reuse instead. The default, 2, only covers loopback; 1 enables it globally.

Bash
sudo sysctl -w net.ipv4.tcp_tw_reuse=1

tcp_tw_reuse lets the kernel reuse a TIME_WAIT socket for a new outbound connection when it's provably safe using timestamps. It only affects the connecting side, so it's the correct knob for a proxy dialing backends. Better still, use keepalive connection pools to your backends so you're not churning connections at all, but when you must, tw_reuse is the safe lever.

Buffer sizes for throughput

On a high-bandwidth, higher-latency link, small socket buffers cap throughput because the sender can't keep enough data in flight to fill the pipe. This is the bandwidth-delay product. Let the kernel autotune within a larger ceiling.

Bash
sudo sysctl -w net.core.rmem_max=16777216
sudo sysctl -w net.core.wmem_max=16777216
sudo sysctl -w net.ipv4.tcp_rmem="4096 131072 16777216"
sudo sysctl -w net.ipv4.tcp_wmem="4096 16384 16777216"

The three numbers are min, default, and max. The kernel autotunes between default and max based on conditions, so you're raising the ceiling, not pinning a fixed size. The kernel docs put the default tcp_wmem max between 64K and 4MB and the tcp_rmem max between 128K and 32MB, depending on RAM, so check what your box already has with sysctl net.ipv4.tcp_rmem before raising anything. The win shows up on high-RTT paths, where the window finally grows large enough to keep the link full. One trap: an application that calls setsockopt() with SO_RCVBUF or SO_SNDBUF disables autotuning for that socket, and these ceilings no longer apply.

Congestion control: switch to BBR

The usual distro default, CUBIC, backs off hard on packet loss, which tanks throughput on links with any loss. BBR models the bottleneck bandwidth and round-trip time instead and holds throughput up far better on lossy paths. It has shipped in mainline kernels for years; check sysctl net.ipv4.tcp_available_congestion_control to see whether your kernel has it loaded.

Bash
sudo sysctl -w net.core.default_qdisc=fq
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

We set the fq queueing discipline alongside BBR so packets are paced. On a lossy long-haul path the per-stream difference against CUBIC can be large, but measure it on your own link rather than trusting anyone's chart. Confirm it took:

Bash
sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = bbr

Making it stick and verifying

sysctl -w is lost on reboot. Put the final set in a file under /etc/sysctl.d/ and apply it with sysctl --system, which reads that directory (plain sysctl -p only reads /etc/sysctl.conf):

/etc/sysctl.d/99-tcp-tuning.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 131072 16777216
net.ipv4.tcp_wmem = 4096 16384 16777216
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Then verify the behavior changed, don't just trust that you wrote the file. When we suspected retransmits were still high, tcpdump on the wire showed us the actual retransmitted segments:

Bash
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-syn != 0' -c 20
nstat -az | grep -i retrans

Watching retransmit counters before and after each change is the only way to know a knob helped rather than hurt. Debugging with tcpdump, ss and eBPF goes deeper on reading those counters. Once the full set was in, the connection ceiling moved well past where it had stalled and the CPU became the limiting resource, which is where you want to be.

Questions people ask

What should net.core.somaxconn be set to?

For a busy proxy or API server, 65535 is a common ceiling, but the number only matters if the application asks for it. The kernel default is 4096 since Linux 5.4. Raise somaxconn, then raise the backlog your server passes to listen(), such as nginx's backlog= parameter, which defaults to 511 on Linux. Confirm with nstat that TcpExtListenOverflows stops climbing.

Is tcp_tw_reuse safe to enable?

Yes, for the connecting side. It lets the kernel reuse a TIME_WAIT socket for a new outbound connection when TCP timestamps make it safe, so it suits proxies dialing backends. The default value 2 enables it for loopback only; 1 enables it globally. Do not confuse it with tcp_tw_recycle, which broke NAT clients and was removed after Linux 4.11.

Should I switch from CUBIC to BBR?

On long or lossy paths, usually yes: BBR models bandwidth and round-trip time instead of treating loss as congestion, so it keeps throughput up where CUBIC backs off. Set net.core.default_qdisc=fq with it. On a clean, low-latency datacenter network the gain is smaller, so measure retransmits and throughput on your own traffic before rolling it fleet-wide.

The call we'd make

Change one thing at a time and watch nstat counters between changes, because half these knobs interact and a blind copy-paste of someone's sysctl.conf will bite you. If you only touch two settings, make them net.core.somaxconn (raise it, and match your app's listen() backlog) and switching congestion control to BBR with fq. Those two fixed the connection ceiling and the throughput ceiling for us, and everything else was fine-tuning on top. Never try to enable tcp_tw_recycle; it's gone for good reason. Prefer connection pooling over any TIME_WAIT trickery when you can.

Sources

Explore topics:Linux
React
Share this post
Cite this article
Plain text

Kiril Urbonas (2026). Linux TCP Tuning for High-Throughput Services. DevOpsNess. Updated 2026-11-15. https://www.devopsness.com/blog/linux-tcp-tuning-high-throughput-services

Markdown

[Linux TCP Tuning for High-Throughput Services](https://www.devopsness.com/blog/linux-tcp-tuning-high-throughput-services) (Kiril Urbonas, DevOpsNess, updated 2026-11-15)

KU

Kiril Urbonas

Founder & Chief Editor

604 articles
View all articles by Kiril Urbonas

You might have missed

Evergreen posts worth revisiting.