Load Balancing Algorithms Explained

A load balancing algorithm picks which backend receives the next request. Round robin rotates through servers, least connections picks the least busy one, hashing pins a key to one server, and power of two choices samples two servers and takes the lighter. For a typical HTTP service, use a load-aware method plus active health checks.

Picking wrong shows up as one server pegged while its neighbors idle. If the vocabulary is fuzzy, start with the networking fundamentals guide and the proxy vs reverse proxy vs load balancer breakdown.

L4 vs L7: where the decision happens

L4 (transport) balancers route on IP and port, never read the request, and balance connections. AWS Network Load Balancer and HAProxy mode tcp work here.

L7 (application) balancers terminate the connection, read headers and cookies, and route per request: AWS Application Load Balancer, nginx proxy_pass, HAProxy mode http.

At L4, one HTTP/2 connection carrying a thousand streams counts as one unit of load.

The algorithms, side by side

Round robin and weighted round robin

Requests go 1, 2, 3, 1, 2, 3. Weighted round robin sends proportionally more to bigger boxes: weight 3 gets three requests for every one a weight-1 server sees. nginx's default weight is 1, and round robin is nginx's default method when you name none. HAProxy's roundrobin is dynamic (weights can change at runtime) and its manual caps it at 4,095 active servers per backend; static-rr removes that limit but ignores runtime weight changes.

It breaks when request cost varies: a backend stuck on slow requests keeps getting its full share.

Least connections

The next request goes to the backend with the fewest active connections, weighted by capacity. nginx's least_conn breaks ties with weighted round robin. It is the right default for WebSockets, streaming, database proxies and anything where duration varies a lot.

One honest caveat from the HAProxy manual: it recommends leastconn "where very long sessions are expected, such as LDAP, SQL, TSE" and calls it "not very well suited for protocols using short sessions such as HTTP." With very short requests, connection counts are mostly zero or one and carry little signal. For short HTTP on HAProxy, random(2) is the better load-aware choice.

Least response time

Route to the backend with the lowest average response time and fewest active connections, skipping servers that are free but slow. nginx's least_time header|last_byte does this, and the nginx docs note it was commercial-only before 1.31.0, so it is now in open-source mainline but not yet in the 1.30 stable branch. HAProxy has no direct equivalent in its balance list; the closest is random(2) combined with hash-balance-factor.

Hashing and consistent hashing

Hash a key (client IP, URI, a header) and map it to a backend, so the same key lands on the same server while the pool is stable. Plain hash % N reshuffles almost every key when N changes. Consistent hashing puts servers on a ring so adding or removing one remaps only its slice; nginx's hash ... consistent uses the ketama method for this.

nginx's ip_hash keys on the first three octets of an IPv4 address, so a whole /24, or every user behind one corporate NAT, lands on one backend.

Random and power of two choices

Power of two choices picks two servers at random and sends the request to the less loaded one, avoiding the unlucky pile-ups of pure random. HAProxy's manual says the default of 2 draws "generally shows very good distribution," and suggests 3 or 4 for large farms with very low load. nginx has it as random two least_conn. ALB's closest option is weighted_random.

Configs you can paste

nginx, with least connections, weights and passive health checks:

# /etc/nginx/conf.d/api.conf
upstream api_backend {
    least_conn;
    server 10.0.1.10:8080 weight=3;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=30s;
    keepalive 32;
}

server {
    listen 80;
    location / {
        proxy_pass http://api_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_next_upstream error timeout http_502 http_503;
    }
}

For cache nodes, swap the first line for hash $request_uri consistent;. For power of two choices, use random two least_conn;. Then validate and reload:

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
$ sudo nginx -s reload

HAProxy, with power of two choices and active health checks:

# /etc/haproxy/haproxy.cfg
backend be_api
    mode http
    balance random(2)
    option httpchk
    http-check send meth GET uri /healthz
    http-check expect status 200
    default-server inter 2s fall 3 rise 2
    server api1 10.0.1.10:8080 check weight 30
    server api2 10.0.1.11:8080 check weight 10
    server api3 10.0.1.12:8080 check weight 10

$ haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid
$ sudo systemctl reload haproxy

AWS ALB, switching a target group to least outstanding requests:

$ aws elbv2 modify-target-group-attributes \
    --target-group-arn "$TG_ARN" \
    --attributes Key=load_balancing.algorithm.type,Value=least_outstanding_requests

The ALB docs list three algorithms: round_robin (the default), least_outstanding_requests, and weighted_random. Neither of the last two works with slow start mode, weighted_random cannot be combined with sticky sessions, and only weighted_random supports Automatic Target Weights anomaly mitigation.

Health checks tie it together

Passive checks watch real traffic: nginx's max_fails defaults to 1 and fail_timeout to 10 seconds, and both are ignored for a single-server group. A few users eat the failures first. Active checks poll an endpoint before users hit it. In nginx they are a commercial (NGINX Plus) feature; HAProxy's check does it in the open-source build, with inter defaulting to 2000 ms. ALB target groups always run active checks, and fail open (send to all targets) when too few are healthy.

Error strings and what they mean

Where this breaks

None of these algorithms fixes a capacity problem; they only spread what you have. Per-balancer state also misleads when you run many balancer instances: each nginx worker or ALB node sees only its own connections unless configured otherwise (nginx needs a zone for shared state). Hash-based affinity collapses under hot keys. And for gRPC or HTTP/2 behind an L4 balancer, connection-level balancing pins whole streams to one backend, so balance at L7 instead; the nginx reverse proxy guide covers that setup.

Questions people ask

What is the best load balancing algorithm for HTTP APIs?

A load-aware one with active health checks. On nginx that is least_conn, on AWS ALB least_outstanding_requests, and on HAProxy random(2), because HAProxy's manual says leastconn suits long sessions better than short HTTP. Plain round robin is fine only when every request costs about the same and backends are identical, which is rarer than people assume.

What is the difference between least connections and round robin?

Round robin hands requests out in a fixed rotation and ignores how busy each server is. Least connections looks at current active connections and picks the backend with the fewest, weighted by capacity. Round robin is simpler and fair for uniform requests; least connections adapts when some requests take seconds and others milliseconds, so slow servers stop getting buried.

When should I use consistent hashing?

When the same key should keep hitting the same backend: cache nodes where locality drives hit rate, or legacy apps with in-memory sessions. Consistent hashing remaps only a small slice of keys when a server joins or leaves, unlike hash % N. Avoid hashing on client IP behind NAT, where thousands of users share one address and one backend.

The call we'd make

For a stateless HTTP service, start with a load-aware algorithm and active health checks: least_conn on nginx, least_outstanding_requests on ALB, random(2) on HAProxy. Add consistent hashing only where cache hit rate or unavoidable session state demands it, and move session data into Redis so you can drop stickiness later. Round robin is fine, but "fine" assumes a uniformity you rarely have.

Sources

• nginx: ngx_http_upstream_module reference
• HAProxy 3.4 Configuration Manual (balance, check, inter)
• AWS: Edit target group attributes for your Application Load Balancer