Load Balancer Testing: 6 Checks, Run Against nginx

Load balancer testing means putting the balancer under real load and checking six things: that traffic spreads across backends, that one load generator is not pinned to one server, that a slow backend does not drag the rest down, that a failed backend is taken out without errors, that a hung backend is detected, and that long-lived connections get rebalanced.

We ran each check against nginx 1.31.8 with three backends and k6 v2.3.0. Two results stood out. When one backend froze, every virtual user was stuck on it within a fraction of a second, nothing completed for about 30 seconds, and k6 still reported 0% errors and a p95 of 3.12 ms. And when a backend process died, GET requests failed over cleanly while 20 POST requests returned errors.

Key Takeaways

• Count which backend answered every request. A response header that names the backend, plus one counter per backend in the load script, turns “it looks fine” into numbers.
• A single load generator is a single client IP. With ip_hash, all 82,964 of our requests went to one backend.
• With round robin, one backend 50 ms slower cut total throughput from 9,135 to 1,088 requests per second. least_conn kept 6,336.
• nginx retries a failed GET on the next server but not a POST. Test failover with the methods your users send.
• A frozen backend is worse than a dead one, and it hides in the summary: watch requests per second over time, not only error rates and percentiles.

What is load balancer testing?

Load balancer testing checks that a load balancer spreads traffic the way its configuration says, keeps working when a backend slows down or fails, and does not become the bottleneck itself. It differs from a plain load test of the application in what you measure: which backend answered each request, how the split changes as backends degrade, and how many requests fail during the change.

The questions it answers are practical. Will a deploy that restarts one server drop requests? Will one slow database replica behind one app server slow every user? Is session affinity sending a whole office to one machine? A load test of the application tells you how much traffic the system takes. Testing the balancer tells you what happens to that traffic on its way in.

What should a load balancer test check?

Six checks cover the behaviour that breaks in production. Each one needs load running while you look, because a balancer that splits three curl requests evenly can still pin a real load test to one server, and a failover that works at idle can drop requests that are in flight. The table lists each check with what we measured on nginx.

How we set up the test

The lab was nginx 1.31.8 from the official nginx:alpine image, running in Docker Desktop on an Apple M1 Max, with three backends that put their own name in an X-Backend header. For the slow-backend check, the backends were small Node.js servers on the host, one of them with a 50 ms delay. For the failure checks, they were nginx containers on one Docker network, so a stopped or frozen backend behaved like one on a real network. k6 v2.3.0 sent the load: 20 virtual users with no think time, usually for 10 seconds.

The load balancer used round robin, the nginx default, with a shared memory zone (the next sections show why the zone matters):

upstream app {
    zone app 64k;
    server be1:80;
    server be2:80;
    server be3:80;
}

server {
    listen 80;
    location / {
        proxy_pass http://app;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

The k6 script reads the header on every response and adds one to that backend’s counter, so the end-of-test summary prints the split:

import http from 'k6/http';
import { check } from 'k6';
import { Counter } from 'k6/metrics';

const hits = { b1: new Counter('hits_b1'), b2: new Counter('hits_b2'), b3: new Counter('hits_b3') };

export const options = { vus: 20, duration: '10s' };

export default function () {
  const res = http.get('http://localhost:18080/');
  const backend = res.headers['X-Backend'];
  if (hits[backend]) hits[backend].add(1);
  check(res, { 'status 200': (r) => r.status === 200 });
}

If your backends do not identify themselves, add a header with the hostname at the app or at the balancer, and remove it again after the test if you do not want it public. The percentages and timings below come from one machine and show patterns, not benchmarks.

Does traffic spread evenly across backends?

Under load, yes: round robin gave the three backends 31,069, 31,066 and 31,062 of 93,197 requests in 10 seconds. A quick manual check can tell a different story. Twelve curl requests, each on a new connection, gave backend one 10 of them, because every nginx worker process keeps its own round-robin position unless the upstream has a shared memory zone.

Our nginx started 10 worker processes, one per CPU. Without a shared zone, each worker keeps its own round-robin position and starts at the first server in the list, so twelve new connections spread over several workers mostly reach that first server. The zone directive moves the group’s “configuration and run-time state” into memory “shared between worker processes”, in the words of the nginx upstream documentation. With zone app 64k; added, the same twelve requests split 4, 4 and 4.

The lesson is about the check, not about nginx being broken. A handful of manual requests says little about distribution. Send a few thousand through the load script and read the counters. The zone also matters beyond the test, because failure counts for max_fails are shared through it too.

Why does a load test send all its traffic to one server?

Because the load test comes from one client IP address, and affinity rules hash on that address. With ip_hash turned on, all 82,964 requests from our single k6 machine went to the third backend, and the other two received nothing. The test result would have measured one backend’s capacity while reporting it as the whole cluster’s.

The nginx documentation adds a detail that widens the problem: for IPv4, ip_hash uses only “the first three octets of the client IPv4 address”. Several load generators in the same /24 subnet, such as cloud instances started together, can still hash to the same backend. Cookie-based sticky sessions have a similar effect inside a script, since every virtual user that keeps its cookie jar stays on the server it first reached, which is correct behaviour but no longer a spread.

To test affinity honestly, either send load from several networks so the client addresses differ, or test affinity and distribution separately: confirm stickiness with a few clients, then switch affinity off for the throughput test. The same single-IP effect trips per-IP rate limits and DDoS protection on the balancer or a WAF in front of it.

What happens when one backend is slow?

With round robin, the whole pool slows to the pace of the slow backend. We added 50 ms to every response from one of three backends. Round robin kept sending it a third of the requests, and total throughput fell from 9,135 to 1,088 requests per second, a drop of 88%, with p95 response time going from 3.13 ms to 51.5 ms. least_conn moved traffic away and held 6,336 requests per second.

Throughput fell further than the latency alone explains because a closed-model test like this one has a fixed number of virtual users, each waiting for its response before sending the next. Every user that lands on the slow backend is out of the pool for 50 ms, so the fast backends sit idle waiting for work. Real users who keep clicking behave more like an open model, where the slow backend builds a queue instead. Either way the cause is the same, and a slow node is a realistic scenario: a noisy neighbour, a cold cache after a deploy, a replica lagging behind.

The least_conn row has a catch: p95 looked healthy, but p99 still showed the 51 ms requests, because 2.1% of traffic still reached the slow backend. That is why percentiles above p95 belong in a load balancer report.

Does failover work without errors?

For GET requests, it did. We stopped the web server process on one backend 5 seconds into a 15-second run, leaving the machine up. k6 sent 234,507 GET requests and none failed. The nginx error log showed 20 connections reset by that backend, and nginx retried each one on another server before answering the client. In a separate run with POST requests, 20 failed.

The difference is documented nginx behaviour. The default proxy_next_upstream error timeout; retries a request on the next server after a connection error or timeout, but since nginx 1.9.13, requests with a non-idempotent method (POST, LOCK, PATCH) “are not passed to the next server if a request has been sent to an upstream server”. A second run with POST gave 20 failures out of 214,407 requests, all inside a 12 ms window right after the stop. After that, nginx marked the backend unavailable (one failure within fail_timeout, 10 seconds by default) and sent it nothing for the rest of the run.

Twenty errors is a small number, but a checkout or a payment is usually a POST. If your failover test only sends GETs, it will report a clean failover that your users will not get. Run it with the same mix of methods as production, and decide whether the POST endpoints can be retried safely before turning on non_idempotent.

Why did the test pass while traffic stopped?

Because a frozen backend produces no errors, only waiting. We paused one backend container as a 20-second test started, so it accepted connections but never answered. Within a fraction of a second, all 20 virtual users were waiting on it, and no request completed for about 30 seconds. The k6 summary afterwards showed 0.00% failed requests and a p95 of 3.12 ms.

Here is what happened, second by second. Round robin kept giving the frozen backend a third of new requests. Each virtual user that drew it waited for a reply, and with requests taking about a millisecond elsewhere, all 20 users drew it within 0.3 seconds. nginx did not mark the backend as failed, because nothing had failed yet: proxy_read_timeout defaults to 60 seconds. When we unfroze the backend, the 20 stuck requests completed after 29.8 seconds each, and the test, planned for 20 seconds, ended at 30.1.

The summary looked healthy because only 20 of 4,205 requests were slow, under 0.5%, so even p99 stayed at 7.14 ms. The signs were elsewhere: 4,205 requests where a healthy run completes hundreds of thousands (the 15-second failover run above sent 234,507), a maximum of 29.78 seconds, and a run that overshot its duration. In production, a backend that freezes this way (a deadlock, a full disk, a stuck garbage collection) would hold every connection nginx gives it for up to 60 seconds before the first timeout counts as a failure.

Two changes help. In the test, watch requests per second over time and set a k6 threshold on the request count or on max, not only on error rate and p95. In the configuration, set proxy_read_timeout and proxy_connect_timeout to what your slowest healthy endpoint needs rather than the 60-second default, and add active health checks if your balancer has them.

Does a TCP (layer 4) load balancer behave differently?

Yes: a layer 4 balancer picks a backend per connection, not per request, so a few long-lived connections can leave backends idle. We put nginx’s stream module in front of the same three backends. With backend connections allowed to last a million requests, 2 virtual users sent 31,505 requests over 2 keep-alive connections, and the third backend received none. With 4 users, one backend got half of the traffic.

The first row is why this trap often goes unnoticed with nginx backends: keepalive_requests closes a connection after 1,000 requests by default (100 before version 1.19.10), the client reconnects, and the new connection is balanced again. Backends or balancers that keep connections open longer, such as gRPC servers or cloud network balancers in front of services with long keep-alive settings, pin each client connection for its whole life.

For a layer 4 test, use many more virtual users than backends. Turning keep-alive off for the whole test is a poor fix: it spread our traffic evenly but cut throughput from 19,158 to 3,684 requests per second, so the test then measured TCP handshakes rather than your service.

How to run a load balancer test with LoadFocus

LoadFocus runs load tests and k6 or JMeter scripts from 25 cloud regions, and you can split one test’s users between regions by percentage, with each region running its share on its own cloud instances. That helps with the single-IP problem above: traffic arrives from many client addresses, closer to what ip_hash, sticky sessions and per-IP limits see in production.

The k6 script from this article, pointed at your balancer’s public address, runs as a cloud load test on LoadFocus, but LoadFocus results show response times, errors and throughput, not custom k6 counters, so read the per-backend split from your balancer’s or backends’ access logs during the run. Because the load generators start on demand, their IP addresses are not known in advance. If your load balancer or WAF throttles heavy traffic from one address, allow the LoadFocus ranges first; the whitelist IPs page lists them and names this case. Run the failover checks during the test by stopping or freezing one backend yourself, and read the per-second charts for the drop that the summary hides.

Frequently Asked Questions

How do you test if a load balancer is working?

Send load through it and count which backend answered each request, using a response header that names the backend and one counter per backend in your load script. Then repeat the run with one backend slowed down, stopped and frozen, and compare the error count, throughput and per-second request rate with the healthy run.

Why does my load test send all traffic to one server?

Most often because the balancer uses source-IP affinity and your test comes from one machine. nginx’s ip_hash keys on the first three octets of an IPv4 address, so even several generators in the same subnet can land on one backend. Cookie-based sticky sessions do the same per virtual user. Generate load from several networks, or switch affinity off for the throughput test.

Does nginx retry failed requests on another backend?

By default nginx retries on the next server after a connection error or a timeout, for idempotent methods such as GET. Since nginx 1.9.13 it does not retry POST, LOCK or PATCH requests once they were sent to a backend, unless the non_idempotent option is set. In our test that meant 0 failed GETs and 20 failed POSTs when a backend process stopped.

What is the difference between testing a load balancer and load testing an application?

An application load test measures how much traffic the whole system handles and how fast it answers. A load balancer test measures how that traffic is shared between backends and what happens to it when one backend slows down, stops or freezes. You usually run both from the same script, adding a counter per backend and a failure step during the run.

How many virtual users do you need to test a load balancer?

Far more than you have backends, and enough to keep every backend busy. With a layer 4 balancer each connection is pinned to one backend, so in our test 2 users on 2 keep-alive connections left one of three backends idle. Distribution figures from a few manual requests are also misleading, so judge the split from thousands of requests.

Sources

• nginx documentation, read 7 November 2026: ngx_http_upstream_module (zone, ip_hash, least_conn, max_fails and fail_timeout defaults), ngx_http_proxy_module (proxy_next_upstream default and the non-idempotent rule since 1.9.13, proxy_read_timeout and proxy_connect_timeout defaults), and ngx_http_core_module (keepalive_requests default 1000, 100 before 1.19.10).
• LoadFocus regions, per-region load split and source IPs: loadfocus.com/k6-load-testing and loadfocus.com/whitelist-ips, read 7 November 2026.
• Test runs: nginx 1.31.8 (nginx:alpine, 10 worker processes) in Docker Desktop 29.4.3, k6 v2.3.0 (official macOS arm64 release, checksum-verified), Node.js 23.11.1 backends, on an Apple M1 Max, 7 November 2026. Timings show patterns, not benchmarks.

Related reading

• Load Testing vs Stress Testing: Key Differences
• Validating Infrastructure Auto-Scaling with Load Testing
• k6 Scenarios Explained: All 6 Executors, Tested
• hey Load Testing Tutorial: Flags, Output and Pitfalls