To reduce API latency with distributed load testing, run the same load against your API from several regions, compare p95 and p99 response times per region, split each request into DNS, connect, TLS and server wait, and fix the largest phase first. Then repeat the identical test to confirm the change worked. A test from one region cannot do this, because it measures one network path and hides the slowest ones your users take.
This post was first published as a case study. The company and figures in that version could not be verified, so it has been rewritten as a how-to. The numbers below are arithmetic examples, not results from a real customer.
Key takeaways
- Distributed generation exposes latency that a single-region test hides entirely.
- The fix comes from finding which phase or dependency dominates the tail, not the average.
- Re-testing after each change is what shows which one actually worked.
Why test API latency from more than one region?
A load generator in the same region as your API sees almost no network delay, so it reports the fastest numbers your API will ever have. Users in other regions pay the extra round trips on every new connection, and their requests are the ones that time out first. Running the same script from a nearby and a distant region separates “the server is slow” from “the route is long”.
LoadFocus runs load tests from 25 cloud regions, with response time, latency and 50th, 90th and 99th percentiles per run (loadfocus.com/load-testing, read 2026-10-23). The method below works with any tool that can run from several locations, including JMeter and k6.
Step 1: Baseline p50, p95 and p99 per region
Run a modest, steady load from each region where you have users, with the same script, the same duration and the same request mix. Record p50, p95 and p99, plus the error rate, per region. Do not look at the average first.
Averages hide the tail. A worked example: 100 requests, where 98 take 200 ms and 2 take 2,000 ms. The mean is (98 x 200 + 2 x 2,000) / 100 = 236 ms, which looks healthy, but the p99 is 2,000 ms. Anyone who hits one of those slow requests waits ten times longer than the dashboard suggests. The post on why percentiles beat averages covers the definitions.
Step 2: Split each request into phases
A slow response has a cause in one of a few phases. Measure them separately:
| Phase | What it covers | Where to read it |
|---|---|---|
| DNS lookup | Resolving the hostname | curl time_namelookup; LoadFocus API monitoring splits out DNS |
| TCP connect | One round trip to open the connection | curl time_connect; k6 http_req_connecting |
| TLS handshake | One to two more round trips | curl time_appconnect; k6 http_req_tls_handshaking |
| Server wait | Time to first byte: server work plus one round trip | curl time_starttransfer; k6 http_req_waiting |
| Download | Receiving the body | k6 http_req_receiving |
curl’s timers are cumulative from the start of the transfer, so the DNS time is the first value and each later timer includes the ones before it (curl.se manual, read 2026-10-23). k6 defines http_req_duration as sending plus waiting plus receiving, and it excludes the initial DNS lookup and connection time (grafana.com/docs/k6, read 2026-10-23). If you read only http_req_duration, you cannot see DNS or connection cost at all.
A quick manual check from a machine in the region you care about:
curl -s -o /dev/null -w "dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n" https://api.example.com/healthStep 3: Read the pattern per region
Compare the phases across regions. The pattern tells you what to fix.
An illustration, using arithmetic only. Suppose the server needs 90 ms for an endpoint, and a request on a fresh HTTPS connection needs about three round trips (TCP, a TLS 1.3 handshake, the request) before the first byte:
| Region (round trip) | New connection: 3 x RTT + 90 ms | Reused connection: 1 x RTT + 90 ms |
|---|---|---|
| Same region (20 ms) | 150 ms | 110 ms |
| Another continent (90 ms) | 360 ms | 180 ms |
| Far side of the world (180 ms) | 630 ms | 270 ms |
The server cost is the same in every row. A distant user pays 360 to 540 ms more per fresh connection purely in handshakes, and connection reuse cuts the distant case by about half. Real measurements will differ; the point is the shape. If your distant regions show a large connect and TLS share, you have a network problem. If every region shows a large server wait that grows with load, you have a backend problem.
Step 4: Fix the biggest phase first
Match the fix to the phase that dominates, in this order of likely payoff:
- Long connect and TLS times from far regions. Reuse connections (keep-alive, HTTP/2 or HTTP/3, connection pools in server-to-server clients), enable TLS session resumption, and terminate TLS close to users through a CDN or edge proxy.
- Cacheable GET responses. Add
Cache-Controlheaders and let a CDN serve them from an edge location near the user. Do not cache personalised or authorised responses unless the cache key accounts for the user. - High DNS time. Check the record TTL, and whether geo-aware DNS sends users to the nearest endpoint. A wrong record that sends one region to a far endpoint shows up as a regional spike only distributed testing can see.
- Server wait that grows with load. Profile the slow endpoints: N+1 queries, missing indexes, serial calls to other services that could run in parallel, and oversized payloads. Compress responses and return only the fields the client needs.
- A slow third-party dependency. Set a timeout, call it asynchronously if the user does not need the result immediately, and cache what you can. The p99 of your API cannot be better than the p99 of a dependency you call synchronously on every request.
- Retry storms. Clients that retry immediately on a timeout multiply load on an already slow service. Use exponential backoff with jitter.
- Whole-region distance. If the dominant cost is physical distance and the data allows it, serve users from a nearer deployment. The posts on multi-region deployment and edge locations explain the trade-offs.
Change one thing at a time. If you change DNS, caching and a query in the same release, you cannot tell which one helped.
Step 5: Retest and set a guard
After each change, rerun the exact baseline test from the same regions and compare p95 and p99, not the mean. A fix counts only if the tail moved. Then keep it from regressing: set pass or fail thresholds on p95 and p99 response time, error rate and throughput so a slow release fails the build. LoadFocus lets you set these per test, run them on a schedule, and gate a deploy from the GitHub Action or the verdict API (loadfocus.com/load-testing, read 2026-10-23). For production, API monitoring runs checks from several locations and splits response time into wait, DNS, TCP connect, first byte and download (loadfocus.com/api-monitoring, read 2026-10-23).
Common mistakes when measuring API latency
- Reading the mean. Use percentiles, per region.
- Using one region and calling it global. Add the regions your users are in, and test each at the same load.
- Reading latency as response time. Latency is the delay before data starts to arrive. Response time adds the server work and the transfer. The post on how throughput and response time relate covers how they move together under load.
- Opening a new connection per request in the test script when real clients reuse them, which inflates every number.
- Starting before the generator is checked. If the load generator’s CPU is saturated, your response times include its own delay.
- Short tests. A five-minute run can miss periodic slowdowns such as garbage collection pauses or cache expiry. Run long enough to see them.
Frequently Asked Questions
What is API latency, and how is it different from response time and throughput?
Latency is the delay before the first byte of data arrives, which includes DNS, connection setup and the network round trip. Response time is the whole request: latency plus server processing and the transfer of the body. Throughput is how many requests the API handles per second. A slow route raises latency and response time without changing the server’s throughput, while a saturated server lowers throughput and raises response time.
Which metrics matter most when monitoring API latency?
p50, p95 and p99 response time per region, the error rate, and the split between connect, TLS and server wait. The median shows the typical user, and p95 and p99 show the slow tail that triggers timeouts and complaints.
How do I reduce API latency?
Find which phase dominates, then fix that phase: connection reuse and edge termination for network cost, caching for repeatable GET responses, query and payload work for server time, timeouts and asynchronous calls for slow dependencies. Retest after every change.
How do I tell a network cause from a server cause?
Compare phases across regions. If connect and TLS times are large and grow with distance while server wait is steady, the network is the cause. If server wait is large in every region and climbs as load rises, the server is the cause.
How long should a distributed API load test run?
Long enough to include the slowdowns you care about. Several minutes at steady load gives stable percentiles, and longer runs catch periodic effects such as cache expiry or garbage collection. Use a schedule to repeat the test and watch the trend.
Sources
- Grafana k6 documentation, built-in HTTP metrics (grafana.com/docs/k6/latest/using-k6/metrics/reference, read 2026-10-23).
- curl manual,
--write-outvariables (curl.se/docs/manpage.html, read 2026-10-23). - LoadFocus load testing and API monitoring pages (loadfocus.com/load-testing, loadfocus.com/api-monitoring, read 2026-10-23).
