What is Latency?

What is latency?

Latency is the delay between sending a request and getting an answer. For web traffic, MDN defines network latency as the time a packet of data needs to travel from source to destination, usually measured as a round trip. Load testing tools often use a narrower meaning: the time until the first byte of the response arrives. Both are measured in milliseconds.

Latency, response time and TTFB: which number is which?

The word is used for three related things, and tools do not agree on it:

• Network latency: the travel time between two machines, one way or as a round trip (RTT). It depends on distance and the route, not on your code.
• Latency in JMeter: the glossary says JMeter measures it "from just before sending the request to just after the first response has been received". Elapsed time runs until the last part of the response has arrived.
• Time to first byte: the k6 metric http_req_waiting is the time spent waiting for the response, which k6 itself calls TTFB. http_req_duration is sending plus waiting plus receiving, and connection setup is reported separately. See time to first byte.

Response time is the whole wait until the response is complete, which is what users feel. When you compare two reports, check which of these definitions each tool reports before comparing the numbers. The blog post on response time in performance testing goes deeper on that side.

Where latency comes from

• Connection setup. MDN notes that a first request includes a DNS lookup, a TCP handshake and TLS negotiation. Later requests on the same connection skip most of it.
• Distance. A user far from your server pays more round-trip time on every step. A CDN shortens that for cacheable content.
• Server work. Application code, database queries and calls to other services show up as waiting time before the first byte.
• Queuing under load. When requests wait for a free connection, thread or database slot, latency climbs even though each unit of work is unchanged.
• Download size. MDN states that receiving time depends on network capacity and asset size.

Latency vs throughput

Latency is how long one request takes. Throughput is how many requests the system completes per unit of time. A system can be fast for one user and still collapse at a hundred, so the two have to be watched together. How latency grows as load grows is what load testing and capacity testing reveal. The blog post how throughput and response time relate shows the shape of that curve.

Why averages hide the problem

Take 99 requests at 100 ms and one at 30 seconds. The average is about 400 ms, which looks acceptable, while one user waited half a minute. Percentiles describe the distribution instead: p50 is the median, p95 means 95% of requests were faster, p99 means 99% were. Google's paper The Tail at Scale explains why rare slow responses matter more as systems grow. Read why percentiles beat averages for the testing side.

Example: a latency threshold in a test

In k6 you can fail a run when the tail gets too slow. The expressions are evaluated at the end of the test, and if one is false the test fails and k6 exits with a non-zero code:

thresholds: { http_req_duration: ['p(95) < 500', 'p(99) < 1000'] }

The numbers are an example. Set yours from what your users and your SLA need, not from a default.

How to measure and reduce latency

• Measure from where users are. LoadFocus load testing runs from 25 cloud regions and reports response time, latency, throughput, standard deviation and the 50th, 90th and 99th percentiles, with pass/fail thresholds on P95 and P99. A test from one region says little about users elsewhere.
• Watch it in production. API monitoring tracks response time on a schedule (free checks run every 5 minutes) and can warn you when an endpoint passes a degraded threshold. The metrics worth tracking are listed under API performance metrics. The blog case study on reducing API latency shows the workflow.
• Cut the cost per request. Reuse connections, cache what you can, serve static files from a CDN, reduce the number of hostnames, and fix slow queries first.

Common mistakes

• Reporting only the average.
• Testing from the same region as the server and generalizing to all users.
• Comparing numbers from tools that define latency differently.
• Ignoring the first request, which carries DNS, TCP and TLS cost.
• Treating a latency target as a throughput target, or the other way round.

FAQ: Latency

What is a good latency?

There is no universal number. It depends on the type of request, the user and your SLA. Set a target on a percentile such as p95, measure it under realistic load and from your users' regions, and tighten it as you improve.

What is the difference between latency and response time?

Latency usually means the delay before the response starts, or the network travel time. Response time is the full wait until the response is complete. Tools define both slightly differently, so check the documentation of the tool you use.

What is the difference between latency and bandwidth?

Latency is delay. Bandwidth is how much data the connection can carry per second. A large download is limited by bandwidth, a chatty page with many small requests is limited by latency.

How is latency different from TTFB?

TTFB measures the time until the first byte of a response arrives, so it contains network latency plus the server's waiting time. Some tools, including JMeter, call that number latency.

Why is p99 latency higher than p50?

Most requests are fast and a few are slow because of cache misses, garbage collection, locks or queues. p99 shows those few, which is why it is the number to watch for the worst experiences.