API throughput is the number of requests an API completes per second, and it is capped by whichever resource saturates first: a worker pool, a database connection pool, CPU, or a rate limit set by a provider you call. To raise it, find that cap with a load test, widen or remove it, and confirm that response time and errors stay flat at the higher rate.
This post was first published as a case study of a financial platform. The company and results in that version could not be verified, so it has been rewritten as a how-to with a worked example. The numbers below are arithmetic examples, not results from a real platform.
Key takeaways
- Throughput, concurrency and response time are tied together by Little’s Law: concurrency = throughput x response time.
- A pool of N connections, each held for t seconds per request, cannot serve more than N / t requests per second, however fast the rest of the code is.
- Past the cap, throughput stays flat while response time grows without limit, because requests queue.
- A rate limit at a provider is a throughput cap you do not control. Design around it.
How are throughput and response time related?
They are linked by Little’s Law. For a stable system, the average number of requests in the system (L) equals the arrival rate (lambda) times the average time each request spends in the system (W): L = lambda x W (en.wikipedia.org/wiki/Little%27s_law, read 2026-10-23). The law holds whatever the arrival pattern or service order, as long as the system is stable.
Read it as a budget. If your service can hold 50 requests in flight at once (threads, workers, connections) and each takes 100 ms, the most it can complete is 50 / 0.1 = 500 requests per second. If response time rises to 250 ms under load, the same 50 slots give 50 / 0.25 = 200 requests per second. Slower responses reduce throughput, which is why the two numbers have to be read together. The basics are in what is throughput in performance testing and how throughput and response time relate, so this post goes straight to the limits.
What caps the throughput of an API?
Four things, roughly in the order teams hit them:
| Cap | How it shows up in a test |
|---|---|
| A fixed pool (threads, workers, DB connections) | Throughput flat, response time rising, CPU not saturated |
| CPU or memory on the hosts | CPU near its ceiling as throughput flattens; restarts or out-of-memory kills |
| A downstream service | Time spent in outbound calls grows with load |
| A rate limit (yours or a provider’s) | A step up in HTTP 429 responses at a fixed request rate |
The first one is the most overlooked, because the hardware looks idle while the API is already full.
Worked example: a connection pool that limits throughput
An endpoint runs two database queries per request. The pool has 20 connections and each query holds a connection for 25 ms.
- Connection time per request: 2 x 25 ms = 50 ms.
- Capacity: 20 connections / 0.050 s = 400 requests per second.
At 300 requests per second, the pool is busy 300 x 0.050 = 15 connections on average, which is 75 percent. It works, but with little headroom.
Now push the arrival rate to 450 requests per second. The pool serves 400, so 50 requests per second join a queue. After 10 seconds, 500 requests are waiting, and a new request waits about 500 / 400 = 1.25 s before it gets a connection. The queue keeps growing for as long as the overload lasts, so response time climbs and clients start to time out, even though no single query is slow.
There is a second effect. If the database slows down under that load and each query now takes 50 ms, connection time per request doubles to 100 ms and capacity falls to 20 / 0.100 = 200 requests per second. The system loses capacity exactly when it is busiest.
The fix depends on where the time goes. Cutting the query time to 15 ms (an index, a lighter query) gives 20 / 0.030 = 667 requests per second. Combining the two queries into one round trip does something similar. A bigger pool helps only while the database has spare capacity: the HikariCP project’s pool-sizing guide suggests starting near (cores x 2) + effective spindle count and load testing around it, since throughput can fall when the pool is far too large (github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing, read 2026-10-23). Test your own database rather than copying a number. The steps for finding the right pool size are in how to optimize database connection pools.
How do you read throughput in a load test?
Raise the number of virtual users in steps and record throughput, p95 response time and error rate at each step. With no think time, Little’s Law gives virtual users = throughput x response time, so you can check the test itself:
| Virtual users | Throughput | Response time (users / throughput) | What it says |
|---|---|---|---|
| 50 | 200 req/s | 0.25 s | Healthy |
| 100 | 400 req/s | 0.25 s | Still scaling linearly |
| 200 | 400 req/s | 0.50 s | Saturated: twice the users, same throughput |
The bend between 100 and 200 users is your capacity. Report the throughput at the knee, not the highest figure the test reached, because the highest figure sits past the point where users were already waiting twice as long. Think time and ramp-up change the numbers, so see how to calculate concurrent users before choosing the load profile.
Run it from the cloud so the generator is not the limit. LoadFocus load testing runs from 25 cloud regions, reports throughput and the 50th, 90th and 99th percentiles, and applies pass or fail thresholds on p95 and p99 response time, error rate and minimum throughput (loadfocus.com/load-testing, read 2026-10-23).
Why does a rate limit cap throughput, and what do you do about it?
If your API calls a provider that allows, say, 600 calls per minute, your sustained ceiling is 10 calls per second per key, whatever your own servers can do. Past it you get HTTP 429, which RFC 6585 defines as “too many requests”, and the provider may send a Retry-After header (RFC 6585 and RFC 9110, rfc-editor.org).
The throughput ceiling for the requests that need that call is the limit divided by the calls each request makes. With 10 calls per second allowed and 2 calls per request, that is 5 requests per second. Ways to raise it without a higher tier: cache responses that repeat, batch calls where the provider offers a batch endpoint, and drop calls the request does not need. Retry with exponential backoff and jitter, never in a tight loop, or the retries use up the limit themselves. A load test against a provider’s production API can break its terms of use and affect its other customers, so use a sandbox, a stub or a mock of the provider, and confirm what the provider allows. See common causes of API rate limiting.
What other changes raise throughput?
- Caching responses that many requests share, so fewer requests reach the database. Check the hit rate with varied test data.
- Asynchronous work. Accept the request, queue the heavy part and return an id, so a worker is not held for the whole job. Clients then poll or receive a callback.
- Smaller responses and pagination, which cut time per request and so raise the Little’s Law ceiling.
- More instances behind a load balancer, once the per-instance cap is understood. If the shared database is the cap, more API instances only add to the queue in front of it.
The post on common API performance issues lists how to detect each cause, and reducing API latency with distributed load testing covers the network side.
Does anything change for financial APIs?
The method is the same. What changes is the cost of a failed or duplicated request. Writes that move money should be idempotent, so a retry after a timeout cannot apply the same payment twice. Test with synthetic data, never with real customer or account data, and do not point a load test at a production payment or market-data provider without their agreement. Regulatory and compliance requirements for your platform come from the applicable standards and your own legal and compliance advisers, not from a performance article, so this post makes no claims about them.
Frequently Asked Questions
What is API throughput?
The number of requests the API completes in a given time, usually requests per second. It is only meaningful alongside response time and error rate at that same rate.
How do I calculate the maximum throughput of a service?
Divide the number of requests it can hold in flight by the time each request holds it. Twenty connections held 50 ms per request give 20 / 0.05 = 400 requests per second. This is an upper bound; the real figure is lower once queueing and other limits apply.
Why does throughput stop growing when I add users?
Because something is full. Extra users then wait in a queue, so response time rises while throughput stays flat. Find the resource that saturates: a pool, CPU, a downstream service or a rate limit.
Is high throughput always good?
No. A high request rate with rising p95 and errors means the system is past its capacity. Report the throughput at which response time and error rate are still within your targets.
How is throughput different from latency?
Throughput counts work completed per second. Latency is the delay before a response begins to arrive. They interact: slower responses reduce the throughput a fixed pool can deliver.
Sources
- Little’s law (en.wikipedia.org/wiki/Little%27s_law, read 2026-10-23).
- HikariCP, About Pool Sizing (github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing, read 2026-10-23).
- RFC 6585, 429 Too Many Requests, and RFC 9110, Retry-After (rfc-editor.org).
- LoadFocus load testing page (loadfocus.com/load-testing, read 2026-10-23).
- All figures in the worked examples are invented for illustration and recomputed in this post.
Published with PostNext
