Key takeaways
- Run two kinds of test: one service at a time with its dependencies controlled, and end-to-end journeys through the gateway. Each finds problems the other cannot.
- Treat dependencies as part of the test: mock them with realistic delay and errors, and check the mock against the real service so it does not drift.
- Watch retries, timeouts, circuit breakers and autoscaling, because most microservice failures under load are interactions between them, not one slow function.
- Use traces and per-service metrics for the same time window as the test, or you will find the symptom and not the source.
How do you load test microservices?
Load test microservices at two levels. First test each service on its own, at a controlled request rate, with its dependencies stubbed or pinned, so you learn that service’s limit. Then test the journeys that cross services through the gateway, with real dependencies, so you learn how latency, retries and scaling interact. Judge both with percentiles per service and per journey, and read them next to traces and resource metrics from the same window.
The tooling comes second. For the LoadFocus workflow (one saved test per service, thresholds, a CI gate and schedules) see Automated Load Testing for Microservices, which this post does not repeat.
Service-level tests or end-to-end tests: which do you need?
You need both, for different questions. A service-level test answers “how much can this service take, and what breaks first?” An end-to-end test answers “does the journey a user takes stay within its budget when everything runs together?”
| Service-level test | End-to-end test | |
|---|---|---|
| Target | One service through its own API | A user journey through the gateway |
| Dependencies | Stubbed or pinned to known behaviour | Real, or real in staging |
| Finds | The service’s saturation point, slow queries, connection pool limits | Cascading latency, retry amplification, mismatched timeouts, shared-resource contention |
| Weakness | Cannot see how neighbours react | Hard to tell which service to blame without traces |
| When | On every change to the service | Before release and on a schedule |
A service that passes alone can still fail in the journey. Suppose the order service handles 200 requests per second by itself, but each order calls the inventory service, which allows 120. The service-level test passes, and the end-to-end test is the one that shows orders stalling at 120.
How should you drive the load: users or request rate?
Prefer a fixed arrival rate for service-level tests. In a closed model, where each virtual user waits for its response before sending the next request, a slow service receives fewer requests per second, which hides the very overload you want to see. The k6 documentation describes this: with a closed model, slower response times mean longer iterations and a lower arrival rate, and it notes that some testing literature calls the problem coordinated omission. In the open model, response times no longer influence the load on the target system. k6 implements the open model with the constant-arrival-rate and ramping-arrival-rate executors.
A service-level test in k6 that holds 50 requests per second against one route, with thresholds scoped to that service by tag:
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
orders_service: {
executor: 'constant-arrival-rate',
rate: 50,
timeUnit: '1s',
duration: '5m',
preAllocatedVUs: 100,
maxVUs: 300,
},
},
thresholds: {
'http_req_duration{service:orders}': ['p(95)<400'], // example limit, set your own
'http_req_failed{service:orders}': ['rate<0.01'],
},
};
const BASE = __ENV.BASE_URL; // your staging gateway
export default function () {
const res = http.get(`${BASE}/orders/recent`, {
tags: { service: 'orders' },
headers: { 'X-Load-Test': 'run-42' }, // lets you find these requests in traces
});
check(res, { 'status is 200': (r) => r.status === 200 });
}If maxVUs is too low the run cannot sustain the rate, so watch the dropped iterations count in the k6 summary. A failed threshold makes k6 exit with a non-zero code, which is what a pipeline gate needs. For gRPC services, k6 has a native k6/net/grpc module (since v0.49.0) for unary and streaming calls.
How do you handle dependencies, contracts and mocks?
Control every dependency you do not own, and keep your stand-in honest. A third-party payment API will rate-limit or bill you if you hammer it, and a dependency that is still in development cannot be tested at all. Replace it with a stub, but make the stub behave like the real thing:
- Realistic latency. An instant 200 OK makes your service look faster than it will be. Replay the response-time distribution you measured from the real dependency, not its average.
- Errors and timeouts. Return occasional 5xx responses, slow responses and dropped connections, because that is what your retry logic will meet.
- Limits. If the real API throttles at a rate, make the stub throttle at that rate and return 429.
A stub drifts. The real service changes a field, adds a limit or slows down, and your stub keeps answering as before, so the test keeps passing against a system that no longer exists. Contract tests, which check that a consumer’s expectations still match a provider’s behaviour, keep the two aligned, and a periodic run against the real service in staging catches what neither does. Avoid these common mocking mistakes in API testing lists the traps. Tools such as WireMock can serve the stubs with configurable delays and faults.
What happens to retries, timeouts and circuit breakers under load?
They decide whether a slow dependency becomes a partial slowdown or an outage, and a load test is the only safe place to see which. Three interactions to test on purpose:
- Timeouts that add up. If service A waits up to 5 seconds for B and retries 3 times, one call can hold A’s resources for 20 seconds (4 attempts of 5 seconds). With a gateway timeout of 10 seconds, the user has already given up while A is still working. Compare every client timeout with the timeout of the caller above it.
- Retry amplification. Retries add load to a service that is already struggling. The Azure Architecture Center’s rate limiting pattern warns that blind or overly aggressive retries can increase load and create retry storms, and recommends a limited number of retries with small random delays and honouring back-pressure signals such as HTTP 429 with
Retry-After. Raise a dependency’s latency during a test and check whether the request rate it receives goes up. - Breakers that trip, and recover. A circuit breaker should open when a dependency is failing, return a sensible fallback or error quickly, and close again once it recovers. A test that only confirms it opens has half the answer.
In a service mesh the breaker is configuration, not code. In Istio, circuit breaking is set in a DestinationRule under trafficPolicy, with connectionPool limits (for example tcp.maxConnections, http.http1MaxPendingRequests, http.maxRequestsPerConnection) and outlierDetection to eject unhealthy endpoints (consecutive5xxErrors, interval, baseEjectionTime, maxEjectionPercent). Istio’s own circuit breaking task shows what a load test sees when the pool limits are exceeded: with the deliberately tiny limits in that task, a run with 2 connections and 20 requests returned 85% successes and 15% HTTP 503, and a run with 3 connections and 30 requests returned 36.7% successes and 63.3% 503, with the proxy statistic upstream_rq_pending_overflow counting the rejected calls. Those limits are set to trip on purpose, so they are a demonstration and not advice for production values. The lesson for your test is to look for 503s that your own application did not produce, and to check the proxy’s counters before you blame the service.
Does autoscaling keep up with your load test?
Not instantly, and a load test is how you learn by how much. Kubernetes’ Horizontal Pod Autoscaler is a control loop that by default runs every 15 seconds, and it computes desiredReplicas = ceil(currentReplicas x currentMetricValue / desiredMetricValue). Several documented defaults shape what a test sees:
| Setting (Kubernetes documentation) | Default | What it means for a test |
|---|---|---|
| Sync period | 15 seconds | Scaling decisions are not continuous |
| Tolerance | 10% | No scaling until the metric is more than 10% off target |
| Scale-up stabilization window | 0 seconds | Scale-up can react at the next loop |
| Scale-up policies | up to 100% or 4 Pods per 15 seconds (largest wins) | Replicas can double every 15 seconds |
| Scale-down stabilization window | 300 seconds | Scale-down uses the highest recommendation of the last 5 minutes |
| CPU initialization period | 5 minutes | A new Pod’s CPU samples are ignored until it is Ready |
| Initial readiness delay | 30 seconds | An unready Pod is treated as still starting |
New Pods also need time to pull an image, start and pass readiness, which comes on top of those numbers. A sharp spike therefore usually shows a window of errors or high latency before the extra capacity arrives, and that window is the number to measure. Run the same ramp slowly and as a spike, and record the time from the start of the rise until latency returns to normal. Validating infrastructure auto-scaling with load testing walks through that measurement.
Run a long enough test to see scale-down as well. With the default 5-minute stabilization window, capacity leaves slowly, and a test that ends right after the peak never shows whether replicas come back down.
How do you find the slow service: tracing and metrics
A journey’s latency is spread across services, and a client-side load test only sees the total. OpenTelemetry defines a trace as the path of a request through an application, made of spans that share a trace ID and form a hierarchy through parent IDs, and it calls context propagation the concept that lets spans from different services be assembled into one trace. For HTTP, the W3C Trace Context Recommendation (23 November 2021) defines the traceparent header as version-trace-id-parent-id-trace-flags, for example 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01, where the last byte’s lowest bit is the sampled flag.
Practical points for a test:
- Tag the load with a header (like
X-Load-Testin the script above) and filter traces by it, so test traffic is separable from real traffic. - Check your sampling rate before the run. A trace sample that keeps one request in a thousand gives you very few traces in a five-minute test, and fewer from the failing minute.
- Break latency down per span. The slowest service in the journey is often not the one that returned the error; look for a span that grows with load while its parent waits.
- Keep per-service metrics for the same window: request rate, error rate, latency percentiles, CPU, memory, connection pool use and queue depth. Resource use that rises while latency stays flat is not the bottleneck yet.
Which metrics and thresholds should a microservices test use?
Use percentiles and error rate per service, and a separate budget for each journey. Averages hide the slow requests that users notice: see why percentiles are more useful than averages.
Percentiles also do not add up across services. If three services in a chain each have a p95 of 100 ms, the journey’s p95 is not 300 ms. It cannot be lower than the largest single p95, and where it lands above that depends on how the slow requests line up. Measure the journey’s percentile directly, and treat per-service limits as a budget to allocate, not a formula.
| Metric | Per service | Per journey |
|---|---|---|
| p95 / p99 latency | Yes, against that service’s objective | Yes, measured end to end |
| Error rate | By status class and by route | By failed step |
| Throughput | Requests per second it sustained | Journeys completed per second |
| Saturation | CPU, memory, connection pool, queue depth | Not applicable |
| Dependency latency | Time spent waiting on each downstream call | Share of the journey spent in each |
Set the limits with the people who own the service-level objectives. When none exist yet, run a baseline, write down what you measured, and set the first limit slightly looser than that.
How close must the test environment be to production?
Close enough on the things that matter to the failure you are hunting: the same instance sizes and replica counts for the services under test, the same autoscaling rules, the same mesh and gateway configuration, a database with a realistic data volume, and network latency between the parts that matches. A scaled-down staging cluster tells you about staging. If you must scale down, scale the load down by the same ratio and say so in the report, and treat the result as a floor for the headroom, not a promise.
How to optimise database connection pools covers one of the limits that only appears with realistic data and concurrency. For templates that fit this topic, LoadFocus lists a resilience test for microservices during dependency failures and a glossary entry for microservices, and its load testing page describes running the tests from the cloud.
Common mistakes in microservices load testing
- Only testing the happy path. The failures are in timeouts, retries and fallbacks, so inject slow and failing dependencies.
- Closed-model load against a slow service. The test slows down with the service and under-reports the overload.
- Mocks that never fail. An instant, always-correct stub makes every dependency look perfect.
- No trace or metric for the test window. You know a journey was slow, and not why.
- A stale scenario. New routes and changed behaviour make an old script test a system that no longer exists. Review the script with each significant release.
Frequently Asked Questions
What is the difference between testing microservices and testing a monolith?
A monolith’s slowness lives in one process and one database, so adding load mostly shows which of the two saturates first. In microservices, latency also comes from the network, from retries and timeouts between services, and from independently scaled components, so a test has to cover journeys across services and observe each hop.
Should I test each microservice separately or all together?
Both. Test services separately at a fixed arrival rate to find each one’s limit, and test the full journeys together to find the interactions. A service that passes alone can still fail when its dependency is slower than your stub.
How do I load test a service that depends on a third-party API?
Replace the third-party API with a stub that copies its real latency, error rate and rate limit, and run a smaller test against the real API at quiet times to check the stub. Do not hammer a third-party service you do not own.
How do I know if autoscaling works under load?
Run the same ramp slowly and as a spike and measure the time from the start of the rise until latency is back to normal. Kubernetes’ Horizontal Pod Autoscaler checks every 15 seconds by default and new Pods need time to become Ready, so a spike normally shows a short period of degraded service before capacity arrives.
Which tools can load test gRPC microservices?
k6 has a native gRPC module (k6/net/grpc) for unary and streaming calls, according to its documentation. For any other tool, check its protocol support against your services before you commit to it.
What should the pass/fail criteria be?
Percentile latency and error rate per service, a latency budget per journey, and a ceiling on saturation such as connection pool use. Set them from your service-level objectives, or from a measured baseline when you have none.
Sources
- Grafana k6 documentation: open and closed models, constant-arrival-rate executor, thresholds, gRPC (grafana.com/docs/k6, read 2026-10-23)
- Kubernetes documentation: Horizontal Pod Autoscaling (kubernetes.io/docs, read 2026-10-23)
- Istio documentation: Circuit Breaking (istio.io/latest/docs/tasks/traffic-management/circuit-breaking, read 2026-10-23)
- OpenTelemetry documentation: Traces (opentelemetry.io/docs/concepts/signals/traces, read 2026-10-23)
- W3C Recommendation: Trace Context, 23 November 2021 (w3.org/TR/trace-context, read 2026-10-23)
- Microsoft Azure Architecture Center: Rate Limiting pattern (learn.microsoft.com/azure/architecture/patterns/rate-limiting-pattern, read 2026-10-23)
