To load test an event-driven system, inject events at the producer at a controlled rate and measure at the consumer or the final side effect, because nothing in the middle returns an HTTP response. The numbers that matter are end-to-end latency from publish to side effect, consumer lag or queue backlog, the dead-letter queue rate, and duplicate or out-of-order deliveries.
Key Takeaways
- A passing request-response test says little about an asynchronous flow. The producer gets its acknowledgement long before the work is done, so the real latency is hidden unless you measure at the far end.
- Track four signals: publish-to-side-effect latency, backlog or consumer lag, dead-letter queue (DLQ) rate, and duplicates plus ordering violations.
- Delivery is usually at least once. Amazon SQS standard queues, SNS-to-SQS fan-out and Lambda event source mappings can all deliver a message twice, so a load test that never checks for duplicates is incomplete.
- Timeouts interact with load. If processing takes longer than the SQS visibility timeout, the message returns to the queue and gets processed again, which adds more load.
- Only inject events into systems you own or are authorised to test, and tag test events so downstream consumers can recognise and discard them.
Why is load testing event-driven systems different?
In a synchronous API the test client sends a request and times the response, so latency and errors are directly visible. In an event-driven system the producer publishes and moves on. The work happens later, in one or more consumers, and failures show up as a growing backlog, retries, a DLQ filling up, or a missing side effect hours later, not as a slow response.
That changes what you inject, where you measure and what counts as a failure. It also means a test can look healthy at the producer while the queue silently grows. For the HTTP side of the same architecture, the microservices load testing guide applies, and the serverless performance testing guide covers function limits, so this post stays on queues, streams and the end-to-end view.
What do queues and streams promise, and what does a load test need to check?
Each broker has delivery and ordering rules that a load test should deliberately stress. The table lists what the official documentation says, read on 2026-10-23, and the check each rule suggests.
| Component | What the documentation says | What to check under load |
|---|---|---|
| Amazon SQS standard queue | Very high, nearly unlimited throughput; at-least-once delivery, so a message may be delivered more than once; best-effort ordering only. | Count duplicates and out-of-order arrivals at the consumer, and confirm handlers are idempotent. |
| Amazon SQS FIFO queue | Order is kept within each message group; deduplication by MessageDeduplicationId or content. Up to 300 API calls per second per action, or 3,000 messages per second per action with batching of 10; high throughput mode raises this. | Do all messages share one group ID? One group serialises processing and caps throughput, whatever the queue limit says. |
| SQS visibility timeout | Default 30 seconds; a message not deleted in time becomes visible again and can be received by another consumer. Maximum 12 hours. | Does the slowest handler, under load, finish inside the timeout? If not, expect duplicate work. |
| SNS fan-out to SQS | A publish to the topic is delivered as a message to each subscribed queue, wrapped in a JSON envelope that carries a MessageId and a Timestamp. | Does each subscriber queue keep up independently? The slowest one sets the age of your oldest message. |
| Lambda with SQS | Polls and invokes with batches, 10 messages by default, with an optional batching window of up to 5 minutes. Processing is at least once. A failed batch becomes visible again after the visibility timeout unless the function reports partial batch failures. | Is the batch size and window adding latency you did not plan for? Do partial failures reappear as repeated retries? |
| Lambda SQS scaling | Starts with five concurrent invocations and adds up to 300 more per minute, to a maximum of 1,250 for a standard queue. A maximum concurrency setting (2 to 1,000) caps it. | How long does the backlog take to drain while the poller is still ramping? |
| Apache Kafka consumer groups | The documentation shows a per-partition view with current offset, log-end offset and lag. Assigning partitions to group members is described as a combination of ordering and scalability. | Is lag growing during the plateau? Are some partitions hot while others sit idle? |
Sources: docs.aws.amazon.com (SQS queue types, SQS visibility timeout, SNS fanout to SQS, Lambda with SQS, Lambda SQS scaling and error handling) and kafka.apache.org (operations, “Checking consumer position”, and design), all read 2026-10-23. For Lambda concurrency and cold-start limits in more detail, see the serverless computing load testing analysis.
What should you measure when there is no HTTP response?
Measure from four places and tie them together with an event ID.
- End-to-end latency. Put a unique ID and a publish timestamp in every test event. At the consumer or the final data store, record when the side effect appeared. The difference is the number your users feel, and you should report it as percentiles; the percentiles post explains why averages hide the tail.
- Backlog or consumer lag. For SQS, watch the queue depth and the age of the oldest message. For Kafka, the consumer groups tool prints lag per partition. Lag that grows through a steady plateau means consumers are slower than producers.
- DLQ rate and retries. Count messages that reach the DLQ and how many receives each one needed. Even a small DLQ rate under load is a defect in how failures are handled.
- Duplicates and ordering. Compare the set of IDs produced with the set observed at the sink. Count IDs seen more than once and per-key sequences that arrive out of order.
On AWS, ApproximateAgeOfOldestMessage is the metric AWS itself points to for noticing that a function is not returning failed messages correctly. A sharp rise there during your test is a signal to look at retries before you look at capacity.
Worked example with illustrative numbers
These numbers are assumptions chosen to show the arithmetic, not measurements. Suppose the producers publish 500 messages per second for 10 minutes, and the consumers can sustain 400 messages per second.
- The backlog grows by 500 - 400 = 100 messages per second.
- After 600 seconds the backlog is 100 x 600 = 60,000 messages.
- A message published at the end of the run waits behind that backlog: 60,000 / 400 = 150 seconds before a consumer even starts on it.
At the producer, every publish succeeded in milliseconds. Only the sink-side timestamp shows the 150 second delay. Now add a second assumption: handling takes 40 seconds for some messages and the visibility timeout is the 30 second default. Those messages return to the queue while still being processed, so they are handled twice and consumer work rises further. A test that checks only publish success would pass both problems.
How do you generate load when the entry point is not HTTP?
Inject at the producer side, using the same path production uses. If events enter through an HTTP API or a webhook, a normal load test on that endpoint is the injection point, and you then watch the downstream signals above. If events are published straight to a broker, use a client that speaks its protocol.
- Through the API in front of the queue. Run the load from a cloud load testing tool such as LoadFocus, which runs JMeter and k6 scripts from 25+ cloud regions, and collect the downstream measurements from the broker and the consumers’ own metrics. LoadFocus measures the HTTP side of that path; the queue depth, lag and sink timestamps come from your cloud provider’s metrics and your own logs.
- Directly to the broker. JMeter has samplers for messaging protocols such as JMS, and k6 reaches other protocols such as Kafka through community extensions, as the k6 vs JMeter comparison explains. Check an extension’s maintenance status before building a plan around it.
- From the broker vendor’s own tools. Kafka ships command line tools for producing and for checking group positions, which is often enough for a first lag measurement.
Shape the traffic like production: steady arrival with bursts, realistic payload sizes, and a spread of keys or message groups. The guide to simulating realistic user behaviour covers how to model it, and spike testing covers the burst. One key repeated for every event, for example, puts all the load on a single Kafka partition or SQS FIFO message group and gives a misleading result.
How do you test failure and recovery?
Failures are where event-driven systems earn or lose trust, so test them under load rather than separately.
- Slow consumer. Add a delay to one handler and watch backlog growth and the age of the oldest message.
- Poison message. Publish a malformed event and confirm it reaches the DLQ after the configured retries without blocking the messages behind it. For FIFO queues, one failing message can hold up its message group.
- Consumer crash and redelivery. Kill a consumer mid-batch. Expect redelivery after the visibility timeout and check the idempotency of the side effect.
- Burst after an outage. Pause consumers, let the backlog build, resume them, and time the drain. The drain time under peak producer rate is the recovery time objective in practice.
- Scaling lag. If consumers scale on backlog, test the ramp separately; the infrastructure auto-scaling validation guide shows how, including why queue depth needs a per-instance metric to work with target tracking.
Is it safe to load test these systems in production?
Only on systems you own or are authorised to test. Synthetic events can trigger real emails, charges or stock changes, and they share consumers and quotas with real traffic. Put a clear marker in every test event, such as a header or field, make consumers skip external side effects for marked events, and keep test volumes inside the provider’s published quotas. A dedicated test queue or topic wired to a copy of the consumers is safer than the production path.
Frequently Asked Questions
How do I load test an event-driven architecture?
Pick the entry point of the flow, usually an API or a topic, and inject events there at a controlled, production-like rate. Measure at the far end: the time from publish to side effect, the queue backlog or consumer lag, the DLQ rate, and duplicates or ordering violations. Ramp up in steps and hold each level long enough for lag to show.
What is consumer lag and why does it matter?
Consumer lag is the distance between the newest message in a log or queue and the position the consumers have reached. In Kafka, the consumer groups tool prints it per partition. If lag keeps rising while the load is steady, consumers are slower than producers and end-to-end latency will keep growing until the load drops.
Does SQS guarantee that a message is delivered once?
Not for standard queues. They provide at-least-once delivery and best-effort ordering, so duplicates and reordering can happen, and your handlers should be idempotent. FIFO queues keep order within a message group and deduplicate by a deduplication ID, with different throughput limits.
Why do I see duplicates in my load test?
The usual causes are processing that outlasts the visibility timeout, a batch that fails after some messages succeeded, and the at-least-once guarantee itself. Lambda’s SQS integration documents that duplicate processing can occur and recommends idempotent code and partial batch responses.
Which tools can generate event-driven load?
Use a tool that can speak the entry protocol. For HTTP entry points, any load testing tool works, including JMeter and k6 scripts run from LoadFocus. For direct broker traffic, JMeter has messaging samplers and k6 has community extensions for some brokers; the broker’s own command line tools also work for small tests.
Can I load test with real customer data?
Avoid it. Use generated events that match the shape of real ones, mark them as test events, and make sure consumers do not send real emails or payments for them.
Sources
- AWS, Amazon SQS queue types, https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-queue-types.html (read 2026-10-23)
- AWS, Amazon SQS visibility timeout, https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html (read 2026-10-23)
- AWS, Fanout Amazon SNS notifications to Amazon SQS queues, https://docs.aws.amazon.com/sns/latest/dg/sns-sqs-as-subscriber.html (read 2026-10-23)
- AWS, Using Lambda with Amazon SQS, https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html (read 2026-10-23)
- AWS, Configuring scaling behavior for SQS event source mappings, https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-scaling.html (read 2026-10-23)
- AWS, Handling errors for an SQS event source in Lambda, https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-errorhandling.html (read 2026-10-23)
- Apache Kafka, Basic Kafka operations, https://kafka.apache.org/42/operations/basic-kafka-operations/ (read 2026-10-23)
- Apache Kafka, Design, https://kafka.apache.org/42/design/design/ (read 2026-10-23)
