To performance test an AI chatbot, measure what a user feels (time to first token and total time per turn), what the system sustains (concurrent conversations, tokens per second, error rate) and what it costs (tokens per conversation). Then find where the provider’s rate limits start returning 429 errors. This page lists those measurements and how to test them. An earlier version of this page, titled “AI Chatbots: Transforming Performance Testing”, was a news-style article about chatbots as testing tools; it carried no real news or sources, so it was replaced with the topic searchers actually ask about, and its short section on AI-assisted test creation is kept below.
Key Takeaways
- A chatbot turn takes seconds, not milliseconds, and most of it is spent streaming tokens. Judge it with time to first token, tokens per second and total time, each at the 95th percentile.
- Streaming responses hold a connection open, so concurrency is users x response duration (Little’s Law), and it climbs fast when the model slows.
- Providers limit requests and tokens per minute and answer with HTTP 429 (and sometimes 503). Test how your app behaves at the limit, not just below it.
- Do not load test the model provider itself. Test your own application, and use a stub that mimics the model’s latency for high-volume runs.
- Cost is a test result too: tokens in and out per conversation, multiplied by the current price on the provider’s page.
What Should You Measure When Performance Testing a Chatbot?
Six numbers cover most of it.
| Measure | What it tells you | How to get it |
|---|---|---|
| Time to first token (TTFT) | How long the user stares at nothing before text appears | Time to the first byte of a streamed reply (http_req_waiting in k6) |
| Total time per turn | When the answer is complete | http_req_duration for the whole request, which includes the stream |
| Tokens per second | How fast the text arrives once it starts | Output tokens (from the provider’s usage field) divided by streaming time |
| Concurrent conversations | How many streams are open at once | Active virtual users, or arrival rate x duration |
| Error rate by status | Whether limits or overload are hit | Count 429, 5xx and timeouts separately |
| Tokens and cost per conversation | What a busy day costs | Usage fields summed per conversation, times the provider’s price |
Report each as a percentile. The average hides the slow turns, which are the ones users remember; see why percentiles beat averages.
Two things differ from a normal web test. First, replies vary in length, so response time varies with the question. Use a fixed set of prompts of known length for comparisons, and record output tokens next to every timing. Second, context grows: each turn resends earlier messages, so input tokens and time rise as a conversation gets longer. Test long conversations as well as single questions.
How Do Streaming Responses Change the Test?
Chat APIs usually stream the reply as server-sent events (SSE) so text appears as it is generated. Anthropic’s Messages API, for example, streams when you set "stream": true and sends the answer as a sequence of events (platform.claude.com/docs/en/build-with-claude/streaming, 2026-10-23). Two consequences for testing:
- The request does not end when the first token arrives, so a connection stays open for the whole reply. At 50 new conversations per second and 20 seconds per reply, about 1,000 streams are open at once (50 x 20). If replies slow to 40 seconds, it is 2,000. Your load balancer, proxy and server need to allow that many idle-looking connections, and proxies that buffer responses can delay the first token.
- First-byte time and full duration are different metrics. In k6,
http_req_waitingis the time until the first byte andhttp_req_durationcovers the whole response. Use both. Per-token timing needs a client that reads the stream chunk by chunk, so treat it as a smaller, separate test.
What Are the Rate Limits, and How Do You Test Them?
Every major provider caps usage per minute. The numbers change by account and tier, so read your own limits from the provider dashboard, and check these facts against the current docs (read 2026-10-23):
- OpenAI documents limits in requests per minute (RPM), requests per day (RPD), tokens per minute (TPM), tokens per day (TPD) and images per minute, and you can hit any one of them first. Throttling returns HTTP 429 (
rate_limit_error) and an overloaded model returns 503. Responses carryx-ratelimit-*headers and aRetry-Afterheader when present. Its docs advise exponential backoff with jitter, and they note that aslow_down429 can occur even within your RPM and TPM limits when the request rate rises too quickly (developers.openai.com/api/docs/guides/rate-limits). - Anthropic measures requests per minute (RPM), input tokens per minute (ITPM) and output tokens per minute (OTPM) per model class, enforced with a token bucket, so capacity refills continuously rather than resetting on the minute. Exceeding a limit returns 429 with a
retry-afterheader when one applies, and responses carryanthropic-ratelimit-*headers. Sharp traffic growth can trigger separate acceleration limits. A monthly spend cap also returns 429, but withoutretry-after, so retrying cannot help (platform.claude.com/docs/en/api/rate-limits). - Google (Gemini API) applies requests per minute, input tokens per minute and requests per day, per project rather than per API key. Exceeding any one triggers a rate limit error (ai.google.dev/gemini-api/docs/rate-limits).
For your test, this means:
- Run a ramp that crosses the limit on purpose, in a test account or against a stub that returns 429 with a
Retry-Afterheader. Confirm the app queues, backs off with jitter, shows the user a clear message and does not retry in a tight loop (which only adds more 429s). - Tell a rate limit from a spend cap. A 429 with no
retry-aftermay mean a budget is exhausted, which no backoff fixes. Alert on it separately. - Ramp gradually. OpenAI’s guidance is to increase traffic in steps once you are at high volume (no more than 50% every 15 minutes above 1 million input TPM), and Anthropic warns about acceleration limits, so a sudden spike test against the real API may fail for reasons that have nothing to do with your code.
- Do not replay a stream blindly. If a stream breaks halfway, a retry repeats the whole prompt and its cost.
The general pattern is covered in API rate limiting causes and fixes.
Should You Load Test the Real Model?
Usually not at full scale. A large run against a provider burns money, fills your quota, and can disturb your other production traffic that shares it. Check the provider’s terms and your limits before you try. A practical split:
- Application test with a stub. Replace the model call with a service that streams a canned reply at a configured speed and delay. This is where you find connection limits, proxy buffering, session storage and database bottlenecks at thousands of conversations.
- Small test against the real model. Run a modest, rate-limited test to measure real time to first token, tokens per second and cost per conversation, with your real prompts and context sizes.
- Limit test with a fake 429. Use the stub to return 429 and 503 at a chosen rate and check your retry and fallback behaviour.
What About Quality Under Load?
A load tool measures speed and errors; it cannot tell you the answer is right. Sample replies during the run and check for truncated answers, empty replies and fallback messages, because some stacks degrade quietly (shorter answers, a fallback model, cached replies) when stressed. Count these as errors alongside the HTTP status.
Can AI Help Build the Performance Test?
Yes, for the scripting part. LoadFocus’s “Create with AI” turns a plain-English description into a k6 or JMeter script that you can read and edit, check, run as a single-user smoke test and have fixed before the full run (loadfocus.com/load-testing, 2026-10-23). After a run, the AI performance summary reports bottlenecks from your results, with allowances per 30 days of 3 analyses on Free, 5 on Starter and Basic, 10 on Essential, 50 on Pro and 100 on Enterprise (loadfocus.com/ai-load-test-analysis, 2026-10-23). Treat generated scripts as drafts: review them, especially auth and the request body, before you run them. More on that in keeping AI-generated load test scripts maintained and how AI is changing automated load testing. To run the resulting script from the cloud, see LoadFocus load testing.
Frequently Asked Questions
What is chatbot performance testing?
It is measuring how quickly and reliably a chatbot answers under realistic and peak load: time to first token, total time per turn, tokens per second, error rates, concurrent conversations and cost per conversation.
What is a good time to first token for a chatbot?
There is no universal number. Set a target from your own users and product, measure it at the 95th percentile under load, and compare each release with the last one. Time to first token matters most because it decides how fast the interface feels.
How do I test a chatbot that streams responses?
Treat the reply as a long-lived connection. Measure first-byte time and full duration separately, size the server for the concurrent open streams (users x reply duration), and check that proxies do not buffer the stream.
What happens when I hit the provider’s rate limit?
You get an HTTP 429 (and sometimes 503 when the model is overloaded). Honour Retry-After when it is present, otherwise back off with jitter, and cap the retries. Test this path on purpose.
How do I estimate the cost of a chatbot at scale?
Measure average input and output tokens per conversation in a test, multiply by the provider’s current price per token, and multiply by expected conversations. Use long conversations in the sample, since context grows with each turn.
Can I use JMeter or k6 for chatbot load tests?
Yes, for the HTTP side: request rates, first-byte time, full duration, errors and concurrency. Plain HTTP scripts do not judge answer quality, and per-token timing needs a client that reads the stream chunk by chunk.
Sources
- OpenAI, rate limits guide (developers.openai.com/api/docs/guides/rate-limits, read 2026-10-23).
- Anthropic, rate limits (platform.claude.com/docs/en/api/rate-limits, read 2026-10-23) and streaming (platform.claude.com/docs/en/build-with-claude/streaming, read 2026-10-23).
- Google, Gemini API rate limits (ai.google.dev/gemini-api/docs/rate-limits, read 2026-10-23).
- LoadFocus, load testing and AI load test analysis pages (loadfocus.com/load-testing, loadfocus.com/ai-load-test-analysis, read 2026-10-23).
