To detect a performance regression automatically, run the same load test on every candidate build, compare its percentiles with a pinned baseline over several runs, and fail only when the rise is both larger than your normal run-to-run noise and statistically clear. When a regression gets through, git bisect finds the commit that caused it in a handful of test runs.
The earlier version of this page was written as a case study of an unnamed SaaS company. Its story and figures could not be verified, so it has been replaced by this how-to. Every number below is either arithmetic you can redo or an invented example, and it is labelled as one.
Key Takeaways
- A regression is a change in the distribution of response times, so compare percentiles (p95, p99) over repeated runs, not one average from one run.
- Pin the baseline to a known-good release. Comparing each build with the one before it lets a 2% slowdown per release go unnoticed: ten of them compound to 1.02^10 = 1.219, about 22% slower.
- Measure your noise first. If five identical runs of the same build spread over 6.6% of the median, a 5% gate will fire on nothing.
- With a rank test, three runs per side can never reach p < 0.05 (the smallest possible p-value is 2/20 = 0.10). Four per side can (2/70 = 0.029).
- Bisecting N commits takes at most ceil(log2 N) test runs: 6 for 64 commits, 7 for 100.
What Counts as a Performance Regression?
A performance regression is a change that makes the same work slower, or less reliable, under the same load. It shows up as a higher p95 or p99 for a journey, a higher error rate at a fixed arrival rate, lower throughput at a fixed number of users, or more CPU and memory for the same traffic. It is not a failing feature, which is why functional tests pass while it ships.
Pick two or three journeys that matter (login, search, checkout) and measure them the same way every time. Percentiles matter more than averages because a slow tail disappears in a mean; percentiles versus averages explains why.
How Do You Set a Baseline and Thresholds?
Use two kinds of gate, because each catches what the other misses.
| Gate | Example | Catches | Misses |
|---|---|---|---|
| Absolute limit | p95 under 500 ms (placeholder) | A build that breaks your service level objective | Slow creep that stays under the limit |
| Relative to baseline | p95 no more than 10% above the pinned baseline (placeholder) | Creep and moderate regressions | Nothing, if the baseline is stale or the environment drifts |
The baseline is a set of runs of a known-good build, taken in the same environment, kept as data (a file or a stored run), and replaced on purpose when performance genuinely improves or the architecture changes. How to build a baseline covers the measurement itself, and performance budgets for web apps covers the absolute side.
In k6 the absolute gate is a threshold, and a threshold that fails makes the run exit non-zero (grafana.com/docs/k6/latest/using-k6/thresholds, 2026-10-23):
export const options = {
thresholds: {
http_req_failed: ['rate<0.01'],
http_req_duration: ['p(95)<500'], // placeholder: set from your baseline or SLO
},
};The relative gate needs data from several runs, which a single k6 run cannot see, so it lives in a small script after the runs (below).
How Do You Handle Noise Between Runs?
Noise is the reason performance gates get switched off. Two identical runs of the same build never give the same p95, and a gate tighter than that spread fires on nothing. Do four things:
- Measure the spread. Run the same build five times and record each p95. If they range from 205 to 219 ms around a median of 212, the spread is 14 ms, or 14 / 212 = 6.6%. Your relative gate has to sit above that.
- Pin the environment. Same instance size, same data volume, nothing else running, and the same load generator location. A shared build agent is a poor load generator; use a separate one, or cloud generators.
- Warm up and use a steady shape. Discard the first minute (caches, connection pools, JIT), and hold a fixed rate or fixed user count so the runs are comparable.
- Interleave baseline and candidate. Alternate a baseline run and a candidate run in the same window, so slow drift in the environment hits both sides equally.
How Do You Compare Runs Statistically?
Run the candidate several times and compare the set of p95 values with the baseline set. Latency data is skewed, so use a rank test such as Mann-Whitney U, which does not assume a normal distribution, rather than a t-test.
An invented example, five runs per side:
- Baseline p95 (ms): 212, 205, 219, 208, 214. Median 212.
- Candidate p95 (ms): 231, 240, 228, 236, 244. Median 236.
- Rise: (236 - 212) / 212 = 11.3%. Every candidate run is slower than every baseline run.
With two samples of five there are C(10,5) = 252 ways to split the ten values into two groups. Only 2 of them (the observed split and its mirror image) are as extreme as this one, so the exact two-sided p-value is 2/252 = 0.0079. The rise is also larger than the 6.6% spread. Both conditions hold, so the gate fails.
The same arithmetic shows how many runs you need. With three per side, the most extreme split has p = 2/C(6,3) = 2/20 = 0.10, which never reaches 0.05. With four per side it is 2/C(8,4) = 2/70 = 0.029, so four is the minimum for a 5% test, and five (0.0079) leaves room for noise.
A small script after the runs makes the call. It reads two JSON lists of per-run p95 values and exits non-zero on a regression, which a pipeline step or git bisect run can use. The 0.10 minimum rise is a placeholder: set it above your own measured spread.
import json, statistics, sys
from scipy.stats import mannwhitneyu
base = json.load(open(sys.argv[1])) # e.g. [212, 205, 219, 208, 214]
cand = json.load(open(sys.argv[2])) # e.g. [231, 240, 228, 236, 244]
MIN_RISE = 0.10 # placeholder: above your measured spread
mb, mc = statistics.median(base), statistics.median(cand)
rise = (mc - mb) / mb
p = mannwhitneyu(cand, base, alternative='greater').pvalue
print(f'median {mb} -> {mc} ms ({rise:+.1%}), p = {p:.4f}')
sys.exit(1 if (p < 0.05 and rise > MIN_RISE) else 0)alternative='greater' tests whether the candidate values tend to be larger, and scipy’s default method uses the exact distribution when a sample has 8 or fewer observations and there are no ties (docs.scipy.org, scipy.stats.mannwhitneyu, 2026-10-23). The script was not run against real load tests, only checked by hand against the example above.
This runs after the load test steps in your pipeline. For where to put those steps and how to fail a build, see continuous performance testing and integrating performance testing with DevOps toolchains rather than repeating them here.
How Do You Find the Commit That Caused a Regression?
Bisect. Give git a known-good commit and a known-bad one, and a script that exits 0 for good and 1 to 127 (except 125) for bad. Exit code 125 means “cannot test this commit”, for example when it does not build, and anything above 127 aborts the bisect (git-scm.com/docs/git-bisect, 2026-10-23).
git bisect start HEAD v1.4 -- # HEAD is slow, v1.4 is the last good release
git bisect run ./perf-check.sh # exits 0 (fast), 1 (slow) or 125 (cannot test)
git bisect resetperf-check.sh builds and deploys the commit to your test environment (exit 125 if that fails), runs the load test five times, writes the p95 values to a JSON file and calls the comparison script against the baseline measured on the good release.
Each step halves the range, so N commits need at most ceil(log2 N) steps: 64 commits take 6, 100 take 7, 200 take 8. At five runs of five minutes each, 6 steps is 6 x 5 x 5 = 150 minutes of load testing, unattended. The same method works on anything ordered: deploys, config changes or dependency versions.
If the regression also needs diagnosing, not only locating, look at what the commit touched: queries, serialization, a new network call, a bigger payload. Common API performance issues and how to detect them lists the usual suspects.
What Is the Simplest Setup That Works?
Start small and add to it only when it fails to catch something.
- One load test of your most important journey, short enough to run on every merge to the main branch.
- An absolute threshold on p95 and error rate in the test itself.
- A baseline file of five runs from the last release, refreshed after each release.
- The comparison script after the test, with the minimum rise set above your measured spread.
- A full run on a schedule (nightly) for the slow tests, with the same comparison.
Where Does LoadFocus Fit?
LoadFocus’s results view lets you set one run of a test as the baseline and load other runs of the same test to compare them against it; the full results comparison shows more than the summary, which matters because averages hide a worse tail (how to compare two load tests with baseline comparison). The load testing page describes a Jenkins CI plugin that can mark a build UNSTABLE or FAILED against limits you set, and says tests run from 25 cloud regions (loadfocus.com/load-testing, 2026-10-23). See LoadFocus load testing. For front-end regressions in page speed, page speed monitoring is the matching tool.
Frequently Asked Questions
How do you detect a performance regression?
Run the same test on a candidate build and on a known-good baseline in the same environment, repeat it several times, and compare p95 or p99 across the runs. Flag a regression only when the rise exceeds your measured run-to-run noise and a rank test says the difference is unlikely to be chance.
How many runs do you need to compare two builds?
At least four per side to be able to reach p < 0.05 with a rank test: three per side can never go below 0.10. Five per side is a more comfortable minimum, because it allows a margin for noise.
What is a good threshold for a performance regression?
There is no universal number. Measure the spread of repeated runs of the same build, then set the relative gate above it. If five runs vary by 6.6%, a 5% gate will fire on noise. Pair it with an absolute limit from your service level objective.
Why compare against a pinned baseline and not the previous build?
Small slowdowns compound. If each release is 2% slower than the last, no single comparison looks alarming, yet ten releases leave the product about 22% slower (1.02^10 = 1.219). A baseline pinned to a known-good release shows the total drift.
Can I find which commit caused a slowdown automatically?
Yes. git bisect run with a script that runs the load test and compares it with the baseline finds the commit in at most ceil(log2 N) steps for N commits.
Should performance regression tests run on every commit?
Run a short test on every merge to the main branch and the long ones on a schedule. Short runs are noisier, so keep their gate loose and let the nightly run enforce the tight one.
Sources
- git-bisect documentation,
startandrunexit-code rules (git-scm.com/docs/git-bisect, read 2026-10-23). - scipy.stats.mannwhitneyu reference,
alternativeandmethodoptions (docs.scipy.org, read 2026-10-23). - Grafana k6 documentation, thresholds (grafana.com/docs/k6/latest/using-k6/thresholds, read 2026-10-23).
- LoadFocus load testing page (loadfocus.com/load-testing, read 2026-10-23).
