Key takeaways
- Setting a baseline turns each run into a regression check instead of a standalone number.
- Update the baseline deliberately when performance genuinely improves.
- Compare full results, not just the summary, since averages hide a worse tail.
The load test comparison feature build-in to LoadFocus Results allows you to compare the results of two different test runs of the same test for the cloud load testing service.
You can visually compare the results between the test run set as baseline and the current test run, just by setting a test run as a baseline, and loading the other load tests runs.
Baseline comparison is important because it allows to easily find differences in performance for the test runs.
Let’s have a look which are the steps to set a test as a baseline and compare with a different test run.
Setting a baseline test run
First step is to set a test run as baseline. All other test runs of the same test will be compared with the chosen baseline test.
When choosing a test as baseline, you should have in mind that you need to start small, with a small number of virtual users, so that in the following tests runs you can increase the load and discover potential issues and bottlenecks.
In order to set a test run as a baseline, open the results of the test chosen as baseline, and click the Set Baseline button.
Setting test run as baseline
After selecting a test run as baseline, you can check that the current test run is successfully set by checking the comparison percentages and the text under the test name.
Successfully setting a test run as baseline
Test run set as baseline is displayed under the test name
Percentages are all displayed as 0% for the test run set as baseline
Comparing test runs
In order to compare a test run of the same test with the baseline, simply navigate to the test run id and the comparison will automatically get displayed.
You can choose the test run by clicking on the id of the test from the dropdown next to the test name.
Response times and percentiles comparison with baseline
Check why using percentiles in load testing are useful in our previous blog post.
Full results comparison
Comparison between two different test results (baseline #1 and current test #2)
Changing the baseline
Changing the baseline is done simply clicking the Set Baseline button on a test run that is not currently a baseline.
Changing the baseline
When you change the baseline, the current visible test becomes the baseline.
Removing the baseline
To remove the baseline, and the comparison for a test, you simply go to the test run which is set as baseline, and click on the green check indicating that the current test run is set as baseline.
Removing the baseline for a test
After removing the baseline, the test has no baseline set, and the comparison will no longer be displayed for any of the test runs.
Written by Bogdan Vazzolla.
LoadFocus is a cloud testing platform, a load and stress testing tool which provides the infrastructure to run tests with thousands of concurrent users, from multiple cloud locations, in less than a few minutes, keep history of the results, compare different runs to inspect performance improvements or performance degradation. It also supports running JMeter load tests from the cloud and monitoring and audit web and mobile performance.
Frequently Asked Questions
What does setting a baseline change?
It turns every later run into a regression check rather than a standalone number. You compare the current run against the baseline visually instead of trying to remember what last week looked like.
When should I move the baseline?
Deliberately, when performance genuinely improves and you want the new level defended. Moving it after every run means you never notice slow drift, because each result is only compared to the last one.
Is comparing the summary enough?
No. Averages hide a worse tail, so a run can look level while the slowest requests have got significantly worse. Compare the full results, including the percentiles.
