{"id":2084,"date":"2021-02-05T12:03:42","date_gmt":"2021-02-05T12:03:42","guid":{"rendered":"https:\/\/loadfocus.com\/blog\/?p=2084"},"modified":"2021-02-05T12:03:44","modified_gmt":"2021-02-05T12:03:44","slug":"understanding-standard-deviation-in-performance-testing","status":"publish","type":"post","link":"https:\/\/loadfocus.com\/blog\/2021\/02\/understanding-standard-deviation-in-performance-testing","title":{"rendered":"Understanding Standard Deviation in Performance Testing"},"content":{"rendered":"<span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\"><\/span> <span class=\"rt-time\"> 2<\/span> <span class=\"rt-label rt-postfix\">minutes read<\/span><\/span><!-- pn-tldr --><h2>Key takeaways<\/h2><ul><li>A high standard deviation means inconsistent response times, which users notice more than a slow average.<\/li><li>Two runs with the same mean can behave completely differently, and this is how you tell.<\/li><li>Pair it with percentiles rather than reporting either alone.<\/li><\/ul><!-- \/pn-tldr -->\n<p class=\"lead\"><strong>Standard Deviation<\/strong> is an important metric in performance testing analysis and informs us how stable the application under test is. In other words, it tells us if the requests that occur during the test are consistent or not.<\/p>\n\n\n\n<p><strong>Standard Deviation<\/strong> measures how the <a href=\"https:\/\/loadfocus.com\/blog\/2013\/07\/what-is-response-time-in-performance-testing\/\" title=\"What is Response Time in Performance Testing?\">response times<\/a> are spread out around the average response time (mean).<\/p>\n\n\n\n<p>A small standard deviation means that the response time of all the requests are close to each other. Which leads us to understand that a smaller standard deviation value the closer the response times are and more consistent the transaction is, and stable the application tested.<\/p>\n\n\n\n<p>It&#8217;s not very easy to calculate the standard deviation, especially for large datasets, that&#8217;s why most of the tools calculate it for you and show a summary for a better understanding on how the application behaves in the real world.<\/p>\n\n\n\n<p>When you run a performance test with LoadFocus, either that you test a website, a REST API or you run a JMeter load test, the Standard Deviation is calculated automatically, and it&#8217;s visible in the Load Test result, under the Insights tab. See here more details about <a href=\"https:\/\/loadfocus.com\/blog\/2016\/03\/standard-deviation-and-percentiles-in-load-testing-metrics\/\" title=\"Standard Deviation and Percentiles in Load Testing Metrics\">Standard Deviation and percentiles<\/a>.<\/p>\n\n\n\n<p><em>Written by&nbsp;<a href=\"https:\/\/coursinator.com\/instructor\/bogdan-vazzolla\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">Bogdan&nbsp;Vazzolla<\/a>.<\/em><\/p>\n\n\n\n<p><em><a href=\"https:\/\/loadfocus.com\/\">LoadFocus<\/a>&nbsp;is a cloud performance,&nbsp;<a href=\"https:\/\/loadfocus.com\/load-testing\" target=\"_blank\" rel=\"noreferrer noopener\">load and stress testing tool<\/a>&nbsp;which provides the infrastructure to run tests with thousands of concurrent users, from&nbsp;<a href=\"https:\/\/loadfocus.com\/locations\">multiple cloud locations<\/a>, in less than a few minutes, keep history of the results, compare different runs to inspect performance improvements or performance degradation.<\/em>&nbsp;It also supports running&nbsp;<a href=\"https:\/\/loadfocus.com\/jmeter-load-testing\" target=\"_blank\" rel=\"noreferrer noopener\">JMeter load tests from the cloud<\/a>.<\/p>\n<!-- pn-faq --><h2>Frequently Asked Questions<\/h2><h3>What does a high standard deviation tell me?<\/h3><p>That response times are inconsistent. Users notice inconsistency more than a uniformly slow site, because they cannot form an expectation of how long anything takes.<\/p><h3>Two runs have the same average. Are they the same?<\/h3><p>Not necessarily, and this is how you tell. Identical means can hide completely different distributions, one steady and one swinging between fast and unacceptable.<\/p><h3>Should I report it on its own?<\/h3><p>Pair it with percentiles. Standard deviation says how spread out the results are; percentiles say where the slow tail actually sits.<\/p><!-- \/pn-faq --><!-- pn-related-reading --><h2>Related reading<\/h2><ul><li><a href=\"https:\/\/loadfocus.com\/blog\/2016\/03\/standard-deviation-and-percentiles-in-load-testing-metrics\">Standard Deviation and Percentiles in Load Testing Metrics<\/a><\/li><li><a href=\"https:\/\/loadfocus.com\/blog\/2021\/10\/why-are-percentiles-more-useful-than-averages-in-performance-testing\">Why Percentiles Beat Averages in Performance Testing<\/a><\/li><\/ul>","protected":false},"excerpt":{"rendered":"<p><span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\"><\/span> <span class=\"rt-time\"> 2<\/span> <span class=\"rt-label rt-postfix\">minutes read<\/span><\/span>Key takeaways A high standard deviation means inconsistent response times, which users notice more than a slow average. Two runs with the same mean can behave completely differently, and this is how you tell. Pair it with percentiles rather than reporting either alone. Standard Deviation is an important metric in performance testing analysis and informs&#8230;  <a href=\"https:\/\/loadfocus.com\/blog\/2021\/02\/understanding-standard-deviation-in-performance-testing\" class=\"more-link\" title=\"Read Understanding Standard Deviation in Performance Testing\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":1,"featured_media":2085,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,48],"tags":[107,333],"class_list":["post-2084","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-load-testing","category-test-automation","tag-standard-deviation","tag-standard-deviation-performance-testing"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/posts\/2084","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/comments?post=2084"}],"version-history":[{"count":0,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/posts\/2084\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/media\/2085"}],"wp:attachment":[{"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/media?parent=2084"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/categories?post=2084"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/tags?post=2084"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}