{"id":2232,"date":"2021-10-20T08:59:53","date_gmt":"2021-10-20T08:59:53","guid":{"rendered":"https:\/\/loadfocus.com\/blog\/?p=2232"},"modified":"2023-09-03T14:09:47","modified_gmt":"2023-09-03T14:09:47","slug":"why-are-percentiles-more-useful-than-averages-in-performance-testing","status":"publish","type":"post","link":"https:\/\/loadfocus.com\/blog\/2021\/10\/why-are-percentiles-more-useful-than-averages-in-performance-testing","title":{"rendered":"Why are Percentiles More Useful than Averages in Performance Testing"},"content":{"rendered":"<span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\"><\/span> <span class=\"rt-time\"> 5<\/span> <span class=\"rt-label rt-postfix\">minutes read<\/span><\/span>\n<h2 class=\"wp-block-heading\">Why Performance Testing<\/h2>\n\n\n\n<p class=\"lead\">We all want our <strong>websites to be responsive and useful<\/strong>. However, the biggest correlation between <strong>website performance<\/strong> and<strong> business results<\/strong> is seen by <strong>improving the performance and speed <\/strong>of your website.<\/p>\n\n\n\n<p>Users get bored if they have to wait for too long to load your website or application.<\/p>\n\n\n\n<p><strong>Performance Testing<\/strong> and <strong>Load Testing<\/strong> your website and <a href=\"https:\/\/loadfocus.com\/blog\/2021\/02\/how-to-load-test-restful-apis-with-loadfocus\/\" title=\"How to Load Test RESTful APIs with LoadFocus\">restful APIs<\/a> will help you understand better how your website and web services behave when a lot of users access your website at the same time. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Performance Testing<\/strong> questions that website owners ask <\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>is my website going to crash?<\/li><li>is everything going to be slow or will freeze when doing certain actions?<\/li><li>will my users encounter <a href=\"https:\/\/loadfocus.com\/blog\/2014\/07\/errors-and-response-codes-in-load-testing-on-loadfocus-com\/\" target=\"_blank\" rel=\"noreferrer noopener\" title=\"Errors and Response Codes in Load Testing on LoadFocus.com\">errors<\/a>? how many errors and of what type?<\/li><li>is my website going to <a href=\"https:\/\/loadfocus.com\/blog\/2018\/12\/what-is-disaster-recovery-testing-in-software-testing\/\" target=\"_blank\" rel=\"noreferrer noopener\" title=\"What is Disaster Recovery Testing in Software Testing?\">recover<\/a> after a heavy load?<\/li><li>which are the most sensitive areas and how will they behave during a load test?<\/li><\/ul>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What are Percentiles<\/h2>\n\n\n\n<p><strong>A percentile is defined as the value where x percent of the data falls below the value<\/strong>.<\/p>\n\n\n\n<p>The <strong>Percentile<\/strong> metric tells you the value at which a certain percentage of your data is included.<\/p>\n\n\n\n<p>The <strong>median<\/strong>, or the middle value, <strong>is also known as the 50th percentile<\/strong> (the middle percentile out of 100). This is the value at which 50% of the data is less than the value, and 50% is greater than the value (or equal to it).<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"784\" src=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-1-1280x784.jpeg\" alt=\"\" class=\"wp-image-2237\" srcset=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-1-1280x784.jpeg 1280w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-1-800x490.jpeg 800w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-1-600x368.jpeg 600w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-1-768x470.jpeg 768w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-1-1536x941.jpeg 1536w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-1.jpeg 1600w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><figcaption>50th Percentile<\/figcaption><\/figure>\n\n\n\n<p>For example, if we call something \u201cthe 10th percentile,\u201d we mean that 10% of the data is less than the value and 90% is greater than (or equal to) the value.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"832\" src=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-90th-1280x832.jpeg\" alt=\"\" class=\"wp-image-2238\" srcset=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-90th-1280x832.jpeg 1280w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-90th-800x520.jpeg 800w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-90th-600x390.jpeg 600w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-90th-768x499.jpeg 768w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-90th-1536x998.jpeg 1536w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-90th.jpeg 1600w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><figcaption>90th Percentile<\/figcaption><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<p>And the 90th percentile is where 90% of the data is less than the value and 10% is greater:<\/p>\n\n\n\n<p>So a 95% percentile tells you the value which is greater than or equal to 95% of your data.<\/p>\n\n\n\n<p>Most people use basic statistics like <strong>min<\/strong>, <strong>mean<\/strong>, <strong>median<\/strong> or <strong>max<\/strong>, which for a large number of data they<strong> hide the truth<\/strong>.<\/p>\n\n\n\n<p>Mean and median tend to hide outliers, wherease min\/max are distorted by a single outlier.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why are Averages so Misleading<\/h2>\n\n\n\n<p><strong>Basic averages<\/strong> are an awful way to paint the world and the numbers around it. Using averages instead of percentiles can lead to false alarms or missed opportunities.<\/p>\n\n\n\n<p>The <strong>median<\/strong> is the middle value, where<strong> half of the data is above<\/strong> and <strong>half is below<\/strong>. The mean (aka average) is defined as the sum(value) \/ count(value).<\/p>\n\n\n\n<p>Let&#8217;s see one <a href=\"https:\/\/loadfocus.com\/blog\/2021\/01\/what-is-a-good-example-of-load-testing\/\" title=\"What is a Good Example of Load Testing?\">example<\/a>: <em>if the average monthly salary in your area is 2000 dollars, but 1 out of 10 people earns 11,000 dollars per month, this does not mean the rest of 9 people out of ten earn 2000 dollars, it means we have an outlier data, which is the 11,000 dollar salary, and 9 other people that earn 1000 dollars per months.<\/em><\/p>\n\n\n\n<p><strong>This is how the averages tend to hide outliers.<\/strong><\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How to Improve your Website Performance using Percentiles<\/h2>\n\n\n\n<p>Using averages can have a dramatic (and negative) impact on how values are reported, while percentiles can help you get closer to the \u201ctruth.\u201d<\/p>\n\n\n\n<p><strong><a href=\"https:\/\/loadfocus.com\/blog\/2016\/03\/standard-deviation-and-percentiles-in-load-testing-metrics\/\" target=\"_blank\" rel=\"noreferrer noopener\" title=\"Standard Deviation and Percentiles in Load Testing Metrics\">Percentiles are great for monitoring <\/a>in improving your website or application<\/strong>. Percentile approximations are very useful for working with large time-series data sets because they offer the benefits of using percentiles (rather than averages or other counting statistics).<\/p>\n\n\n\n<p>If your website is too slow, and you want to improve the response time, or an API load time is slow when it&#8217;s requests by a large number of concurrent users, you can focus on improving the 90th percentile.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/loadfocus.com\/load-testing\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"742\" src=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-7-1280x742.jpeg\" alt=\"\" class=\"wp-image-2240\" srcset=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-7-1280x742.jpeg 1280w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-7-800x464.jpeg 800w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-7-600x348.jpeg 600w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-7-768x445.jpeg 768w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-7-1536x891.jpeg 1536w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-7.jpeg 1600w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><\/a><figcaption><a href=\"https:\/\/loadfocus.com\/load-testing\">Load Testing your Website<\/a><\/figcaption><\/figure>\n\n\n\n<p><em>Improving the 90th percentile will ensure that the overall response time of your website goes down.<\/em><\/p>\n\n\n\n<p>However, your 90th percentile can look very good, but we see a lot of websites with very bad 95th or 99th percentiles.<\/p>\n\n\n\n<p>These kind of observations are only possible using percentiles, with basic performance stats like averages, min, max these observations would&#8217;ve not been possible.<\/p>\n\n\n\n<p>Hopefully, now you are beginning to see applications of percentiles all over your data and your performance metrics.<\/p>\n\n\n\n<p>People often use averages and other summary statistics more frequently than percentiles because they are significantly \u201ccheaper\u201d to calculate over large datasets, both in computational resources and time.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/loadfocus.com\/load-testing\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"672\" src=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-4-1-1280x672.jpeg\" alt=\"\" class=\"wp-image-2245\" srcset=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-4-1-1280x672.jpeg 1280w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-4-1-800x420.jpeg 800w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-4-1-600x315.jpeg 600w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-4-1-768x403.jpeg 768w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-4-1-1536x806.jpeg 1536w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-4-1.jpeg 1600w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><\/a><figcaption>Percentiles in Load Testing<\/figcaption><\/figure>\n\n\n\n<p>The point is that the <strong>averages<\/strong> don&#8217;t give us a good way to <strong>distinguish<\/strong> <strong>between outliers and real effects<\/strong> and can give odd results when we have a long-tail or asymmetric distribution.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/loadfocus.com\/load-testing\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"726\" src=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1-1280x726.jpeg\" alt=\"\" class=\"wp-image-2246\" srcset=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1-1280x726.jpeg 1280w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1-800x454.jpeg 800w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1-600x341.jpeg 600w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1-768x436.jpeg 768w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1-1536x872.jpeg 1536w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1-968x550.jpeg 968w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1.jpeg 1600w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><\/a><figcaption>Raw Response Times in Load Tesitng<\/figcaption><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why You Should Monitor your Website<\/h2>\n\n\n\n<p>To understand how our website performed and find ways to improve you need to test and monitor at a scheduled basis.<\/p>\n\n\n\n<p>Monitoring an API is a common scenario whcih can help you catch and fix issues before they impact too many production users, which may cause a lot of losses to the bottom line of your business.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Percentiles in Load Testing<\/h2>\n\n\n\n<p>Using the <a href=\"https:\/\/loadfocus.com\/load-testing\" title=\"load testing\">Cloud Load Testing tool<\/a> from <a href=\"https:\/\/loadfocus.com\/\">LoadFocus<\/a>, you can easily inspect raw response times, min, max, along with all percentiles:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>50th percentile<\/strong><\/li><li><strong>90th percentile<\/strong><\/li><li><strong>95th percentile<\/strong><\/li><li><strong>99th percentile<\/strong><\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"638\" src=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-6-1280x638.jpeg\" alt=\"\" class=\"wp-image-2239\" srcset=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-6-1280x638.jpeg 1280w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-6-800x399.jpeg 800w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-6-600x299.jpeg 600w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-6-768x383.jpeg 768w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-6-1536x765.jpeg 1536w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-6.jpeg 1600w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><figcaption>Load Testing results using <a href=\"https:\/\/loadfocus.com\/\" title=\"cloud load testing\">LoadFocus<\/a><\/figcaption><\/figure>\n\n\n\n<p>Similar results can be inspected when using JMeter scripts and running them in the cloud using the <a href=\"https:\/\/loadfocus.com\/jmeter-load-testing\" title=\"jmeter cloud testing\">JMeter Cloud Load Testing Tool<\/a>.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p>Hopefully, now you are beginning to see applications of percentiles all over your data and your performance metrics.<\/p>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<p><em>Written by&nbsp;<a href=\"https:\/\/coursinator.com\/instructor\/bogdan-vazzolla\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">Bogdan&nbsp;Vazzolla<\/a>.<\/em><\/p>\n\n\n\n<p><em><a href=\"https:\/\/loadfocus.com\/\">LoadFocus<\/a>&nbsp;is a cloud testing platform,&nbsp;a&nbsp;<a href=\"https:\/\/loadfocus.com\/load-testing\" target=\"_blank\" rel=\"noreferrer noopener\">load and stress testing tool<\/a>&nbsp;which provides the infrastructure to run tests with thousands of concurrent users, from&nbsp;<a href=\"https:\/\/loadfocus.com\/locations\">multiple cloud locations<\/a>, in less than a few minutes, keep history of the results, compare different runs to inspect performance improvements or performance degradation.<\/em>&nbsp;<em>It also supports running&nbsp;<a href=\"https:\/\/loadfocus.com\/jmeter-load-testing\" target=\"_blank\" rel=\"noreferrer noopener\">JMeter load tests from the cloud<\/a><\/em>&nbsp;and&nbsp;<a href=\"https:\/\/loadfocus.com\/page-speed-monitoring\">monitoring and audit web and mobile performance<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p><span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\"><\/span> <span class=\"rt-time\"> 5<\/span> <span class=\"rt-label rt-postfix\">minutes read<\/span><\/span>Why Performance Testing We all want our websites to be responsive and useful. However, the biggest correlation between website performance and business results is seen by improving the performance and speed of your website. Users get bored if they have to wait for too long to load your website or application. Performance Testing and Load&#8230;  <a href=\"https:\/\/loadfocus.com\/blog\/2021\/10\/why-are-percentiles-more-useful-than-averages-in-performance-testing\" class=\"more-link\" title=\"Read Why are Percentiles More Useful than Averages in Performance Testing\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":1,"featured_media":2235,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,279,6,48],"tags":[359,361,360],"class_list":["post-2232","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-load-testing","category-modern-performance-metrics","category-performance-testing","category-test-automation","tag-percentiles-vs-averages","tag-performance-testing-with-percentiles","tag-response-time-using-percentiles"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/posts\/2232","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/comments?post=2232"}],"version-history":[{"count":1,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/posts\/2232\/revisions"}],"predecessor-version":[{"id":2754,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/posts\/2232\/revisions\/2754"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/media\/2235"}],"wp:attachment":[{"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/media?parent=2232"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/categories?post=2232"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/tags?post=2232"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}