{"id":2232,"date":"2021-10-20T08:59:53","date_gmt":"2021-10-20T08:59:53","guid":{"rendered":"https:\/\/loadfocus.com\/blog\/?p=2232"},"modified":"2023-09-03T14:09:47","modified_gmt":"2023-09-03T14:09:47","slug":"why-are-percentiles-more-useful-than-averages-in-performance-testing","status":"publish","type":"post","link":"https:\/\/loadfocus.com\/blog\/2021\/10\/why-are-percentiles-more-useful-than-averages-in-performance-testing","title":{"rendered":"Why are Percentiles More Useful than Averages in Performance Testing"},"content":{"rendered":"<span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\"><\/span> <span class=\"rt-time\"> 5<\/span> <span class=\"rt-label rt-postfix\">minutes read<\/span><\/span><!-- pn-tldr --><h2>Key takeaways<\/h2><ul><li>An average hides the slow tail, and the tail is what users complain about.<\/li><li>p95 and p99 describe the experience of your least fortunate users.<\/li><li>Report a percentile with the median; the mean alone can hide a serious problem.<\/li><\/ul><!-- \/pn-tldr -->\n<h2 class=\"wp-block-heading\">Why Performance Testing<\/h2>\n\n\n\n<p class=\"lead\">We all want our <strong>websites to be responsive and useful<\/strong>. However, the biggest correlation between <strong>website performance<\/strong> and<strong> business results<\/strong> is seen by <strong>improving the performance and speed <\/strong>of your website.<\/p>\n\n\n\n<p>Users get bored if they have to wait for too long to load your website or application.<\/p>\n\n\n\n<p><strong>Performance Testing<\/strong> and <strong>Load Testing<\/strong> your website and <a href=\"https:\/\/loadfocus.com\/blog\/2021\/02\/how-to-load-test-restful-apis-with-loadfocus\/\" title=\"How to Load Test RESTful APIs with LoadFocus\">restful APIs<\/a> will help you understand better how your website and web services behave when a lot of users access your website at the same time. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Performance Testing<\/strong> questions that website owners ask <\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>is my website going to crash?<\/li><li>is everything going to be slow or will freeze when doing certain actions?<\/li><li>will my users encounter <a href=\"https:\/\/loadfocus.com\/blog\/2014\/07\/errors-and-response-codes-in-load-testing-on-loadfocus-com\/\" target=\"_blank\" rel=\"noreferrer noopener\" title=\"Errors and Response Codes in Load Testing on LoadFocus.com\">errors<\/a>? how many errors and of what type?<\/li><li>is my website going to <a href=\"https:\/\/loadfocus.com\/blog\/2018\/12\/what-is-disaster-recovery-testing-in-software-testing\/\" target=\"_blank\" rel=\"noreferrer noopener\" title=\"What is Disaster Recovery Testing in Software Testing?\">recover<\/a> after a heavy load?<\/li><li>which are the most sensitive areas and how will they behave during a load test?<\/li><\/ul>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What are Percentiles<\/h2>\n\n\n\n<p><strong>A percentile is defined as the value where x percent of the data falls below the value<\/strong>.<\/p>\n\n\n\n<p>The <strong>Percentile<\/strong> metric tells you the value at which a certain percentage of your data is included.<\/p>\n\n\n\n<p>The <strong>median<\/strong>, or the middle value, <strong>is also known as the 50th percentile<\/strong> (the middle percentile out of 100). This is the value at which 50% of the data is less than the value, and 50% is greater than the value (or equal to it).<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"784\" src=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-1-1280x784.jpeg\" alt=\"\" class=\"wp-image-2237\" srcset=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-1-1280x784.jpeg 1280w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-1-800x490.jpeg 800w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-1-600x368.jpeg 600w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-1-768x470.jpeg 768w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-1-1536x941.jpeg 1536w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-1.jpeg 1600w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><figcaption>50th Percentile<\/figcaption><\/figure>\n\n\n\n<p>For example, if we call something \u201cthe 10th percentile,\u201d we mean that 10% of the data is less than the value and 90% is greater than (or equal to) the value.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"832\" src=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-90th-1280x832.jpeg\" alt=\"\" class=\"wp-image-2238\" srcset=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-90th-1280x832.jpeg 1280w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-90th-800x520.jpeg 800w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-90th-600x390.jpeg 600w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-90th-768x499.jpeg 768w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-90th-1536x998.jpeg 1536w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-90th.jpeg 1600w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><figcaption>90th Percentile<\/figcaption><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<p>And the 90th percentile is where 90% of the data is less than the value and 10% is greater:<\/p>\n\n\n\n<p>So a 95% percentile tells you the value which is greater than or equal to 95% of your data.<\/p>\n\n\n\n<p>Most people use basic statistics like <strong>min<\/strong>, <strong>mean<\/strong>, <strong>median<\/strong> or <strong>max<\/strong>, which for a large number of data they<strong> hide the truth<\/strong>.<\/p>\n\n\n\n<p>Mean and median tend to hide outliers, wherease min\/max are distorted by a single outlier.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why are Averages so Misleading<\/h2>\n\n\n\n<p><strong>Basic averages<\/strong> are an awful way to paint the world and the numbers around it. Using averages instead of percentiles can lead to false alarms or missed opportunities.<\/p>\n\n\n\n<p>The <strong>median<\/strong> is the middle value, where<strong> half of the data is above<\/strong> and <strong>half is below<\/strong>. The mean (aka average) is defined as the sum(value) \/ count(value).<\/p>\n\n\n\n<p>Let&#8217;s see one <a href=\"https:\/\/loadfocus.com\/blog\/2021\/01\/what-is-a-good-example-of-load-testing\/\" title=\"What is a Good Example of Load Testing?\">example<\/a>: <em>if the average monthly salary in your area is 2000 dollars, but 1 out of 10 people earns 11,000 dollars per month, this does not mean the rest of 9 people out of ten earn 2000 dollars, it means we have an outlier data, which is the 11,000 dollar salary, and 9 other people that earn 1000 dollars per months.<\/em><\/p>\n\n\n\n<p><strong>This is how the averages tend to hide outliers.<\/strong><\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How to Improve your Website Performance using Percentiles<\/h2>\n\n\n\n<p>Using averages can have a dramatic (and negative) impact on how values are reported, while percentiles can help you get closer to the \u201ctruth.\u201d<\/p>\n\n\n\n<p><strong><a href=\"https:\/\/loadfocus.com\/blog\/2016\/03\/standard-deviation-and-percentiles-in-load-testing-metrics\/\" target=\"_blank\" rel=\"noreferrer noopener\" title=\"Standard Deviation and Percentiles in Load Testing Metrics\">Percentiles are great for monitoring <\/a>in improving your website or application<\/strong>. Percentile approximations are very useful for working with large time-series data sets because they offer the benefits of using percentiles (rather than averages or other counting statistics).<\/p>\n\n\n\n<p>If your website is too slow, and you want to improve the response time, or an API load time is slow when it&#8217;s requests by a large number of concurrent users, you can focus on improving the 90th percentile.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/loadfocus.com\/load-testing\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"742\" src=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-7-1280x742.jpeg\" alt=\"\" class=\"wp-image-2240\" srcset=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-7-1280x742.jpeg 1280w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-7-800x464.jpeg 800w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-7-600x348.jpeg 600w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-7-768x445.jpeg 768w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-7-1536x891.jpeg 1536w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-7.jpeg 1600w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><\/a><figcaption><a href=\"https:\/\/loadfocus.com\/load-testing\">Load Testing your Website<\/a><\/figcaption><\/figure>\n\n\n\n<p><em>Improving the 90th percentile will ensure that the overall response time of your website goes down.<\/em><\/p>\n\n\n\n<p>However, your 90th percentile can look very good, but we see a lot of websites with very bad 95th or 99th percentiles.<\/p>\n\n\n\n<p>These kind of observations are only possible using percentiles, with basic performance stats like averages, min, max these observations would&#8217;ve not been possible.<\/p>\n\n\n\n<p>Hopefully, now you are beginning to see applications of percentiles all over your data and your performance metrics.<\/p>\n\n\n\n<p>People often use averages and other summary statistics more frequently than percentiles because they are significantly \u201ccheaper\u201d to calculate over large datasets, both in computational resources and time.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/loadfocus.com\/load-testing\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"672\" src=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-4-1-1280x672.jpeg\" alt=\"\" class=\"wp-image-2245\" srcset=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-4-1-1280x672.jpeg 1280w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-4-1-800x420.jpeg 800w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-4-1-600x315.jpeg 600w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-4-1-768x403.jpeg 768w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-4-1-1536x806.jpeg 1536w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-4-1.jpeg 1600w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><\/a><figcaption>Percentiles in Load Testing<\/figcaption><\/figure>\n\n\n\n<p>The point is that the <strong>averages<\/strong> don&#8217;t give us a good way to <strong>distinguish<\/strong> <strong>between outliers and real effects<\/strong> and can give odd results when we have a long-tail or asymmetric distribution.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/loadfocus.com\/load-testing\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"726\" src=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1-1280x726.jpeg\" alt=\"\" class=\"wp-image-2246\" srcset=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1-1280x726.jpeg 1280w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1-800x454.jpeg 800w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1-600x341.jpeg 600w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1-768x436.jpeg 768w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1-1536x872.jpeg 1536w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1-968x550.jpeg 968w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-5-1.jpeg 1600w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><\/a><figcaption>Raw Response Times in Load Tesitng<\/figcaption><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why You Should Monitor your Website<\/h2>\n\n\n\n<p>To understand how our website performed and find ways to improve you need to test and monitor at a scheduled basis.<\/p>\n\n\n\n<p>Monitoring an API is a common scenario whcih can help you catch and fix issues before they impact too many production users, which may cause a lot of losses to the bottom line of your business.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Percentiles in Load Testing<\/h2>\n\n\n\n<p>Using the <a href=\"https:\/\/loadfocus.com\/load-testing\" title=\"load testing\">Cloud Load Testing tool<\/a> from <a href=\"https:\/\/loadfocus.com\/\">LoadFocus<\/a>, you can easily inspect raw response times, min, max, along with all percentiles:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>50th percentile<\/strong><\/li><li><strong>90th percentile<\/strong><\/li><li><strong>95th percentile<\/strong><\/li><li><strong>99th percentile<\/strong><\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1280\" height=\"638\" src=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-6-1280x638.jpeg\" alt=\"\" class=\"wp-image-2239\" srcset=\"https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-6-1280x638.jpeg 1280w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-6-800x399.jpeg 800w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-6-600x299.jpeg 600w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-6-768x383.jpeg 768w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-6-1536x765.jpeg 1536w, https:\/\/loadfocus.com\/blog\/wp-content\/uploads\/percentils-load-testing-50th-results-6.jpeg 1600w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><figcaption>Load Testing results using <a href=\"https:\/\/loadfocus.com\/\" title=\"cloud load testing\">LoadFocus<\/a><\/figcaption><\/figure>\n\n\n\n<p>Similar results can be inspected when using JMeter scripts and running them in the cloud using the <a href=\"https:\/\/loadfocus.com\/jmeter-load-testing\" title=\"jmeter cloud testing\">JMeter Cloud Load Testing Tool<\/a>.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p>Hopefully, now you are beginning to see applications of percentiles all over your data and your performance metrics.<\/p>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<p><em>Written by&nbsp;<a href=\"https:\/\/coursinator.com\/instructor\/bogdan-vazzolla\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">Bogdan&nbsp;Vazzolla<\/a>.<\/em><\/p>\n\n\n\n<p><em><a href=\"https:\/\/loadfocus.com\/\">LoadFocus<\/a>&nbsp;is a cloud testing platform,&nbsp;a&nbsp;<a href=\"https:\/\/loadfocus.com\/load-testing\" target=\"_blank\" rel=\"noreferrer noopener\">load and stress testing tool<\/a>&nbsp;which provides the infrastructure to run tests with thousands of concurrent users, from&nbsp;<a href=\"https:\/\/loadfocus.com\/locations\">multiple cloud locations<\/a>, in less than a few minutes, keep history of the results, compare different runs to inspect performance improvements or performance degradation.<\/em>&nbsp;<em>It also supports running&nbsp;<a href=\"https:\/\/loadfocus.com\/jmeter-load-testing\" target=\"_blank\" rel=\"noreferrer noopener\">JMeter load tests from the cloud<\/a><\/em>&nbsp;and&nbsp;<a href=\"https:\/\/loadfocus.com\/page-speed-monitoring\">monitoring and audit web and mobile performance<\/a>.<\/p>\n<!-- pn-faq --><h2>Frequently Asked Questions<\/h2><h3>Why not just report the average?<\/h3><p>Because it hides the slow tail, and the tail is what people complain about. A comfortable average can sit above a meaningful share of requests that were unacceptably slow.<\/p><h3>What do p95 and p99 tell me?<\/h3><p>The experience of your least fortunate users. If the p99 is several seconds, one request in a hundred was that slow, which at scale is a great many people.<\/p><h3>What should a report contain?<\/h3><p>A percentile alongside the median. The mean on its own can conceal a serious problem, and the two together describe the spread honestly.<\/p><!-- \/pn-faq --><!-- pn-related-reading --><h2>Related reading<\/h2><ul><li><a href=\"https:\/\/loadfocus.com\/blog\/2016\/03\/standard-deviation-and-percentiles-in-load-testing-metrics\">Standard Deviation and Percentiles in Load Testing Metrics<\/a><\/li><li><a href=\"https:\/\/loadfocus.com\/blog\/2026\/06\/reducing-api-latency-distributed-load-testing-case-study\">Reducing API Latency With Distributed Load Testing<\/a><\/li><li><a href=\"https:\/\/loadfocus.com\/blog\/2026\/04\/optimize-api-performance-testing-faster-results\">10 Ways to Optimize API Performance Testing<\/a><\/li><\/ul>","protected":false},"excerpt":{"rendered":"<p><span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\"><\/span> <span class=\"rt-time\"> 5<\/span> <span class=\"rt-label rt-postfix\">minutes read<\/span><\/span>Key takeaways An average hides the slow tail, and the tail is what users complain about. p95 and p99 describe the experience of your least fortunate users. Report a percentile with the median; the mean alone can hide a serious problem. Why Performance Testing We all want our websites to be responsive and useful. However,&#8230;  <a href=\"https:\/\/loadfocus.com\/blog\/2021\/10\/why-are-percentiles-more-useful-than-averages-in-performance-testing\" class=\"more-link\" title=\"Read Why are Percentiles More Useful than Averages in Performance Testing\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":1,"featured_media":2235,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,104],"tags":[359,361,360],"class_list":["post-2232","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-load-testing","category-website-speed-testing","tag-percentiles-vs-averages","tag-performance-testing-with-percentiles","tag-response-time-using-percentiles"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/posts\/2232","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/comments?post=2232"}],"version-history":[{"count":1,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/posts\/2232\/revisions"}],"predecessor-version":[{"id":2754,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/posts\/2232\/revisions\/2754"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/media\/2235"}],"wp:attachment":[{"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/media?parent=2232"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/categories?post=2232"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/loadfocus.com\/blog\/wp-json\/wp\/v2\/tags?post=2232"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}