Well into the pandemic and with the E-commerce boom rapidly accelerating due to people needing to stay inside, you’ll need to make sure that your website and mobile applications are performing well under high load for the holiday shopping season and especially for Black Friday and Cyber Monday sale events.
We have written a handbook, in order to help businesses from small to medium and large with this assessment, handbook (or testing plan) that can be used for setting up and running such a performance testing session. This would be useful, so that no site crash happens for you and your team during the Black Friday or Cyber Monday events.
Why should I care about performance testing my web and mobile applications ?
The answer is simple: if you don’t consider performance testing it’s important, it may affect your customers experience, it may hurt your sales and revenue in a bad way.
With that in mind, let’s take a deep dive into performance testing.
By using performance testing as a tool, will allow to avoid crashes and system slowness, which in turn means more people will have a better experience accessing your application, better UX, which translates to bigger sales and higher revenue for your company.
There are some examples online with companies that did not bother to run a performance assessment for their website before a major event like Black Friday. Check the links more a more in-depth understanding:
The list is much longer, but let’s focus on what to do to prevent these undesirable situations.
When should I start load and stress testing my application?
The short and correct answer is, all the time.
In reality, performance testing is time consuming and most of the companies, or the teams of engineers inside most of the companies, just do it before major events when the load on their applications is expected to shoot up.
From our experience, if you are in this category (and most of the companies are) you should plan the load or stress tests ahead of the Black Friday event with at least 2-3 weeks, so that in case something pretty bad is found during the test the engineering teams have time to fix the issue and rerun the test session.
Who should be involved in the testing process?
For the testing session to be successful, you need to involve people from multiple departments and teams.
- Developers (Backend, Frontend and Mobile Developers): they are the one closest to the application internals, they know the ins and outs of how the application is implemented; they are the first people to know if some performance issue arises as a result of an application fault or wrong configuration
- QA engineers: they represent the main part of the entire process as they are involved not only in running the test session, but also in planning, discussing and agreeing with all the involved parties, the discussion should take into account:
- which is the success criteria?
- what scenarios should be run?
- when is the best time to run the tests?
- which environments should be used (a lot of times the real performance test can only be run in production as is not feasible to have a development or testing environment at the same scale with the production one)?
- which are the tools that are needed?
- which are the metrics and logging information that is needed?
- DevOps: they are the best people to know what happens at the infrastructure level and the networking level, on cloud or on premise instances, they should handle all the provisioning necessary for meeting the performance session goals.
- Product Managers: should be the ones that know the business goals and have data and metrics about normal traffic and traffic from previous Black Friday or Cyber Monday periods, and can help with potential insights on current year’s expected levels of traffic based on marketing campaigns and growth.
What type of performance tests should we run?
An important first step for getting a head start would be for the QA team to have an initial plan proposition for the type of tests that they think should be run based on mapping them to the business goals agreed with the product managers.
Below is of load test types that can be performed during the assessment session:
- Load testing: gives information on how the system performs under an expected load
- Soak testing: gives information about how the system behaves under an expected load for a continuous period of time
- Stress Testing: gives information about the maximum capacity limits of the application / system
- Breakpoint testing: these is similar to stress testing; an incremental load is applied and the system is monitored for predetermined failure conditions; this type of testing is also referred to as Capacity testing because it determines the maximum capacity under which the system will perform to its required specification or SLAs.The results of the breakpoint tests gives information about the initial capacity allocation and the scaling strategy that is required.
- Configuration testing: this type of testing is performed to determine the effects of configuration changes to the system’s components on the system’s performance and behaviour; an example would be trying different memory allocations for applications, different methods for load balancing and check the performance impact
- Isolation testing: this type of testing involves repeating a test execution that resulted in a system problem as this testing can often isolate and confirm the fault domain
Which are the systems that will be under load and which are the ways of monitoring their behaviour?
Now that the team has agreed on the type of tests that will be run and the business goals that the test session needs to answer to, the next important step is to identify which are the system parts that will be under a heavy load and make sure that there is monitoring in place that will be able to provide to us insights into what is happening with our system in real time (you would like this to be true especially in the case of testing in production as you would not want to take down the entire production environment – even though probably less expensive than to be taken down during Black Friday).
Which is the information that we need to have access during the test session?
- Relational and NoSql databases metrics
- Physical machine metrics
- Network metrics
- Load balancer metrics
- Application metrics
- Logging information
Most of the metrics from above can be achieved by using products like this can be achieved using products like Datadog, SolarWinds, DynaTrace, NewRelic, AppDynamics, CloudWatch and others; all these give you the option to send information to a central point where all that information can be easily viewed on charts and dashboards.
The logging information is normally present in systems like Splunk, ElasticSearch; be sure to have queries, charts and dashboards already set up before the tests are running.
Normally a fairly mature company has all these systems set up way before getting to performance testing.
The information that is already on the dashboards before the test is started gives the team the baseline that they need to compare against when running the tests.
Which are the exact metrics that we should be considered during performance testing ?
This is quite a long discussion and can be checked out in a future blog post.
How can one person know all this stuff ?
Most of the time there is no person that knows all this stuff on his/her own. If there is such a person probably is someone from DevOps or the QA team, but most of the time the entire picture can be created from the information that Developers, QAs and DevOps hold collectively. So this is why you need someone from each of the teams present (make sure that is someone fairly knowledgeable in their area of expertise).
Which are the scenarios that you should consider?
Choosing the scenarios is not as straightforward as people might think. At a very simplistic view one would say, let’s test:
- “Login / Sign In” flow
- “Sign Up” flow
- “Add to cart” flow
- “Checkout” flow
And this is all correct, these should definitely be the first scenarios to be run when performance testing your application but there are more scenarios that you should take into consideration; below we have added some of the ones that should also be part of your test plan:
- “Add to wish list” flow in case the user missed the product but you will have a future customer for that item
- “Notifications” flow for new products available from your wish list: you would not want potential users to not know the products are back in stock
- “Forget password” flow as you would not want your potential customers remain logged out
- “Apply coupon” flow in case you web application has this functionality implemented
Which are the tools to be used to generate the needed load on the system?
There are different open source tools that the team can use for load testing. The most used ones are: Apache JMeter, Gatling, Locust. All these tools are similar with little variations.
JMeter Load Testing tool from LoadFocus is a cloud load testing platform the offers the possibility to define your tests using directly the web UI or by uploading your Apache JMeter scripts, and run these test at scale with thousands of concurrent users.
Besides the very easy configuration of the tests the platform offers large scalability, the possibility to load test your application from more than 15 different cloud geolocations with custom or predefined scenarios.
LoadFocus offers an easy way to understand the dashboards and reports The management and stakeholders can easily to understand the results of the tests ran by the engineering teams.
Which are the next steps ?
It’s fairly simple (in theory):
- run the tests
- analyze the results
- rerun the tests once the fixes for issues found during the initial performance testing session are done
- increase the load and rerun previous steps
More details on the actual running and analyzing the results in a separate post, as it is quite a lengthy discussion.
Ultimately, your customer experience and satisfaction comes down to a couple of things: UX, Performance, Site Reliability and Availability (taking into consideration the technical part of the business).
With this in mind, make sure that the Development, DevOps and QA teams are all aligned and have gone through the steps needed to make sure your application can withstand the unexpected events and will not crash or show slowness during the shopping season as this may hurt the bottom line of your business.
Written by Cristian Vazzolla.
LoadFocus is a cloud Performance, load and stress testing tool which provides the infrastructure and the ability to run all these tests with thousands of concurrent users, from multiple cloud locations, in less than a few minutes, keep history of the results, compare different runs to inspect performance improvements or performance degradation.