In case you are not familiar with the term, an assertion “is a test / a condition that must be tested to confirm conformance to a requirement”. Basically a test assertion is – a condition that compares the expected value with the actual value for a specified use case.
Why should I use assertions ?
Before we deliver a product, we need to run many test assertions to make sure the product meets the customer’s expectations. Also, you can check our intro blog post regarding how to run a load test with Apache JMeter.
Should we use assertions in the JMeter tests ?
Usually assertions are not used when we create the scripts for load or stress testing an application. Most of the tester that run these kind of tests base their results only on the response status codes that the application under test returns. This might be a problem and might make the final results not accurate.
What are the cases in which not using an assertions may affect your final results ?
Not all the time a successfull response from the server (200 OK) means that the application responded as expected. You may ask yourself and the a few cases are mentioned below:
the application redirects to a maintaince page
the application responds a 200 OK but sends an error message in the body of the response indicating a error message
the application redirects to an error page and sends back a 200 OK with and error message indicating why the error returns
To avoid these kind of errors to affect your performance report is a very good idea to use assertions:
in GET/POST/PUT/PATCH methods
in API performance testing
in functional testing
Be aware that if your are creating a long running test or a test with a lot of requests is good not to use a lot of assertions as they tend to consume quite a bit of memory which could make your JMeter tool crash and means starting the test again.
What type of assertions does JMeter support ?
There are several types of assertions that you can use in your test scripts. The most used ones are:
“Response Assertion“: the response assertion control panel lets you add pattern strings to be compared against various fields of the response
“XPath Assertion“: the XPath Assertion tests a document for well formedness, has the option of validating against a DTD, or putting the document through JTidy and testing for an XPath (if that XPath exists, the Assertion is true)
“HMTL Assertion“: the HTML Assertion allows the user to check the HTML syntax of the response data using JTidy.
“XML Assertion“: tests that the response data consists of a formally correct XML document; it does not validate the XML based on a DTD or schema or do any further validation.
“Compare Assertion“: can be used to compare sample results within its scope; either the contents or the elapsed time can be compared, and the contents can be filtered before comparison. Compare Assertion MUST NOT BE USED during load test as it consumes a lot of resources (memory and CPU). Use it only for either functional testing or during Test Plan debugging and Validation.
“Size Assertion“: verifies that each response contains the right number of bytes in it; you can specify that the size be equal to, greater than, less than, or not equal to a given number of bytes
How much do assertions consume in terms of CPU and memory ?
If the test that your are using is a long running test or makes a very large number of requests you may encounter unexpected failures related to “out of memory” errors or similar.
In terms of resources the HMTL Assertion, XML Assertion or Compare Assertion consume a lot of resources so is a good idea to avoid using those in long running tests. The other assertions does not have a very bad impact on the resources so they can be used without a problem.
Which is the scope to which the assertions apply ?
There are 3 cases related to which scope an assertion applies to:
1. applies to all the child requests
2. applies to all the sibling requests
3. applies only to one specific request
How to use a “Response Assertion” in JMeter ?
Response Assertion is one of the most widely used assertion as it allows to verify the content of the response, status code or headers.
It can be configured to verify:
the different segments of the response like text (response body)
the document type like pdf, doc etc
the headers
the response code (2xx, 4xx)
response message (description of code) and response headers
tests the response against a pattern string which can be a plain text or a Perl5 regular expression; the specified pattern is matched against the selected segment of the response
Let’s start by adding a response assertion that expects that the response code is 200.
We will now add another assertion that expects that the text “wikipedia” is present in the response of the server.
Running the script will show that that the request succeeded without any problem.
Now we will change the text of the second assertion to “dummy_text_inexistent” and run the test again.
In the next tutorials we will show you how to use the rest of the assertions that are supported in JMeter.
