Recently, Aurion has made large changes in how we perform regression testing. This has seen vast improvements in productivity and quality. However, before we get to the good stuff let’s discuss: what is regression testing?
You and your team will always need to write new features or fix bugs. With that, there’s the chance that the changes you’ve made can and will affect the rest of the product. Regression testing is ensuring that your software continues to perform to expectations set by those who develop the product. In addition, those who use the product! From an industry standpoint, it is both the most boring, and the most interesting part of modern software testing!
Let’s Break It Down
Regression testing is simple. It’s having a set list of expectations the product needs to achieve to be customer ready. Having to perform this list of tests over, and over, and over again, isn’t very exciting. Manually testing generates lack of interest because you’ve run these tests multiple times before. And they’ve passed before, which leads to complacency and disinterested testers. Leading to a higher chance that actual bugs in this process can be missed. Introducing automation testing allows us to perform the same tests repeatedly without these risks. Freeing up our testers to work on new features without having to worry about regression testing.
With automation testing, there’s a lot of innovation and excitement going on. And, for Aurion’s quality assurance team we’ve recently matured. Previously we were using Selenium to create one front-end web framework that exercised both user interface, and business logic of Self Service. Following this method granted us some coverage, but over time our regression tests became slow, bloated, and unmaintainable. Some of the reasons for this include:
- Each test requires an instance of a browser to be created and then destroyed.
- Loading a web page takes time. Figuring out how much time changed in different environments, and programmatically tracking down where wait times were needed often caused tests to fail falsely.
- Updates to the product in one area may break the test despite not being the area under test.
The Aurion Process
Where we’ve matured is that our new process has us running two frameworks: a Selenium one that exercises the front-end user interface, and an API framework that checks the business logic of the product. This new framework circumvents all of the issues we had previously with the one Selenium framework. With this API framework we no longer need to wait for browsers or pages to load, and we don’t have to do front end navigation that would cause us errors as before reaching the area under test.
In practice, our old Selenium framework ran 30 tests in 33 minutes, by comparison moving to an API framework we’re now able to run 700 tests in 14 minutes. This allows us to have an unprecedented level of coverage, and confidence in our product as it is released.
We’re still running our Selenium framework alongside the new one, but now it’s more targeted on GUI elements instead of business logic. We’re no longer running field or input validation using this framework, instead, we’re making sure that when a user uses our web product that all events are firing as expected. This simplifies the tests performed with Selenium and lets us have increased confidence in their outcomes. It also increases the speed at which these tests run while providing us with the coverage that we’ll miss moving to a full API solution.
