It’s pretty hard to manage what is out of your control. When it comes to regression it’s much more efficient to measure it than to control. Dealing a lot with testing and reviews, I’ve noticed some all-too-common and insidious symptoms that lots of regression tests suffer from: they are either repetitive or redundant, which significantly limits code coverage. Most engineers are misled by the observed simplicity of regression. This affects both manual and automated strategies.
Let’s jump right into the things essential for regression test management.
Identifying the value of regression tests
It’s necessary to figure out what exactly this very value consists of and what is the correct direction to put our attention to. The main ingredients are:
- Reliability. The level of your confidence in the system under test defines the value of regression testing. Once you are sure what the expected behavior is, the testing won’t be performed for nothing.
- Duration. Always pay attention to how much time it takes you to create, run and maintain the tests.
The success formula is evident: if a test takes little time and provides high reliability, this is what you need.
Also, keep in mind that “reliability” doesn’t equate to “get rid of work”. Though running automated regression tests doesn’t require constant attention and physical presence of an engineer, it still takes expenses to analyze the received data and to maintain the tests. So, be alert.
Managing regression tests
More is not necessarily better. Avoid executing multiple regression areas for whatever it may be worth. Assess potential weak points of the system and use appropriate tests to reveal them. Tagging cases by the functional block can become the solution.
Control release regression time leaks. There is also no need for a whole regression pack for testing a new release or changes in configuration. Just a number of regression tests per user story will be enough. While performing change-impact analysis or analyzing new commits chances are you already know what parts of the regression suite you will run.
Measure time since the first run. As far as the system under test is getting more complicated, it may require more complicated tests to maintain the necessary level of test value. Consider useful and time-saving integrations and system-level adjustments.
Changes in functionality. Once you added new functionality, make sure it is reflected on the related tests. It doesn’t mean creating new tests, but reviewing their reliability will be a wise move.
Test level. Your choice of regression tests will depend on the lifecycle of software development itself. Be reasonable when including new tests into the pack and remove those that have little value at the moment.
Test case retirement. Every time you archive a part of a system or an application, mark the related tests accordingly. It will contribute to testing management efficiency a lot. Distinguishing between “active/working” and “retired” regression tests will save your time and efforts.
Develop a flair for tracking down the coverage of regression tests. Be aware of the value of the regression tests you create, run and maintain. Measure and analyze all the time.
Conclusion: these simple steps will allow you to get the most out of regression testing:
- Decompose. It must be clear what requirement is under test to state clear business priority.
- Measure. If you don’t track the coverage of regression tests performed, then you have no idea about their actual value.
- Monitor. The timeframe required for running tests either adds value to your QA or cuts it off. As the amount of regression testing grows, consider using reporting tools, such as Zebrunner.
- Update. Make sure your tests don’t just work correctly but meet the actual state of the app.
- Manage. Get to know the time and costs required to create, run or maintain your regression tests.