Introduction.
This article seeks to describe an approach to software regression testing that can be used in the typical commercial application setting. The strategy presented here uses a canned data approach as well as a simple regression test selection (RTS) technique.
The strategy itself is targeted at commercial data processing environments, such as Sales Order processing, CRM, Payroll, General Ledger etc.
Regression Testing, a quick definition.
There are many definitions of software regression testing services on the web but for our purposes regression testing refers to a systematic re-testing of code that has previously worked in order to detect any new errors that may have been introduced during code changes.
The problems with Regression Testing are:-
Typically there is a narrow time window, just after full system testing has been completed and prior to production release, during which regression testing can be performed on a stable system.
Regression testing itself, for any significantly large system, is an expensive process of preparation and execution.
The need for a Regression testing strategy.
Even with test automation services tools there is a need to reduce the regression testing effort scope and at the same time optimize the effectiveness of this testing effort.
The identification of those tests that would give the most ‘pay back’, in terms of finding software defects, requires the use of a regression testing strategy that includes building out a comprehensive set of test cases and selecting an appropriate subset to use.
The Retest all strategy should be self explanatory and that is to run the entire library of regression tests you have built. If time and money are not an issue, this is the strategy to select.
The deep dive when changed strategy.
Most commercial application testers will already use this strategy in one form or another and what is presented here is an attempt to formalize what many already practice.
The idea is to select a subset of test cases from the regression test library that will ‘exercise’ a single entrysingle exit block of code, which we will refer to as a component. The technique could be applied to Web Services (as a component) but the idea is to ‘touch’ every component that makes up the system. The modified component itself should be tested to a deeper extent.
By way of example, for a commercial application, lets say we have a Sales Order Processing system that schedules finished parts and creates an order and booking in Accounts Receivable.
We would have a regression test library for every component, or function, e.g. enter customer details, check customer credit, check parts availability, create sales order etc. In the test library we would also have deeper test cases that went beyond a basic touching of the component.
When a component is changed, for example credit check, then that component is subjected to the extensive test cases whilst the other components are tested with the minimal set of regression test cases.
Learn more about performance testing services
Some testers refer to this as ‘basic end to end testing’ for the entire system. The important point of using this strategy is that the full regression suite has to be constructed in a way that each single entry and exit point, for all components, has to be identified. By way of example if there is a ‘preferred customer’ route thru the system, with its own component (code module), then this has to be tested all the time during regression.
In essence this regression testing strategy requires an extensive set of test cases to match the various Use Case paths for the system.
The basic two level approach.
The two level approach is the simplest in terms of constructing the regression test suite. With this approach two test suites for each component (single entry, single exit) are constructed, one is the simple path thru and the other suite tests multiple paths thru the component.
Selecting the appropriate test suite(s).
A cross reference from the test cases to the components is constructed and for each release the changed components are known and the appropriate test suite (simple or complex) can be selected. That is the process selects the simple test suite for unchanged components and the deeper test suites for the changed components.
Data dependencies, the canned data approach
There is an obvious issue with this approach and that is data dependencies. By way of example lets say I have a component that has changed and to test it I need extensive data from another component (that has not changed), in this case the simple test of the first component may not present the second component with sufficient data to exercise the extensive test paths. There are other more complicated variations of this but the problem of data dependence between the components is an issue with this strategy.
The answer is building out and keeping an extensive database that is restored just prior to the regression tests being executed. This strategy, of building out a known state for a given database is widely used for training environments and other testing.
DbUnit is an open source data preparation utility that could be used for the purpose of setting up the requisite data for this regression testing strategy. There are other resources referenced for data quality as well as regression testing in the software testing directory.
Issues with the canned data approach
The above canned data approach provides a simple strategy to data dependencies between components but there will be cases when this strategy is not appropriate. For example there may be significant data base changes, or changes are made in several key components where the dependent data has itself been changed.
In these cases the canned data may not be representative (of what will happen in production) as the component that created the data has been changed in a way to compromise that assumption. In these cases impact analysis has to be undertaken and a decision needs to be made in terms of whether or not an extensive regression (deep dive with many components) needs to be executed.
Conclusion
Many of the decisions as to how much regression testing should be executed are dependent on the extent (and nature) of the changes to the system and the amount of time and resources available to test those changes.
The above regression strategy provides a basic framework to plan and build regression suites (and corresponding data). This strategy should facilitate informed decisions, and actions, on the many trade offs involved in the regression testing phase of a project.
As with most regression testing strategies a well thought out automated approach is essential, including the set up and tear down of the appropriate data.
This article seeks to describe an approach to software regression testing that can be used in the typical commercial application setting. The strategy presented here uses a canned data approach as well as a simple regression test selection (RTS) technique.
The strategy itself is targeted at commercial data processing environments, such as Sales Order processing, CRM, Payroll, General Ledger etc.
Regression Testing, a quick definition.
There are many definitions of software regression testing services on the web but for our purposes regression testing refers to a systematic re-testing of code that has previously worked in order to detect any new errors that may have been introduced during code changes.
The problems with Regression Testing are:-
Typically there is a narrow time window, just after full system testing has been completed and prior to production release, during which regression testing can be performed on a stable system.
Regression testing itself, for any significantly large system, is an expensive process of preparation and execution.
The need for a Regression testing strategy.
Even with test automation services tools there is a need to reduce the regression testing effort scope and at the same time optimize the effectiveness of this testing effort.
The identification of those tests that would give the most ‘pay back’, in terms of finding software defects, requires the use of a regression testing strategy that includes building out a comprehensive set of test cases and selecting an appropriate subset to use.
The Retest all strategy should be self explanatory and that is to run the entire library of regression tests you have built. If time and money are not an issue, this is the strategy to select.
The deep dive when changed strategy.
Most commercial application testers will already use this strategy in one form or another and what is presented here is an attempt to formalize what many already practice.
The idea is to select a subset of test cases from the regression test library that will ‘exercise’ a single entrysingle exit block of code, which we will refer to as a component. The technique could be applied to Web Services (as a component) but the idea is to ‘touch’ every component that makes up the system. The modified component itself should be tested to a deeper extent.
By way of example, for a commercial application, lets say we have a Sales Order Processing system that schedules finished parts and creates an order and booking in Accounts Receivable.
We would have a regression test library for every component, or function, e.g. enter customer details, check customer credit, check parts availability, create sales order etc. In the test library we would also have deeper test cases that went beyond a basic touching of the component.
When a component is changed, for example credit check, then that component is subjected to the extensive test cases whilst the other components are tested with the minimal set of regression test cases.
Learn more about performance testing services
Some testers refer to this as ‘basic end to end testing’ for the entire system. The important point of using this strategy is that the full regression suite has to be constructed in a way that each single entry and exit point, for all components, has to be identified. By way of example if there is a ‘preferred customer’ route thru the system, with its own component (code module), then this has to be tested all the time during regression.
In essence this regression testing strategy requires an extensive set of test cases to match the various Use Case paths for the system.
The basic two level approach.
The two level approach is the simplest in terms of constructing the regression test suite. With this approach two test suites for each component (single entry, single exit) are constructed, one is the simple path thru and the other suite tests multiple paths thru the component.
Selecting the appropriate test suite(s).
A cross reference from the test cases to the components is constructed and for each release the changed components are known and the appropriate test suite (simple or complex) can be selected. That is the process selects the simple test suite for unchanged components and the deeper test suites for the changed components.
Data dependencies, the canned data approach
There is an obvious issue with this approach and that is data dependencies. By way of example lets say I have a component that has changed and to test it I need extensive data from another component (that has not changed), in this case the simple test of the first component may not present the second component with sufficient data to exercise the extensive test paths. There are other more complicated variations of this but the problem of data dependence between the components is an issue with this strategy.
The answer is building out and keeping an extensive database that is restored just prior to the regression tests being executed. This strategy, of building out a known state for a given database is widely used for training environments and other testing.
DbUnit is an open source data preparation utility that could be used for the purpose of setting up the requisite data for this regression testing strategy. There are other resources referenced for data quality as well as regression testing in the software testing directory.
Issues with the canned data approach
The above canned data approach provides a simple strategy to data dependencies between components but there will be cases when this strategy is not appropriate. For example there may be significant data base changes, or changes are made in several key components where the dependent data has itself been changed.
In these cases the canned data may not be representative (of what will happen in production) as the component that created the data has been changed in a way to compromise that assumption. In these cases impact analysis has to be undertaken and a decision needs to be made in terms of whether or not an extensive regression (deep dive with many components) needs to be executed.
Conclusion
Many of the decisions as to how much regression testing should be executed are dependent on the extent (and nature) of the changes to the system and the amount of time and resources available to test those changes.
The above regression strategy provides a basic framework to plan and build regression suites (and corresponding data). This strategy should facilitate informed decisions, and actions, on the many trade offs involved in the regression testing phase of a project.
As with most regression testing strategies a well thought out automated approach is essential, including the set up and tear down of the appropriate data.
No comments:
Post a Comment