In this blog post I pose a question that few ask or fully address:
What are the requirements for an automated testing framework?
Years ago the automated ‘requirements’ question would be confined to an individual testing tool (whether functional or performance) and looking at features such as recordplayback, GUIprotocol support, HTTPS and authentication support etc.
Today there is a whole context, or environment, in which an test automation services framework operates and it is this broader context that extends the ‘requirements’ consideration beyond the standard feature set of a given testing tool.
The requirements for an automated testing framework now include:-
Interoperability with:-
Continuous Integration systems (such as Hudson and Jenkins)
Test Management Systems, including Agile Story based systems (such as Testopia, Rally and Jira).
Source revision control systems (such as Perforce, GIT and Subversion).
The important point is that we have now moved the requirements discussion to a testing ‘framework’ rather than an individual testing tool.
Indeed one of the requirements of a ‘testing framework’ is to be able to plug and play (orchestrate) a variety of individual testing tools that target a specific technology that is being subjected to verification, for example SOAP, REST testing tools as well as Browser GUI testing tools.
The expression ‘plug and play’ as a requirement, stated in the previous paragraph, applies not only to testing tools but to any of the testing framework components or systems that the framework interfaces with (e.g Jenkins C.I.).
In this way the testing framework itself becomes an open architecture (not necessarily open source) that allows for extensions and the mixing and matching of appropriate components in order to contribute to the efficient and effective verification of the given System Under Test (SUT).
The above paragraph, including a definition of a testing framework, positions the scope of a set of requirements. By way of clarification, lets look at a requirements template for a testing framework.
The testing framework must be able to utilize the ‘appropriate’ individual testing tool(s).
Here the word appropriate is key, and it potentially refers to SOAPUI, Selenium, Jmeter or any other individual testing tool that has been deemed appropriate (useful) for testing a given aspect of the System Under Test.
In an actual set of requirements the list of testing tools would be stated as would the ability of the framework to be extended (to use other testing tools) in the future.
To know more about Software testing services
The testing framework must be able to integrate with:-
As noted above, and for each integration point there is a process requirement i.e. for Test Management integration: Report on the total number of tests executed (passfail) including automated testing, for a given Jira story.
The testing framework should be maintainable, useable, supportable……
These requirements are concerned with the overall flexibility of the framework, since the main advantage of a framework is that is can change and adapt to new environments (interfaces) and support new testing tools. When we examine these type of requirements (sometimes referred to as ‘non-functional requirements) we need to illustrate them with examples (or scenarios). By way of example:-
Adaptability:
We need to be able to swap out the Test Management system at a minimal cost.
Although the expression minimal cost could be replaced with a dollar amount, the idea here is to evaluate a number of design alternatives then estimate the cost to substitute a given component. In this way the Design choice that gives the lowest ‘adaptability’ cost is selected.
What are the requirements for an automated testing framework?
Years ago the automated ‘requirements’ question would be confined to an individual testing tool (whether functional or performance) and looking at features such as recordplayback, GUIprotocol support, HTTPS and authentication support etc.
Today there is a whole context, or environment, in which an test automation services framework operates and it is this broader context that extends the ‘requirements’ consideration beyond the standard feature set of a given testing tool.
The requirements for an automated testing framework now include:-
Interoperability with:-
Continuous Integration systems (such as Hudson and Jenkins)
Test Management Systems, including Agile Story based systems (such as Testopia, Rally and Jira).
Source revision control systems (such as Perforce, GIT and Subversion).
The important point is that we have now moved the requirements discussion to a testing ‘framework’ rather than an individual testing tool.
Indeed one of the requirements of a ‘testing framework’ is to be able to plug and play (orchestrate) a variety of individual testing tools that target a specific technology that is being subjected to verification, for example SOAP, REST testing tools as well as Browser GUI testing tools.
The expression ‘plug and play’ as a requirement, stated in the previous paragraph, applies not only to testing tools but to any of the testing framework components or systems that the framework interfaces with (e.g Jenkins C.I.).
In this way the testing framework itself becomes an open architecture (not necessarily open source) that allows for extensions and the mixing and matching of appropriate components in order to contribute to the efficient and effective verification of the given System Under Test (SUT).
The above paragraph, including a definition of a testing framework, positions the scope of a set of requirements. By way of clarification, lets look at a requirements template for a testing framework.
The testing framework must be able to utilize the ‘appropriate’ individual testing tool(s).
Here the word appropriate is key, and it potentially refers to SOAPUI, Selenium, Jmeter or any other individual testing tool that has been deemed appropriate (useful) for testing a given aspect of the System Under Test.
In an actual set of requirements the list of testing tools would be stated as would the ability of the framework to be extended (to use other testing tools) in the future.
To know more about Software testing services
The testing framework must be able to integrate with:-
As noted above, and for each integration point there is a process requirement i.e. for Test Management integration: Report on the total number of tests executed (passfail) including automated testing, for a given Jira story.
The testing framework should be maintainable, useable, supportable……
These requirements are concerned with the overall flexibility of the framework, since the main advantage of a framework is that is can change and adapt to new environments (interfaces) and support new testing tools. When we examine these type of requirements (sometimes referred to as ‘non-functional requirements) we need to illustrate them with examples (or scenarios). By way of example:-
Adaptability:
We need to be able to swap out the Test Management system at a minimal cost.
Although the expression minimal cost could be replaced with a dollar amount, the idea here is to evaluate a number of design alternatives then estimate the cost to substitute a given component. In this way the Design choice that gives the lowest ‘adaptability’ cost is selected.
No comments:
Post a Comment