Test Automation Tool Selection

The process of determining which tool will best be suited to our needs is not an easy one. Each organization will have different needs that should be met by software test automation, which will be complicated by the various applications to be tested and operating systems used.

The steps to software test automation tool selection are as follows:

1. Determine our requirements

    

    These are some of the important questions we will need to ask.

Ø  Compatibility issues

                   Any testing tool we choose will need to be compatible with:

ü  The operating systems our application supports

ü  The development environments used to create our application

ü  Third party software with which our application integrates

Ø  Tool audience

ü  Who in our organization will be using the tool?

ü  Will our organization allow for staff training?

Ø  Business requirements

                     Vendors need to conform to certain requirements before they can become suppliers to the organization.

Ø  Testing requirements

                  What types of testing problems do we want the tool to address?

ü  Manual testing problems

ü  Time constraints

ü  Shorter regression testing timeframes

ü  Test data setup

ü  Defect tracking

2. Identify our constraints

 These factors will allow us to exclude certain tools from our selection process at an early stage:

Ø  Environmental constraints

This category will include hardware and software constraints. Consider the operating systems the tool needs to work on as well as the hardware requirements for the tool.

Ø  Should the tool be co-resident with test applications?

This needs to be considered to ensure future use of the test automation software.

Ø  Supplier constraints

ü  Is the supplier a bona fide company?

ü  How mature is the company and their product?

ü  Do they have adequate technical support?

ü  How many other organizations have purchased the tool?

ü  What is the history of the tool?

Ø  Quality constraints

ü  Skill level required to use the tool

ü  Multiple user access

ü  Support and help documentation

ü  Integration with other tools

ü  Frequency of failure during realistic use

Ø  Budget constraints

Keep in mind that the budget amount available will have to cover the purchase of the tool as well as licensing, training and implementation costs for the use of the tool. The cost of the tool will also directly affect our Return on Investment (ROI) calculations in the proposed business case to management.

3. Compile a possible list of candidates

Ø  Feature categorization:

ü  Mandatory features: These are the features that are essential to accomplish our goal in meeting our requirements within the constraints.

ü  Desirable features: These are features that will distinguish the best tools from the others.

ü  Irrelevant features: Features that are not important and will not provide any real benefit to our situation.

Ø  Rate these features.

                  This may be an iterative process.

4. Make selection

Ø  Feature comparison

Our feature list will allow us to compare the tools in terms of performance of the features advertised in marketing material against the requirements of our own system.

Ø  In-house demonstrations

                 We can ask vendors to perform an in-house demonstration on our system.

Ø  Test of script maintenance

It is important to evaluate how easy it is to maintain the automation scripts on these tools.

Ø  Competitive trial

      Run all our tests on each of the tools on our short list using the same test applications. This might require more work from our side, but will give us a very good idea of how each tool compares against the others.