Expert Advice on Implementing Test Automation
“Deciding what to automate might be the most important factor in the success or failure of your automation project.”
Each organization has its own factors that determine how much automation can be accomplished within a given period of time. Instead of attempting to apply the rule of thumb that may be irrelevant to your project, it is best to select a pilot project and conduct a time-study in your own environment. Choose a project that you believe can be completed in less than a month and keep track of the number of hours spent on developing automated tests. This time-study will provide measures that will be useful for setting goals for future projects:
1. Time taken to automate a feature –By keeping track of the time taken to automate a feature and comparing it with the duration required to manually execute the same tests, you can determine a multiplier that will allow you to estimate future automation projects.
2. Amount of automated test coverage – By comparing the relative size of future projects to those already completed, it is possible to predict the amount of coverage that can be expected from test automation and its impact on the duration of test cycles.
Hiring and training the Staff
Experience and training are necessary to ensure a good ROI from test automation. Most automation efforts that fail do so because of a lack of experience. Prepare to send the test automation staff to the full set of courses offered by the test tool vendor. In addition, look for a course on test automation methodology. Be sure that it is based on practical experience.
It is helpful to have at least one staff member with prior experience on the automation team. If you are unable to hire someone already skilled in test automation, consider hiring a consultant, but be aware that there are downsides to that too. First, do not allow the consultant to create a complex framework that will require additional consulting to maintain. Make certain that your staff is fully engaged in the process. Domain experts should participate in the development of the automated tests. Finally, make sure that a training plan for your staff is part of the turnover process.
Evaluating test tools
You can develop poor automated tests with a good testing tool but you can’t write good automation with a poor one. It makes sense to spend several weeks evaluating a testing tool. Be certain that the tool under consideration can drive the application under test through a method that is similar to a real user using the keyboard and mouse, so that application failures are not being averted by going through a ‘back door’. The tool must also be capable of collecting information about the state of the application for verifying test results.
When evaluating ‘ease of use’, keep in mind that what appears easy upon initial usage might not be easy when it comes to maintaining tests in the future. It is a mistake to judge ‘ease of use’ based on the robustness of a test tool’s recorder. Instead evaluate the testing tool based on features like:
• Test planning
• Object recognition
• Powerful scripting language
• Built-in support for your environment (e.g. Web, Java, .Net, PowerBuilder)
• Reusable components
• Test data definition and management
• Test maintenance
• Transition from manual to automated testing
• Multiple machine execution
• Built-in test management
Deciding what to automate
Deciding what to automate might be the most important factor in the success or failure of your automation project. Look for aspects of your application where data is persistent so that expected results can be easily predicted. Identify opportunities to develop data-driven tests that can be reused with different sets of data.
Designing good test cases
Good test case design is important for both manual and automated testing. A test case should have a single objective and result in only one of two dispositions: pass or fail. The disposition of a test should always be determined by comparing the actual results to the expected results. A well designed suite of tests can be run in any order and adhere to the following rules:
• Tests are independent
• No test relies on the successful completion of another test
• Tests start and stop at a common ‘base’ state
Building a strong foundation
A good set of automated tests can be developed quickly and can be readily maintained over time with requirements for a strong architecture:
o If it takes weeks to develop the test framework (before the first test is written), it is probably over-engineered and will be difficult to maintain by all but the original author.
o Some testing tools automatically generate the test framework, but if your selected tool does not, opt for something simple.
o Tests should be constructed of reusable components.
o Data should be separated from automated tests so that the tests can be executed with multiple sets of test data.
o Test data should also be encapsulated so that it can be used for multiple tests.
o Object-oriented test architecture will insulate tests from changes in the application’s user interface.
o Building tests with reusable components minimizes maintenance with the changes in the application under test.
Managing test execution
Automated tests provide the best ROI when they are run frequently. Ideally, tests should be run after each new build of the application under test, to provide immediate feedback. It is much easier to find and fix a bug when developers are notified right after it is introduced. When tests are run regularly, it is easier to maintain them as small changes are made to the application. Doing so also provides the test team with feedback regarding what has changed in each build. To ensure that tests are run on a regular basis, consider executing them as part of the build process for the application under test.
Test automation can reduce costs and improve software quality if it is implemented using good software design principles. Following the guidelines outlined above should help ensure that your next project gets off to a good start.