Success with Test Automation

Bret Pettichord

Revised version of a paper originally presented at Quality Week, San Francisco, May 1996.
Version of 28 June 2001.


This paper describes several principles for test automation. These principles were used to develop a system of automated tests for a new family of client/server applications at BMC Software. This work identifies the major concerns when staffing test automation with testers, developers or contractors. It encourages applying standard software development processes to test automation. It identifies criteria for selecting appropriate tests to be automated and advantages of a Testcase Interpreter. It describes how cascading failures prevent unattended testing. It identifies the most serious bug that can affect test automation systems and describes ways to avoid it. It circumscribes reasonable limits on test automation goals.


Over the past several years, tools that help programmers quickly create applications with graphical user interfaces have dramatically improved programmer productivity. This has increased the pressure on testers, who are often perceived as bottlenecks to the delivery of software products. Testers are being asked to test more and more code in less and less time. They need to dramatically improve their own productivity. Test automation is one way to do this.

This paper presents advice on how to staff, plan and design a test automation system for GUI applications. I will present some ideas that have helped me make testsuites that are reliable and easy to maintain. These concepts and suggestions will be demonstrated by reference to the system built to test BMC's MetaSuite family of products. These client/server applications provide an easy-to-use interface for administering open systems databases.

Taking Test Automation Seriously

Software testers, under pressure to do more testing in less time, often find themselves rushed and eager for anything that will give them a hand. Their fear and desperation lead them to seek a "silver bullet" that will help them regain a rational pace to their work. Test automation is often this silver bullet. The fantasy is that it will make their job simpler and easier and help them contend with unrealistic schedules.

Automating tests for a graphical user interface presents significant difficulties not found in character based interfaces, much less command line interfaces or programming interfaces (APIs). Graphical user interfaces tend to be made of complex components and tend to be constantly redesigned during the development process. Significant successes have been made in delivering test tools that are able to identify and manipulate graphical user interfaces. QA Partner by Segue Software and Xrunner and WinRunner by Mercury Interactive are examples.

Most experienced software testers have excellent insights into the kinds of practices that are critical for software development. They see the consequences of their developers coding before designing, neglecting code reviews or ignoring feedback: more bugs and slipped schedules. However, testers' clear insight into the development process often fades when they undertake test automation. They fail to realize that test automation is itself a software development activity and that it therefore needs to adhere to critical development practices. Like developers who are stressed for time, testers are prone to skipping steps, taking the big leap, and blindly hoping to come out with a success at the other side. But frustration and disappointment are more likely consequences.

Some testers even try to develop test automation in their spare time. I have rarely seen this deliver testsuites that can bear much weight.

Major challenges for GUI test automation are maintainability and reliability. These challenges demand that a software engineering approach be taken to address them. Different teams define the software development process differently. This is fine. The important thing to remember is that this process should be used with test automation as well.

Let's look at some of the ways testers and developers are tempted to underestimate test automation. First off, the name "test tools" makes them sound simple and easy to use. But they are really development environments specialized for creating testing programs.

Many testers do not have strong programming skills. This combined with the repetitive nature of much testing, leads people to use record and playback techniques. There indeed are many tools that allow scripts to be recorded and then played back, using screen captures for verification. The problem that always crops up is that the layouts are changed, invalidating the screen captures and then the interface controls change making playback fail. Now the scripts must be re-recorded from scratch. Record and playback tools provide an easy way to create throwaway testsuites.

It is particularly hard for people who have had so much success using these techniques with character-based interfaces to understand the difficulty of using them with graphical interfaces.

Recording tests and reviewing the created code is an excellent way to learn how to use a test tool. But most successful test automators move on to more generalized testing methods.

Someone using a test tool, whether she has the title of tester or developer, needs to understand how to design, develop and maintain software. Test automation systems will need to be tested themselves and should be subjected to frequent review and improvement to make sure that they are actually addressing the testing needs of the organization.

Who Should Automate Tests?

I have been a GUI test automation specialist for a couple different testing groups in the past four years and currently head up a small team of automation specialists. I think it makes a lot of sense to have someone focus on these aspects of the testing. These are some of my thoughts on how to select someone to do this task.

A test automator needs to have good testing and development skills. She needs to understand testing requirements and the situations testers face. Automating tests should not be an opportunity to impose a particular testing methodology on testers. They will find fault with it and refuse to use it. Rather it needs to build from existing testing methodologies.

If the test automator has background as a tester, you will need to ask if she will show the necessary discipline. Sometimes testers who really want to be programmers seize on test automation as a way for them to develop these skills. It is important that they have good judgment and not get carried away with the programming. Be wary if they are hoping to automate all of their testing. They need to be focusing on the big wins. They may focus on improving the automation when it is actually good enough for the job.

A test automator needs to know how to develop software. She needs to be particularly aware of issues such as maintenance and reliability. Making the system easy to update with changes to the product under test should be the priority.

If her background is as a developer, you will need to ask if she has understanding and respect for the testing process.

Sometimes you can find independent contractors who have well-matched backgrounds. With them, you will have to ask who will be maintaining the testing system after they have left. Maintenance will be a critical challenge.

If you have access to good training in test automation, take advantage of it. Developments in test automation are being made very quickly. It's often cheaper to pay to people learn from someone else's mistakes than to have to make them make the mistake again themselves.

Don't assign automation to rejects from programming or testing. Unless test automation is done by someone who is motivated and working closely with the rest of the development group, it will not succeed.

Choosing What To Automate

A colleague once asked me if I thought it was theoretically possible to automate 100% of testing. This question threw me. Theoretically, testing should never be necessary in the first place. The programs should be coded correctly first off! But we're not really talking about the theoretical. Testing is the art of the pragmatic. Good software testing requires good judgment.

Look at where you are spending a lot of time testing manually. This is what you should consider automating. If you are a conscientious tester, you are often aware of things you wished you only had time to test. Don't focus your automation efforts on these tasks that may otherwise go untested. It's usually a mistake. For one thing, you only want to code automation after you have a determined testing procedure. If you've run through your manual tests a couple times, you probably have a solid sense of how to proceed with the testing. You don't want to automate tests you haven't run much manually and then realize that there is a more-effective procedure. This may mean re-working your automation or just giving up on it. Another problem with automating tests you haven't found the time to run manually is that you're not likely to find the time to maintain them later. Test automation always breaks down at some point. Make sure the tests are important enough that you will be devoting the time to maintain them when the opportunity arises. First, get your testing procedures and practices standardized and effective. Then, look at how you can use automation to improve your productivity.

Testing can be boring. Sometimes people want to automate even casual tests that will only be executed a couple times. The thought is that automation may allow them to avoid the tedium. But there are snags. Complications arise. Test automation will itself have to be debugged. This may often take as much time as it would to just execute the tests manually in the first place. I use a rule of thumb that says test automation will take at least ten times the time it would take to execute the tests manually. Don't fall into the temptation to automate simply to make your job more exciting.

Many testers really want to be programmers. Test automation may provide these people with an opportunity to develop their programming skills. If you find yourself in this circumstance, try to stay clear on your goals and how to sensibly use your programming skills to accelerate the testing. Don't let your attention get caught into the programming for its own sake. Don't try to automate all of your testing. Don't be a perfectionist. If your program does the testing, great. It can have a couple bugs. You're not creating a commercial product; if there are fatal bugs, you'll be around to fix them. Later I'll discuss some advice regarding the parts of test automation that must be reliable. If you are intent to become a programmer, hone your testing skills while you seek a programming position. They will be extremely valuable when you get programming work.

Performance is an area where wasted effort can be applied to test automation. Performance improvements generally depend on assumptions about the product under test. But since maintainability is usually fostered by making as few assumptions about how the product works as is practical, improving performance often reduces maintainability. Don't do this. Make maintainability a priority. I have seen performance improvements to test automation that had to be ripped out when the product changed. They made it harder to maintain the testsuite and didn't last long anyway. The best way to allow more tests to be run in the day is to design your testing system to allow for unattended testing. I have more say about this later.

Test automation efforts have failed by trying to do too much. You are better off trying to get first results quickly. This has several advantages. It will allow you to quickly identify any testability issues. These may require cooperation from developers and may take some time to resolve. The sooner they are identified, the better off you are. You may also wish to just automate the most laborious part of the testing, leaving such items as setup and verification to be done manually.

Starting small and building from there will allow you to validate your testing strategy. Getting early feedback from testers, programmers and build engineers will allow you to grow your testsuite into something that will benefit many people. Demonstrate to your programmers the assumptions you are depending on.

If you've been asked to specialize on the test automation, you may find a tendency to try to get a big chunk all worked out before handing it off. Fight this tendency. The sooner you hand off bits to others that they can use in their daily testing, the better off you all will be. Test automation may require testers to rethink how they are doing their job. The sooner they start this, the better. Late in a testing cycle, they may find themselves putting all their energy into keeping up with the product changes and the bug list. They may not put much energy into learning how to use the automation and you may find yourself frustrated when it goes under used and unappreciated.

First, try to get one test to run. Then build up your testsuite. Realize that the people using test automation don't care much code you've written to support the testing. All they will notice is how many tests are automated and how reliable the testsuite is. After you have a small suite, you can work on generalizing code and putting together a more general testing system.

Build acceptance tests are the tests that are run before software is moved into testing. These should be able to be run quickly, often in less than an hour. These tests are excellent candidates for automation. These tests are run frequently. These tests should try to cover as many different functions of the product as possible. The aim is breadth, not depth.

It's worth the investment to make your acceptance tests easy to setup and run. Once the acceptance testsuite is put in place, smart programmers will want to run it on their code before checking it in. Finding their own bugs will help them avoid embarrassment and lot of rushing around. As a tester, you will want to do all that you can to make this happen.

In my experience, making good decisions about what to automate can be critical to successful test automation. There are often many simple things that can give big paybacks. Find them.

Building Maintainable Testsuites

One of the biggest challenges to using automated testsuites is keeping them functional as the product interface changes. The BMC Meta testsuite uses several techniques to allow it to be easily maintained as our product interface changes.

We use QA Partner for our test automation. This includes tools for creating "window declarations" which map window controls to logical names. If the name of a control changes, we can update the window declaration. All of our test scripts will now work with the revised product. Another nice feature of this tool is that it can often locate moved controls. In these cases, we don't need to make any changes to our testsuite. Window declarations are just one technique we use to keep our testsuites easy to maintain.

When we have different dialogs with largely the same controls, we use QA Partner's class hierarchy to set up a superclass that contains the common controls. Only the unique items are defined for the individual dialogs. This also simplifies maintenance.

It is very important for us to be able to anticipate user interface changes. We keep in close communication with our developers on this. They understand that late and unannounced changes to the user interface may delay testing. By knowing what parts of the interface remain subject to design changes, we are able to use common routines that can be easily updated later.

We also use common code for testing equivalent features in the different products in the Meta product family. Different products administer different databases, such as Oracle, Sybase, DB2 and the like. Generalizing the common aspects has made it easy for us to port our testing apparatus to new products. It has also reduced the total lines of code, thus reducing the amount of code to maintain and debug.

Probably the most significant thing we do to reduce the maintenance burden is we write our testcases in an abstract testing language. Testcases only indicate the objects to be manipulated in the testcase. We build an interpreter and test driver to actually execute the testcases. This has many advantages, only one of which is easing maintenance. I'd like to explain how we do this in more detail.

Building Test Interpreters

A test interpreter and driver allow testcases to be easily specified by a domain expert. Many testers do not want to have to deal with the various details of a testing tool. A test interpreter allows them to focus on testing requirements rather than automation implementation. The testcase indicates the details of what to test. The test interpreter and driver actually do the testing. They know how to do the testing. This arrangement is particularly effective when different people are responsible for the testcases and the test automation.

Here is an example of one of our testcases.

TEST CASE ID: dtbed101
EDIT TABLE: SA3.TB03								
Note: Column name is of maximum length and is of type char.
EDIT TABLE: SA2.TB03								
Note: Column is of type integer.
EDIT TABLE: SA3.TB02								
Note: Column is of type clob.  Logged is the default.

The format for this testcase was originally based on documentation that was meant strictly for use by other people. We formalized it and made it be the actual input language for our test driver. Let me review some of the advantages to using this kind of format for specifying testcases.

Testcases are independent of implementation details. Many of our testcases are specified long before our testers know what the user interface will look like. Also, when interface changes are made later, the testcases don't have to be updated. The testcases only need to be changed when the product requirements change.

Testers don't have to know test tool details. We leave this for our automation specialists and those testers who have an interest in the testing tools.

Testing can focus on requirements. We are able to leverage the knowledge of our domain experts. We document the testcase format and this is what they need to know.

Tests are self-documenting. Since the format was originally based on documentation, the tests are easy to review and understand.

Let me give some more information about how we develop our interpreters and drivers.

The testcase format which I gave an example of above is actually the second generation. An example of the first generation format is given below.

| |dtbed101|E|TB||SA3|TB03||C|011|datatype||CHAR(100)
| |dtbed101|E|TB||SA3|TB03||C|011|null||Y
| |dtbed101|E|TB||SA3|TB03||C|011|default||N
| |dtbed101|E|TB||SA2|TB03||C|011|colname||NEW_INTEGER_COL
| |dtbed101|E|TB||SA2|TB03||C|011|datatype||INTEGER
| |dtbed101|E|TB||SA2|TB03||C|011|null||Y
| |dtbed101|E|TB||SA2|TB03||C|011|default||N
| |dtbed101|E|TB||SA3|TB02||C|035|colname||NEW_LOB_COL_AT_END
| |dtbed101|E|TB||SA3|TB02||C|035|datatype||CLOB(5K)
| |dtbed101|E|TB||SA3|TB02||C|035|null||Y
| |dtbed101|E|TB||SA3|TB02||C|035|default||N
| |dtbed101|E|TB||SA3|TB02||C|035|logged||Y

This format is easier for our automation to parse and execute but is more difficult to write. It was difficult for our testers to create these files and they would often make mistakes like putting items in the wrong column. This would lead to a long process of debugging testcases. This was often frustrating for testers, who would rather be finding bugs in the product than in their own test data.

This testcase was actually created from the first by means of a translator that converts the information from the first format to the second. We wrote our translator in Perl. The column format is then the input to the test driver written in QA Partner.

Let me review several components of our testing system that allows us to support easy-to-read testcase formats.

Translator. This converts the testcase into a format that is easier for a program to read. Our translator is written in Perl. QA Partner does not support the kind of string-matching commands (regular expressions) that this required.

Testcase Reader. This QA Partner function reads and parses the intermediate format. Errors in the test data are reported.

Test Driver. This QA Partner script starts the testcase reader and executes the lines of the testcase. It embodies the testing methodology and conventions. The driver finishes by triggering the product to generate a work script. Our test drivers also test things like memory leaks.

Window Declarations. This QA Partner file identifies the controls. Any special handling for non-standard controls can be specified here.

Verification. A separate Unix shell script compares the generated work script against a pre-defined baseline. The script automatically ignores insignificant differences such as time stamps.

Keeping Your Testsuite Reliable

You will want to be able to depend on your automated testsuite. You will want it to be able to run it new builds need to be tested. You will want to trust its accuracy.

The absolute worse thing that can happen to an automated testsuite is that it reports that tests have passed when there are in fact problems with the functions they should be testing. This situation is called a false positive. If your testsuites get a reputation for false positives, no one will want to use them. They'd rather do the testing manually.

Your test automation will have bugs in it. You will be able to live with this if automation bugs either result in aborts (the test didn't run) or false negative (reported failure but no product bug). Generally, you will want to be manually reproducing reported problems anyway.

The goal of test automation should be to reduce the number of tests that need to be run manually, not to eliminate manual testing entirely. As long as no more than a small portion result in false negatives, automation will have saved you significant amounts of time. Now you know the likely places to find bugs: the testcases that failed.

When you are coding your testsuite, you will want to take some measures to ensure that errors are not hidden or ignored. That is generally the cause of false positives. The easiest way of inserting this kind of problem is to make a mistake with exception handling. Double-check any exception handling code you write or better yet have someone else review it. The rule of thumb is "When in doubt, call it out." What this means is that unless your code is sure of the cause of an error, it should not suppress the reporting of the error. I have also seen false positives result from switch statements that did not include a default clause.

Being very careful to avoid false positives will allow you experiment more with other parts of your testing system. It does not have to be perfect to be useful.

I have also found that usability is important for perceived reliability. If it is easy to set up the tests incorrectly without getting good error messages, testers won't think well of the testing system. They will be frustrated if they review the results of a test run only to realize they forgot to set a parameter, meaning that they tested the wrong thing. If this happens repeatedly, they will realize that test automation is not saving them time. Be sensitive to these issues. Think about how to reduce the likelihood for these kinds of problems.

The biggest way to keep your testsuites reliable is to design them so that they can be run unattended. This will allow you test at overnight or while you are at meetings. It will also allow you to be testing on multiple machines at once.

Using Error Recovery Systems

A common problem that prevents productive unattended testing is cascading failures. This is what happens. One of the tests fails. The product is left in an unexpected state (perhaps an error dialog is displayed). Subsequent tests attempt to run but can't because the error dialog is still displayed. To run the testsuite, the product must be reset and the testsuite must be restarted after the failed test. Successive failures will require the testsuite to be rerun again and again.

An error recovery system is the solution to this problem. It automatically records the error and then restores the application. This allows successive tests to run reliably. Cascading failures are avoided, allowing unattended testing to occur.

A recovery system needs to know what the "base state" of the product is. After each testcase, it will check to see if it is in this state. If not, it will reset the product.

Testcases must be set up properly in order to take advantage of a recovery system. Each testcase must start and end at the predefined base state. The base state for our products is just the main window that appears after starting it. This is a somewhat different approach from manual testing. Typically, manual testers don't reset the product before each test, but rather run several tests in succession, only resetting the product if a problem arises.

One consequence of this is that tests cannot depend on earlier tests. This principle is called "testcase independence." If a test is meant to pick up where another finished, it will have to be redesigned. One option is to include the repetition of the earlier test as part of the second test. Adhering to testcase independence will allow your tests to work with a recovery system and run unattended. They will also be able to be run singly or in a group.

Testcases that are not independent can cause difficulties unrelated to unattended testing. One may fail when run as part of a battery, but pass when run alone. You may decide that the bug is irreproducible, when the problem is really with the testcase.

We've built our error recovery system from code included with QA Partner as well as code we've written ourselves. Our recovery system has been customized to recognize our products' error dialogs and to be able to close down various product dialogs.

We began building ours very early. It helped us debug our test drivers and test data. Errors that our recovery system logs and handles include scripting errors, product error dialogs, unexpected product behavior, and product crash.

We also included code to handle time-out situations, but we are planning to remove this. The recovery system has not been able to cleanly shut down the product during a time-out. We've decided it's better just to wait.

Don't try to engineer your recovery system to recover from all "possible" errors. Instead, make it handle the actual errors you are encountering.

Living with Test Automation

Here are some recent results on our use of test automation during a busy testing cycle. In one week in March of 1996, we ran the following number of testcases of the type described above.

Product    Unique testcases run    Total runs
Product A       399                   1005
Product B        43                    155
Product C        54                     82

You can see that on average tests were run about three times during that week.

By dedicating people on test automation, we have multiplied the other testers' productivity and have focused on the testing areas where the big wins are.

We have accepted "Good Enough" test automation. We've realized that just like with products we sell, software quality is a combination of reliability, features and timeliness. Understanding the importance of avoiding false positives has allowed us to make reasonable quality trade-offs.

We have tightened our testing cycle. Our testing is more consistent and repeatable. We are able to test on more configurations. We are constantly improving our test battery.


QA Partner, Segue Software (

WinRunner and Xrunner, Mercury Interactive (

An Approach to Functional Test Automation, Kerry Zallar (

Testing Computer Software, Second Edition, Chaper 11, Cem Kaner, Jack Falk and Hung Quoc Nguyan (Van Nostrand Reinhold).

About the Author

Bret Pettichord is a consultant, writer and trainer specializing in software testing and test automation. He edits the Software Testing Hotlist, writes a column for, and frequently speaks at industry conferences. He is the founder of the Austin Workshop for Test Automation.