April 24, 2009

Three Key Tools for Automated Testing: Language, Driver, Harness

When I started working on Watir, I was pushing a vision for automated testing. This vision still motivates my work with Watir. A couple of years ago, Brian Marick told me open-source developers should share their vision and let the community know why they built the tool. Here's the vision.

There are three essential elements of an automated testing system: language, driver and harness. Your tests need to be written in a language and you need a language to extend your testing system. I've long believed in scripting languages, such as Perl, Python and Ruby. I find that testers are more productive with them and find them easier to understand and use. I have reasons why I prefer Ruby, but there are other good automated testing systems written using other scripting languages. In fact over the years, I have built testing frameworks in Perl, Python and VB, usually building on languages that were already in use at the client. My emphasis on full-featured programming languages was a contrast to the proprietary languages that had been commonly used in commercial testing tool suites. I had no patience for them.

There are lots of languages to choose from. Finding a suitable driver, however, was often more difficult. A driver is what you need to drive your application. Watir is an example of a browser driver, suitable for web applications. Often the driver determines the language you use. Years ago, I used Expect as a driver for a command line application. Expect used TCL as its language, so the suite was written in TCL. I developed Watir because I wanted a browser driver in Ruby. Watir was our solution for Web Application Testing in Ruby.

The third thing you needed for a testing system was a harness. When we started, we all used Test::Unit, a Ruby test harness designed for unit testing. We found that it could be adapted for functional testing. More recently, Watir users have been using Rspec or Cucumber as their test harness. A test harness is responsible for executing tests and collecting and reporting results. Some people like to build their own test harnesses instead of using one of these.

I developed this vision while working with commercial testing tool suites like SilkTest and WinRunner. These packages are often referred to as tools, but I found that they actually comprised a collection of integrated tools. They had predefined usage expectations that often didn't match actual tester needs. Data-driven testing, for example, required us to break down the suite and then reassemble it in a more suitable configuration. I wanted testers to be able to identify the individual tools that made up a testing system.

Many new Watir users have trouble identifying where Watir (the driver) ends and Ruby (the language) begins. But learning this is key to becoming a proficient user. I'm told that some Java teachers don't like their students to use an IDE (such as Eclipse or Netbeans). Instead they want their students to learn to use the required tools separately -- the editor, the compiler. This way they learn the function of each tool. In the same way, I've also wanted testers to understand the functions of the different tools and components used in a testing system.

We often get questions from new Watir users whose answers are obvious once you understand this. Can Watir read CSV files? Can it do date math? Can it read from a database? The answer to all of these questions is that Watir can't do it, but Ruby can. No browser driver could do these things (it would be like asking whether your compiler supports search and replace), but any full-featured programming language can. It's just a matter of finding the library to do it. These questions are raised in the first place because testers are used to using commercial tool suites that were closed systems. So you only got the ability to do these kinds of things if the vendor added it to the package. Watir is part of an open system, so all the libraries are there already and more are being written all the time, often by people who are not part of the Watir community, per se, but rather part of the larger Ruby community.

Want to learn more? I am teaching classes on Watir, Ruby and the new WatirCraft framework that supports both Rspec and Cucumber.

Posted by bret at 11:37 AM | Comments (5)