August 20, 2004

Making WATIR

Paul Rogers and I are leading the development of WATIR, an new open source tool for testing web applications. It is a part of the Web Testing in Ruby (WTR) project, initiated by Chris Morris. Other contributors have been Brian Marick, Jonathan Kohl and Andy Tinkham.

We plan to make this tool the leading solution for testing browser-based business applications. This note summarizes the current status of this project and announces plans for further development.

Web Testing with Ruby

WATIR is based on three earlier versions of test tools in the WTR project. They all drive Internet Explorer (IE) using Ruby, utilizing Ruby’s support for COM and IE’s extensive COM interface to the the document object model (DOM). You are welcome to review our current work.

  • Morris created the initial version, called IEC. It uses several clever techniques to provide an extremely compact api. It is packaged with MoTunes, a demonstration application. This version is labelled IEC.
  • My version modified IEC, allowing it identify controls by means other means than names. We’ve also developed tutorial materials for the tool, based on Marick’s TimeClock application. Unit tests exercise most of the modifications and tutorial materials. We’ve been teaching this version of the tool in workshops over the past year. The tutorial and the tool are packaged as Scripting101 and depends on IEC.
  • Rogers created a more drastic modification, including detailed logging, to meet the needs at Wireless Matrix (WMX), where it is currently being used in production. A stripped down version of the WMX tool is contained in the Scripting101 distribution in the “contrib” directory. Samples scripts for using it to drive TimeClock are included.

Comparison with Alternate Tools

Our work demonstrates a superior approach to automating web-based applications than http-based test tools such as jWebUnit, HttpUnit or Canoo WebTest. These tools run into complications with JavaScript and Cookies because they must take responsibility for simulating the browser. Our approach avoids these complications because we use a real browser. We also have found that our approach, using interactive Ruby (IRB) and a visible browser, facilitates the development of scripts by testers with limited technical expertise.

Our approach is also superior to commercial tools such as WinRunner and SilkTest. We use a standard scripting language, facilitating library development. Test scripts can easily be stored in a configuration managment tool such as CVS. And our tests can be run either with the browser visible, which aids debugging, or in the background, allowing other work to occur while tests are running. The commercial tools do not support background operation.

And our tool doesn’t cost an arm and a let, allowing it to be deployed widely. Any one who wants to test a build can use our tests; they can be including automated build tools such as CruiseControl.

We note that our approach does not support testing the wide variety of technologies supported by the commercial tools. We don’t support Java Applets or ActiveX or Flash. But our approach does cover the kinds of web interfaces that we have found to be most prevalent in business applications. Our approach has also been used to test WinForms interfaces.

The Plan for WATIR

The Agile XP Universe conference in Calgary this week provided an opportunity for us to get our approach reviewed (we taught a tutorial) and have face-to-face discussions for the first time. We’ve developed plans to develop a new tool, which we are calling WATIR (Web Application Testing in Ruby). Looking at the existing tools in the WTR as spikes, we plan to build a tool from scratch using test-driven development, reusing code from WTR, and building on the unit testing techniques developed in the Scripting101 material. This tool will have 100% coverage by unit tests.

We have sketched out the API to the tool and have secured support from managment at both Wireless Matrix and ThoughtWorks for our participation in this project.

Our initial goal is to build a tool sufficient to support the Scripting101 tutorial. This entails support for buttons, text fields, check boxes and forms, accessing them by name, action or value. Next will be supporting WRX’s needs, entailing adding support for radio lists, frames, links, and selection lists. We’ll also be adding a logging interface, including selectable output to the console or xml suitable for cruise control.

During this period, we will be seeking critical review from the testing, open-source, and Ruby communities. I will be demonstrating our work at upcoming conferences in Austin, Minneapolis, Denver and Portland. Rogers and I will be jointly teaching a tutorial based on WATIR at STAR West in November in DisneyLand. All of the teaching materials for this class are open-sourced, and we are encouraging others to use them to teach classes in their communities. I am particularly interested in opportunities to teach this material publicly in Australia, India and the United Kingdom (where ThoughtWorks has offices). We are also working with the Center for Software Testing Education and Research to tailor these materials for individual self-study.

Language Choice

We have examined several languages and technologies that could serve as a basis for our tool. We believe that full-featured scripting language is a key element for our tool. We’ve considered Ruby, Python, Groovy and Perl.

We want something that will be easy for testers who are not expert programmers to learn and use. An interactive interpreter is a key tool for learning; this leads us to reject Perl.

Another key element of our solution is to use the COM/DOM interface of Internet Explorer. Unfortunately, this is hard to do from inside the Java virtual machine that serves as the foundation for Groovy. Thus we reject it (as well a the jRuby and Jython cousins to Ruby and Python). We also believe that the long-term respectability of this tool in the open-source community will require us to support Mozilla (if for no other reason). We’ve reviewed some approaches for Mozilla integration and have concluded that it won’t be easy. Nonetheless, we believe that we’ll have the best chances if we stick to language that can easily extended with C++ (which is what Mozilla is written in). This means no Groovy. Sorry.

We believe that both Ruby and Python could serve as suitable platforms for this project. We chose Ruby because of personal preference. After we have developed a stable and acceptable implementation in Ruby, we will be encouraging a port to Python.

Learn More and Get Involved

Questions and suggestions are welcome on the wtr-general mailing list, where we also discuss new developments and announce new releases. And talk about other stuff.

New releases will be posted to the WTR project on rubyforge.

We use a wiki to track stories and progress. This is a good place to look for more technical information about our project.

Please let us know if you would like to contribute.

Posted by bret at 05:40 PM | Comments (0)