August 17, 2005

Watir Update and Plans

Over the past several months, i’ve been spending most of my free time on Watir. It’s becoming more and more popular. I taught a Watir tutorial at the Agile conference last month in Denver. The class had over-filled (the conference itself was sold-out) and my original co-instructor had cancelled. I was really worried. The tutorial is a hands-on class where people load Watir on their laptops and learn how how to create tests. It’s a lot of hands-on work, and i normally won’t agree to teach a class of more than 12 people myself. This time, 50 people were signed up for the class. But when i got to the conference i was surprised to find Watir fans left and right. Several volunteered to help with the tutorial, and we actually ended up with more help than we needed.

Watir is attracting a lot of people to Ruby; some have said that it is the second biggest attraction encouraging people to learn Ruby after Rails. And many of these people need help learning Ruby. Most of the books on Ruby assume you already know another object-oriented programming language, such as Java, but many Watir users are new to object-oriented programming, don’t understand how to use Ruby’s modules and libraries, and are often confused by Ruby’s Test-Unit framework. Nathaniel Talbott did an excellent job porting the standard xUnit framework to Ruby; the problem is just that xUnit in general presumes a greater understanding of object-oriented programming than most of our users have. In fact, understanding Test-Unit has pushed me to increase my understanding of many object-oriented design patterns.

I don’t know if any published books that present Ruby for beginners; they all assume you already know another object-oriented language. There are a couple of online books that do address beginners, but they don’t fully cover all the issues that our users are running into. Brian Marick is writing a book for testers that should address these issues, but we could really use even more books that present Ruby from the ground up.

One of our goals with Watir has been to make it easy to use by new users who are unfamiliar with Ruby. We’ve found that people find basic Ruby to be very intuitive. But it’s been equally important for us for Watir to gain the respect of serious programmers and work well with other Ruby libraries and the Ruby community as a whole. This has lead to some recent announcements regarding Watir add-ons that other programmers are developing.

Scott Hanselman recently announced a prototyped recorder for Watir called WatirMaker. He’s released the C# source for the initial prototype and is now developing it further in Ruby with Michael Kelly (from Portland; we now have two Michael Kelly’s working on Watir—the other lives in Indianapolis). I’ve actually had mixed feelings about WatirMaker. It’s great to have sharp programmers using Watir as a platform. But i’ve been skeptical about the utility of recorders for a long time. Watir users have often asked for a recorder, but i’ve generally discouraged people from building them. Why?

I actually once thought recorders were a good idea. I worked at Segue before they created their first recorder. There was a lot of debate at the company about it at the time. The developers thought it was a bad idea and the salesman really wanted it. I was in the middle. I thought it would be handy, but didn’t have a strong need for it. But once the recorder came out, we saw a whole new class of users try to use our test tool that hadn’t used it before: non-technical people who couldn’t understand why their scripts wouldn’t work. Recorders made automated testing technology available to non-technical people who weren’t inclined to really learn what they needed to know to be successful. In a way, recorders have only recapitulated the woes of wizard-driven programming

Over the years, i’ve also noticed that recorders are fussy things. They don’t always capture events correctly, and although test tools often have to be customized to handle particular controls (“custom” controls), the recorders invariably can’t also be made to record these controls correctly. In the end, these means that recorders only work about 80% of the time. This is good if you are only using them as an assist, but it’s doom if you were counting on recorder to create the scripts all by itself. A test that has 20% of its steps missing or incorrect will fail every time.

So recorders in general breed dependency and tend to break. But there is also something to be said for learning from experience. For years, i taught lecture-based classes on automated testing—how the tools work, how to use them effectively—but now i teach the Watir-based class and in only a couple hours people learn for themselves why they need name or ID tags on the controls they are testing to avoid maintenance problems. I’d been telling people this for years, but they really need to be able to learn these kinds of things for themselves, in their own terms, in a direct way that makes sense for them. Watir was originally intended as a training tool to help people learn principles of automated testing. It was unanticipated that it would become so popular in its own right.

So we will see what we learn about test script recording. I had already presumed that C# would be a better language for a recorder than Ruby because of limits in Ruby’s support for COM events. But they report good progress with Ruby—that’s a surprise for me. I also suspect that recorders will need to be tuned for the apps they test. They will need to know whether to record identifiers using names or ID’s or some other attributes. It will be interesting to see what happens here as well. Nothing teaches better than experience. And if it doesn’t work, it won’t be the first time sharp programmers crash against the shoals of test script recording.

What i’d really like to see is simply a tool that would show you the Watir syntax for identifying a control or page element you point to. In our training, we use QualityForge’s SpySmith tool. This reveals the key information, but in HTML, rather than in Watir’s syntax.

Another exciting project that experienced programmers have built using Watir as a platform is WET. This is a open-source testing framework that Raghu Venkataramana and Satti Shankar at Qantom have built atop of Watir. WET provides a lot of the features that Watir users have been asking for: a comprehensive logging facility, better support for modal dialogs, and the ability to read test data from Excel.

In many ways the framework they’ve built illustrates the concept of homebrew test automation that i’ve advocated. It’s not really so much a matter of using open-source software as it is having the ability to really take control over your test harness and make it work the way you want it to. They’ve put together a complete framework that isn’t probably what i would create—but it doesn’t matter. They’ve been able to use and extend Watir to make the framework they want.

To some degree this is simply a matter of seeing Watir as a library rather than a tool. Although we want to grow Watir and make it into something that is reasonably able to be the basis of a testing framework, we also want it to be flexible enough to allow people to adapt it to their own needs.

We’ve designed Watir so that it it is reasonably easy for non-technical people to learn and use, but at the same time we also want to make sure that it is well structured enough that more savvy programmers can use Watir as a component in a larger frameworks. WET is an example, but looking at how they did it has made me realize how much more work we really need to do in order to make it easy to use Watir as a component. They had to tie into a lot of “internal” interfaces that we didn’t realize others would want to use. Many of these are misnamed and inconsistent and seeing how others are using it has embarassed me enough that i’ve already renamed many of them. We’ll be working with with WET team to define better interfaces for logging and other areas.

In fact, these kinds of extension interfaces are needed not only for WET, but also for various users who need to customize the behavior of particular objects. For example, many of our users have redefined our method for setting the value of text boxes. Our standard method is fairly bulletproof in terms of making sure that any possible JavaScript method that could be tied to the text box is allowed to be triggered. But this takes a performance hit that seems to be particularly large with web pages that have lots of fields. I’m starting to realize this is an area where instead of having one method that we expect to work for everybody, we instead are going to have to make it easy for people to tune Watir for their particular app.

The WET team actually had to redefine Watir classes in several places. Note that the ability to do this is unique to Ruby. It’s a big area of difference from Python and other languages. If Watir and WET were in Python, they would have had to negotiate changes to Watir before they could could written something like WET. I note this because many people often ask why Ruby over Python. This is a big advantage to Ruby. At the same time, we’d like to make it so this doesn’t have to happen, and will try to rework Watir so that it can be extended satisfactorily using more conventional methods.

Did i say how much i like WET’s approach for handling modal dialogs? This has been an ongoing issue with Watir and we’ve tried several approaches. WET’s is clearly superior and will see if we can’t use their dialog code in Watir directly. It still doesn’t fully and cleanly solve the thread-locking issue that we’ve been struggling with, but it is clearly the best way forward.

Enough talk, time to code. If you want to know more about upcoming changes—or would like to volunteer to help out—please join us on the wtr-general list where we’ll be hashing out the details.

Posted by bret at 11:09 AM | Comments (7)