January 31, 2005

The Best Testing Curriculum in the World

Twenty years ago, Cem Kaner was managing testers and was completely dissatisfied with the texts and training on testing then available. He developed his own training material and eventually published it as Testing Computer Software, the best-selling testing book of all time. That book was last revised in 1993, but Cem has been continuing to update his teaching materials year after year, in collaboration with James Bach.

Their curriculum is now available online, as open-source. It includes detailed slide decks, examples, review questions and other supporting material. Cem has also begun recording several of the lectures on video, which you can watch on your own computer.

Thinking about getting certified in testing? You'll learn a lot more by taking a look at this material.

Posted by bret at 09:36 PM | Comments (2)

January 12, 2005

AWTA6 Report: Open-Source Web Test Tools

Over the weekend, i hosted the sixth annual meeting of the Austin Workshop on Test Automation. Elisabeth Hendrickson co-hosted the event and Pat McGee and Scott Barber helped with the organization. We had 10 participants this year. Our theme was open-source web testing tools. And despite my efforts to encourage participants to consider alternatives, the particpants became quite enamored with the Watir testing tool. It was compelling enough to convince a few to learn Ruby so they can use it. Watir may actually become the killer app that makes Ruby the de facto language for testing.

We used a new format for the three-day conference. In the mornings, we had presentations. The afternoons were open and participants were encouraged to pair up with others to work on projects related to the material presented. At the end of the day, people presented their results. In many cases, these were as interesting as the formal presentations. I’ll describe the presentations and then the projects they inspired.

Chris McMahon presented WWW-Mechanize and HTTP-Recorder, Perl libraries for recording and executing HTTP-based test scripts.

I presented a basic overview of Watir, a Ruby library for automating the IE web browser. I’ve led the Watir development effort with Paul Rogers (well actually Paul wrote most of it). About half the participants had used this library before arriving. This was followed up with two presentations describing what people had done with it. Watir currently uses IE’s COM interface to access it. This limits it to only being able to test IE on Windows.

Karlin Fox presented two libraries that his employer, Atomic Object will soon be making available through open-source. One was a simplified interface to Watir that they have found easier to use by non-technical people. The other library assisted with the creation of a domain-specific language in Ruby (and could work with any Ruby testing library). Karlin is lead developer for these projects. Putting the two together resulted in a really great way to write tests. We are keenly looking forward to the release of these libraries. The first will probably be released as part of the Watir library, likely to be named watir/simple. The second will be separate and is tentatively named test/system.

Michael Kelly presented a test that used Watir to collect performance information, along with a spreadsheet with macros that provided a slick way of presenting performance data with both a high-level overview and the ability to drill down to specifics. These are the basis for an article he wrote that will soon be published in InformIT.

Ryan Wilcox presented Selenium using a demo of it testing PetSoar. The scripts ran on his Mac using the Firefox browser. Selenium uses an innovative method for driving a browser: it uses a javascript engine that actually runs inside the browser itself. This allows it execute on any platform, and support any browser that supports javascript (with some customization for the ways in which different browsers support javascript differently). Selenium currently supports testing with IE, Mozilla and Firefox on Windows, Linux and Mac. To get this benefit, however, Selenium must be installed with the web server — an obstacle for some people. Ryan’s demo showed the Fit-style interface to Selenium in which test scripts take the form of action words in a HTML table. This interface is fully supported. I demonstrated a prototype of the driver that delivers the Selenium functionality to Ruby. Ryan and i are both lead contributors to Selenium, a project sponsored by ThoughtWorks, our employer.

Pat McGee described a tool that executes random repeated regression testing and how it has been used at one company. This was an update to his report at AWTA5. He is doing this work as part of his dissertation at the Center for Software Testing Education and Research at Florida Tech and the tool will be published there as open source soon.

Martin Taylor described a testing system they use throughout the Texas Instruments calculator division. It consists of an action word framework in Python that is driven from Mercury Test Director. He’s been tasked by his management to make parts of this system open-source and we provided him with some suggestions.

There were lots of different activities that took place during project time. Following is a description of some of those.

Elisabeth and Mike wrote unit tests for Watir that tested the tool in isolation from the IE browser itself, using a mock IE object. They checked these tests into the Watir repository (unittests/ie_test.rb). Several others also spent some time writing tests for Watir. (Who knew so many would be interested in testing a testing tool?)

I worked with Chris to write a script demonstrating how Watir can be used to test concurrently, with multiple browsers testing at the same time. It turned out that the Watir library is thread safe and Ruby’s thread library allowed this script to be written with only about 6 lines of code. It’s also been checked in the the Watir repository (examples/concurrent_search.rb).

Karlin and Martin wrote a SOAP interface to connect Martin’s Python action word framework to the Ruby-based domain language Karlin wrote. Both languages have SOAP libraries that made this pretty simple.

Ryan worked with Chris to update the documentation for Selenium. Scott worked on an article on performance testing, based on discussions with Chris and Pat and others. And Pat worked on a curriculum for a class he’ll be teaching on software maintenance, bouncing ideas off various participants.

Many of the projects stretched into the evenings. The presentation + project format was a big hit, and i think we’ll be doing it again.

It was exciting for me to see the interest in Watir and to realize that it’s reached a maturity level that makes it attractive for testers, and indeed that some are actually using it as a platform for specialized testing frameworks that will also be released as open-source. We’ve also recently seen several developers join our project. I’ll be working to solidify the library and integrate it with Selenium, thus providing the ability to test browsers besides IE with existing Watir-based tests. Stay tuned for more on that.

Posted by bret at 11:23 AM | Comments (0)

January 02, 2005

WATIR Reviewed

Danny Faught published a review (sub req'd) that compares Watir with Samie and has Watir coming out ahead. This is especially noteworthy because Samie is written in Perl and Danny is an avowed Perl affectionado. Watir, of course, is in Ruby. The highlights:
The most noticeable difference betwen the two is that Watir does more of the synchronization automatically, waiting for IE to complete an operation before allowing the script to continue. I had to do this manually with Samie....

Overall, I preferred Watir over Samie for several reasons. Most importantly, I could not get the Samie scripts to run reliably. While sometimes it works fine, other times it would simply hang in the middle. And several times I hit a fatal error....

Samie is poorly documented, while there is a decent user manual for Watir....

I didn't find a way to print out a full DOM representation of a web page with Samie.... So i ended up using Watir's showAllObjects method to help me write the Perl script.

The review appears in Open Testware Reviews, a subscription-based website.

Posted by bret at 11:24 PM | Comments (0)