November 24, 2007

Harry Potter, Test Manager

Jerry Weinberg describes how he’d write a Harry Potter story:

I suspect my Harry Potter would be a test manager, expected to do magic but discounted by software developers because "he's only a tester." As for Voldemort, I think he's any project manager who can't say "no" or hear what Harry is telling him.

The entire interview is quite interesting.

Posted by bret at 01:18 PM | Comments (0)

November 06, 2007

Thoughts on FireWatir

Over the past week, I've taken a close look at FireWatir, with two questions in mind. First, can we make use of it at Dovetail to execute our tests? We have a Watir-based test suite that has over 50 tests and runs in nearly 5 minutes. We've developed it over the past couple of months for a new product. Many of our developers are developing on Macs, so they have a particular interest in using Firefox for testing. Our app needs to support both IE and Firefox.

Secondly, what is the best way to get FireWatir in sync with Watir 1.5 and how should we maintain it going forward? This is a harder question, for a number of reasons that I will try to explain.

FireWatir is based on Watir 1.4. Angrez Singh and the Firewatir team took a copy of the Watir 1.4 code base and then changed it so that it would work with Firefox instead. A lot of the Watir code was reused. It essential amounts to a fork of Watir 1.4. In the past two years since, the Watir code base has evolved significantly. We've cleaned up a lot of the code (both library and test code), refactored much of the code to remove duplication and added support for new attributes and methods (including multiple-attribute support). Few of these changes have been backported to the Firewatir code base. For me, reading through the Firewatir code during the past week has often felt like a trip back in time.

The more that I've reviewed it, the more that I've realized that maintaining FireWatir as a separate code base is a dead end. Some of the classes are direct copies of the original Watir classes (some of which have since changed), some have minor modifications, others are completely changed. Clearly the right way to support this going forward would be to share the classes that can be the same, and refactor the code to maximize the use of shared classes. Indeed, Watir needs a pluggable architecture that will allow adapters for particular browsers, thus supporting SafariWatir and Selenium-RC integration as well. Figuring out how to do this is not trivial. At the same time, it is consistent with the kind of technical refactoring we've been doing for some time in the Watir 1.5 code base. Incremental refactoring of both the code bases is possible because they both have comprehensive test suites that we can use to ensure that the necessary architectural changes don't break the software.

The place to start integrating the two code bases is probably with with tests themselves. Very few changes should be needed to allow the tests to run against either Watir::IE or FireWatir. After reviewing the code, it is pretty clear that running the Watir tests against Firewatir would result in dozens of failures, mostly indications of gaps between where Watir and FireWatir stand today. But synchronizing the test suite and providing a means to mark individual tests as specific to IE or Firefox is really the only way to ensure that the two versions stay in sync.

Some have asked about the core mechanism used in FireWatir. FireWatir requires a Firefox plugin called the Javascript Shell or JSSH. This plugin provides a console interface to the Javascript running in the browser. Once installed, you can actually telnet into Firefox and execute javascript commands. FireWatir connects to this interface using a socket. The basic mechanism is powerful, because it means that anything that can be done in Javascript can be done with FireWatir. However it does require that the plug in be installed first. JSSH is currently provided for Windows (32 bit), Linux and Mac. It is not currently available for FireFox 3.0 (Gran Paradiso) currently under development. Althought it was apparently development independently, it seems that FireWatir users are currently the only users of JSSH. The JSSH code base is open-source, and indeed part of the Firefox code base, but it is unclear to me whether anyone is committed to maintaining this code going forward. This seems to me to be the greatest long-term risk as regards FireWatir. Also the various builds of JSSH that are available appear to have been compiled by different people and different times and they do not seem to have a common or consistent convention regarding versioning. I would feel a lot better if this had some more organizition behind it. Perhaps this could be sponsored by the Mozilla Foundation.

Going forward with developing a single code base supporting two browsers will require us to have a stricter testing policy that we've had so far. Installing CruiseControl.rb on a Windows virtual server will allow us to automatically run our tests on both IE and Firefox whenever code changes are made. The results would be public, so any one could see them. Having this in place makes it easier for us to coordinate the work of multiple contributors who may be using different development platforms. A hosted Windows virtual server, which is what we would need to run our tests, costs about $50 a month. This makes me think that it may now be time to pass the hat and ask users to help fund Watir development going forward. I'm not sure if I should simply ask people to send me money (say, using a paypal button), or whether we should go through the formality of setting up a non-profit or selling memberships in a society. I'd appreciate your thoughts on this. Jason Darling suggests T-shirts sales.

In working with FireWatir this week, I spent a lot of time struggling with a speed issue. My tests would run very slowly, especially when typing. They seemed slower than what a manual tester would do. Tests would run fine for a bit, and then would suddenly slow down, making me think that some kind of resource had been exhausted. Although it showed up slightly differently in different circumstances, I reproduced the problem on three different machines, with both XP and Vista, and both with the Dovetail test suite and the FireWatir unit tests. I have also spoken with both Paul Rogers and Jim Matthews, both of who have made significant use of FireWatir and have not seen this problem. Indeed, Jim says that FireWatir is faster than Watir for him. I eventually found a workaround for my speed problem, but it remains an issue of concern for me. I don't know whether the problem lies in FireWatir itself or in the JSSH plugin. The next step would be to add a debugging/logging layer to FireWatir to help debug these and other issues.

In running the Dovetail tests, I ran into five specific errors, blocking about 90% of the tests. Three were obviously caused by missing features new to Watir 1.5. I suspect the other two were also in this category, but did not have time to track them down far enough. In investigating these issues, I noticed that the attach command was missing. I use this all the time to debug tests. Without it, my ability to investigate the source of errors was limited. This feature was included in Watir 1.4; I'm not sure why it isn't in FireWatir. The missing features I looked into could be added to FireWatir trivially. Merging the code bases would accomplish this as a side effect.

Looking forward, I think it will be a long term project to integrate Watir and FireWatir. We don't have many contributors, all of us mostly work on Watir on our own time. It took us two years to go from 1.4 to 1.5 and I think it will be a similar timeframe to see the complete integration of Watir and FireWatir.

Posted by bret at 01:49 PM | Comments (3)