April 26, 2007

Watir, Nant and Cruise Control

Lately, I’ve been spending a lot of time hooking up Watir tests to Cruise Control, both at Dovetail and before that, at DataCert. At DataCert, we scheduled Cruise to run our Watir tests every night. At Dovetail, Cruise runs them every time our developers update the source code, so they often run several times a day.

Here at Dovetail, we are also running our Watir tests from Nant. This makes it easy for our developers to run Watir tests before committing new code. I was slow to see the value of this at first—why can’t they run them directly instead?—but it really does help our developers incorporate automated testing into their daily workflow. They have a single Nant target that automatically compiles their code and then tests it. They do this before they check in the code.

In my view, making testing automatic and available on demand is key to getting significant ROI. I’ve seen too many automated testing efforts underutilized and thus underfunded because the tests still had to be run by the automation guy. Indeed, the sad fact is that most of the commercial testing tools are still focussed on this model. Every developer who writes code needs to be able to run the tests.

In the process of integrating Watir with Cruise and Nant, I’ve run into a number of issues with getting our tests to run correctly and, just as important, report results accurately and clearly. The purpose of this post is to share some of the things I’ve learned to help you with similar integrations.

We’re using CruiseControl.net version 1.1.0.2527 at both DataCert and Dovetail (we probably need to update, since this is not the most recent). At DataCert we were using Ruby 1.8.2; at Dovetail, 1.8.5. We’re using Nant v. 0.85 at Dovetail and at both places we’ve been using recent versions of Watir 1.5.

A key issue has been making sure that errors propigate from the Watir tests to the tools that are “wrapping” them. When this isn’t working right, your Watir tests can run and fail, but Nant or Cruise will incorrectly report “Success”. This is very bad. Fundamentally, error status is propigated through return codes, and we’ve run into some bugs in Ruby where it has not been propigating return codes correctly.

Let’s say you are running a watir/ruby script called “my-tests.rb” from a DOS command line. You can easily tell from the output whether it passes or not. Presumably you are using Test-Unit or maybe Rspec as the test-runner itself (we are currently using Test-Unit). But how can you tell whether the correct return code is being returned by Ruby? After running the test, use this command: “echo errorlevel”. If your test fails, this should not be zero. If it is, the error will not be propigated correctly.

We are actually using Rake to run our Watir tests, which actually adds yet another tool to the chain. Normally, when you execute rake test, this starts up a batch file (rake.bat) that then starts Ruby (since Rake is written in Ruby). We found that this batch file found in Ruby 1.8.5 for Windows does not propigate return codes. To work around this problem, we found that we could start up rake with this command instead:

ruby -x c:\ruby\bin\rake.bat test

This is, in fact, the call that the batch file generates. By running it directly, we correctly get actual error code. (To put these commands in Nant, you have to break them into pieces and wrap each piece in xml. It’s a mechanical and ugly process, which is why I’m not showing the xml here.)

I did take a look at Ruby 1.8.6, and found that this problem with the batch files has been fixed. Unfortunately, there is a separate problem. In Ruby 1.8.6, error codes set by the Ruby command exit are not returned by the Ruby executable when the exit command is contained within an at_exit block. And it so happens that this is exactly what test-unit does. So we are not using Ruby 1.8.6.

The main reason that we are using Rake is because it provided the simplest way of incorporating CI Reporter. This is a gem that automatically generates the XML reports that Cruise needs to include test results in its build report. It makes really nice reports.

So far, we’ve discussed propigating errors from the bottom of the tool chain (Watir) to the top (Cruise). But you also need to propigate options and settings from the top on down. Specifically, we wanted Nant to run Watir scripts invisibly by default. There is a ”-b” command-line option in Watir to support this, but version 1.2.2 of the CI Reporter was preventing this option from propigating to the tests. We provided a patch for this problem, and it has been included in version 1.2.4. We are also seeing failure messages being truncated in the build report, but have submitted a patch for this problem and expect it to be included in a future version of CI Reporter.

Another problem that we ran into at DataCert and which others have run into as well is getting Watir scripts to work under Cruise when they use the IE.attach method. This method normally works fine, but will fail when the tests are running under Cruise. At root, the problem stems from the fact that this method tries to cycle through the windows in the desktop object, but services don’t have access to the desktop. Sometimes you can rewrite your tests so they don’t use the attach method. At DataCert this was not an option: we made significant use of modal dialogs, and under the covers, Watir uses attach to gain access to these. The simplest solution is simply not to run Cruise as a service, but instead just log on to the machine and start it up under a user account. Of course you will have to remain logged in.

I’ve seen several other, more elaborate, suggestions for how to solve this problem, but in the end they all amounted to getting a non-service process to kick off the tests.

The final comment I have on the topic of Cruise regards editing the ccnet.config xml file that Cruise uses. I found it easy to make mistakes with this file and ultimately realized that I needed to install cruise itself on my test machine. If I need to make changes to this config file, I change them first in test, test them out, and then migrate them to the official cruise server that everyone’s CCTray is watching.

Posted by bret at 04:15 PM | Comments (1)

April 12, 2007

Rearranging the Furniture

When I first visited the Dovetail office, I told them that they had their desks arranged wrong. They were scattered around a large room, mostly facing the outside of the room. What I had seen was that Agile teams generally preferred to be facing each other, rather than facing the walls. In fact, I’d twice seen Agile teams start with a facing-out arrangement only to eventually change to a facing-in one. Once was a couple years ago when I visited ThoughtWork’s Bangalore office, where they replaced fancy partitions with large farm-kitchen tables. I was there when the work crews came to replace the furniture, and it was fascinating to see the staff so excited that their fancy, expensive custom-designed partitions were being replaced by droll tables.

More recently, the most-Agile team at DataCert’s Houston office rearranged their office so that they were facing each other rather than the walls.

On my fourth day at Dovetail, I got my team to rearrange our desks into a central cluster, facing each other. This has increased communication, and also chatter. I’ve also encouraged my collegues to feel free to use headphones when the chatter is getting too much for them. We may also have to learn how to minimize some of the purely social conversation.

Alistair Cockburn gives a sophisticated explanation for why this is better. But I think the reason may be more simple. I can explain with a story.

We’ve recently integrated our Watir-based smoke tests into the development build. This means that the default Nant target will compile the software, run the unit tests, and also run the browser-based smoke test. Our developers do this many times a day. And they have been complaining about the browser that pops onto their desktop when the smoke test runs. Several days ago I started to fix this. You can pass a ”-b” switch to your Watir/Ruby script to force the test to run invisibly (in the background). I set up Nant to pass this argument to Rake which would then pass it to the test script, but I found that a flaw in the ci_reporter package that we are using to publish our test results back to CruiseControl.net is blocking the switch from getting to the script. I figured this much out the other day, but put the project on the back burner when I realized that I was going to have patch ci_reporter. So our Nant target still pops up a browser window during the build. When this happens, since I am now facing our developers, I can see their annoyance. Clearly they had context switched to email or IM or something else while they were running the build, and then in the middle—pop goes the browser, interrupting whatever they were doing. And I can see the annoyance on their faces. It’s not horrible, but it is one of those petty annoyances that just build up. And I can see that. They don’t have to bring it up at a stand-up meeting, or go get my attention and explain this to me and dramatize the effect for me. I can just see it on their faces. And they can just let me know it bugs them without dramatization.

I’ll get to it first thing in the morning.

Of course, I also get the benefits of this kind of immediacy. When stuff they’ve done bugs me, or when they are in a position to help me with something that I’m stuck with, they help. If we were in separate offices, I’d probably spend 15-20 minutes on a problem before asking for help. If we were on opposite sides of the same room, I’d probably work solo for 5-10 minutes. But facing each other, it can often be only a couple minutes before I try to get help.

With us facing each other, we share our annoyances. The petty annoyances of computers are vast. I think a major reason for the increased productivity of agile teams is simply that we tackle these annoyances as a team.

Posted by bret at 12:33 AM | Comments (1)

April 10, 2007

Back in Austin

I've just started a new job at Dovetail Software. I'm very excited about it. It's a hard-core agile shop with a small team that uses Watir and other open-source testing tools. The company even encourages blogging. Everyone on the development team is also a blogger: Gary, Jason, Kevin, Melissa, and Scott. We also have several openings for developers, so let us know if you want to join us. After seven years of frequent travel, I am now working for a company in Austin, my home town. I'm very happy to not only be spending more time with my family, but also to have a chance to reconnect with the Austin testing and development communities.
Posted by bret at 12:18 AM | Comments (0)