September 24, 2007

What’s Your Aha?

I’ve organized various meetings, workshops and user groups over the years and am always interested in learning techniques for making them go better. I recently wrote up the meeting protocol for an Agile group in Salt Lake, and after reflecting on the feedback, I realized that brightest gem among their practices was asking each attendee to share their Aha at the end of the meeting. They require everyone to do this; people aren’t allowed to leave early without sharing their Aha first.

I watch a lot of movies, and my general test of whether a movie is good is whether I’m still thinking about it days later. Aha’s are like that. They are the things you are still thinking about after the meeting is over. It could be something to try, an article to read, an analogy that gives you a new perspective, or a story someone shared and the lesson it carries.

The Salt Lake group said that the sole success criterion for the meetings was whether the conversation was interesting enough to make you want to come back again next month, and the Aha’s are really a test of that.

I used this technique for the first time last week, with a small group of developer-testers that meets monthly. I was surprised at what I heard. Many of the things that were Aha’s for people were not what I would have predicted. That tells me that I don’t have great models for selecting what people will find to be valuable. But it also reminds me of the value of feedback, and now gives me and everyone else in the meeting a new tool for learning what others find valuable.

At AWTA this year, I closed the meeting with set a commitments. People shared commitments with the group about what they were going to do in the months ahead. This is something I do a lot on my own, but one problem with this is that some people feel pressure to overcommit, and then feel guilty that they don’t meet the commitment. This can generate negative energy that actually decreases achievement. Next year, I think I will close the meeting with Aha’s instead.

Indeed, in a couple of weeks, I am co-hosting the Alt.Net open-space conference. To close the conference, I think we should share Aha’s. And I don’t think we should let any one leave early without first sharing an Aha.

Posted by bret at 11:43 PM | Comments (2)

September 19, 2007

IE will not fetch some pages if your tests run too fast

I’ve spent most of the past day or two tracking down an intermittent bug in our automated tests.

At first I thought the bug was with Watir. Then I thought the problem was with how our application was caching pages. In the end my collegues and I realized that the problem was actually with the IE browser. Even if you turn off client-side caching, it will still sometimes cache pages. I suspect that other Watir users are also seeing this bug, so I am providing some details.

Failing Watir Tests

Here are the tests that failed intermittently.

describe "Show case error handler" do
  include ScenarioHelper
  before :all do
    @browser = open_browser

  it 'should present an error when using a url for a non-existant case' do
    @browser.goto @@app_root + "cases/show/123456789"
    @browser.p(:class, 'error').text.should == 'Case 123456789 was not found'
  it 'should present an error when no case id is provided' do
    @browser.goto @@app_root + "cases/show"  	
    @browser.p(:class, 'error').text.should == 'Case was not found'

Specifically, the second test would sometimes fail, saying that we actually got this error message: ‘Case 123456789 was not found’

My first thought was that maybe Watir wasn’t waiting long enough after the second call to goto. We’ve had some intermittent reports from Watir users that they needed to add delays after calls to goto, and in fact we did have a bug in this area with Watir 1.4. So I added some instrumentation to Watir to see if i could find out exactly where it was going wrong. But I couldn’t find any problem with Watir.

Server-Side Caching

My second thought was that the problem was with server-side caching. We recently started working with Rails, so Chris Didyk and I learned more about that. We also rearranged our test harness so that we could monitor the server logs as our tests ran. What we saw was that both of the navigations (“cases/show/123” and “cases/show”) were reaching the server, which was redirecting them to an error page. But sometimes, the server would not see the second request for the error page. And these were the cases where the tests were failing.

These results verified the the problem was not a timing problem with Watir, but we still weren’t sure where the problem was.

Client-Side Caching

Scott Bellware helped us track down the problem to IE. We created toolbar links for the URLs in our test and then clicked them in succession. If we were fast enough, we could reproduce the same failure. The second click would get a redirect, but the page would not be fetched from the server. Using the same technique, we were not able to reproduce the bug with Firefox.

We changed the cache settings of IE, but this had no effect.

Temporary Internet Files

We also deleted the previously saved files from the cache.

We also checked the headers of the pages, but they clearly indicated that the page should never be cached by the client.

Response Header 
Cache-Control private, max-age=0, must-revalidate

Thus we learned that when the inputs came fast enough, IE would display a cached version of the error page, even though the header said that it should never be cached, and we had manually turned off all the client-side cache settings.

It’s an IE bug and the only workaround we could find is to add delays to the tests. Or use Firefox. We reproduced this with IE 7. We haven’t checked IE 6.

Why didn’t we see this before?

I’ve been using Watir for years, but this is the first time I’ve seen this problem. Once it started showing up, it was regularly failing, even if it wasn’t 100%. I’ve seen scattered reports from Watir users that suggest they have seen this before, but why haven’t my collegues and I?

We’ve recently moved our development over from .Net to Ruby on Rails. Beforehand, we’ve been working on .Net applications. I think the reason we haven’t seen this problem before is because our .Net apps have been a lot slower that Rails. Slow enough to keep this IE bug from showing up.

Posted by bret at 12:57 PM | Comments (4)

September 16, 2007

The Salt Lake City Agile Round Table

A couple of years ago, I had the opportunity to attend the Agile Roundtable in Salt Lake City. I really liked the way the meetings were run. The group meets monthly, in the afternoon at a bookstore coffeeshop. It is hosted by Alistair Cockburn and others.

Lately I’ve been thinking about hosting a meeting with a similar format and have recently interviewed Alistair, Zhon Johansen, Kay Johansen and Jeff Patton, all regulars, about their meeting and how they run it. This note summarizes these interviews.

Their meeting is held from 2 pm until 5 pm the first Thursday of each month. Apparently it is never cancelled or rescheduled, although Alistair sometimes arrives late. Each meeting has a facilitator. Zhon facilitates most frequently. Alistair, Jeff and Jonathan House also facilitate.

The meeting starts with people going around the table giving their name and mentioning whether there is a topic that they would like to discuss. Not everyone who attends will have one. When Zhon facilitates, he passes around an index card with his name written at the top and asks the others to add their names. Alistair has a better memory of names and does not pass around a card.

After every one has said their bit, the list of topics is reviewed and every one is asked whether there are any topics that they are particularly eager to see discussed. Based on these responses and their own judgment, the facilitator then selects one of the topics to discuss first. A timebox for each of the topics is also defined. Sometimes they don’t have time for all of the topics suggested.

Zhon says: “You wanted to say something about X, can you set that up for us?” When time is up, he asks the initiator whether their topic was addressed. Then they move to the next topic.

They have a break in the middle. It is “scheduled” for ten or fifteen minutes, but is often allowed to run longer (up to thirty minutes) when everyone is engaged in private conversations. The facilitators consider this break to be a key part of the format, not just as a biology break, but also as a chance for people to chat informally.

At the conclusion of they meeting, they go around the table again and ask for “Aha’s.” This seemed to be Kay’s contribution. Everyone is asked to say, in thirty seconds or less, what was their “Aha” from the day’s discussion—what struck them as notable? During the Aha’s there are no interruptions. If someone has to leave early, they have to give their Aha to the group before they are allowed to leave. Sometimes people will come so late that they only get the Aha’s but this in itself can be worth the trouble of driving out to the meeting.

There are six regulars at the meeting (presumably those I interviewed, plus John Major and Jonathan House). Meeting typically have 10-15 people. If the meetings have over 16 people, at the end of the meeting, Alistair will tell the people that the group works best with fewer people and that they should self-select a smaller group to attend next month.

John Major takes detailed notes at the meetings and posts them to the group mailing list.

Comments are open for any additional notes that attendees want to share, or questions that readers may have.

Update: These notes have been copied over to Alistair’s Wiki, where he and others are annotating and updating them.

Posted by bret at 10:09 PM | Comments (1)

September 06, 2007

Updated Watir Tutorial

I taught a Watir tutorial at the Agile 2007 conference last month. Watir actually grew out of a tutorial that I taught regularly between 2003 and 2005. It was a Ruby-based tutorial that Brian Marick originally designed for teaching testers scripting in Ruby. Brian eventually developed those ideas into his book Everyday Scripting in Ruby. I took over the original class, focussed it on web testing, and we developed Watir as a spin-off of the class. Watir’s intuitive API is largely the result of feedback from early prototypes in these classes.

Anyway, after a two-year hiatus, and after rewriting the course materials from scratch, I taught the Watir tutorial again last month. I have also posted the class workbook and published the entire course materials, including all the software you need. Any one may download these; there is no charge. The tutorial is more self-contained than in the past. If you are interested in learning Watir, you might find this useful. Special thanks to Charley Baker, Steven List and Lisa Crispin for helping with the class.

Posted by bret at 04:59 PM | Comments (1)