Testing Hotlist Update

Agile is All About Feedback

Agile methods allow your team to get feedback regarding the software you are building. That’s the point. The feedback works on several levels. Pair programming gives developers instant feedback on their code. Stories represent units of work where testers and analysts can give feedback to developers. Iteration releases facilitate feedback from outside the team. Most agile practices are valuable because they create feedback loops that allow teams to adapt.

A lot of teams adopt Agile with a grab-bag approach without quite realizing the point of the practices. They pair-program without discussion or changing drivers. They send code to QA that the testers can’t test because the story boundaries are arbitrary; they can’t tell whether they found a bug or just the end of the story. Iterations become schedule milestones rather than real opportunities to improve alignment and adjust objectives.

The reason Agile teams can do with less planning is because feedback allows you to make sure that you are on course. If you don’t have meaningful feedback then you’re not agile. You’re just in a new form of chaos.

On my last project, we defined our stories so that they made sense to everyone on the team. Our analysts, testers and developers could all understand and review individual stories. But we found that we had to create a larger grouping, which we called called features, to facilitate meaningful review from outside our team. We made sure all the stories in a feature were complete before soliciting feedback from outside the team.

Being able to give and receive meaningful feedback is often a challenge for people. Yet it is crucial to success with Agile.

Agile teams get into terrible binds when executives or clients hand them a list of requirements at the start, tell them to use Agile (because its faster), and then don’t want to participate in the feedback process.

Agile isn’t faster all by itself. Agile is only a benefit in a world that acknowledges the value of adapting. And that adaptability needs to go all the way to whoever is funding the project. It is not enough for the team to be Agile. The sponsors need to be Agile too. Are all of the requirements really required? Do we know exactly what the software needs to look like from the start?

Agile is faster because feedback allows you to find and focus on the most valuable features. If you are certain you know what needs to be built, don’t use Agile. If you don’t have time to gather and act on feedback from customers, then don’t use Agile. If you are sure that everyone understands exactly what needs to be done from the start, then don’t use Agile.

Agile practices build a technical and organizational infrastructure to facilitate getting and acting on feedback. If you aren’t going to adapt to feedback, then this infrastracture is waste that will only slow you down.

Previous: Lessons Learned in Agile Development
Next: Agile Mostly Means Something Between Scrum and XP

FireWatir 1.2.1 is Released

We have a released a new version of FireWatir. This includes some fixes related to Firefox 3. It also marks the merger of the FireWatir and Watir projects. Unlike Watir, FireWatir works on Windows, Mac and Linux.

To install:

1. "gem install firewatir"
2. Install the plugin for Firefox.

Lessons Learned in Agile Development

This begins a series of short essays on Agile development. I am writing these for several reasons.

We are using many Agile methods in the development of Watir. We are using such methods as continuous integration and emergent architecture in our development. Many Watir users are curious about our development, but unfamiliar with some of our methods. I want them to understand.

Many Watir users and other testers are trying to understand Agile. Many are on teams that have recently begun using Agile methods, and are wondering whether it works. It does, but there are also lots of ways to fail. Testers, always quick to find faults, have a hard time with much of the Agile literature, written as it is with developers or managers in mind. I’m always thinking about the testers.

WatirCraft is also using Agile methods. Explaining Agile gives me a chance to explain to the current and future WatirCraft team some of the rationale behind our methods.

I began studying Agile testing six years ago and for the past four years have worked exclusively with talented, high-performing Agile teams. In the past few years, I’ve also been coaching my collegues on Agile methods in areas beyond testing. In that time, I’ve compiled quite a pile of notes on the topic, including key ideas and common misunderstandings. I’m happy to have this chance to share them.

Next: Agile is all about feedback.

Private Blogs

Yesterday, I spent yet another evening researching private blogging options. Several months ago, I started a private blog to share with the people who are helping us launch WatirCraft. I've had these before at previous companies. I think I was the first to start an internal blog at ThoughtWorks. At Dovetail, many people already had them when i arrived. Some used it as a training ground to get more comfortable with the format before blogging publicly. I liked it as a way to publish essays in a way that encouraged a public discussion. Something similar can be done with email, but there are a few drawbacks.

Invariably, email discussions leave someone important out, and can be difficult to add people to the discussion later. Internal blogs don't have this problem: any one in the company who finds out about the discussion can join in, sometimes even months later. I also find myself reluctant to send out emails because they can be so interrupting. With a blog, it seems easier to write something that is important but not urgent, and you'll get more thoughtful replies because people know that they replies will be archived.

I remember pushing the idea that QA people should be called “Testers” (and not “QA”) on my ThoughtWorks blog. We had pretty much come to agree on this point in the Context-Driven community before I joined ThoughtWorks, but I came to see that ThoughtWorks, like the industry at large, still labelled the role “QA”. Because I was then in a position of some standing (Director of Testing Practice), making an issue of this on my public blog would have been a positional move, likely polarizing both me and those who disagreed. By publishing my position internally, I essentially acknowledged that I wanted feedback from my collegues on this topic. I remember Rebecca Parsons suggested “QA/Tester,” which, though awkward, didn't run into the various objections I had laid out against QA, and since then I've been more comfortable with the “QA” term, even though I still usually prefer “Testers”.

At both ThoughtWorks and Dovetail, we made our blogs private simply by publishing regular blogs on our internal networks. You either had to be in our offices or connect over the VPN to see them. Web-based feedreaders (like Google Reader) wouldn't work, but you could use any client-based feedreader (Outlook has one) when you were connected to the company network.

I looked closely at Blogger and WordPress and also more generally across the field and was not able to find any ready-to-use blog engine that supported private blogs and provided a feed. It is possible for feeds to be password protected. For example, we use Highrise and it provides a password-protected feed containing recent changes. But i could not find such a thing for any blogs.

Apparently several blog engines added support for private blogging a couple of years ago, but left public feeds on them. These were crawled by search engines and people were finding their private posts on Google! The quick fix was to remove the feeds, and apparently the issue hasn't been addressed since. One option that might work for some would be to have a blog that isn't strictly private, but is protected from search engines, so you'd have to know the URL to read it. This option is offered by WordPress.

At WatirCraft, we have not yet set up a VPN, so that is not an option for us. For now, it seems the best way to run an internal blog is to make one private and then to set up an accompanying mailing list that you post to when there are updates on the blog.

Setting up Watir with Cruise Control

Last week I set up Watir so that its unit tests are run automatically whenever any changes are committed (“checked in”) to the development repository. This is done with Cruise Control, a tool which watches repositories, runs tests and posts results. The Watir tests are run at watirbuild.com and you can check the latest test results yourself. Indeed, the page has a button called “build now.” Click it and the tests will be run immediately on the server.

The general term for this process is continuous integration—a common practice with Agile teams. I’ll write more about why this practice is important for the Watir project another day. Today’s post is about what I did to set this up.

First, I got a server, a virtual private server (VPS) running Windows from VPSLAND. I would have preferred a machine running XP, since most Watir users use that, but the closest I could get was 32-bit Windows 2003 Server. Cost: $19 a month, courtesy of WatirCraft. This VPS is named watirbuild.com.

Next I installed the Google Toolbar, because I can’t find anything without it. I also had to loosen up security on IE so that I could use it to download the software I needed. (The server VPSland provided came with security really locked down.)

I then installed Ruby 1.8.6-26. The next step was to do a “gem update --system” to update the gem installer to the latest version (which Watir needs), but this command failed with this error.

c:/ruby/1.8/yaml.rb:133: [BUG] Segmentation fault

A little research showed that this was a common problem on RAM-constrained VPS’s. Apparently the pre-update gem installer is a memory hog and the VPS’s don’t have any swap space. The common workaround has been manually download the gems before installing them. (Only the remote gem installer runs into memory problems.) But I used a different approach.

I wanted our test environment to match, as much as possible, the environments that other people are using for testing, so I really wanted to have the latest gem installer. I also knew that it had better memory handling. So what I did was install Ruby on one of my own systems with more RAM, and then did a “gem update --system” on it. Then I zipped up the resulting “ruby” directory (using 7zip), uploaded it to watirbuild.com and then unzipped it there over the original installation. I was then able to do remote gem installs without any further trouble.

Because Watir now depends on several other gems. I did a “gem install watir” and then uninstalled the Watir gem itself. This left all the dependent gems installed.

Watir uses Subversion (SVN) as its development repository, so we needed a client to access that code. I installed the command line client.

The next step was to install the continuous integration tool itself. I chose CruiseControl.rb because it is Ruby-based and easy to configure. Pre-deployment testing had shown, however, that the latest release (1.3.0) did not work on Windows (After starting the server, it would get repeated "Access is denied." errors from SVN.) Instead I needed to use the latest from its development repository. They recently migrated that project from Subversion to Git, so that meant I needed to install a Git client. Once it was installed, installing Cruise was simply a matter of doing

git clone git://rubyforge.org/cruisecontrolrb.git

With Cruise installed, I added the Watir project:

cruise add watir https://svn.openqa.org/svn/watir/trunk/watir

I also needed to add a “cruise” target to Watir’s rakefile. Which I did with the following code:

desc 'Run unit tests'
task :test do
  load 'unittests/core_tests.rb'
end

task :cruise => :test

(This code is now part of the Watir project, so you wouldn't need to provide this code if you were running Watir. But you will need something similar if you are setting up your own project with CruiseControl.rb.)

Then I started Cruise using “cruise start -p 80”, and this immediately kicked off Watir’s unit tests. Some failed on first run, and I had to modify a couple of IE’s security settings to fix this.

To test everything, I checked in a minor documentation change to Watir and verified that the unit tests were automatically kicked off.

WatirCraft is Betting on Test Automation

Pete Dignan and I are founding a company called WatirCraft. WatirCraft is a for-profit company that is supporting and maintaining Watir and providing additional products and services based on Watir. We are betting that we can build a business around making testers successful with automated testing.

Our top priorities right now are providing ongoing support for Watir as well as improving Watir’s ability to operate with more browsers and platforms. We recently released an updated version of Watir that included a number of patches contributed by the community, as well as a performance improvement.

Watir’s number one weakness today is its focus on Windows and IE. We are working closely with Angrez Singh and the FireWatir team to bring our code bases together, rejoin the projects and improve the compatability between our implentations. Testers have every right to expect tests they write for one browser to able to be executed with another— without the use of browser-specific logic.

FireWatir supports Firefox on Windows, Linux and Mac. We are also looking further. We have recently had serious inquiries from test architects working on the development teams of two other important browsers, seeking advice on how to make Watir work with their browsers. And we also have plans to integrate Watir with Selenium-RC.

Many people today think of Watir as something that works with Windows and Internet Explorer. We seek to change that. We think Watir’s defining characteristic is its intuitive API for operating a browser. This API is sufficiently appealing that it has been translated to C# (Watin) and Java (Watij). And of course there are a couple of implementations of this API in Ruby (Watir, FireWatir, SafariWatir, Celerity). But we need to keep these Ruby implementations in sync, which is why we are developing a browser-independent test suite to validate Watir compatability between these code bases.

Making Watir more portable and more supportable will mean changes, some of which will be disruptive. We will be making some features more robust, and deprecating others. We plan to continue to support Watir as a community project. In fact, the Watir community is the key asset that we are building our business on. Every day, Watir users are helping each other, sharing their work and their visions, and making it easier for the next person who comes along. Pete, new to the Watir community, is amazed at the patience, good faith and dedication he has seen.

By founding this business, Pete and I are making a bet on Watir— not just the tool as it is today, but also the vision that inspires it. It is a vision that realizes that automation is about code and success requires understanding code, and making code understandable.

We are also betting that people want and will be willing to pay for a framework that is easier to use.

It has been great to learn about all the frameworks that have been built for Watir. Watir itself was always conceived as a library that could be used with any framework. Originally we mostly used Test::Unit, which was convenient and well-supported. But with time, other unit testing frameworks (such as Rspec and Shoulda) have arisen and it is just as easy to use Watir with them. It has been great to learn from all the frameworks that Watir users have created and shared.

Watir was concieved as much as an embodied critique of the then-prevailing testing tools as it was as a foundation for successful automation. It was thus designed around a standard scripting language, not a vendorscript. It was designed to be easy to write, rather than easy to record. And it was designed to be a library and not a framework. Many of the commercial tools have been burdened by tight bindings to leaden frameworks.

WatirCraft is developing a framework that will allow testers who aren’t wizards (and who don’t have wizards around to help) to nonetheless hit the ground running with Watir. Our framework won’t be for everybody and we will continue to encourage Watir’s many users to use whatever framework they wish, whether it be open-source, home-brewed or ours. Our goal, really, is to make Watir-based testing accessible to more people.

We are also betting on Ruby. We still sometimes hear regrets from testers that Watir isn’t written in Python. I don’t regret it one bit. I am thankful that Brian Marick pulled me on to the Ruby train with him years ago. I believe that we can build a better framework with less effort in Ruby than in Python. This is because Ruby has a set of features called meta-programming.

The advantages of meta-programming have been hard to explain to testers. “Meta-programming” is actually a new label for a set of advanced language features that Ruby has had for a long time. But these features are hard to explain and understand and most testers won’t be using them. Nonetheless, Ruby’s meta-programming features make it easier to build consise, intuitive frameworks. Indeed, this is a big reason behind the success of Rails.

We are also betting that the growing use of Agile Development practices will continue. Testers on agile teams are being hammered by shorter iterations. Regression testing is more and more necessary and yet there is less and less time to do it. At the same time, Agile Development is getting testers and developers to work together more closely—exactly what needs to happen if successful test automation is going to happen. We will continue to support automated testing that works not just for testers but for the whole team, and that integrates with agile practices such as developer testing, continuous integration and frequent commits. Personally, I’ve been working exclusively with Agile teams over the past four years. Only when you run tests daily (or more frequently) will you really do what is necessary to make automated testing work.

Today, most of the innovation in automated testing today is coming from agile developers. I find it quite exciting and hard to keep up with, there is so much of it. Yet many of these tools are written for developers, and testers have trouble using them. Indeed developers in some companies are already taking over responsibility for automating testing. Will this be the trend? Will there be a role for testers in the future of automated testing? Will this role be defined by developers?

We think testers do have a significant role in the future of automated testing. By working closely with both testers and developers we can make Watir an even better tool and sell additional tools and support that will make testers successful in an increasingly Agile world. We’re betting on it.

Watir 1.5.6 released with "Zippy" speed

Last week we released a new version of Watir. We've mostly been accumulating bug fixes and minor enhancements. However there is one new feature that may be worth the upgrade. We've added a speed improvement to Watir, which we call "Zippy" speed.

Installation

Watir now has a two-step install process (assuming you've already installed Ruby). Type this at the command prompt:

gem update --system
gem install watir

This will update your version of the Rubygems installer and then install the latest version of Watir. Some people have had install problems with 1.5.4 and 1.5.5. Those problems are now fixed.

We are also now recommending that people use Ruby 1.8.6-26 final with Watir. A Ruby bug affecting many Watir users in earlier versions of 1.8.6 is now fixed. If you are upgrading your version of Ruby, you basically have to do a fresh install. There is no upgrade option. Sorry.

Zippy Speed

When you first use Watir, it is rather slow. This is intentional. We made Watir run slow at first because we found that new users wanted to be able to watch their scripts and make sure they were working correctly. So we added some delays to slow Watir down to an auditable speed.

However, once you have confidence in your tests, you probably want to run them faster. This is just one command.

ie = Watir::IE.new
ie.speed = :fast

I know this is a surprise to some Watir users: we haven't done a great job of letting this trick be known. This feature has been in Watir for a long time, certainly as far back as Watir 1.4.

What is new is that we now have an even faster way to run Watir. We call this "zippy" and it works the same way.

ie = Watir::IE.new
ie.speed = :zippy

Normally, when Watir enters text into a text field, it does so letter by letter. Two reasons. One, it looks cooler. Two, there are rare cases where you have to do this to realistically engage Javascript attached to the text field that, say, limits the characters that can be entered. But with tests that had lots of data entry, this could really slow things down.

The new :zippy speed speeds up tests by enter text fields all at once. If you have tests that require character-by-character text entry, you'll need to continue to use :fast.

In the past, many Watir users realized that they could speed up their tests by using value instead of set. Thus, instead of

ie.text_field(:id, 'payment').set '25'

They would use

ie.text_field(:id, 'payment').value = '25'

The new :zippy speed does exactly the same thing, only behind the scenes so you don't have to modify your individual scripts.

Zippy speed was originally included in Watir 1.5.4, but I've been reluctant to brag about it until we got our install problems fixed. It is based on a proof of concept by Jonathan Kohl.

Should You Upgrade?

Other features new since 1.5.3 (released last fall) include support for Chinese character input and the ability to locate more elements using multiple attributes. However, multiple attribute support for text fields, buttons and other input elements has yet to be implemented. We know you are waiting for it.

You can now call the row method from within a table, also frequently requested. There are a number of other fixes. (See the Release Notes for details.)

I know many of you are still using Watir 1.4.1. There is only one reason I know of why you should not upgrade. We have had several reports that Watir 1.5.3 and later is slower with some complex pages. We recently got a reproducible case of this problem and will be looking at fixing it in 1.5.7.

The Next Advance in Testing

Ward Cunningham says:

I suspect there will be a tool at the center of the next advance in testing. The tool will enable the advance, but it won't be the advance. Like with jUnit, the tool will be useful before the advance has happened, but, when the books are written, they will devote far more attention to the practices surrounding the tool than the tool itself.

Next Watir Austin Meeting: Jim Matthews' Link Checker

The next meeting of the Austin Watir Users Group will be this Wednesday, May 21.

Jim Matthews will be speaking about his link checker, a web crawler optimized for testing that he has released as open-source. He needed a script that would crawl a website looking for certain special conditions.  He built a tool using Ruby, Watir, and Mysql that he has been using to test the website at Callaway Golf. Jim will discuss the problems encountered and the evolution to the tool.

I met Jim years ago during a consulting assignment at Tonic, where he had built his own browser tester in VB. I later developed Watir partly inspired by Jim’s work. Later we worked together at DataCert, where we both used Watir to test different products.

We are meeting at Uplogix again. Try to get to the meeting by 6:30 if you can. We’ll have pizza for you. Complete driving directions.

Update: Use the following Subversion command to check out the link-checker:

svn checkout svn://rubyforge.org/var/svn/test-system/link_checker/trunk

What I've Been Using Lately

On my last project we used Watir 1.5.4 (only recently publicly released, but we'd been using it since last fall) and Ruby 1.8.5. There are several problems with Ruby 1.8.6 and therefore I will not use it.

We used Rspec on this last project, and I really liked it. We started with Rspec 0.8, and at that time we looked at the Story Runner, but it clearly wasn't ready for production testing. We later upgraded to Rspec 1.1.1. but did not have time to revisit using the Story Runner. The code seems solid now, and I'm eager to try it out on a real project.

We used Eclipse as our IDE, version 3.3 with the Aptana plugin. There is a new version of Aptana that I plan to install (standalone) when I get a moment. Our developers used IntelliJ and really liked it. I was able to get a comp license for it, but haven't actually installed it yet. I think I'm reluctant to use a tool that might be beyond the reach of our users.

Reintegrating Watir and FireWatir

Right now the top priority of the Watir project is to reintegrate with FireWatir and improve the compatability between the two implementations. I've begun serious discussions with Angrez Singh, the FireWatir lead, and we've agreed on this basic objective. There are, however, lots of logistical and architectural issues that need to be discussed further. We've agreed to continue this discussion on the Watir Development list, wtr-development@rubyforge.org.

If you would like to be part of this discussion, or learn more about our plans, please join this list. This is a public list that has been quiet lately, but has been used for similar discussions in the past.

Rasta supports data-driven testing

Rasta is a data-driven testing framework that pulls data out of Excel spreadsheets and executes it using simple "fixtures" that you write in Ruby. Your fixture methods could use Watir or another Ruby-based library (Selenium-RC, Mechanize, Soap4R) to execute against the target application.

There are several things that I think are simply terrific about Rasta. The most dramatic feature is that the spreadsheet is annotated in real-time as the tests execute. Each cell is colored red or green to indicate whether that step passed. If there are errors, they are attached to the cell as a comment that appears when you hover over it.

There are a lot of data-driven testing frameworks that work off of tables these days. Most use wiki's to create the tables, which feels somewhat barbaric. Rasta allows you to use a spreadsheet instead.

Another great thing about Rasta is that it allows test cases to be defined either horizontally or vertically. I've noticed that most data-driven frameworks (Fit, TestFrame, Certify, QTP) specify test cases horizontally. However, most testers, if they are starting from scratch will organize their test matrices vertically. I'm sure there is some psychological reason for this. In any case, Rasta allows you to organize your tests either way. To me this is simply a great of example of how test tools should be adapted to the way tester's think rather than the other way around.

Another feature along these lines is the ability to stop the tests and then pick up where they left off, instead of having to restart from the beginning.

Right now, Rasta only supports Excel, but the developers are working on adding support for Open Office and Google spreadsheet. They've put together a great website, with screen shots and examples. It really gives a much better impression of what the tool is like than I can convey here.

There has been a lot of discussion in the Watir community about data-driven testing frameworks. Any one interested really should give Rasta a look. It is simple, effective, and fun.

Supporting Watir

Pete Dignan and I are founding a company to support Watir and other open-source testing tools. We are still working on the business plan and name of the business. Normally, I would have wanted to wait until we had a few more details decided before making an announcement, but our plans were leaked out early so there is really no point in keeping quiet.

Pete is the president of ProtoTest, a Denver-based testing services company, with both financial success and an excellent track record of client satisfaction. One of the reasons that I’ve agreed to join Pete in this venture is because of his strong commitment to open-source software and continuing to respect and support the Watir community.

One of the first things we’ll be doing is to get a new release of Watir out the door. This has been overdue; me and the other core committers have been very busy over the past six months with other work. The most important thing that we’ll be working on is making Watir work better with Firefox. Specifically, we would like to merge the FireWatir code back into the original Watir code base. I still think that the best way forward on this project is to instrument our test suite so that it can be run against either project. This means some intermediate work on the test suite, as well as putting the Watir project itself under continuous integration using CruiseControl.rb.

Austin Watir Users Group, Weds Mar 19, Keyword Testing with Rasta

I am very please to announce that the first meeting of the Austin Watir Users Group will be this Wednesday, March 19. I've been working for some time with several people to make this happen.

Keyword Driven Testing With Rasta, Hugh McGowan, Convio

Hugh McGowan is presenting his work on Rasta a framework for building keyword-driven tests using Ruby and Excel. This open-source framework fills an important gap for the many testers who are looking for a way to easily put their Watir tests into spreadsheets.

Sponsors

TestCo is providing food and Uplogix is providing beverages. If you are planning to attend, please RSVP to Brent Lavelle so we can have an accurate head count when we order the pizza.

Agenda

6:15 Pizza
7:00 Rasta, Hugh McGowan

Location

Uplogix, Inc.
7600B N. Capital of Texas Hwy.Suite 220
Austin, TX  78731-1189

Driving Directions

This is located on 360 between 2222 and Spicewood Springs Road.

Uplogix is next to the apartment complex with the big American flag so if you are coming from the south you will need to make a U-turn on 360 at this complex and get over to the right lane fast. Traffic on 360 from 5 till 7 will be bad.

Austin Watir Users Group

We will be planning additional meetings of the Austin Watir Users Group. To recieve future announcements, please subscribe to
watir-austin@yahoogroups.com

Looking for Chief Architect and Senior Rails Developers

Due to recent turnover, my team has senior development leadership positions open. These are great opportunities for people who are committed to Agile development and Ruby on Rails.

We are developing a new product line, mostly following XP, but also influenced by Crystal and Lean. We use test-first development, continuous integration, and automated acceptance testing.

The chief architect needs to define and promote a technical vision for our product line, effectively balancing competing concerns such as customizability and upgradeability. We are looking for someone who knows how to work with an Agile team, developing the architecture incrementally, and work with our business stakeholders, understanding the technical requirements and promoting their technical vision.

The Rails developers we are hiring need to be well-versed in test-driven development and related practices of design patterns and object-oriented design.

These are great opportunities to work with a high-performance team on a challenging project. Contact us if you are interested.

Update: Actually these weren't such great opportunities after all. The project was cancelled and the team was let go.