January 31, 2004

GUI operations are essentially impossible to script

It is much more difficult to automate administrative or testing tasks on Windows than on Unix. Unix makes ubiquitous use of command-line interfaces that are easy to script. There is no simple equivalent on Windows. This difference is clear to anyone with significant experience on both platforms. Windows users who are skeptical of this don't have to take our word for it. Now they can refer to a Microsoft internal white paper. The difficulty with scripting is but one of the difficulties detailed in this account of changes needed to convert Hotmail from BSD to Windows after Microsoft acquired the company in 2000. According to the paper, "GUI operations are essentially impossible to script." This paper's findings were originally reported by the Register.

I regularly encourage developers to include programmatic, scriptable interfaces to their product software. It makes the software much easier to script, which is essential both for automated testing as well as to facilitate administration by customers. Internally at least, Microsoft sees the wisdom of this as well.

Posted by bret at 12:20 PM | Comments (2)

January 20, 2004

Python Books

python_books.jpg I learned Python last year, trained several testers to use it, and then lead the development of a Python-based testsuite for a client. Python is a great language for testing. Mostly, i learned it from books, reading eight different books in the process. My favorite is Python in a Nutshell by Alex Martelli (O'Reilly). I found it well-organized and with extremely accurate and detailed descriptions of the inner workings. Mostly you don't need to know this kind of thing, but when you do, you have it. Unlike most Python books, this gives equal treatment to the two slightly different versions: the one written in C ("CPython") and the one written in Java ("Jython").

Another excellent book is the Python Cookbook by Marelli and Ascher (O'Reilly). It contains lots of meaty Python examples. Some are useful because they illustrate how to use certain features or libraries. Others can be reused in whole to solve problems you have. I did.

The Visual QuickStart Guide to Python by Chris Fehily (Peachpit) is an excellent book for helping anyone understand the basic syntax of the language. I think it would help any one who wanted to learn Python. It uses an easy to read split-page format with code examples (often interactions with the interpreter) on one side and explanation on the other. Some users may find that it is the only book they need.

I learned a lot from The Quick Python Book by Harms and McDonald (Manning). It has a nice introduction to the language that will be helpful to people who already know a couple other languages. It also has a useful reference section, although after learning the basics of the language, i found myself depending on Martelli's Nutshell book instead.

There are also two online books that teach Python to people who don't know how to program at all. I haven't read them, but if you are a manual tester looking to get into automation, one of these might just be the thing. You can read them online for free or purchase a bound version. They are How to Think like a Computer Scientist and Learning to Program.

Posted by bret at 02:54 PM | Comments (0)

January 19, 2004

AWTA5 Report

awta5_group_small.jpg Last weekend i hosted the fifth annual meeting of the Austin Workshop on Test Automation. Co-organizers were Carl Erickson, Cem Kaner and James Bach. We had 21 people attend from around the country. This year's theme was open source test tools and we had presentations from a variety of tool developers. Many are saying that this has been our best AWTA ever. I'm very excited about it.

Carl Nagle, Bill Nasuti, Bob D'Antoni and Yuesong Wang presented SAFS, a data-driven engine. It parses tests written using keywords in a spreadsheet-based format and then executes them using a GUI test tool. It was initially developed for Rational Robot but has since been extended to also support Rational's RobotJ, which uses a completely different architecture (it is built using Java and Eclipse). In the process, they developed a tool-independent layer. As such, they believe it could be extended fairly easily to support other GUI test tools. The speakers represent three different companies (SAS Institute, Management Science Associates and Claritas). We got a good look at how and why test teams are creating new kinds of alliances in developing their automated suites. This was actually the first time that they had all meet face-to-face. The tool independence layer was implemented using STAF -- another tool presented at the workshop. Indeed some of us are wondering whether the test case parser and the tool independent layer could be logically separated, thereby giving tool independence to tests expressed in other languages or formats.

Charles Rankin presented STAF and STAX. These are tools used widely at IBM that were made open source because of their use in IBM's Linux testing. This was Charles' first presentation outside of IBM. STAF is a remote test agent. Many commercial testing tools include proprietary test agents that allow them to control tests on multiple machines. Now you can build the same capacity into your home-built automated systems. STAF has several services built in, including the ability to run executables, store local variables, and manage resource lists. It is designed as platform technology, allowing you to plug custom services into the framework. STAF runs on most operating systems: Windows, Unix, AS/400, MVS. The most obvious limitation is that it currently hasn't been ported to the MacOS and other BSD Unices, although Charles doesn't think that would be too hard. One of the great things about STAF is that the same services are available from a wide array of languages, including C/C++, Java, Perl, Tcl, Python and Rexx. There is even a command-line interface which looks for handy for analyzing test failures. STAF was the big hit of the workshop. Many of the attendees appear to be spending time this week playing with it and figuring out how to incorportate it into their test suites.

STAX is one of the richer services built on STAF. It is a test harness and test execution language with an xml-based grammer. The language has a number of unique logic primitives. For example, it has a parallel iterate command. This is like the for iterator found in many languages, except that each iteration is run concurrently in a separate thread. I'd never thought of making something like that a language primitive, although i'd recently added a similar feature to a python-based testsuite. That took me three days to get all threading right. I could have probably done the same thing with STAX in a couple hours. It's a bit of wonder that these tools ever made it public. This happened in the window of time between IBM's adoption of Linux and their acquisition of Rational. I expect that Rational is developing commercial testing tools that use STAF as their agent technology, just as they've already embraced Eclipse. STAF and STAX both come with extensive training materials.

Todd Bradfute presented TestMaker, a tool he is a contributor for. TestMaker has a proxy service that allows it to record the communication between a webbrowser and a server. It spits this out as a Python script. This is typically used for creating load tests, but it really could be used for any kind of protocol-based testing. TestMaker itself is written in Java. It comes with support for HTTP, SOAP, SSL and several other protocols. And it's designed to be extensible to easliy support other protocols. Frank Cohen, the lead developer, has published a nice book describing how to use the tool. Todd also impressed the workshop with his new mohawk hairstyle.

Chris Morris demonstrated the latest version of WTR, a Ruby library for testing web-based applications. It really amounts to convenience functions for accessing the COM interface to Internet Explorer, mostly through access to the DOM. Brian Marick and i have been using this tool in our Scripting for Testers hands-on training. Chris and i gave a paired, impromptu presentation. While Chris demo'd the tool (including some new features i had seen for the first time just beforehand), i talked about why i thought it made an effective testing tool. For example, you can type WTR commands in an interactive Ruby session, and see their effect immediately on an adjacent browser window. Chris also showed a script that didn't use his library, but made calls directly to the IE COM interface. This really helped the workshop understand what was involved in the technique. Of course, a similar library could just as easily be written in Python or Perl or VB or any other language with COM support. Well, actually, Chris makes heavy use use of Ruby's "method-missing". This effectively allows methods to be created only after they've been called! I think Python has a similar facility, but Perl or VB wouldn't be able to provide quite as elegant an interface as WTR.

Carl Erickson presented Haste, a test harness for system tests in Java. With Haste, a new JVM is spawned for each test, preventing side effects from one test from affecting others. Of course, Haste itself runs in a JVM as well, so you get two JVMs. He demonstrated several tests, showing how they used other tools like Abbot and Marathon as the GUI drivers. He also showed how he uses "narc" classes. He's built a code-generator that creates special narc-versions of classes with public methods. This allows tests to call private methods and inspect private attributes without making changes to the base classes. He refers to this system of alternate testing classes a "narcitecture."

David Vydra presented ESL, Java-based parser technology for creating domain-specific languages. Keyword-driven tests are really just test expressed in a simple grammer. This grammer has one statement per line, with the first element representing a function and the remainder the arguments to the function. The more sophisticated keyword-driven frameworks (such as SAFS) also allow assignments of values to variables and additional loops and control logic. But if you are going to end up creating a grammer anyway, maybe you should consider using some of the standard tools and priciples of computer language design. That's what David has done with ESL. With ESL, you can have a statement in a test that looks like "assert Customer with id = 100 has creditLimit <= 10,000.00". Is this an easier language for expressing tests? Does this language allow for the definition of more complex tests? I dunno, but now that we have an open-source implementation of this technology, i hope some comparisons will be made. Personally, i'm very interested to see what can be done with this.

James Bach presented ALLPAIRS. Unlike the other tools, this is a test design tool. A classic testing problem is combinatorial explosion. There are so many different valid combinations of inputs to a system, that you just can't test all possibilities. ALLPAIRS uses an algorithm to create a subset of the combinations that meets a particular coverage requirement: that all pairs of combinations will be tested in one test case or another. This technique has been in the literature for some time, but it is tedious to do by hand. We actually included a description of the manual process in Lessons Learned, but the awkwardness of it lead James to develop this tool. This is the first easy-to-use open source implementation of the algorithm.

Pat McGee gave a presentation on a technique he is developing for his Ph.D. at Florida Tech under Professor Cem Kaner. The technique consists of taking existing regression tests and rerunning in them random sequences over long durations. I've heard people suggest this technique over the years, but i'd been pretty skeptical. Pat presented some pretty convincing evididence, however, that it can be very effective at finding important bugs that are missed by other standard forms of testing. He calls the technique "extended run regression," and hopes to develop a open source tool for it, perhaps using some others as components.

Those were the formal presentations, but i know that the best parts for some participants were some of the impromptu discussions and debates that were fostered by the presentations. AWTA has a loose schedule and policy to explore digressions that seem to capture the interest of the workshop.

James Bach raised the issue of tool bias: how does using a particular tool bias us in favor of some types of testing and against other types? He convinced us that we needed to be thinking of this in any fair assessment of a tool and it became touchstone throughout the workshop. Another interesting discussion came in the context comparing a protocol-based testing, such as might be done with TestMaker or HttpUnit, with browser-based testing, such as might be done with WTR or Samie. We also were hoping to compare different testing formats, such as data-driven, script-based, or domain-specific languages, but we didn't quite have time to do it. Maybe next time.

Pete TerMatt brought in CD's with Knoppix. This is a bootable version of Linux. You don't have to install it. You just boot up with it in the drive and you are running Linux. I've been doing a lot in the past year with teaching people to use test tools they install on their own laptops. We've stuck with Windows, because that's what people have, but the existance of Knoppix opens up the possibility of Linux based classes. It would be neat to teach how to test Unix command line programs using Expect using the Knoppix version.

Several workshop participants visited Austin's Computer Museum. I didn't really realize we had one, but we do. It's behind the Goodwill Computer Store, and has a lot of neat stuff that i had completely forgotten about.

Most of the participants were experienced software testers, but about a quarter were actually test-driven developers. A lot of test tools have come from the TDD community lately. And i've been hosting many get togethers for people from these two communities over the past year or two. But this is the first time that we just assumed that everyone basically knew about both context-driven testing and test-driven development. That was never made explicit, but it was nice to have the groups working together without much attention to the methodology debates that have often embroiled both groups.

Posted by bret at 03:32 PM | Comments (2)