June 19, 2003

My Atomic Watch

I recently received a watch as a gift. This watch contains a radio receiver that automatically callibrates to the official US atomic clock in Boulder Colorado. At first i was mainly happy to have something that looked nicer than my previous watch -- a cheap Timex with steel showing through the worn goldish patina and a ratty watchband. I found it functional, but had become aware that it might not give the right impression. My new watch would help give me nerd credibility -- important for a software consultant.

But i've actually grown quite fond of it. I soon found that nearly all of my timepieces were five or ten minutes off: my old watch, the clock on my laptop, the dashboard clock on my car, the clock on my PDA, the clock on cell phone, my wife's watch. I expected some to be off by only a minute or two. I was wrong. They were worse.

I've now synchronized them all to my La Cross Technology radio controlled watch.

Posted by bret at 10:17 PM | Comments (6)

June 09, 2003

My History As A Certified Software Quality Engineer

The last substantial change that we made to Lessons Learned in Software Testing was my removal of my recounting of my personal experiences with certification. As published, the section (Lesson 271) only recounts Cem's experiences.

Certification is a complicated topic, and one that i've become less and less interested in. But it keeps coming up. People want to know my thoughts on the matter, which are hard to summarize. I thought i'd start by telling the story i'd pulled from the book.

In 1996, i drove to Dallas to take a test. As a result i was certified by the American Society for Quality Control as a software quality engineer. The exam had a lot of questions about the CMM and ISO-9000 and process auditing. It also had a few questions on testing. I didn't find the test to be difficult. They provided a long bibliography. I read most of the testing books on the list, and looked through a sampling of others from the other sections. It was an open book test and most of the questions could be answered by simply knowing where to find the answer in one of the books. For example, they showed a diagram and asked you to label it. It was a fishbone, or cause-effect, diagram and the exact same diagram was contained in Watts Humphrey's book with a caption telling you the official label.

At the time, i had decided to focus my career on software testing. I thought the certification might help me to establish my credentials. At the time i was fan of the writings of Cem Kaner and James Bach, and i knew that they were on the board that produced this certification. The exam i took was the first time it was publicly offered.

I don't believe it helped me much. Mainly it helped me understand a popular conception of "quality assurance" sufficiently well enough to realize that i didn't like it. Since then i have refused to take jobs with a "quality assurance" title, prefering "testing".

Quality control is also known as statistical process control. It's a respectable discipline that requires a fairly advanced understanding of mathematics. And it's a key technique in manufacturing: Japanese auto manufacturers used it to improve the quality of their cars and take a large part of the American market. On an assembly line, being able to make the quality control calculations is percieved as a technical skill.

None of this can be said for software quality assurance. It is not a technical skill. It does not require mathematics. And it does not have an underlying theory. Rather it is a set of nostrums, common sense and scattered techniques that are intended to allow a relatively non-technical person provide management support for software development projects.

Sadly, the ASQC has even renamed itself the American Society for Quality, focussing its mission on sanctimony rather than a valuable technical practice.

I allowed my certification to lapse for two reasons. One was that it stood for a set of practices that i did not believe in. I might be mistaken for someone who believed in the value of the "body of knowledge" supporting the certification. I don't. Indeed, i had mistakenly thought that Cem and James supported it when i took the exam. I later found out that they had serious reservations with the process that came up with the certification. By claiming the certification, i feared that others would take me as endorsing it.

The second reason concerned the title "software quality engineer." Claiming to be an engineer when you aren't actually a licensed professional engineer can get you into legal hot water. This is particularly true if you are an independent consultant (as i am), and if you live in a state that licenses software engineers (as Texas does). It seems quite irresponsible for the ASQ to encourage people to put themselves into this kind of legal jeopardy. Specifically, a dissatisfied client could accuse me of malpractice and sue for triple damages.

So i let it lapse.

Posted by bret at 06:14 PM | Comments (0)

June 03, 2003

Why Ruby

We used Ruby in our Scripting 101 class. It's a scripting language that competes with Perl and Python. We've also chosen to use it for other testing projects as well. Why?

First off, we think that scripting languages are the right kinds of languages for most kinds of testing. And most can be extended in C/C++ for those occassions where they won't do. But there are lots of scripting languages. Why not use Perl or Python? Or TCL?

The greatest thing about Perl is its libraries. It has libraries for everything. But it also has a nasty syntax. Although a major improvement over the shell scripting languages it replaced, it is still hard to read. This makes it an especially poor choice for testers who are not highly technical. It also makes code harder to review -- which is a problem because i always worry about bugs in test code. Cryptic syntax makes it harder to notice these.

TCL is popular among testers of embedded software. One reason is that it's more compact than any of the other languages. But its syntax, which is arguably cleaner than Perl, still leaves many confused.

Python has a cleaner syntax than Perl or TCL, and could make a good choice. It is certainly better established than Ruby. And both Python and Ruby have interactive interpreters, which make them easier to learn and experiment with. And they both use exceptions, which i see as essential. Now Ruby does show signs of growing popularity, but it's main advantage over Python is that a community of people interested in testing has already gathered around Ruby.

Why has this community developed around Ruby? The developers with an interest in testing are mostly known for their interest in XP and agile development, but many of them have a strong interest in object-oriented programming, with roots that go back to Smalltalk. And Ruby is more thoroughly object-oriented than Python, and it has better support for closures and lambdas and other programming constructs that programmers in Smalltalk grew in love with. I was exposed to these in Lisp. A lambda is a function without a name. It sounds trivial, but having things like that can make it much easier to create compact, well-structured code. That's what i think, but it's also the opinion of Hal Fulton and Dave Thomas and Andy Hunt, and John Dell'Aquilla that language features like this make a difference. Even if you don't care about such things, you probably do want to use a language that other people are using. And we're using Ruby.

I'm very excited to have a language for testing. When i first went independent, i wanted to find a language to standarize on, so that i could develop libraries and training materials. I had been a long-time SilkTest user and had recently been impressed with WinRunner. I considered basing my training on either of these tools, but their companies (Segue and Mercury, respectively) were most uncooperative about providing me with training/demo copies of their tools. And frankly i was already growing dissatisfied with vendor-specific scripting languages. I found that about half the training that people needed in order to use these tools was simply learning these scripting languages. Later, i was able to get courtesy copies of WebLoad and Robot (from Radview and Rational). I was also happy to see WebLoad use JavaScript for its language and to see that Rational was moving towards using Java and Visual Basic for its testing tools. But i was also concerned that when i needed to get the updated versions of these tools (which happens nearly every six months) i might no longer have enlighted or generous people at these companies to deal with. I was also concerned with how i'd be able to provide hands-on training. That is a lot of licenses, and it's impossible without special terms.

So i am using Ruby, which raises none of these concerns. But then it isn't a testing tool. At least not yet. But i plan to fix that. And i'll be converting my examples in my test automation class into Ruby.

Posted by bret at 01:00 PM | Comments (10)

June 02, 2003

Scripting 101

Two weeks ago i taught a one-day class with Brian Marick called Scripting for Testers 101. Scripting, of course, is simple programming. Our goal was to teach some basics of programming and test automation to testers who didn't know how to program and indeed may have had some trepidation about it. Nonetheless, the class was a bit of a struggle for some such people. We have several ideas on how to make it easier for them and will be teaching it again.

We used Ruby as our scripting language. We had the students install it and a test application on their own laptops for the class. Having students use their laptops was a new idea, and it worked very well. We encouraged the students to pair up. Most did and we had plenty of laptops on hand. We also had a classroom network for students to hook into. Most hooked into the wired network, but a few hooked into the wireless network we also had. With time, i expect the size of the wired network to shrink as more people get Wifi cards.

Brian spent three quarters of the class in the front, while i was floating around answering questions and helping people out. It was an eye-opener for me see people struggle with stuff that i saw as basic. For example, our materials presumed familiarity with the windows command line interface. But many of our students said that hadn't used it in years, and had trouble with simple things like knowing when to use absolute versus relative pathnames.

In the past, my one-day classes have been mostly lecture with some discussion. To lecture effectively, i need a fair amount of discussion to help me gauge the level of understanding in an audience. Observing hands-on exercises gave me a much better sense of whether our students were keeping up. Many weren't, and i now suspect that the same was true of my lecture-based classes -- only i had no comprehensive mechanism for getting this feedback during the class.

We also had several experienced and programming-savvy testers in our class. I'm not sure why they took it, but they seemed to get a lot from it. Most of our exercises were pretty open-ended, so they allowed the advanced testers to try out more things.

The class generated a lot of interest for Ruby. The syntax is pretty quick to learn. It also has a rudimentary driver for Internet Explorer that allowed us to automate tests of a web-based interface by controlling the browser. I'm planning to continue development of this to make it well-suited as a test tool that can be used in training environments. It is open-source.

Indeed, we'll probably make all the code open-source. That will facilitate getting students to install the software beforehand so that we can have more time for the testing exercises.

Posted by bret at 01:40 PM | Comments (1)