The power of one little automated test

by Andy Lester

Automated testing is the #1 investment we can make in software quality today. Here's a simple example of one simple test accidentally uncovering an existing bug. The code is in Perl, but applies to any project in any language.


It all started easily enough. There was a little note sent to the Perl
bug database:



perlop man page mentions: Binary "x" is the repetition operator
... repeated the number of times specified by the right operand.

It should mention what about if the right operand is negative, e.g.,
print '-' x -80

I figured I could make a quick documentation fix, and maybe even add
some automated tests to the Perl test suite.


The x operator in Perl does repetition on a scalar or list, as
appropriate. For example:



$a = "abc" x 2; # $a = "abcabc";
@a = ("abc") x 2; # @a = ("abc","abc");

If the right-hand operator is 0, then you get an empty scalar or list,
as appropriate. If the right-hand operator was negative, it was the
same effect as having it be zero. As the bug said, the documentation didn't say anything about negatives, so I decided to investigate, and document appropriately.


I added a little sentence to the paragraph describing the operator, and
then I added some tests. If it's worth documenting, it's worth testing.
Documentation and tests are as much a part of the code as the code itself.


The t/op/repeat.t file in the Perl distribution already had a lot of tests in it, like:



is('-' x 5, '-----', 'compile time x');
is('-' x 1, '-', ' x 1');
is('-' x 0, '', ' x 0');

So I added the obvious add-ons:



is('-' x -1, '', ' x -1');
is('-' x undef,'', ' x undef');

And then went to add them to the list-related sections:



@x = qw( a b c );
is(join('', (@x) x -14), '', '(@x) x -14');

Before I sent the patch in, I ran a full make test and found that
the last test didn't pass. In fact, it caused a Panic in Perl, and the
program died. I boiled it down to a simple:



@x = (1,2,3);
@y = (@x) x -1;

Turns out that that case of a negative or zero operand wasn't handling the stack correctly. Fortunately, this was only in the latest development version of Perl, but at least we found the problem.
A quick patch made it all better.


Some morals to this story:



  1. Never underestimate the power of one little test.


  2. There is no such thing as a dumb test.


  3. Your tests can often find problems where you're not expecting them.


  4. Test that everything you say happens actually does happen.


  5. If it's worth documenting, it's worth testing.





What bugs have you accidentally stumbled across, and how did you find them?