What You Test Changes How You Test

by chromatic

I practice annoyance driven development. I set my threshold of annoyance low such that everytime I feel frustrated by a technical limitation, I notice consciously. My intent is not to find technology endlessly frustrating (though that happens sometimes), but so that I can identify the next most important thing to fix.

For example, Parrot has a large test suite. Several of those tests exercise the source tree as a whole, checking for copyright notices, Subversion ID strings, and metadata properties. I call these non-functional tests, because they exercise externalities of the project, not features of the code. Having accurate copyright notices and repository metadata (especially "Make sure these files have the proper platform-specific line endings") is useful... but analyzing thousands of files in dozens of directories isn't instantaneous.

Because we have several contributors, we attempt to keep all of our tests passing on all of the platforms to which we have regular access all of the time. (Exotic platforms like Windows aren't always so fortunate. Porters wanted.) To achieve this, committers must be able to run the test suite before checking in changes.

Everything so far is obvious. What wasn't immediately obvious to me was that there's a threshold beyond which people will not run the entire test suite.


4 Comments

Reedo
2008-02-07 06:52:25
"Exotic platforms like Windows..."


That gave me a chuckle. Thanks.

Robert Fischer
2008-02-07 09:40:37
Why not have the non-functional tests run on the CI server, and let the developers have at the functional tests? And why is it so hard for you to run a subset of tests?


The CI server is a good idea in any case. Sometimes developers get non-virtuously lazy and/or tired and forget to run their test suite before they check in, and it's good to catch that ASAP. Plus you get exotic platforms. :)


Drop me an e-mail. I've got a spare AMD 64 box kicking around, and I might be able to spare some clocks on a Windows machines. The two of them are shortly going to be turned into local repositories, CI servers, and BOINC clients. Might as well donate them for good use.

Ovid
2008-02-07 12:42:58

I'd also recommend people check out Improving Test Performance. That quickly covers about a third of what the proposed talk will cover.


Your comments also highlight something I plan to bring up at the beginning: trade offs. Just about every technique I introduce will involve trade offs (particularly process separation issues) and developers will have to know those and weigh them carefully against proposed benefits.

Roboticus
2008-02-27 06:00:48
It sounds like the testing isn't divided into appropriate domains. I think you should keep your tests close to the problem at hand. So perhaps you should have your standard make/ant target run the full compile/run tests, and defer the nonfunctional tests to your checkin target.


This gives you the benefits of maximum speed during your most-frequent activity (edit/make/run), and you're still covered during your checkins. You may find this a bit less frustrating, as when you're checking in your code, you're at a natural pause point anyway.


...roboticus