Improving Test Performance

by Curtis Poe

Just before I started my job at the BBC, one of our developers committed code which reduced our test suite run time from an hour and twenty minutes down to twenty-two minutes. One of my first tasks was to improve that. However, improving performance begs the old question of "cpu or developer performance?" Both are equally important, but I'll just talk about making the tests run faster. Right now, it looks like we're on track to get our test suite to run in under ten minutes. Here's how we did this.


4 Comments

Michael Peters
2007-12-29 17:27:08
All good suggestions. I especially like what I'm seeing so far with Test::Aggregate.


But what happens when you can't make the test suite faster without investing loads of time that you simply don't have? What if you've done these things and your test suite still takes 40+ minutes to run? That's the situation at $work now.


Our solution: Run a build server that is constantly (every hour) looking for updates to SVN. Then doing a checkout, build and then running the tests. Test results get uploaded to our Smolder server and everyone on the team get's notified within the hour if a commit broke the tests. This means that all of our "police" tests (testing pod, stricture, etc) also get run all the time.


Developers still make sure the immediately relevant tests get run by hand before checkins, but at least they don't kill 40 minutes playing online games while the tests are running :)

Robert
2007-12-31 05:47:18
@Michael


You are doing what I would do. Developers only test what they have changed and everything goes up to a continuous testing server.

chromatic
2008-01-03 14:17:36
I have trouble believing that "use a continuous integration server" is a good answer to any question other than "How can we hide technical debt in our tests and increase the amount of time and effort and context-switching we spend debugging?"


As soon as you admit that it's okay not to know if your code works when you check it in, you're admitting that it's okay to break the build for other people. It's difficult for me to believe that you have the time and resources to work around that more than once if you don't have the time and resources to make your test suite faster.

Robert
2008-02-08 11:49:35
"As soon as you admit that it's okay not to know if your code works when you check it in, you're admitting that it's okay to break the build for other people."


I don't recall saying or implying that but you may have a different experience than me. If you check code out, change it, test it and check it back in. I don't see how having a CIS building the *whole* shebang is a bad thing.