Participatory Testing: The SpikeSource Approach

by Murugan Pal

Did you ever wonder why one of the most-used pieces of open source software was named Apache? Aside from the obvious nod to the Native American tribe, it is also an acronym for "A PAtCHy server" as it was based on some existing code and a series of "patch files." (Editor's Note: The link to was inadvertently left off when this article was originally published). This patchiness is an important characteristic of open source software.

Open source software is built to be resilient and to evolve organically. The first release of a new open source component may not cover all the failure scenarios, but its source code transparency and the wide use of these components help to improve the quality.

Traditional software vendors have adopted various testing methodologies and techniques that are broadly categorized as white-box, black-box, and in some cases gray-box testing. Typical tests include unit, regression, load, integration, boundary, negative, functional, standards, and security tests. For a good overview of the testing that one major commercial software company puts its products through, look at how Microsoft tests its software.

Open source software testing faces some additional challenges because of factors like source code transparency, velocity mismatches between component releases, and risks due to security and IP rights. The projects and source code in and of themselves are superior to many of the commercial products, but customers want them to be "tested, certified, and supported" for production-quality usage. Why? Because many enterprises run into some of these classic challenges when using open source:

  • How do I resolve version mismatches and upgrades across major releases?
  • How do I know which bug impacts other dependent components?
  • If there are multiple fixes for the same bug, which fix is the right fix to use in my situation?
  • How do I contribute back my specific bug fix for a generic use case?

There needs to be an "underwriters lab" for open source components:

For enterprise customers to

  1. Validate component interoperability and end-to-end integration
  2. Certify functional readiness for production-quality usage
  3. Verify bug fixes and avoid regression

For open source developers to

  1. Verify cross-platform compatibility
  2. Help run benchmark tests and compile uptime data
  3. Test backward compatibility and migration requirements
Code Zoo.

CodeZoo is a site for any developer who wants to avoid writing code. We believe the best code is the code you don't have to write -- the pieces already done for you, as well or better as you would do them yourself.

Learn more

Testing different versions of components on different hardware for different types of functional usage brings up the challenge of combinatorial explosion. For example, a typical SpikeSource release integration involves optimizing 272 parameters distributed across 189 configuration files across 63 components yielding 7 preconfigured stacks for developing applications in Java, PHP, Python, Perl, C, and C++.

We need a new approach in testing to address these challenges. Our recommendation is to apply the very nature and strength of the "architecture of participation" to open source testing. To quote Tim O'Reilly, "we need a system that allows for a real free market of ideas, in which anyone can put forward a proposed solution to a problem; it becomes adopted, if at all, by acclamation and the organic spread of its usefulness."

Hence the requirement to build a Participatory Testing System that allows developers to participate by testing their applications as the payload. We here at SpikeSource have adopted the following initiatives as part of building this participatory testing system:

  1. Testing Framework Transparency

    Leverage existing and popular open source testing tools, integrate them into a common framework to automate tests, and publish the results. Additionally, we develop new tools and contribute to the community. Today, we use 18 open source components as part of our testing framework.

    Screenshot of PHP Code Coverage Tool (first ever of its kind) contributed by SpikeSource to the community:

    Screen shot.
    Click image for full-size view.

  2. Test Results Federation

    An initiative and published interface for open source project committers, ISV developers, and customers to share and federate their test results.

    Chart - click for full-size view.
    Click image for full-size view.

  3. Managed Testing Services

    Let customers and developers write tests, archive them into an "uploadable" format, and run the tests in a remote and managed fashion. This helps customers and developers to verify or validate a patch with their functional applications even before the patch is officially released.

  4. Advanced Testing Techniques

    Develop new techniques to test open source projects from an inside-out perspective. A few initiatives that can be leveraged, like Binary Regression Testing, and next-generation testing, exist today.

  5. Everybody Tests

    Develop methodologies and frameworks to leverage people with different skill sets to test the software at different levels. For example, a big open source project could be formulated in subcategories, with committers doing unit testing, ISV developers testing interoperability, IT developers testing functionality, system administrators testing stress and load, and academic students contributing to testing that software in their own respective areas of expertise.

Our goal is to leverage the power of participatory testing to:

  • Validate a fix from any corner of the globe as quickly as possible-- within hours.

    Automate mechanisms that can be built to identify source repository changes, to spider different project home pages, and to crawl multiple bug databases. Today, we pick up our bug fixes by tracking 57 mailing lists, 5 USEnet groups, and 62 RSS feeds from the open source community. In addition, the test bed must be flexible enough to accommodate different granularities of testing, ranging from minimal acceptance tests to complete suites of interoperability tests, thus optimizing the "time to test."

    Here is a screen shot from our internal build and bug validation harness depicting the latest Geronimo progress:

    Geronimo build test chart.

    We internally track bug validation processes through our automated alert dashboard:

    Screen shot - click for full-size view.
    Click image for full-size view.

  • Guarantee functionality and backward compatibility across different versions of integrated software components.

    This matrix explains some of the complexity involved in testing interoperability between functional constructs and component layers:

    Screen shot.
    Click image for full-size view.

  • Certify interoperability and manage exponentially increasing combinations stemming from alternate platform choices like:

    Operating Systems: 6 Linux distributions, Windows, Solaris, ...
    Database: PgSQL, MySQL, Oracle, SQLServer, ...
    Web Server: Apache
    App Server: JBOSS, Tomcat, Geronimo, Jonas, ...
    Languages: Java, C, C++, PHP, Perl, Python, ...

As an interesting note, here are some of the statistics we have compiled within SpikeSource:

  • Building and validating 73 components on 6 platforms across 6 languages by running 22K native tests, 205 SpikeSource tests yielding 41.5 percent code coverage in 14 hours
  • 88 components can be identified and tracked through Spike Asset Manager
  • We track and watch 106 components as part of the information services
  • "Native validation framework" automated to cover 4 major components on 4 major platforms
  • 152K+ entries in SpikeLibrary with 400+ original articles (growing at a rate of 3.94 per day)
  • Seventeen components used to deploy and manage Spike online services
  • Thirty-one open source components used for Spike intranet and network operations

To conclude, the traditional and existing open source testing methodologies are necessary and useful, but they do not take full advantage of that "architecture of participation", which makes open source development so powerful. SpikeSource's goal is to facilitate the adoption of open source software in the enterprise through testing, certification, and support services. We innovate, automate, and optimize advanced testing techniques as part of our core competency.

Murugan Pal is the founder and CTO of SpikeSource, Inc.

Return to the O'Reilly Network.