Meeting User Requirements--How Free Software Achieves Quality

by Andy Oram

Most observers see free software development as a vine--and not a
particularly pretty one. They watch as the code for various free
software projects extends tendrils in odd directions, puts down roots
wherever it finds a friendly resting point, and generally seems to
grow with no thought or planning. Linux, for instance, was criticized
a few weeks ago for lacking a "roadmap."

True, few free software projects can boast a formal structure of user
requirements, schedules, test plans, and so forth. But does this mean
that user requirements are not specified, that the software does not
benefit from thorough testing, and that in general there is no point
of contact between open source and traditional software engineering?

This past weekend I had a talk with Andrew Stellman, a programmer and
project manager, on this subject. He is also working on a book for
O'Reilly along with a co-author, Jennifer Greene. Like many
sophisticated developers, Andrew finds that his job requires him to do
most of his work with .NET and Windows, but he has a great affection
for free software. "When open source is well done," he told me, "it's
the best software--no doubt about it."

In this blog I'll discuss the ideas Andrew and Jennifer have codified,
and the implications of Andrew's work for free and open source

Back to software engineering basics

Andrew's and and Jennifer's ideas for process improvement are familiar
to anyone who has read the classic texts in software engineering. What
they have tried to do is find simple solutions to tasks such as
developing use cases, and to strip down the planning process to the
essentials that 90% of software shops need. User requirements, test
plans, unit testing, and regression testing--those are key to making
and maintaining high-quality software.

User requirements determine what succeeds and what fails. If only one
product offers you the features you need, you'll use it no matter how
ill-designed or difficult to use it is (and no matter how onerous the
commercial license). On the other hand, you'll walk right by the
best-designed, award-winning software if it doesn't do what you want.

Just writing down user requirements, Andrew says, is a great way to
eliminate bugs and speed up the development process. This is because,
done well, it impels the designer to state specifically what inputs
and outputs he or she expects.

He also has an interesting way of summarizing the goals of software
engineering: to ensure that all users' needs are met.

This is a fairly simple idea when you consider user requirements
documents: only if you specify the features required by the user will
they get into the final product. But the same driving force also
affects testing. Consider this: each different use of software
represents a different path through the source code. An untested path
means a potentially buggy feature, which means a user's needs that go
unmet. Thus, specific requirements followed by exhaustive testing are
synonymous with meeting users' needs.

As many have observed, the ad hoc development cycle typical of open
source tends to produce tools suited to programmers, or structured in
such a way that programmers are comfortable with them. Jennifer points
out that most users don't like the multiplicity of choices in such
tools and just want the software to follow the familiar steps they
already go through to accomplish a task.

Open source--a different approach

While Andrew wishes free software developers would do more of the
things that software engineering experts advise, he is still very
enthusiastic about the quality of free software. Somehow it achieves
many goals of formal software engineering in its own, less structured
fashion. We spent some time exploring this mystery, and come up with
the following ideas:

Mailing lists

Free software projects are conducted openly, which means that
every change is debated on mailing lists that everybody can
join. This effectively allows every user to contribute ideas for
features--and more importantly, makes sure the request is
heard. In proprietary development, feature requests are funneled
up through management channels and inevitably get filtered along
the way. One person, or a few people, may semi-arbitrarily drop a
request. On open mailing lists, by contrast, feature requests
receive whatever attention they deserve (and usually more); those
with merit get implemented. The same goes for bugs.

Parallel development

In Tracy Kidder's classic Soul of A New Machine, Data General sets
up parallel development teams racing to produce the best possible
chip. This kind of parallel, competitive development (without the
inhumane pressure) goes on all the time in free software. Two or
three people may be simultaneously working on a new virtual memory
system for Linux, for instance. One person may do his best to
solve the device numbering problem over several years, while
another says, "No, you're doing it all wrong, I'm going to do it
differently." Where there's interest, a wealth of solutions are
available to choose from. (Of course, sometimes there's not enough
interest among developers and there are few or no choices.)

Widely distributed testing

This phenomenon was described by Eric Raymond in his famous
dictum, "Given enough eyeballs, all bugs are shallow." This
statement has certainly been disputed (particularly where security
vulnerabilities are concerned) and seems somewhat dubious in the
face of long-dormant bugs that sometimes surface in major free
software projects. But such testing often does work, very
powerfully. Andrew says, "As long as thousands of students are
eager to take each new revision of the Linux kernel and load it on
their hardware, testing is going to make Linux strong. I don't
know why they're willing to do it, but it's great for Linux."

Design is not as much fun as coding. Almost everybody starts to code
too soon; that's an industry-wide observation. But hopefully more
highly experienced free software developers will decide it's time to
try something new and take the next step into design. They'll write
out those user requirements, based on mailing list feedback. They'll
lay out tasks for parallel development. And they'll formalize a lot of
the testing. Then we'll really see what free software can accomplish.


2005-01-19 18:40:35
process improvement improvements
I have yet to find success in open source software development, but for every large project I've done on the job I write a development plan, requirements specification, and a design document using my own evolving templates. Often I do a (acceptance) test plan and sometimes I produce a development review, too. Customers don't always read these documents as they should, but I find them invaluable... even though I'm the one that wrote them!

I designate a significant amount of project time to analysis and design. I track effort against estimates, and compare actuals for future estimation. I retain and review all project artifacts, even the scribbled notes. I comment the code so that extracting comments produces pseudocode. The more times I go through these motions, the better I get at following them.

The open source process appears to be a very different beast, and I still am not sure if I've got what that takes. I don't know if all the traditional rigor and discipline will pay off when I give cooperative development another shot, but I will say it sure does make for some nice software.
2005-02-09 03:10:16
For engineers by engineers
For me the two defining factors that help explain why free software achieves its mysterious high level of quality despite the lack of rigorous methodology are motivation and skill.

In my opinion, formal methodology is necessary when teams are carrying dead weight. When the majority of the team has average skills and probably low motivation, structure gives them the support needed to produce higher quality code.

However, in the case of someone that is volunteering his time to work on free software, it is typically out of the love of the art--and recognition as well. So in this situation, the individual usually brings high-levels of motivation to their accepted responsibilities. Some people refer to someone with a high level of motivation on a particular task as possessing a passion for his work. Passion is very good for quality.

The corollary attribute of a free software project is that they are free to choose whether to accept a contribution. This freedom from compromising interests, affords the leaders of a free software project the liberty to evaluate (and test) contributions until they feel that the contribution meets the high standards of the project. On the contrary, when the constraints on a project's objectives involve the compensation of the project team, and where deadlines must be met, managers end up accepting contributions that are of average quality.

My current thinking supports your premise--that user requirements lead to better code. However as in project planning, the value is not in the written plan but in the process of planning. I believe that the value of user requirements is not in the actual written requirements, but in assuming the position of the user as foremost in the design process. When a development team takes a user-centered perspective to their analysis and design, they will produce a product where all the users needs are met. Maybe this even begins to explain the mystery of free software. The most successful projects seem to be where the development community become actual users of the product--for engineers by engineers.