Do-It-Yourself Documentation? Research Into the Effectiveness of Mailing Lists (Part 1)

by Andy Oram

Mailing list membership is a must for anyone who works with
complicated computer technology, such as a programming language, a
server, or a professional software package. On popular mailing lists
for difficult topics, such as Linux distributions, messages stream in
at every hour of the day and night.

How many messages on these lists get satisfactory answers? How long
does it take to resolve the questions? These are just two of the
simpler ways to measure a mailing list's effectiveness (we will
encounter others as we proceed).

In a series of blogs I'll present the results of a modest research
project of mine to measure the effectiveness of two mailing lists,
which will be the start of what I hope to be a larger study.


Karl Fogel
2006-08-23 17:08:26
It's unfortunate that some of the most important things about mailing lists are also the hardest to measure. A great thing to track would be the audience's learning rate, for example, and one way to measure that would be to track how long it takes for "the same" question to get answered each time it appears. I put "the same" in quotes because, of course, a human would have to classify when a question is the same as some question already seen on the list.

Related to this number would be the FAQ generation rate: is the information produced on the mailing list being organized into more permanent reference forms, such as a FAQ? If so, how many times must a question appear before it gets into the FAQ? After that, how often is the new FAQ item's URL given as the answer to questions on the mailing list?

As long as random musings are permitted: I wish there were a way to subscribe to a thread, rather than a whole mailing list. Any entity that goes through state changes should be subscribable, whether a mostly static web page, a wiki page, a ticket in a bug tracker, or a thread in a mailing list. If I see a thread on a list, and it doesn't get resolved, I can at least subscribe so that a) if an answer does appear, I'll be notified, or b) if I stumble across the answer later, I can easily post the answer to the interested parties. Of course, b) is already possible, but our interfaces think in terms of mailing lists (which are really just filters to keep certain threads grouped together) and therefore thread-based manipulation is still primitive. The thread, not the list, is the fundamental unit of discussion, but our tools are only beginning to adjust to this.