Better Books through Literate Programming

by Ben Hammersley

The moral first: Knuth Knows All.

It's fun to see how we move on. I'm writing two books at the moment and preparing two more. One of these is the sequel to my Content Syndication with RSS, that I wrote two years ago. During that time, I was in London, and for most of book - until a total hard drive failure pushed me over the edge to Apple righteousness - working on PC. An enormous tower with about six fans: it was like writing inside a busy hairdressers.

Anyway, I'm now in Italy, on a Mac, and infinitely more organised, more prepared, better than I was then. And more aware of the process of writing one of these sorts of books. For me, at least, the average syndication specification document is not interesting enough to dedicate a few months of my life to rewriting in O'Reilly style. But using that chapter as an excuse to hack on a load of problems and serve them up as examples most certainly is. Anyone who starts writing one of these books understanding everything about the subject, and knowing exactly how to code the examples, is a fool. And, frankly, nowhere near ambitious enough. As Disraeli said:

The best way to become acquainted with a subject is to write a book about it.

But, with so much work to be done, I've been drawn to thinking about the process of writing. What is it that I like to do and what things are drudgery? Talking this through with some other authors, I find some agreement: you spend so much time working out the solution to a problem, that by the time you come to write it up, you're both bored with it and resentful of the future readers for not knowing the answer themselves. The fun leaves at this point, especially as you then have to wrangle with the obligatory and baroque Word template thrust upon you by the publisher. O'Reilly aren't the only ones to do this.

The answer, at least it seems for me, lies in Donald Knuth's concept of Literate Programming

Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.

The practitioner of literate programming can be regarded as an essayist, whose main concern is with exposition and excellence of style. Such an author, with thesaurus in hand, chooses the names of variables carefully and explains what each variable means. He or she strives for a program that is comprehensible because its concepts have been introduced in an order that is best for human understanding, using a mixture of formal and informal methods that reinforce each other.

Literate Programming isn't a new idea - Knuth's paper quoted above is from 1984. But it is a powerful one. And reading through it, I've developed my own system of documenting the scripts as I go using POD. (I'm a Perl guy. I know, I know.)

The POD is more than extensive commenting, because you can give it some form of structure, and by using the myriad POD modules on CPAN, can parse it into pretty much anything. This means that I now have a suite of scripts that check my code, check the POD to see that everything in the code is written up, tidy the code, format the whole thing nicely and then dump it into an OpenOffice document, with the correct O'Reilly template already embedded. This is heaven.

For sure, POD is Not Literate Programming, but it's close enough, and when jiggled around with, is exactly what I need to go from concept to manuscript with as little fuss as possible. The ease in workflow alone probably saves me half an hour a day - but more to the point, the enforcement of a certain thinking pattern makes writing scripts for books (a different thing, I would argue, than scripts for real world use) all the easier.

I'm rather excited about the way this is working out, and the increase in productivity it's giving me. I'll let you know how it evolves, but in the meantime, is there anything else I could be doing to ease the way from brain to paper? Use the Talk Back below and share your ideas...

Do you automate your writing in any way? Is there anything else I can do to ease my efforts?


2004-08-16 09:21:58
Docbook Lite
I agree with your sentiments but was fortunate enough not to have to struggle with Word to get there.

Learning PHP 5 and PHP Cookbook were both written in Docbook Lite.

I accumulated a bunch of elisp macros to help with some of the tagging, and was able to write simple programs to automate tasks like generating the downloadable code example archive.

2004-08-16 22:01:53
POD Works for Books Too

We used POD for BSD Hacks as The Making of BSD Hacks explains. It's worked even better on subsequent projects.

2004-08-18 23:06:44
Leo is an interesting project in the veign of literate programming: