Editor Andy Oram on O'Reilly's First

by Tara McGoldrick Walsh

Related Reading

Harnessing the Power of Disruptive Technologies
By Nelson Minar, Marc Hedlund, Clay Shirky, Tim O'Reilly, Dan Bricklin, David Anderson, Jeremie Miller, Adam Langley, Gene Kan, Alan Brown, Marc Waldman, Lorrie Faith Cranor, Aviel Rubin, Roger Dingledine, Michael Freedman, David Molnar, Rael Dornfest, Dan Brickley, Theodore Hong, Richard Lethin, Jon Udell, Nimisha Asthagiri, Walter Tuvell, Brandon Wiley

McGoldrick: This is O'Reilly's first book on peer-to-peer (P2P) and, indeed, the first book on P2P by any publisher. How did you approach such a new and revolutionary topic?

Oram: With a topic that's so fast-moving and--no way to deny it--poorly defined, I wanted to get a lot of opinions from a lot of knowledgeable and thoughtful people. They came through with flying colors. I also was determined to cover the field on many different levels: the kinds of problems P2P could solve, the kinds of problems P2P raises, its impacts on users and businesses, and so on. I really had to jump on the topic and let it carry me where it wanted; not try to capture it and cage it and wrap it up with a pretty bow to deliver it.

One critical goal stayed uppermost in my mind. I knew the field was loaded with hype. I also knew there was something significant going on behind the hype, and that O'Reilly was the best publisher to show the public that significant core. While I wanted to talk a bit about collaborative networking's social meaning and its potential impact on people, I knew that was good for only a few dozen pages worth of text. The rest had to be solid technical issues and solutions.

McGoldrick: P2P gets a lot of criticism as well as hype. In fact, just last week the P2P concept got slammed twice on the same day by major writers and publications: Lee Gomes in the Wall Street Journal (reprinted on ZDNet and on MSNBC) and Jon Katz on Slashdot.

Oram: I guess we should get used to Internet time, and be thankful at O'Reilly that the book had five weeks to circulate in a relatively positive medium before the sharks converged. Of course, neither article criticizes the book. Gomes doesn't mention it at all. Katz actually writes that "Peer-to-Peer: Harnessing the Power of Disruptive Technologies does a great job of explaining how P2P works." He questions not the book's quality, but its relevance. I respect what both these writers have said, and I do have comments that put their questioning in perspective. Clay Shirky also offers a valuable and balanced response, Backlash!, on O'Reilly's openp2p.com Web site.

The Wall Street Journal is responsible for letting its readers know where to invest and what the prospects are for success in a given industry. Gomes must feel acutely the risks his key readership is facing, from venture capitalists through pension fund managers. He has to let them know that P2P is no "exception to the dot-com downturn." Definitely a valid concern, but it's only one aspect of a complex field.

I have to admit that I'm not a regular WSJ reader, but I trust it will also tell another side of the story: not just whether businesses can succeed at creating P2P applications, but about the benefits and drawbacks of businesses using P2P applications. There's a lot of intriguing potential in P2P, along with organizational and funding issues that businesses should start considering.

And while I wouldn't tussle with Gomes's assessment of investor prospects, I have talked to lots of business people creating P2P applications. They have strong stories to tell. You can read about them by visiting my P2P Profiles on openp2p.com. I haven't invested a dime in any of these companies, and I don't consider myself an investment expert. But if you come to the site every week or so and see who we're interviewing, you'll have a chance to judge whether the idea of P2P is fertile.

For technical, analytical, and news coverage of peer-to-peer technology, visit O'Reilly's openp2p.com Web site.

On to Jon Katz. He has some of the same "show me" attitude as Gomes--a healthy attitude--but he also stakes out the strange claim that not many readers will use P2P. To declare that an idea has no practical application is the most dangerous kind of technology prediction--all the more so when, as Katz accurately points out, the major pieces of robust P2P infrastructure haven't fallen into place yet.

We have to remember (as some of the respondents on Slashdot remembered) that lots of great computing ideas have entered everyday use under thick layers of simplifying technology. How about the complicated mathematical graphical functions that are used to paint our monitors (and the psychological theories of researchers such as Ben Shneiderman, which informed the GUI revolution)? How about structured programming, which harks back to Edsger Dijkstra's "Go To Statement Considered Harmful" letter in 1968 and was generally ignored by programmers, but now underlies point-and-click component technologies used by thousands of Visual Basic users every day? [Editor's Note: A reprint of Dijkstra's letter is available online.]

By the way, Katz's claim that "In most of the world, inventors identify a need and wear themselves out creating innovations to meet it" is directly refuted by the famous book Guns, Germs, and Steel (by Jared Diamond, 1999). Diamond states that most inventions start as tinkering and take a long time to become useful. I think P2P is one of those slowly unfolding advances that will have repercussions on how people work and interract.

2001 P2P Industry Overview: An O'Reilly Research Report is a comprehensive look at peer-to-peer from business and technical perspectives. The report, available at the end of May, includes company profiles, in-depth explanations of the underlying technologies, estimates of mind share and community size, and our forecasts.

McGoldrick: Back to the book itself, why did you decide to publish an anthology as opposed to, say, a Definitive Guide to P2P or P2P in a Nutshell?

Oram: The field of peer-to-peer, which is at such a formative stage, requires a different approach. First, there's no single path to developing or deploying a P2P application. You don't just follow the pull-down menus or even write programs following a strict sequence of API calls. There are multiple applications, multiple APIs, multiple levels to work at. So, instead of trying to fashion a step-by-step guide for a field that's not ready for one, we offer three kinds of information in the book:

  • Part 1: Historical and social context.

  • Part 2: Descriptions of real-life projects.

  • Part 3: Technical lessons drawn from those projects in such areas as performance, security, and accountability
  • .

There's material in there to let people develop new P2P projects, to evaluate the value or risk of existing ones, and to judge the viability of the whole endeavor.

Because the P2P terrain is wide open and multifaceted, it's worth hearing the different viewpoints of many productive people who have explored that terrain. I have wanted to let as many of them speak as possible.

Some people dismiss peer-to-peer as just a buzzword or too vague a concept. But what united all the projects in the book was the problems the project's leaders faced. Each one had to accomplish certain tasks to get their project off the ground. Let me tell you that every person I approached to contribute to the book (including the ones who declined because they were too busy) recognized the relevance of the topic to their work and the value of coming together to do this book. Some of them protested that their applications weren't true peer-to-peer (the Publius team, for example--a team that contributed two fascinating and fundamental chapters), but they all could still see how their research fit the book.

Luckily, O'Reilly had a precedent for an anthology about an emerging field: our Open Sources: Voices from the Open Source Revolution. That book, like Peer-to-Peer, contains far-ranging essays by a wide range of leaders (some of them quite well known) doing various work in a big space. In both books, the authors all worked together to define their spaces well.

McGoldrick: Why do you think this will be an important book?

Oram: I'm glad we're publishing this interview now, in the shadow of the media storm clouds concerning peer-to-peer, because now it would be silly for me to answer, "The book's important because of the hype around P2P." Instead, the book rests firmly on its solid analysis of an important technological movement.

I mentioned earlier that I've talked to a lot of people running companies that are trying to develop P2P products. Invariably they tell me they're reading this book.

The book tells people key information they can't get anywhere else about computing projects that are widely regarded as important models for other developers: Jabber, Groove, Gnutella, SETI@home, and so on. Some projects will stand and some will fall, but the technologies they're using and the things they're trying to accomplish will provide important lessons for the future.

The largest part of the book (Part 3), therefore, comprises technical topics, but the smaller social analysis section (Part 1) is valuable, too. For P2P to spread, there has to be a lot of changes in thinking and practice among businesses, IT staff, and users.

McGoldrick: The subtitle of this book is "Harnessing the Power of Disruptive Technologies." What do you mean by 'Disruptive'?

Oram: That's well worth an explanation. I did not choose this subtitle. It was chosen by marketing. In fact, I balked for a while at the subtitle, and a number of the authors protested the use of the word "disruptive." They were afraid that it would further swell the bad image that the legal controversy over Napster was casting on their own work. Still, calling P2P a set of disruptive technologies is extremely apt, and I'll be glad if that subtitle causes some less technically-minded readers in business or the general public to take a closer look at the book.

Clayton Christensen popularized the term "disruptive technology" in his book The Innovator's Dilemma. These are technologies that change how people and organizations do what they're doing day to day. To quote from my own text (in the "Afterword" to Peer-to-Peer), their "impacts can fundamentally change the roles and relationships of people and institutions." I expect that many P2P applications that are beginning to hit the market will fit that description.

Instead of people pushing documents to each other (often over email in a dozen evolving versions), they may sit at their PCs and see a relevant document come to them automatically. Instead of depending on some programmer at a central site to envision and code up a service, users may create it themselves using their own data. Groups may be able to form and regroup more spontaneously and efficiently, crossing organizational boundaries. There's a lot of potential forms of interaction ahead that can excite people willing to try something new--and scare people who depend on old ways of controlling the flow of information.

McGoldrick: How did you choose the contributors to this book?

Oram: O'Reilly tries to find and stay friendly with all kinds of people doing new and promising projects. So when we decided we should write this book, we drew on our contacts in the computing field, both academic and commercial.

Personally, I had contacts with Freenet creator Ian Clarke (who was too busy to write, so three other members of that team contributed chapters instead) and with Gene Kan of Gnutella fame. These contacts grew out of two popular articles I wrote last July, Gnutella and Freenet Represent True Technological Innovation and The Value of Gnutella and Freenet.

People we knew suggested others whose work they respected, and we gradually built up an impressive roster that includes writer Clay Shirky, product designer Dan Bricklin, privacy researcher Lorrie Faith Cranor, consultant and O'Reilly author Jon Udell, and W3C researcher Dan Brickley.

I don't want to suggest that the chapters by famous people are the most important ones. I like every chapter, and in fact some stunning contributions were turned in by people who were pretty unknown before the book was published.

Like most communities working together, we see each other all over the place. I started recruiting authors in August 2000; a lot of them were invited to the O'Reilly Peer-to-Peer Summit that took place in September 2000. We saw each other again at the first-ever O'Reilly Peer-to-Peer Conference and I've been in touch with many of them regarding other projects.

O'Reilly & Associates is dedicated to providing you with information about critical trends and innovations in computer technology. We're pleased to announce our second Peer-to-Peer and Web Services Conference (September 17-20, 2001, Washington, D.C.), an event exploring the technical, business, and legal dimensions of these technologies.

McGoldrick: Did you define topics for chapters and assign them to the writers or did you solicit writers and let them define their topics?

Oram: This was the most difficult and most inspiring stage in the development of the book. It was a very rich collaborative process--I have to break down and call it a peer process--because I knew certain topics were important but wanted to choose the right topic to excite each writer. I spent days and days talking to some of the authors in order to hone each idea into a topic that would be engrossing for readers, stimulating to the authors, and focused enough to fit in a single chapter. We also wanted to avoid overlap, so the writers talked to each other to establish boundaries between their chapters. A few reviewed each others' drafts.

McGoldrick: How did the finished book compare to what you had envisioned?

Oram: It astonished me. I gave the authors--all busy people with demanding projects to develop--just a month to write chapters and a couple more weeks to incorporate reviewer comments. But when the drafts arrived, I was taken aback by their depth, their comprehensive understanding of background research, their philosophical and historical richness--and often, the cleverness of their writing style. (Some authors required substantial rewriting, but the clarity and relevance of their vision was never in question.) This book is more than a snapshot of current work; it is a weighty statement about a field in rapid motion. Weight plus motion lends the book substantial kinetic energy.

McGoldrick: Because there is so much disagreement about what P2P is, was there a lot of passionate discussion about the content in the essays?

Oram: Certainly. As I said earlier, some contributors didn't really consider their technologies peer-to-peer, although their work was still highly relevant because the projects all tended to face the same problems. Many of the authors in Part 1 of the book grappled with "What is peer-to-peer?" And perhaps even more interesting, with: "What is the part of peer-to-peer that's worthwhile?" Authors didn't always agree. Clay Shirky has been evolving and refining his own answer to the question "What is peer-to-peer?" for almost a year.

Authors disagreed about minor technical points as well; their polite but candid review of each other's work definitely improved the book. By no means did we resolve all differences. To illustrate how diverse the viewpoints are, I'll reveal that Lucas Gonze, moderator of the Decentralization mailing list, tried near the end to pull together a glossary for our book. But the authors realized they were using the same terms in slightly different ways, so that for this edition at least, a glossary would be confusing rather than illuminating. Luckily, Gonze could publish his list of terms, which he calls a MemeBag, on O'Reilly's openp2p.com Web site.

McGoldrick: You recently returned from O'Reilly's first Peer-to-Peer Conference. Did you hear about any new developments at the conference that you wish you could have included in this edition?

Oram: Of course, I heard about lots of juicy projects that, in some ways, were going beyond what the projects in the book had achieved. Many of the authors who wrote in the book about the projects they were working on in October or November of 2000 are currently starting new projects. Most fundamentally, I realized that the field of P2P is moving beyond the heroic early-experimentor stage and is giving rise to hard-nosed, secure, scalable products.

McGoldrick: The Peer-to-Peer book hit bookstore shelves in March, but a small print run was made available at the Peer-to-Peer Conference. How was it received?

Oram: People were very glad to have it, naturally, and the book required only a few hours to sell off. (We put out part of the stock on the first day of the conference, and the rest on the second.) We wish we had more books, of course, but the print schedule was so tight we were lucky to have the 200 that we got. At any rate, we didn't expect to get 1,000 people at the conference! (Not counting the ones we had to turn away.)

It was quite a trip sitting at a row of tables with about a dozen of the authors, watching conference participants troop by to collect signatures from each in turn. And the authors were even more excited than the other attendees--excited to see the book and excited to be able to meet each other in the flesh.