OSCon: Developers and testers as heroes

Email.Email weblog link
Blog this.Blog this

Andy Oram
Aug. 03, 2005 11:38 PM

Atom feed for this author. RSS 1.0 feed for this author. RSS 2.0 feed for this author.

URL: http://conferences.oreillynet.com/os2005/...

The software developer is the hero of our day. This is not only because of what developers have accomplished, but because of their integrity and transparency. They can't fake what they do.

A developer can't make his code perform better by ingesting dangerous medicines. He can't improve benchmarks by carrying out shady accounting deals. He doesn't get anywhere by wearing glitzy clothes and appearing on TV shows after two-hour make-up sessions. He's not likely to make his project progress by splitting and polarizing a population with appeals to deep-seated fears. We're disgusted by all the other public figures society has thrown at us; only software developers are pure.

The developer as hero was celebrated several times over in the morning keynotes at OSCon. Tim O'Reilly and conference coordinator Nat Torkington shared a presentation celebrating the progress of programmable Web services, popularly called Web 2.0. Tim pointed out that major Web sites such as Apache, Google, and eBay have provided programming interfaces (Web Services) for a while, but that now we're seeing "the first true Web 2.0 applications"--meaning applications that create something new rather than just enhancements to the underlying web site. He illustrated the trend with a hack that combined craigslist with Google maps to show real estate offerings for given neighborhoods.

But another set of heroes emerged in the next keynote, given by Kim Polese of SpikeSource.

SpikeSource and the ascent of testing

Polese started with some crackling one-liners illustrating the trend she's seen in the evolution of open source: From DIY to DIT (do it together). From Egosystem to Ecosystem. From an architecture of participation to building of participation.

Basically, Polese said businesses have come to accept open source and are performing their own integration in order to get it to work for them. She recognized and articulated a concern I've had for some time, and which by chance I tried to turn into a blog last week but never quite pulled together. So I'm going to plunk down that not-yet-published blog right here and then return to her view of the situation.

The lament of the Linux knowledge worker

I've been living with and struggling with a mental disconnect in the free software arena for several years.

On mailing lists, news sites, and trade journals I read glowing reviews (or anticipations) of wonderful new packages and support for the most advanced computing uses. Then I return to my little desktop systems, where my wireless card and power management still don't work, my GNOME desktop often thinks my left-most mouse button is depressed when it isn't (leading to wild errors), and remote file transfers have been crippled by an apparent bug in the latest distribution. (I'm always trying new distributions in the hope of finding a happy moment where some version has all the critical functions I need. The closest I came was a year ago when I paid $99 for the first version of Red Hat Professional Workstation.)

These are just the problems that hold my productivity back every day; there are others I haven't bothered to list.

I'm not surprised at the configuration problems on my system, and I don't blame anybody. It's the fallout from open source development. I'm getting the kernel from one project, the desktop from another (both KDE and GNOME--I'm desktop fluid), drivers from still other projects, middleware such as the ESD sound daemon from their projects, and of course, applications from their various producers. No matter how stable and carefully developed all this software becomes, getting it to work together is hard.

GNU/Linux is the most volatile of the major operating systems. That's partly because it's the youngest (certainly compared to Unix and BSD, and even compared to NT and its derivatives). But it also attracts a user community that's always pushing it ahead; this is what brought it to the astonishing success it has today.

Most major distributions have come to face the tension that rapid development produces between the goal of widespread adoption and the goal of continuous improvement. Debian split into a stable and unstable distribution, Red Hat split into Red Hat Enterprise Linux and the more community-focused Fedora project--even Gentoo is considering an Enterprise version.

Maybe I should just go ahead and pay $99 a year for a commercial distribution, but I think there would always be something out of sync. You can never rest with the system you've installed: before the CDs were even shrink-wrapped, some of their contents had become out-of-date, and the user must routinely update a system for the sake of security as well as functionality.

In his keynote today, Andrew Morton (former Linux kernel maintainer) pointed out that the readiness of Linux for the desktop is routinely underestimated because the people who evaluate and report on it are the power users or knowledge workers. Naturally, they have the most varied needs and are the hardest to please. He repeated the generally accepted view (see, for instance, my report from the Desktop Linux Conference) that desktop Linux will start with kiosk application, move up to people using one or two major applications, and gradually continue up the knowledge chain as it appears in more and more places and companies have reasons to enhance it.

So I wonder whether the computer industry will pass through a period during which people see it as prestigious to have a PC running Windows. ("I see you're stuck with Linux, Jones, but after all, you don't have documents to post through SharePoint.")

More on Morton's talk later in this blog.

The SpikeSource approach to free software inconsistency

Large organizations like open source, but they find that it takes a lot more internal support, especially for precisely the problems I encountered on my desktop: the integration of components that weren't previously tested together. According to Polese, companies are internally creating "formalized processes" such as review boards, support desks, and incubation centers. But it's time-consuming and takes away attention from whatever their core business is.

SpikeSource aims to improve the situation by recruiting volunteers to do this drudgery, and providing better tools to help them do it. To see whether they were really following through on this tall promise, I visited their demonstration center, which was quite fancy (it shouldn't be a surprise, given that they're partnering with O'Reilly on CodeZoo).

Their work did look pretty spiffy. Their download and configuration utilities for selected stacks of products, such as the servers and language tools in a LAMP system, were very easy to understand and well integrated. Their automatic test generation utility for Java code did a very full job (producing comprehensive code coverage in a set of XML files) but it seemed to me it could serve only as a first step for programmers who customized the XML to run the tests they really need.

More keynotes

The morning was rounded out by several other presentations, each interesting for very different reasons.

Andrew Morton

Morton gave a short speech in a manner so low-key and unprepossessing that one could miss his irony and incisiveness unless one listened carefully. He struck a balance between praising the contributions of the large companies participating in OSDL (the company that pays his salary, as they pay Linus Torvalds), and acknowledging the importance of the wider free software community. The latter folks are often suspicious of OSDL, rarely for any well articulated reason. Morton labeled the OSDL member companies "blundering elephants" but insisted they were dealing sensitively with the Linux community.

The same winning cynicism came out when he explained why Linux was spreading everywhere. He was almost dismissive is saying the operating systems and other infrastructure had become "commoditized," that their implementation was "common knowledge," and that they "have no intellectual property value left in them." (So why was there so much argument about implementing features at the Ottawa Linux Symposium I reported on two weeks ago?)

He also claimed that free software developers are "significantly better programmers on the whole" than their equivalents in proprietary software companies. But he pointed out that OSDL member companies contributed very important developers to Linux.

He made three major comments on the development process. (He wrote me email to help clarify his statements.)

  • First, any major kernel developer practically has veto power over any decision, but they manage to move ahead nevertheless.

  • Second, he validated the complaints of some people in the wider community that Linux was being loaded down with enterprise features.

  • He said that the free software community was not good at collecting input from end-users (by which I assume he meant unsophisticated end-users among the general public), and that companies had processes in place for doing that.

Jeremy Zawodny, Yahoo!

Jeremy gave a brief talk on two themes: how Yahoo! makes use of free software, and what it's doing these days.

The role of free software at Yahoo! is pretty much the same as in other organizations. They have lots of servers, and lots of desktop users running the usual office suites and other things. The servers mostly run FreeBSD, but they use Linux to run commercial software packages because these tend to be ported to Linux rather than FreeBSD. Jeremy reeled out the usual list of benefits of using free software: flexibility (it's easy to change), quality (it's reliable), good documentation (including often the source code), availability, and support on community forums.

He said not to worry whether free software is scalable. Yahoo! is handling billions of page views per day, and a hot item can get a million views in the hour after it first goes up.

He also promoted Yahoo! for opening up its APIs with the goal of becoming a full-fledged participant in Web 2.0. The Yahoo! Developer Network and Yahoo! Widgets (originally Konfabulator) are the main exhibits.

Jonathan Schwartz, Sun

I was nervous about attending Schwartz's presentation, partly because I've been annoyed by a lot of his published statements (as I pointed out in one blog which happened to be written when I was feeling relatively kindly toward Sun) and partly because I was afraid of what the audience would do to him.

Everything turned out fine: Schwartz was personable and natural-looking, and Nat Torkington carried out an informative interview. Nat gave Schwartz plenty of room to make his points, while lassoing him on his occasional slipperiness, such as sliding from discussing "free as in speech" software to "free as in beer" software. Another example was an exchange about software patents, where Schwartz played to the audience's abhorrence of the patents (saying they're "largely inapplicable") while allowing that patents could be used both to promote innovation and to squelch it.

I would have liked an exploration of Sun's relationship to the OpenOffice.org project and their controversial decision to include some Java code in it.

I was impressed with Schwartz's defense of Solaris. He said Solaris should not be compared to Linux (a software project) but rather to particular distributions of Linux being offered in the marketplace. I can acknowledge his proud claim that OpenSolaris could provide competition that spurs other free operating systems to improve. He also vigorously defended OpenSolaris's CDDL license, saying their license is open to linking with other software and that it's the GPL that it is prohibiting collaboration. And he claimed that a large community of contributors were flocking to OpenSolaris.

He said that "just because the software or service is free doesn't mean it doesn't have value," and pointed out that it can provide value by becoming a substrate for support, update networks, and ads. He also boldly declared that "ultimately, every single product at Sun will be free and open"--but of course, his definition includes the current state of Java.

Notes on other sessions

I got around to at least parts of a number of sessions today.

Theo Schlossnagle on database replication

During the part of the talk I attended, Schlossnagle made the interesting point that database administrators really want to replicate only the results of an update, not the update itself. He used, as an example, a complex query that involves a huge search bringing in multiple databases, potentially requiring hours--but ending up with a change to only five records. I didn't stay to find out whether he had a way to propagate the five changes rather than to rerun the update.

Ted Leung on Chandler

I sat with about 18 attendees for the beginning of Leung's demonstration of how to build a new component (called a "parcel," just to be different, I suppose) for the emerging Chandler office suite from Mitch Kapor's Open Source Applications Foundation. I didn't find out the answer to the main question I've had all along about Chandler--why? Why start a whole new project from scratch to produce yet another office suite?

I can guess the answer, though, from the little that I saw. I imagine that anyone who learns Python could extend Chandler (the point of the talk) and therefore provide new features to the community fast. The parcels are supposed to allow easy integration with Web Services and other web sites. Chandler's visual components remind me of XUL.

Some general notes

Various O'Reilly staff and Safari staff held well-attended sessions today. I was pleasantly surprised by the interest attendees had in our work and thinking. But I realized it made sense, given the strong interest in search engines nowadays. The very same needs--that is, the aspiration toward reliable sources of knowledge in this chaotic information world--also leads people to look for high-quality publishers.

But there are others making advances in the publishing industry. One up-and-coming organization is Osoft, makers of ThoutReader, whose extraordinarily clever designer Gary Varnell posed with me in a photo today.

Andy Oram is an editor for O'Reilly Media, specializing in Linux and free software books, and a member of Computer Professionals for Social Responsibility. His web site is www.praxagora.com/andyo.