The Bacula Philosophyby Nathan Valentine
An often overlooked but essential ingredient of any smooth-running IT environment is reliable data backup and restore capabilities. Thankfully, the free software/open source community has provided several centralized over-the-network backup and restore solutions. Dan Langille covered the technical details of one such solution, Bacula, in Bacula: Cross-Platform Client-Server Backups. Those just discovering Bacula will find that its features are very competitive with those of other proprietary and FLOSS backup solutions. Bacula is particularly capable in multiple-platform environments where flexibility and custom scripting are essential.
Nathan Valentine recently swapped email with one of Bacula's founders and head developers, Kern Sibbald, to discuss the history, goals, and upcoming features of the project.
Nathan Valentine: Kern, please tell us about how the Bacula project started. What drove you to decide to write your own network backup solution?
Kern Sibbald: Believe it or not, it all began with the October 1998 Microsoft "Halloween" document that was leaked to Eric Raymond, who then published it on the internet. At that time, I was "retired," but still using a Sun Solaris 2 Workstation and a Win 95 machine. I was looking for a Unix solution on a PC because I was not happy with Windows, and Minix did appeal to me for a number of reasons. When I saw that Microsoft was concerned about Linux, I realized it was serious. After looking around a bit, I chose the Red Hat distribution (version 5.2, if I remember correctly). Red Hat Linux really opened my eyes to what open source software was.
Until late 1999, I was busy learning Linux, setting up a server, and delving into all the new "toys" (Apache, Perl, DNS, ntp, etc.), when I started looking for a serious project and sent a few ideas to John Walker asking his opinion. He responded with a number of ideas, but on the top of his list was a network backup solution comparable to Legato NetWorker. He had attempted to use Amanda but gave up, if I remember correctly, principally because it could not span tapes.
Just about that time, the motherboard and hard disk of my main development PC were fried by a lightning strike, wiping out a month or so of work, so a backup solution interested me a lot. I looked at Amanda and afbackup and a few other Perl solutions, and decided to write my own program. This was in January 2000.
Nathan: What were the original goals of the project?
Kern: The original goals (and they are still valid) were to write a network backup solution that could back up any client from a Palm to the largest mainframe computers with similar functionality to a commercial package such as NetWorker. Also, we made a commitment to upward compatibility of backup media. DLT tapes have a retention time of 30 years, and it's only reasonable to expect your software to be able to read them if the hardware can--the software problem is a lot easier to solve! This seems totally obvious to John and me, because Univac programs still run 30 years after they were last compiled, but it may be a revelation to younger programmers. A backup solution represents a potential multigeneration undertaking between those who provide the solution and those who depend on it to preserve their intellectual capital. An open source solution is best in such an environment, because there's no way the collapse of a vendor or the discontinuing of a product can orphan your backup media containing all of your data.
Nathan: Did you start the Bacula project alone or with a team?
Kern: John and I worked on it together, with John concentrating mostly on the Storage daemon and me on the other aspects. However, quite early in the project, John was overwhelmed by other projects and responsibilities.
Nathan: The more familiar one becomes with Bacula, the more apparent it is that a considerable amount of design went into the architecture: a centralized Director, distributed Storage daemons, distributed Client/File daemons, a flexible scheduler, and the ability to tie into multiple SQL database back ends. Is the architecture based on experience with other network backup solutions?
Kern: The idea of a Director, Client/File, and Storage daemons seemed rather obvious to me at the time, probably because of my long computing experience. John Walker contributed a lot to the early design process as well. NetWorker, or actually Solstice Backup 5, introduced me to the concept of pools and retention periods. However, their concept of savesets, pools, and storage node affinities goes way beyond my ability to understand it. At that time, I already had a lot of experience with simple nonnetworked mainframe backup programs. My first professional job was maintaining the backups for the University of Maryland's computer science Univac computers. This was long before Amanda.
I did the main architectural design while on vacation in a hotel room in Italy that was previously a convent. I remember lying in bed looking out on the lake a few meters away through vaulted arches, and the ideas on the separation of the daemons and how they communicate just started streaming in.
Nathan: As a quick aside, I hope you see the humor in the situation that you just described: the inspiration for daemon-to-daemon communication in a system called Bacula coming while vacationing in a former convent. ;)
Kern: Yes, pretty weird.
Nathan: Were there weaknesses in existing network backup solutions that you were hoping to address? New ideas that you were hoping to bring to the table?
Kern: The main weaknesses of the two existing open source network backup solutions in early 2000 were: an inability to span multiple volumes in Amanda, and a license that forbade porting the server portion of code to Microsoft in afbackup. The afbackup license has since been changed to remove this restriction. I made a number of design decisions that were difficult (perhaps unpopular or risky, or had performance implications), but with time they have proved to be good choices, at least for Bacula:
To make Bacula a network backup solution. That is, all communications between the five components (Director, Storage and File/Client daemons, Console/GUI, and Catalog Database) are done over the network.
To use an open source SQL database engine rather than write my own indexing scheme, which would be faster but surely less flexible.
To centralize all the control in the Director. Clients do not define their own FileSets or schedules.
To make Bacula threaded rather than using independent processes.
To use a custom but well-documented backup format.
Each of the above was a nonobvious decision, at least for me, and each posed its own hurdles.
Nathan: Let's talk about the Bacula development team. From looking at the web site and mailing lists, it appears that Bacula is completely volunteer driven. Can you give us a run-down of the major components, the primary developers, and how development is managed? For instance, is it similar to Linux kernel development, where many people contribute code, perhaps specific to their area of expertise, and you are responsible for blessing releases? Or more like the FreeBSD "core" model? Or perhaps you write most of the code and others submit patches as they can?
Kern: Yes, Bacula is completely volunteer driven. In the early stages in 2000, John Walker contributed to the design and the code. Then from about mid-2000 until the first public release in April of 2002 on Source Forge, I did all the programming. Aside from the SQL database engine, I tried to write most of the code myself, but owe a lot to being able to look at open source code as well as incorporating a bit of it in Bacula. Until recently, the way it worked out was that I would write most of the code and others would submit patches. However, in the last year, there have been some very significant contributions by other developers many of whom commit directly to the CVS. Some of their contributions are:
a PostgreSQL driver
the wx-console GUI console program
the GUI tray-monitor applet
a bimagemgr program for backing up to CDR media
a native Windows Client/File daemon that is no longer dependent on Cygwin
Unix Access Control List support
release builds including RPMs and debs
I am responsible for reviewing all commits and blessing all releases.
I suspect that the Bacula model is more like Linux kernel development than FreeBSD, but I would be hard-pressed to compare it to either one of those projects.
Nathan: One area in which Bacula excels, and especially in regard to many other open source software projects, is the documentation. The documentation is so thorough, you might conclude that as much time has gone into documentation as development. The Bacula web site has links to a fantastic manual and to a "beta" manual that is actively being updated. Who is responsible for keeping the documentation current?
Kern: Thanks for the compliments. I am responsible for the documentation, and generally spend at least 20 percent of my time on documentation.
Nathan: What advice would you give to other open source software projects about how to generate and maintain good documentation?
Kern: If you want people to use your software, and if you don't want to spend all your time on repeating the same support responses, you need to write lots of clear, detailed documentation. When I respond to a support request or a problem, I often put the response into the manual, or I update the manual to include the problem. Documentation is not a task to be left to the end of the project or just before the release. It is an ongoing "project" in itself--at least that is my view of it.
Nathan: This is an interesting take on the role of documentation. In addition to being helpful to the user, it also helps the developer spend more time developing and less time dealing with usage questions.
Kern: I think most of the large projects understand this, but a good number of smaller ones, unfortunately, do not.
Nathan: Back to the technical side of things. Can you elaborate on the road map for upcoming releases of Bacula? What features and improvements are you most excited to see going into the next release?
Kern: In general, Bacula is starting to move into the enterprise environment, which means mainframe servers, multiple simultaneous copies, Redundant Array of Inexpensive Tape (RAIT), multiple drive autochangers, and the like. The main features planned for the next release are:
Migration of backups; i.e., moving a backup from one volume to another, copying it, or archiving it
Embedded Python scripting with "events" in Bacula that give control to the script; e.g., start of Job, end of Job, and need new Volume
Allowing the user to specify multiple storage devices for a single job; Bacula will select the first free device
Allowing a single Job to back up the same data to multiple devices (two or more copies) at the same time
There are lots of other smaller projects that are being implemented, such as ANSI labels, support for SQLite3, Mac OS X resource forks, support for ACLs on systems other than Linux, multiple drive autochanger support, and adding an index to the documentation. I try to spend 10 to 20 percent of my effort for each release, in addition to the main projects, on adding new features requested by users and on cleaning up the code.
Nathan: In regard to design, implementation, and documentation, Bacula has a strong foundation. For open source hackers who are interested in the project, in what areas would you most like to see new members contribute?
Kern: Though not essential, it is helpful for a developer to have some knowledge of C++. Pretty much every area of Bacula has potential for willing contributors, depending on the experience of the developer. I'd suggest that those who are interested should look at the files projects and the kernstodo file in the main source directory to get an idea of tasks I have noted; these files by no means document all possible contributions. Then read the developer's chapters of the manual. Finally, submit ideas or indicate your willingness to contribute to the bacula-devel list or directly to me.
Areas where help would be particularly welcomed are:
improving the regression scripts and testing
Windows programming and support in general
adding support for the Windows VSS API (Volume Shadow Copy--back up open files)
working on the Bacula GUI programs
documentation of new features and enhancing the manual (isn't this the dream of every programmer?)
Nathan: Thanks for taking the time to talk with us, and good luck with the next release of Bacula. I'm certainly looking forward to it. :)
Kern: Thank you for your interest in Bacula and for giving me the opportunity to talk about its origins and goals.
I've enjoyed working on Bacula for the past five years. It has been by far the most diverse, challenging, and interesting project I have ever worked on. There is no doubt that I will remain with the project for quite some time to come, but I also welcome and look forward to the continuing evolution toward more and more participation from other contributors.
Note: Some of the names mentioned in this interview, such as Bacula, Legato NetWorker, Linux, Mac OS X, Microsoft, Red Hat, Solstice, Univac, and possibly others, are trademarks of their respective owners.
Return to the Linux DevCenter.