Creating a Data Mirror

by Dru Lavigne

Related link:

On Saturday, I'll be giving a tutorial on "Installing, Securing and Maintaining FreeBSD Servers" at meetBSD 2005 in Krakow, Poland. The gist of the talk is how to install the minimum possible required to keep a server operational and fully patched.

Since this is a hands-on tutorial and we weren't sure how reliable the Internet access would be in the lab, I promised to mirror the necessary files on my laptop. My personal goal was to mimic Internet availabilility within the classroom environment. For example, pkg_add -r should just work; so should cvsup.

I'm providing my working notes as they may be of use to others who find themselves with multiple FreeBSD systems behind a slow or intermittent Internet connection.

On the system hosting the files:

I initially experimented with /usr/ports/net/cvsup-mirror, but it seemed overkill for my purposes. (Michael Lucas wrote a how-to on this utility in his Big Scary Daemons column.)

My laptop already uses cvsup to keep the entire ports collection and all src up-to-date. Meaning, I already had all the tools I needed to create a mirror without installing additional software.

In addition, the other FreeBSD systems in the lab will only require the minimal ports tools but will need full src in order to rebuild world and compile a custom kernel. I wanted to be able to easily control which files students could receive via my mirror.

I started by creating a directory structure to hold the 2 required mirror config files:

# mkdir -p /usr/meetbsd/sup/test
# cd /usr/meetbsd/sup/test

# vi releases
cvs list=list.cvs prefix=/usr

# vi list.cvs
upgrade ports/Mk
upgrade ports/Templates
upgrade ports/Tools
upgrade ports/Makefile
upgrade src

Note that releases should be appropriate for any site wishing to mirror files contained somewhere within /usr. list.cvs should be customized to
indicate exactly which subdirectories you wish to be available via cvsup.

To make the mirror available, simply refer to your base like so:

# cvsupd -b /usr/meetbsd

On each system to receive the files:

Prepare the cvsup client. Note that the supfile is similar to the examples found in the handbook except for the sections I've bolded. You'll want to replace IP_address with the IP of the system hosting the mirror:

# pkg_add -r cvsup-without-gui
# mkdir /usr/local/etc/cvsup

# vi supfile
*default host=IP_address
*default base=/usr/local/etc/cvsup
*default prefix=/usr
*default release=cvs delete use-rel-suffix

You'll also want to prepare the missing directories:

# mkdir /usr/ports /usr/src
# chmod 755 /usr/ports /usr/src

Then, run cvsup:

# rehash
# cvsup -L 2 supfile

There you go. You now have your own customized cvsup mirror. If you ever feel like adding or removing available files, simply edit your list.cvs.

Hosting Packages

Next, I wanted to prepare my laptop to host the packages students would need in the class.

On the system hosting packages, create the packages directory:

# mkdir /usr/ports/packages

Then, make the desired packages:

# cd /usr/ports/misc/cvsup-without-gui
# make package-recursive

Note, you'll have to make deinstall first if this application is already installed on the system on which you're building the packages.

Once you've installed the necessary packages, you'll want to configure anonymous FTP since pkg_add connects to FTP servers. This can be easily done using sysinstall:

Anon FTP
ftp (group)
/usr/ports/packages (FTP root directory)
remove upload subdirectory
21 (for group ID)
No (to welcome message)

When you're finished, start the FTP server in daemon mode, for anonymous access over IPv4:

# /usr/libexec/ftpd -D -A -4

Note: I haven't demonstrated how to create a secure FTP server. These directions are suited for internal LAN use where FTP is not allowed through a firewall. If this isn't your scenario, read up on how to secure your anonymous FTP server.

Accessing the package repository

On the systems you wish to install packages, change the default environment. Again, substitute IP_address with the IP address of the system hosting the packages:

# setenv PACKAGESITE ftp://IP_Address/Latest/
# pkg_add -r cvsup-without-gui

Instead of going on the Internet, your packages will install flawlessly from your own package server. Since you created those packages using the make package-recursive command, all dependencies are taken care of for you.

Well, I need to finish packing for the trip. I'll be blogging about meetBSD as I get a chance. I'll also let you know when the PDF for the tutorial is available, as well as the slides and PDF for the talk I'm giving on BSD Certification on Friday.