O'Reilly Hacks
oreilly.comO'Reilly NetworkSafari BookshelfConferences Sign In/My Account | View Cart   
Book List Learning Lab PDFs O'Reilly Gear Newsletters Press Room Jobs  

Buy the book!
Linux Server Hacks
By Rob Flickenger
January 2003
More Info

Keeping Parts of Filesystems in sync with rsync
Use rsync over ssh to mirror exactly what you want, to any number of servers
Listing: Balance-push.sh
[Discuss (1) | Link to this hack]

Listing: Balance-push.sh


# balance-push - Push content from the master server (localhost) 
# to multiple front- and back-end servers, in parallel.

# $FRONT_END lists the servers that receive the front-end (e.g. static content) updates.
FRONT_END=$(cat /usr/local/etc/servers.front)

# $BACK_END lists the hosts that receive the full back-end (e.g. everything) updates.
BACK_END=$(cat /usr/local/etc/servers.back)

# $TARGET specifies the filesystem root on the remote host to push to.
# Normally, you want this to be /, unless you're doing testing.

# $EXCLUDE specifies the prefix of the per-mode rsync exclude files.
# For example, if your exclude files are /usr/local/etc/balance.front and
# /usr/local/etc/balance.back, set this to "/usr/local/etc/balance". The 
# per-mode extensions will be added.

# $LOCK_DIR specifies a path to put the lock files in.

######## Ignore the shell functions behind the curtain. ########


lock (  ) {
local lockfile="$LOCK_DIR/balance.$1.lock"
if [ -f $lockfile ]; then
if kill -0 $(cat $lockfile); then
echo "$0 appears to be already running on $1."
echo "Please check $lockfile if you think this is in error."
exit 1
echo "$0 appears to have completed for $1 without cleaning up its lockfile."
echo $$ > $lockfile

unlock (  ) {
rm -f $LOCK_DIR/balance.$1.lock

push_files (  ) {
local mode=$1 host=$2

if [ ! "$mode" -o ! -r "$EXCLUDE.$mode" ]; then
echo "$0 $$: mode unset for $host!"

if [ ! "$host" ]; then
echo "$0 $$: host unset for push $mode!"

lock $host

rsync --archive --rsh=ssh --delete --ignore-errors --whole-file \
--exclude-from="$EXCLUDE.$mode" / ${host}:${TARGET}

unlock $host

push_tier (  ) {
local mode=$1 host_list=$2

for host in $host_list; do
$SHELL -c "push_files $mode $host" &

export -f lock unlock push_files

[ "$FRONT_END" ] && push_tier front "$FRONT_END"
[ "$BACK_END" ] && push_tier back "$BACK_END"

# Fin.

This script (call it balance-push) will manage your rsync jobs for you, ensuring that servers don't "lap" themselves on each run. It will push the proper files to each group, depending on whatever you specify in the files in /usr/local/etc/. If it finds a server that hasn't finished its last run, it will continue to skip the server until it has finished (and issue you an email to that effect via cron's MAILTO feature).

The load on your master will likely go up a bit, depending on how many machines you're syncing to (as each server requires both an rsync and an ssh session.) But practically speaking, on a production network serving millions of hits a day, the load introduced to each individual web server is negligible.

O'Reilly Home | Privacy Policy

© 2007 O'Reilly Media, Inc.
Website: | Customer Service: | Book issues:

All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners.