BSD Success Stories

by Dru Lavigne

At this year's OSW (http://www.osw.ca), one of the booths had an interesting little booklet which was free for the taking. It was the 3rd edition of "Perl Success Stories", a collection of a dozen or so testimonials written by real users who have implemented the benefits of Perl.

Intrigued, I did a google search and discovered that this initiative has been around for about 2 years (http://www.oreillynet.com/pub/wlg/1438). Of course, I immediately envisioned a BSD equivalent, suitable for trade shows, conferences, user groups, and other such avenues.

After a discussion with my editor and a query to the three BSD advocacy groups, the first batch of Success Stories has arrived, which I'll post at the end of this blog. Even better, the first booklets are being printed and I'm keeping my fingers crossed that they'll arrive in time for both BSDCan (http://www.bsdcan.org) in Ottawa and GUBUG's Installfest in Salt Lake City (http://www.gubug.org/meetings.php).

I'd like to thank everyone who took the time to submit a BSD Success Story. If you don't see yours posted yet, don't worry, it's being formatted and will show up shortly.

If you have a success story to share, send it to either myself or chromatic@oreilly.com so it can be included on the site and in upcoming editions of the booklet. If you have an event where you'd like to give away the booklets, drop me a line and I'll make sure it is sent to the appropriate contact.

Here are the first 4 stories:

*****

I am sometimes asked "Why do you use FreeBSD?" My usual 'auto-response' is
"Because it is the best!" While I truly think it is the best, this reply does
not really answer the question. I hope this article will better explain my
feelings.



Way back in 1994 I became the manager of a hardware laboratory at my old
University. I was to manage the labs in all aspects, from computers down to
data sheets, pliers, and students. With over 300 students attending each year,
this was a very time-consuming task. Of the three labs, only two had computers
and networks in them. In the third lab, 16 serial lines were connected through
a Xyplex terminal server to a computer of unknown pedigree. (It later turned
out to be a Sun 3.) I had been into computers since the early 80s and PCs since
1989, but I had never touched a network of any kind.



The two labs were connected by a Lantastic LAN. Only one area of the
server's disk was shared and every student stored his own home-directory on it.
As long as everything worked fine, and it generally did, I had better things to
do than trying to understand what this network setup was all about. The odd
downtime was mostly due to students tampering with the RG-58 coax that ran from
one machine to another continuing on into the next lab. Problems were easy to
locate. I knew the coax had to be terminated with a 50-ohm BNC terminator and
moving this terminator to strategic places along the cable always led me to the
culprit.



Then there was a software upgrade. The labs had been running MS-DOS with a
DOS-based compiler for many years. The new version of the compiler required MS
Windows. (At the time this meant Windows for Workgroups 3.11.) I could not get
the old LAN software to run under Windows. I don't remember why, perhaps I was
a victim of my own ignorance. After reading the Windows docs I managed to set
up a common workgroup for the two labs. With a 486/66 Windows box working as a
file server we again had a fully functioning lab... or so we thought!



I found that it was possible to access the server's shared area and was
quite pleased with my doings. I compiled some typical code from a client
machine and everything ran as expected. I did not test network performance
under a higher load. The idea never crossed my mind.



During one compilation, several large, multi-megabyte files were created in
the project's home directory. This was in itself not a problem as the
network's bandwidth was not saturated. However, all this data moving in and out
of the server turned out to be detrimental to its health. With 10-15 clients
chewing away, the load on the server increased. After a period of time ranging
from a few minutes to several hours, the server would unexpectedly crash. A
reboot got everything working again. All source files had been auto-saved
before entering the compilation stage so it was easy to start over and hope for
more success the second time. Still, the situation was less than
satisfactory.



At this point, OS/2 entered the story. In the outside world, Windows 95 was
being deployed. I got my hands on an old OS/2 2.1 demo CD. After several
problems I got it up and running. I thought it was cool and ordered my own OS/2
Warp soon after. Warp was my favourite desktop for years to come. I really
liked the smooth user interface and enjoyed the increased stability compared to
the Windows version I had used earlier. A co-worker with a similar interest in
computers used Windows 95 and had to re-install every three to four months.
The system would somehow clog itself up beyond recognition. My OS/2 machine ran
happily all the time. Great! Except for the RSA DES Challenge. Being outside
of the United States I took an interest in the Swedish-run SolNet DES-attack.
With SolNet's limited resources there was quite some time before an OS/2
executable was published. The situation annoyed me.



So what to do? I was unwilling to give-up OS/2 and surrender to Windows 95.
I had started to experiment with Linux, a Unix-like system but felt very little
enthusiasm for it. Because of an earlier experience with HP-UX, I was under
the impression that Unix only represented an extremely complicated way of doing
things, and therefor, Unix was ruled out.



At this time, there were executables for FreeBSD available. Not knowing much
about FreeBSD, I made an FTP install of 2.1.7-STABLE. The DES-client ran as
expected. I figured out nice commands; kill -STOP, kill
-CONT
and that putting an ampersand (&) after the
command line ran a job in the background. Cool! I could manage every aspect of
the program. A taste of a new world! This was very enticing.



My success was short-lived. There was soon a new DES client requiring newer
libs, which forced me to install 2.2.2. This time I ordered the 2 disc FreeBSD
set from Walnut Creek sometime around June 1997.



Before the summer I had plans to replace the lab server with OS/2, but
during the summer, I experimented with FreeBSD and learned about TCP/IP, Samba,
FTP and Unix.



The client machines in the lab were using Windows 95 and since we had to get
rid of Windows for Workgroups anyway, I installed FreeBSD on each and every
machine. I made further experiments with the r*-services (rsh,
rcp, ruptime, etc.) among other things. All in all,
the experiments made me confident with managing FreeBSD. In the beginning of
August, when I had to wire-up the lab's new infrastructure, I made FreeBSD run
the lab for me. That turned out to be a wise move; not one spontaneous reboot
occurred during the years to come.



With a busy lab, reliability is of major importance. The logs show students
accessing the server around the clock. FreeBSD really is reliable; the server's
longest uptime to date is 220 days.



By now, things were moving fast. The third lab was also computerized. (The
introduction of Microchip's PIC line of processors necessitated this.) I added
a couple of hubs and the labs were interconnected. The next summer I set up a
local name server and a private domain on a FreeBSD box with two NICs —
one for the internal net and one for external access. There was no forwarding
between the two interfaces. This setup allowed me, as admin, to download
software which I then installed on the clients. Each project team in the lab
had its own home, with permissions fixed so they could not peek at each other's
work. They could also edit files at home and upload via FTP. I added the
system user as a way to keep a central repository of programs and
files that every client would need — Adobe Acrobat for Windows for
example, printer drivers, upgrades and other stuff.



Until now all data sheets had been in a folder in my room. The projects'
supervisors had access to the folder and make copies which they handed out to
the project teams. There were 95 steps to the photo copier, and with 95 steps
back from it, handling the data sheets became more and more cumbersome as a
variety of components were added. What to do? Install Apache! Putting the data
sheets on the server in PDF-format has eliminated the actual paper handling.
This was a truly brilliant move that I wish I had thought of earlier. Now each
project group could peruse the available components' data sheets, both from
home and when in the lab before committing the device into their project. Some
students printed out data sheets of course, but I see a trend of more and more
students reading the specs directly off the screen.



There is a lot of activity in the labs. With up to 75 students there at the
same time, not all of them can be busy with their assignment; installing MP3s
on the clients has become a popular pastime. It is quite possible that the
especially zealous students fiddle around with the system settings of Windows
95 and cause damage! To simplify the Windows re-installation, we have made a
complete 2 gigabyte image of the Windows partition which easily can be
downloaded and written onto the machines. While this may sound rather brutal,
it works well in practice. In fact, every machine in the lab was born this
way.



Needless to say, the client machines dual-boot between Windows 95 and
FreeBSD. It is a particular pleasure to find the students shutting down Windows
and rebooting into FreeBSD because they feel more at home with it.



Earlier I mentioned we had a Sun 3 box. That particular machine has gone to
the scrap heap and the FreeBSD server has replaced it. Using the ports, I
installed the m68k-coff cross-compiler and spent some time rebuilding the libs
needed for our particular hardware. We were able to get rid of gcc-1.40 and now
use gcc-2.8.1 for our in house built 68008 computer card.



Some small bits and pieces complete the picture. The server has been endowed
with a Tandberg SLR-5 tape backup. Once a week I do a level 0 dump and every
night a cron job initiates a level 1 incremental backup. The machines' disks
are mirrored with vinum(8). Thus the entire /usr is
safeguarded from disk failure. To ease new installations and administration,
each client has a P:\ (P for program) directory with all
non-standard Windows 95 software. It is now sufficient to install new software
on only one client; it will become available to the other clients at the next
log-in. Every home share is attached as Q:\ on the client. To this
end, Samba is configured as an NT domain controller. Log-in scripts handle the
actual attachment of the devices at log-in time. On the FreeBSD side I use NFS
and amd(8) to automount the user's home directories upon
log-in.



I have unfortunately left OS/2. I really liked the Workplace Shell, but the
functionality built into even a basic FreeBSD system makes me more productive.
I have now been using FreeBSD exclusively for over two years for all my desktop
needs. My experience is best summarized by something I saw someone post to one
of the FreeBSD mailing lists: "I'm home."



Michael Josefsson is a research engineer at Linköping University,
Linköping, Sweden.



Originally published at href="http://www.daemonnews.org/200107/adventure.html">Daemon News: Adventures
in BSD.



*****

You haven't had email since when?



SOHO: FreeBSD saves a dot-org, and maybe me, too!



The phone rang ... "Hello, this is Susan* ... this may seem
strange, but I was thinking about some problems we've been having, and I
thought you could help us. Could you come down sometime and take a look at our
computers? We're really struggling here...."



She went on in some detail about the state of things at her office. I
assured her I'd think about it, and hung up ... like Samuel Morse, thinking
"what hath God wrought?" I'd created an HTML page or two, but I was no
computer genius. I had a computer and I had friends and a smart brother who
worked on them and with them, and that was about it. My training was in a
totally unrelated field; although I was currently in a "career path crisis", I
was neither computer repairman nor network consultant.



The caller was a friend, personal secretary to the pastor of the largest
church in a nearby city; I was on the ministerial staff at the smallest church
in the next town up the road, and as far as she knew, our computers were always
working. (Of course, she didn't know that we never asked them to do
much....)



I went down there; things were a mess. It was September 2001; in addition
to frightful things in the world at large, Code Red and Nimda were ravaging
computers globally. The machines at Big Church* ran that
"Wonderful" OS (well, it starts with a "W", anyway) and some were infected.
Furthermore, although everyone had high-speed Internet, almost no one was using
e-mail to communicate. It was strange; an organization with their own domain
name, over a dozen paid staff members, and a budget of nearly a million dollars
still depended on sticky notes for intra-office communication and spent a lot
of phone time just hashing out details of upcoming activities with their
members. I didn't think that was possible in the 21st century.



After removing Nimda from a couple of machines, I asked about their e-mail.
"Oh, that hasn't worked for a while ..." Curiously, I opened a couple of
outboxes on their POP clients. The most recent sent items were from November
2000, ten months past!! Inboxes? Same story....



Timidly, I asked, "Where's the server?"



"The server? Hmm, that would be Brenda*'s machine."



Brenda was in charge of payroll and financial and membership records. Her
computer was a 500 MHz Athlon with 128 MB of RAM. The OS looked a tad
different from everyone else's. Whatever it was, it didn't run so well, and
there were some obvious reasons. This machine was running an "Advanced"
version of that same very popular OS, and between being infected with Nimda and
attempting to operating as a PDC, along with IIS SMTP, and (for some strange
reason) anonymous FTP servers, plus Brenda's browsing (she had been smart
enough to get a mail account at Yahoo!) and the large financial package she
used, this machine was taxed past the limit of its resources. Furthermore, I
couldn't believe that they would spend so much money just to run Web and Mail
for 12 users. Of course, once I removed the virus I discovered they hadn't
spent quite enough! There was no POP/IMAP service on this box, it was not
firewalled, and most of the services it did offer were badly configured or not
working properly.



I called a friend of mine who worked for a small ISP. He'd been kind enough
to do a little open-source evangelism with me, and had given me a shell account
on a virtual server that hosted the company's website. I'd learned a little
about telnet and ftp, knew ls and cd, and could use
pico to edit text or HTML files. I was certainly no guru.



"What is that OS I've been talking to on your web server? Some kind of
Linux?" I asked. (His kids teethed on little penguin toys...he had a five-box
LAN in his house...these days I'd call him an übergeek....)



"No, that's a little more 'hard-core'. Probably a -BSD. Maybe FreeBSD.
Why?"



Hard-core? (Brief shiver) ... I swallowed hard and gave him an overview.
He told me I should do something about the situation (besides letting this
server spew viruses all over the place) and that an open-source OS like
FreeBSD would probably be just the thing I needed. "You can read everything
you need to know on the 'Net." So, with at least a bit of trepidation, I
pointed a browser at www.freebsd.org. "All you need is a pair of freshly
formatted floppies and these instructions. ...."



I went home and read up a bit (I grabbed the entire FreeBSD Handbook in
.rtf format.) I printed several pages. I formatted the floppies
and read the instructions.



The next day I was back. I found, in an upstairs closet at Big Church, an
old clunker that wasn't up to modern specs. It booted, and had a rather small
HDD, but it didn't clack too much and the fans sounded OK. I managed to stick
a NIC in it and cable it to their switch. Booting from the floppy, holding
tight to a couple of pages I'd printed from the online Handbook, I clunked away
though sysinstall. If I remember correctly, it took two, maybe
three attempts to navigate the menus correctly — it was a little
different then I was used to, but the alternative was paying thousands of
dollars to someone who didn't even care to come and check up on the machines
they had installed and (apparently) hadn't been there for 10 months....



To make a long story shorter, once I navigated the installation menu
properly, FreeBSD installed itself. I found myself in a rather familiar CLI
environment. In less than a day, we had SMTP and POP3 capability inside the
building. Then we moved the HTTP service to this little clunker. Now Brenda's
financial app would work better because she wasn't sitting on an overworked box
with no resources. Later we started in-house DNS to ease the load on the ISP
and router. As a bonus, all this was accomplished with free software and
hardware that was waiting for the dumpster.



That was almost three years ago. Today, that church hardly ever calls me
with problems, and when they do, they're never server-related. I do think I've
proactively changed every piece of hardware on that box, as parts have become
available, and as their website has grown way past the size of the original
HDD, but the server still bears the same name, and today provides even more
services both to the LAN and the outside world.



As for me, I still have that part-time ministerial position, but also have a
growing business in networking, troubleshooting, and web application
programming that has added substantial income to the family budget, and given
me some direction for that career path crisis.



This is no testimony of "what FreeBSD can do for your Fortune 500 shop".
Frankly, unless I become a better businessman, I'll never make a killing. But
there are lots of little problems everywhere that people need good tools to
solve, as well; I found a good tool in FreeBSD.



I imagine FreeBSD can offer the same things to anyone that it offered to me,
and even much more. But I know that because of its stability, excellent
documentation, helpful user community, highly versatile design (and of course
its cost), FreeBSD was a seminal force in this growth in both my life and the
life of Big Church. Guess what pretty desktop I'm writing this on?



C'mon, you already know!!



(*names changed to protect the naive [except for me])



*****

BSD in a Panic



My employer's main business is designing Web applications, but once those
applications are built our clients turn around and ask "Where should we host
this?" That's where I come in, building and running a small but
professional-grade data center for custom applications.



As with any new business, our hosting operation had to make the most of the
resources we had. Our resources were strictly limited to cast-off hardware
from the web developers and free software. The only major expense was a
big-name commercial firewall, purchased for marketing reasons rather than
technical ones. With FreeBSD and a whole mess of open-source software, we
built a reliable network management system that provides the clients with a
great deal of insight into their equipment. The clients, of course, pay for
their own hardware and so have fancy high-end rackmount servers with their
chosen applications, platforms, and operating systems. We've since upgraded
the hardware — warranties are nice, after all! — but have seen no
need to change the software.



One day, a customer that had expected to use very little bandwidth found
that they had enough requests coming in to use close to twice the bandwidth we
had for the entire datacenter. This affected every customer, slowing the
entire hosting environment to speeds comparable to a snail in molasses. If
your $9.95/month web page is slow you have little to complain about, but if
your $50,000/month Web application is slow you pick up the phone and scream
until it stops.



To make matters worse, my grandmother had died only a couple days before.
Visitation was on Tuesday, and the funeral was Wednesday morning. Monday
morning I handed the problem to a minion and said "Here, do something about
this." I knew bandwidth could be managed at many points: the Web servers
themselves, the load balancer in front of them, the commercial firewall, or
even the router.



Tuesday after the visitation I found my cellphone full of messages.
Internet Information Server can manage bandwidth — in eight
megabyte increments and only if the content is static HTML and JPEG files.
With several Web servers behind the load balancer, that fell somewhere between
useless and laughable. The load balancer did support traffic shaping,
if we bought the new feature set. If we plopped down a credit card number, we
could have it installed by next Sunday. Our big-name commercial firewall also
had traffic shaping features available, if we upgraded our service level and
paid an additional (and quite hefty) fee for the feature set. That left the
router, which I had previously investigated and found would support traffic
shaping with only a flash upgrade.



I was on the phone until midnight Tuesday night, making arrangements to do
an emergency OS upgrade on the router on Wednesday night. I had planned to go
to the funeral in the morning, give the eulogy, go home and take a nap, and
arrive at work at midnight ready to rock. The funeral turned out to be more
dramatic than I had expected and I showed up at work at midnight sleepless,
bleary-eyed, and upright only courtesy of the twin blessings of caffeine and
adrenaline. In my email, I found a note that several big clients had
threatened to leave unless the problem were resolved Thursday morning. If I
hadn't already been stressed out, the prospect of choosing a friend to lay off
would have done the trick.



Still, only a simple router flash upgrade and some basic configuration stood
between me and relief. What could possibly go wrong?



The upgrade went smoothly, but the router behaved oddly when I enabled
traffic shaping. Over the next few hours, I discovered that the router didn't
have enough memory to simultaneously support all of our BGP feeds and the
traffic shaping functionality. Worse, this router wouldn't accept more memory.
At about six in the morning, I got an admission from the router vendor that
they could not help me.



I hung up the phone. The first client who had threatened departure would be
checking in at seven thirty AM. I had slept four hours of the last
forty-eight, and had spent most of that time under a fiendish level of
emotional stress. I had already emptied my stash of quarters for the soda
machine, and had been forced to pillage a co-worker's desk for his. The
caffeine and adrenaline that had gotten me to the office had long since worn
off, and further doses of each merely slowed my collapse. We had support
contracts on every piece of equipment and they were all useless. All the hours
of work I had put in, and my team before me, left me with a sum total of
absolutely nothing.



I made myself sit still for two minutes simply focusing on breathing, making
my head stop sliding around loose on my shoulders, and ignoring the loud
ticking of the server room clock. What could be done in ninety minutes —
no, now only eighty-eight?



I really had one only option. If it didn't work, I would be choosing
someone to lay off or filing for unemployment myself.



6:05 AM. I slammed the floppy disk into the hard drive and started
downloading the OpenBSD install floppy then grabbed a spare desktop machine,
selecting it from amongst many similar machines by virtue of it being on top of
the pile. The next few minutes I alternated between hitting the few required
installation commands and dismantling every unused machine unlucky enough to be
in reach to find two decent network cards. By 6:33 AM I had two Intel
EtherExpress cards in my hands and a new OpenBSD 3.5-snapshot system. I logged
in long enough to shut the system down so I could wrench the case off, slam the
cards into place, and boot again. OpenBSD's built-in PF packet filter includes
all sorts of nifty filtering abilities, all of which I ignored in favor of the
traffic-shaping functions. By 6:37 AM I was wheeling a cart with a monitor,
keyboard, and my new traffic shaper over to the rack.



Here, the killer problems manifested. I didn't have a spare switch that
could handle our Internet bandwidth. The router rack was jammed full, leaving
me no place to put the new shaper. I lost almost half an hour finding a
crossover cable, and when I discovered one it was only two feet long. The
router, of course, was at the top of the rack. Fortunately, if I put the
desktop PC on end and left it sitting on the cart, the cable just
reached the router. I discovered this about 7:10 AM. I stacked everything so
it would reach and began re-wiring the network and reconfiguring subnets.



I vaguely recall my manager coming in about 7:15 AM, asking with taut
calmness if he could help. If I remember correctly, as I typed madly at the
router console I said "Yes. Go away."



At 7:28 AM we had an OpenBSD traffic shaper between the hosting area and our
router. All the client applications were reachable from the Internet. I
collapsed in my chair and stared blankly at the wall.



While everything seemed to work, the proof would be in what happened as our
offending site started its daily business. I watched with growing tension as
that client's network traffic climbed towards the red line that indicated
trouble. The traffic grew to just short of the danger line — and
flatlined. Other clients called, happy that their service was restored to its
usual quality. (One complained that his site was still slow, but it turned out
that bandwidth problems had masked a problem with his application.) The
offending client complained that their web site was even slower than before, to
which we offered to purchase more bandwidth if they'd agree to buy it.



Today, I have two new routers and new DS3s. The racks are clean again,
without extra cables from thrown-together solutions. The desktop machine has
been replaced by two OpenBSD boxes in a live-failover configuration, providing
protection for our big-name commercial firewall as well as shaping traffic.



My thrown-together OpenBSD desktop machine is sitting in the corner of the
hardware room. The sign on it says "DO NOT TOUCH: EMERGENCY USE ONLY." Should
the clock tick down on some other problem, well, at least I won't have to spend
the thirty minutes it took to install.



*****

FreeBSD at Shannon Medical Center



Shannon Medical Center is a not-for-profit trauma center in West Texas that
acts as the safety-net hospital for the surrounding area.



As the health care industry is under pressure from increased costs of
providing health care, coupled with reductions in reimbursements for providing
that care, the importance of providing timely information to hospital
management has never been higher.



Shannon Medical Center uses a Decision Support System (DSS) from a
commercial vendor. In 2000, a supplemental decision support database was built
using FreeBSD and PostgreSQL. This system integrates key data from the
commercial DSS, other sources of internal data that cannot be imported into the
commercial DSS, and external data for benchmarking purposes. Analysts access
the data using preexisting database clients via ODBC, eliminating the need for
additional training. The initial system was built on an existing desktop
computer for proof-of-concept. Due to the project's success, the system's
hardware was upgraded to a database server in 2001.



In the words of Andrew Gould, Manager, Clinical Decision Support:





The FreeBSD/PostgreSQL combination is extremely flexible and robust.
FreeBSD provides a stable and efficient environment for the database server.
The relational data model facilitates flexible analysis that is unsupported by
the commercial DSS. The database is used primarily for financial, clinical and
market share analysis. Some of the results of this implementation include:





  • Hospital management receives information that was previously
    unavailable.


  • Increased efficiency — highly focused, clinical data analysis reduces
    the number of medical charts to be reviewed manually.


  • Unplanned downtime since/including installation = 0.

  • License fees since/including installation = $0.

  • Support expenses since/including installation = $0.




The flexibility and low cost of this project reduce our dependency upon
vendors and free us from many budgetary constraints. We can adapt the system
whenever we want, however we want, to meet our changing needs.





*****



2 Comments

jwenting
2004-05-14 04:26:54
preaching to the quire
I never fail to be amazed at people spreading the gospel among themselves.


Is it really needed to increase the self-gloss of BSD (or Perl, or...) users by giving them booklets where they tell eachother their successstories?


Most glaring example of this I've seen was a major product release aimed according to the marketeers squarely at users of a competing product yet all advertising for the product was done in media aimed at existing users and all invitations for the event were sent to those same existing users.
As a result out of over a hundred people in the audience only 3 were not existing customers and of the 3 2 had come in company of an existing customer...


What they should have done of course is advertise in professional magazines aimed at competing products and in those invite the users of those products to attend...

chromatic
2004-05-17 12:05:40
preaching to the quire

The intent of the booklets is to give them to people who need to hear the good news, not to hoard them among true believers. For example, it's always nice to have a supply to hand out at installfests, where people who wandered in and don't know how nice a free operating system can be can take home a booklet, think about things, and may show up next time ready to take the plunge.