Operations Workers Unite!

by Brian K. Jones

Ok. Here's the deal. As the term "Web 2.0" makes its way into more and more glossy ads printed prominently in the largely irrelevant technical drivel outlets (er, magazines), more PHBs are going to want it. In many cases, this is probably a justified desire, given all that Web 2.0 encompasses. Some of those who have a clearer view of the technological landscape have commented recently that operations could well be *the* differentiating factor, the "value add", that determines success or failure in Web-2.0-land.

I've seen very little, even from the usually outspoken developer community, to dispute these comments. Of course, I'm an operations person, and I think that most of the comments are spot on. They are also cautionary in their tone, which is also spot on, because operations is sort of a forgotten landscape in the technological ecosystem. It's worthy of concern.

The only questions that seem to be going unasked, then, are "What do the
operations folks need, and what plans exist to meet these needs?"


Commentary has gotten us to the point of consensus. It's time to get moving on
a plan of attack now. I haven't a clue exactly where to start. For my part,
I've tried to clean up a lot of my code, and post any snippets that can be of
use to others on my website - http://www.linuxlaboratory.org. Unfortunately,
what's there is hardly even a drop in the bucket.

(NOTE: if you have code you want to put there for operations folks, drop me a line)

In addition to working in operations, I'm also someone who loves discovering
and exploring new technologies (and old ones, too), and I love to write and
help others understand how things work, the context that different
technologies operate in, the patterns that exist in different technologies and
operating paradigms, and the like.

This, I'm told, is something that a vast majority of people working in
technology despise. Nobody likes to document things. Maybe part of it has to
do with r0ml's 'bozo' theory: documenting things means somebody will read it,
and perhaps you'll be found to be a bozo for one reason or another. Maybe part
of it is sheer laziness: nobody's found a way to automate documentation for a
service deployment that I know of. Whatever reasons exist for operations
people tending to stay away from vi for anything other than config files or
code, I'd like to help fill the gap in whatever ways I can.

So here's how this works:

I'll ask you, the operations people at large, some questions. You will either
leave replies as comments to this post, or you can email them to

I'll monitor the responses and email, and I'll use any and all resources at my
disposal to come up with some directions that I and others can take to address
the needs of the operations community. If it means development work, we can
bring them into the conversation. If it means more documentation is necessary,
I'll try to do my part there, and share the information I get with other
authors and experts whose documentation I can help develop if need be. My
hunch is that there's plenty of things needed by operations people that a
whole team of smart people who have access to resources themselves haven't
even thought of. That's what makes this whole experiment interesting.

So, without further nonsense:

What is it that you need? Do you need tools? What kind of
tools? What tasks need automating for which there is no good tool? Why do you
think the current tools suck? What exactly sucks about them?

What things lack the requisite integration? What services are lacking in
features? What features do you want to see? How would these features help you
in your work?

What services don't even exist yet in a form that you can use? Why can't you
use what's available now? Why does it suck?

What are you currently writing code to do yourself? What platforms are you
working with? What is the "holy grail" for your environment? Why aren't you
there yet?

What things have you left behind because they've failed to scale or perform?
What have you moved to? What decisions have you made that you've been forced
to revisit in light of new requirements? What's the driving factor of the new
requirements? Is there a theme or patterns in the requirements you're seeing
now as opposed to two or three years ago?

What tools have you been happy with? How can they be even better?

Basically, in a nutshell, "What's causing you pain, why is it painful, how can
it be alleviated, and why do you think the right solution doesn't exist

Be as specific as you can in your replies. Name names. Bring on the links. If
you think that application x is only useful to you if it can integrate with
service y, and it doesn't, then say so, and also say anything else you can
think of that would make either app x or service y more useful.

If you work with software vendor foo and you need tool bar, but you can only
get it if you sign your life away, then SAY SO! Chances are, some of us who
work with nothing but open source solutions aren't even aware that these
commercial proprietary tools exist, and if we were aware, we'd say "hey,
that's cool, I need a free tool that does that". If anyone has proven that
they have the power to make it so, it's the open source community.

I look forward to your responses.


2006-08-09 07:10:42
I'm only really a sysadmin in name. I'm actually our only real software developer. Right now, perl is causing me pain, but on the other hand it also has a lot of modules which I like. The code I'm writing is a lot of glue code to gather data from many sources (queuing software, fedora ds, process accounting, web forms, etc). I'm using an object relational mapper called rose::db::object which is great. I'll probably be using prototype once I start work on the web frontend.

In an ideal world what I would really like is a programming language similar to java, but without the annoyances like no unsigned types. It would have perl's install base, distribution, and extensive module collection. Equivelant eclipse integration as current java code. Extensive exception handling, OO support, and event handling would be nice. It would also need to be well supported in eclipse (IE as good as java is).

For now, I'll live with perl5 and java.


2006-08-10 13:56:38
I work for a large business process outsourcing company and am responsible for all of the interfaces between our clients' accounting or ERP systems and their vendors, suppliers, and banks so file transfer management and data transformation is a big headache for me on a daily basis. I would love to see some open source tools and frameworks that address these areas.

In the area of file transfer management I would love to have open source software similar to Tumbleweed's SecureTransport or IBM's WebSphere Business Integration Connect. Right now most of this is done using a series of shell scripts so there is minimal logging, almost no reporting, too many configuration files, and encryption and decryption is difficult to manage with all the different keys.

Once we get the file, we need the ability to convert the data from the vendor's format to the ERP system's format (i.e. XML to CSV, CSV to Fixed Width, etc). Here again we don't have a standard way to do it. Sometimes its shell scripts or Perl scripts, sometimes custom Java code and other times we use SQL Server's DTS if it is an option. Here I would like to see an open source framework that let you define the workflow of a data transformation task and it would be great if it could integrate with the file transfer management software. DataSift is a data transformation framework I have researched recently that is a good start.

All of these processes need a standard way to be scheduled and executed like a Web 2.0 version of cron. The ROC Enterprise Suite is a good example of the kind of functionality I want.

With all of these interfaces and scripts another weekly headache for me is Configuration and Change Management. In this area there are many good version control systems available for source code and script changes but I would like to see a system that covered the entire change workflow:

1. Submit the change request
2. Approve the change request
3. Define, develop, and document the change
4. Test the change
5. Generate and manage all needed documentation for auditing and compliance
6. Backup of the current system's configuration
7. Deploy the change

Tripwire is one example of this type of software.

These are issues many businesses face everyday and it would be a great place for open source software to make more inroads into Enterprise IT.

2006-08-17 06:36:50

i am a part time sys/netadmin (on Linux). I need ZFS for Linux! I know there is NexantaOS/GnuSolaris ... but is it ready for production use?
Thanks for such a great post/idea!


Depressed sysadmin
2007-04-30 14:16:52
I got a sysadmin job, not realizing very well what I was getting myself in to. I'm tired of feeling unapreciated and looked down by developers. I get less resources than all the other tech people at the company, I get to have the phone on 24/7 and I get to do all the boring IT work, such as trying to explain to clueless business people why they shouldn't use their name as their password and clicking through some Norton or Symantec piece of crap anti virus because they *have* to use Windows. To top it off, I get to put with the crap the in house developers produce. Hurray!

I'm quickly backpedalling for a career change into security or development as fast as I possibly can. As far as I'm concerned operations can go to hell, people only seem to value it when things go wrong and maybe they will go wrong more and more, considering how attractive the field is.