A sysadmin tip that could save you hours...

by Brian K. Jones

There's a rule of thumb that's been passed down to me from earlier generations of systems administrators that I will occasionally forget. I'm always sorry when I do. Not only that, but I'm also often sorry when another administrator forgets it, because it makes helping them very hard (and I like helping other admins where I can). Here's the scoop:

Let's say an employee has just left your firm. Let's say, for giggles, that he was a webmaster who maintained several LAMP-driven applications. Well, you'll need to change a few passwords on a few databases, in a few files for the applications, maybe change a pg_hba.conf file here, a slapd.conf there, etc., etc. Well, don't do any of this until you perform this essential step: without making ANY CHANGES to the configuration or anything else, simply shut down and then restart the service.

In a perfect world, this is a waste of time. It is not a perfect world. Suppose the ex-employee made a last minute change and figured it would be reflected after the next downtime - a downtime they will now not be present to see. Suppose the employee was disgruntled and planted a present there for you. The point isn't to determine whether malice was involved or not, the point is that you, as an administrator, need to get to a place where you're confident that the service is still healthy.

If you DO NOT restart the service before making the requisite changes, and the service doesn't restart properly, well, how can you be sure the service wasn't borked before you made your changes? And so how are you going to determine where to start troubleshooting? Looking at your own changes may or may not be a waste of time. Who knows?

Well, I hope this has helped someone. Share your war stories about a time when this bit you!

2 Comments

bryan
2006-03-13 18:12:53
regression testing, often ignored at your peril.
Dave Harding
2006-05-11 17:41:15
I remember when you offered this advice at the April LUG/IP meeting. It was then that I decided you were a genius.


-Dave