A "cutover" is when you move from one system to another. It's a very common task regardless of the particular field you work in. For example, a cutover could be changing your company from one Internet provider to another, upgrading to a different model of PBX, or replacing a web server with a more powerful machine.

Cutovers usually have some impact on the company, such as loss of a service (Internet, phone, web site, etc.), and disruption for end users or customers. The goal of the cutover is to move to a new system while minimizing the loss of service. To achieve this, you need to have a planning strategy.

I have been involved in many cutovers and have tried capture this experience in a checklist form. The following items explain important concepts and then give examples for various types of cutovers, including Internet, phone, and web site. Keep in mind that the cutover process is in no way limited to the ones listed, but they should give you enough information to apply the concept for your particular area of expertise.

Understand the task

Make sure you fully understand what needs to be done. Do not assume you can figure how to do something during the actual cutover. This is not the time or place for learning on the job or for "amateur hour". Ideally, you should be consulting with a peer who has prior experience. People involved in technical fields frequently think it is a sign of weakness or a loss of face to ask others for help. This is exactly the wrong mindset for mission-critical cutovers. You should try to come up with a plan that is the culmination of input from the most experienced people.

Break up the task

Cutovers should be broken down into discrete steps. For example, a likely step in the process of changing Internet providers would be changing an IP addresses on a router interface. For changing a PBX, a step may be to move the station wiring from the old PBX to the new. For replacing a server, a step may be to copy over data from the old server to the new.

If you are unsure of how to break a task into steps, the following criteria may help:

Try to insure "reversibility" of task steps

If the cutover involves ten steps, plan ahead for how you will back out of the entire cutover if you get to step eight and can go no further. Do you know how to get back to step seven, then six and so on? Ensuring reversibility means that you'll avoid "burning bridges behind you". For a phone example, when moving from one PBX to another, do not start tearing out old wires until you are assured the new system is working. If you are moving wires or cables from the old device to the new one, ensure that you have labeled them so you know how to put them back. You can also take Polariods, or use a digital camera, to record the position of cables, connectors, and wires. If you are upgrading a web server, write down or print out system information that you will need to put back if the upgrade cannot be accomplished.

Do as much as possible in advance

If something does not require downtime or interruptions, try to do it in advance of the actual cutover. If you have minimized the amount of work required to perform the actual cutover, this frees you up and gives you more time to handle unplanned or unforeseen events. If equipment needs to go in a rack, install the rack in advance. If you need new cables, make them beforehand. If you are plugging new equipment in, make sure the correct type of power outlet is available. Open the boxes and inventory all parts to make sure you received everything you ordered. Do not assume anything.

Perform "dry-runs"

When the cutover is complex and involves multiple people, it may be worth practicing the cutover in advance. This is a great way to expose problems and make sure everybody knows their role. Try to get specific time estimates for each person's part of the job. This will enable you to create an accurate time line for the entire procedure. It may help to pantomime the entire event, with everybody "walking through" their roles. This may seem silly to the participants, but it exposes problems such as one person needing to be in two different places at the same time.

Make sure resources are in place

Each step typically requires interaction with different groups of people. For example, when working on connections to the Internet, you may have to communicate via email with the InterNIC to change a DNS record before the change, call the ISP installation engineer during the cutover, and then call people at remote offices after the cutover to verify that the new connection is working.

For a server cutover, you may want the system administrator to be available, and then have key individuals available to test software on the new server. The absence of any one of these people could prevent the entire process from happening. Do you have everybody's cell phone, beeper, after-hours numbers and email addresses? This is especially true when working with vendors and remote people across different time zones. Make sure you have after-hours access to every facility that you may need physical access to. Do you have all the alarm codes, keys, smart cards, and emergency numbers? Have you actually tried them?

Figure out dependencies ahead of time

Knowing which steps depend on other steps will prevent cascading failure. For example, if you have to take down a voice circuit, does this also prevent you from calling someone if you get into trouble? Does it prevent someone from reaching you while you are performing the cutover? I was once present at a voice T1 cutover where the engineer cut off his own conference call to the carrier. Luckily, the machine room happened to have an outside analog phone line that he could use to call the carrier back with. If you need a resource on the Internet while your Internet connection is down, how to do you work around this situation? If you take down your Internet router to upgrade it, you cannot reach Cisco's web site if you get into trouble. Perhaps you should get an account at a local ISP, so you can dial into the Internet while your company Internet connection is down.

Pick a good time for cut over

Only schedule a cutover if you know you will be available for the next couple of days after the cutover date. For the actual cutover event, set a time when you will suspend the cutover work and evaluate the progress you have made. If you are not assured that you are going to succeed, back out of the steps, reschedule the cutover, and take into account the problems that you encountered. Set some maximum amount of time for the cutover. People will get tired and make mistakes if they work too long. Avoid the situation where you are stuck in a noisy machine room at 2 AM with a group of hungry and tired people.

The details will either make or break the cutover

There has to be at least one person involved in the cutover that makes sure the details get taken care of. Most people are content to think in general terms, but someone has to worry about which wire gets plugged into which type of connector. Do not get 10 hours into the cutover, and then start trying to figure out if you need a DTE or DCE cable. Vendors typically only think about their specific areas. It is up to you to fill the gaps between vendor responsibility.

If possible, have both old and new running at the same time

The ideal situation is to have both the old and new systems running in parallel. This way, you only move services to the new system when you are assured it is working, or you can back out of the entire process if something goes wrong. If you have moved from the old system to the new one, keep the old one intact for a period of time. A classic example of this would be backup media. If the old system is the only one capable of reading backup tapes, you might have to keep it around in case you need to read the old tapes. In some cases, this will not be an option, due to the cost or uniqueness of the system, but it is highly desirable. If you can have both old and new systems running at the same time, you may be able to perform several of the "dry runs" mentioned earlier, before doing it all for real.

Establish a "test suite" to verify operation

Make sure you have a mechanism to verify that everything is working after the cutover. This typically involves becoming familiar with the operation of the current system before the cutover takes place. For example, come up with a list of tests which prove that all major functions are working. For a PBX, this could involve placing an internal call, an outgoing local call, and an outgoing long distance call, and then an incoming call from an outside line. For a website, test connectivity to the site from inside the company and from outside using another ISP. For network equipment, it may be helpful to note the color and position of status lights, LEDs, and other indicators so you know when something has changed.

Thinking through all of these strategies ahead of time will greatly increase the chance of a smooth cutover. Sun Tzu's famous maxim "Every battle is won before it is fought" can be applied directly to the cutover process. The outcome will be pre-determined by the depth of advance planning, not the actual event.