Case Study: Data Center Move with No Service Disruption

CASE STUDIES
from the files of
Networking Unlimited, Inc.

WebQuery@NetworkingUnlimited.com
14 Dogwood Lane, Tenafly, NJ 07670
Phone: +1 201 568-7810

Data Center Move with No Service Disruption

Normally, moving a data center requires a "flash cutover" of the network to provide remote systems with access before and after the move. Networking Unlimited, Inc. developed a series of router configuration changes which allowed the data center routers to be "leisurely" moved one at a time, bringing down the old lines only after their replacements at the new data center were proven, providing uninterrupted user support for the duration of the move, and minimizing the risk to ongoing operations.

Background

A retail chain needed to move their data center to a new headquarters building several miles away. The challenge of moving all the headquarters personnel at the same time was daunting enough, but the data center also supported warehouse operations and retail sales. While the warehouse only functioned six days a week, allowing one day to move an AS/400 and bring up new SNA links, many of the stores were open 24 hours a day and any significant down time would quickly find its way to the bottom line.

From the data processing viewpoint, the move included a large UPS, an AS/400, multiple RS/6000 AIX application servers, and Novell and NT PC servers. The network consisted of approximately 75 stores connected via frame relay (with ISDN backup) to three routers at the data center running a common LAN shared by HQ staff, the data center, and the warehouse.

Needless to say, the prospect of shutting everything down on a Friday night, running backups on every server, dismantling all data center and HQ employee systems, reconfiguring all warehouse systems which were not moving, and having everything up and running for the warehouse to begin picking at 6:00 AM on Monday morning caused the MIS manager to suffer from many sleepless nights.

Technical Approach

Rather than even attempt to pick everything up and hope that it would all work when deposited in the new location, Networking Unlimited, Inc. worked up a network migration plan which would allow the move to occur in phases, allowing the MIS staff to concentrate on one critical factor at a time and minimize the danger of excessive downtime at any time during the move. Key to the plan's success were the availability of a T1 link between the two sites to support remote bridging and development of configurations for all the remote routers which would allow them to support both HQ sites simultaneously.

Essentially, by breaking the network move into multiple steps and moving only one data center router at a time, developing appropriate configurations for each step along the way, the various units dependent upon the data center could be moved independently. The network hardware moves and associated router reconfigurations were actually performed during the week, so that all staff were available each weekend for moving application systems and testing could proceed without concern about incomplete or incorrect network migration.

The data center and HQ move timeline worked as follows:
1- Reconfigure stores connecting to the first router to move to support two more PVCs to the new location.
2- Reconfigure the last HQ router planned to move to support the new warehouse IP addresses.
3- Reconfigure all store routers to dial backup to either old or new location numbers.
4- Move first router and bring up & test frame T1 to old location, frame T1 for store access, and two ISDN PRIs for backup (one for stores, the other for the remote bridging).
5- Weekend #1: Move all data center server systems except the AS/400 and the retail sales support backup system, run them off of individual UPSes (the main UPS moves with the AS/400).
6- Move the second router to the new location. Bring up the second store frame T1 and the third PRI. Move a skeleton MIS staff and support systems.
7- Reconfigure all store routers to prefer links to the new location, retaining the old for backup.
8- Weekend #2: Move the AS/400 from the old data center to the new data center. No other move activities are scheduled for this weekend to allow total concentration on this task.
9- Move the rest of MIS, leaving behind a skeleton staff to support HQ users and move activities.
10- Weekend #3: Move all HQ personnel and their systems from the old site to the new site.
11- Reconfigure the use of the "old to new" link from remote bridging to normal routing (for IP and IPX) and DLSw (for SNA support).
12- Move the third data center router and the backup retail server to the new data center, leaving just the remote routers to support the warehouse.
13- Reconfigure all store routers to only support the new HQ location.

Bottom Line Results

The data center move took place on schedule and with no detectable disruption of retail support services. As could be expected, there were numerous glitches and problems involved with moving individual systems and network connections, and it was recognized that there is no way the move could have been completed in one weekend, even a very long weekend. In addition to the expected frame relay configuration errors given the number of moves and changes, there were a number of system hardware problems that required extra work to repair.

ISDN backup continued to work correctly throughout the move, so the normal frame relay glitches had no impact on store connectivity. The only networking problems encountered were due to the need to support IPX over the remote bridge between the time the Novell servers moved and the HQ personnel followed. As it turns out, one workgroup used an inefficiently written data base application which could not tolerate the added delays across the bridge, and one of them had to move ahead of schedule to get a PC closer to the server.

The primary failure during the move occurred when the T1 link between the old and new sites failed during peak time, and ISDN backup kicked in. Although the ISDN line came up correctly, its 64Kbps data rate was too low to adequately support all the HQ Novell traffic, which caused SNA connections to time out and disrupted warehouse operations. Shutting down the Novell servers reduced the loading on the link sufficiently to bring the SNA sessions back, but there was still over an hour of degraded operation until the T1 link was restored.