I’ve never been so busy in my chosen career. The entire month of Feb has been hellish for me and has done wonders to deteriorate my social life. The reason – try relocating 10 units of HP 42U racks to another datacenter. Yes. You’re reading it right. 10 Racks.
Ten in itself is just a small number after all, so let’s complicate things a little bit shall we? Inside the racks we have at least 5 production servers that takes 3-5U’s each. In addition, each server has it’s own corresponding storage array comprises of 16 disks each and occupying 3U of space by itself. Around 50+ servers, not including any tape drive and media library mounted inside selected racks. To make matters more interesting, each server has 3 to 4 network points. I wouldn’t even begin with the IP addresses.
The amount of coordination and planning involved in the project was insane. We were going through the design phase in months, even had the implementation date delayed several times due to high risk factors. At one point I honestly thought that the entire project will be cancelled, but eventually we had the Feb 21 and 28 as the final dates for implementation.
The last 2 weeks pre-D day was more engaging than I could ever imagine. My brain was split into three sections: 1. Reserved for my most basic bodily functions – breathing, blinking, eating, muscular movements 2. Allocated for work related activities like meetings, planning, coordination, last minute checks and 3. High Availability (HA) for brain section number 2. Yes, for the past few weeks I was nothing short of being dubbed as a corporate zombie.
To take things to the next level, another PM was pushing me for completion and sign-off of my design document for another project. There’s also another PM pushing for a sign-off and completion of an older project’s handover document. Times like these, I’d wish THIS was real.
Surprisingly, it wasn’t too difficult on the day itself. We moved a couple of messaging servers and domain controllers without a sweat. A pair of bluecoat servers was also moved with minimal effort. I’ve planned for 3 hours per rack to be moved to a different datacenter, and at the end of the day it took us way less than that complete the migration process. The entire move was completed in 2 weekends as planned. Casualties include 2 faulty disks resulting in a broken storage array, and failed backup which resulted from a simple cabling fault. I feel I should list down a few things that my fellow engineers should take note of, in case you find yourself in a similar project:
1. Get the details – Server serial numbers, warranty information, application green zone, power requirements of individual devices, existing cabling structure, device owner and support – just to name a few. I can’t stress how important it is to master the details. Get all the details of everything, and put them somewhere you can reach with ease. Come up with a spreadsheet with all the details, and sit down with your team to brainstorm anything you might have missed. Memorize the server names, existing location, new location, rack labels, and cabling details if you have to. You’ll thank yourself later if things get out of hand.
2. Write down a solid task plan and stick to it – Once you have all the details sorted out, visualize the actual move and write it down in sequence. Play it over and over in your head until you can see it happening in front of your eyes. Think of it until you dream about it and you see things that may need to be improved or added to your plan. Use all the details you have in conjunction with visualizing the entire activity. Think of several scenarios that might occur and plan on it accordingly.
3. Cover yourself – Always, always take into consideration the impact of whatever change that you’ll be performing. Make sure that you have a solid Backout plan (revert to original state pre-change) before the activity. Make sure that you’re not taking unnecessary risks. Downtime caused by bumping into another rack during production hours is a sure fire way to get upper management’s attention.
4. Get everyone in sync - Make sure that everyone knows what they’re supposed to do. Talk to all the people involved and make sure they know their corresponding roles during the implementation. It’s pretty standard to get all the contact information of all people involved, but with all the information you’re being fed daily you might miss this one. Don’t make that mistake. Ensure that everyone is in sync of the the plan and the timings. Make sure to inform and update your stakeholders every now and then. Oh, and don’t forget the users affected.
5. It’s not over till it’s over – Once everything seems complete and you’ve passed all the necessary tests, continue to be alert. Double check if you have to. There may be some issues that may pop up later and that point to the activity as the cause. Monitor the hardware and ensure that everything is in order, get a sign-off or some form of handover document to get the owner’s concurrance that everything has been done according to plan and your equipment in working order. If you’re changing any ip addresses make sure that hostname resolution is working as well.
Overall, I’d say the entire project team did an excellent job to ensure the overall success of the project. There’s a certain satisfaction that I felt upon completing the project that I hadn’t felt before. It was a great experience and made me feel a little bit more mature in terms of my professional career. I’ll be doing another move in the succeeding weeks, albeit lesser in scope but the impact is just as much if not greater.
Strict security policy works within the datacenter as expected, so no visuals for this entry. I’ll save it for another post.
Posted by blackfury 
Posted by blackfury 
Posted by blackfury 
I’ve been blog-tagged by 