Nominis expers

I’ve been blog-tagged by the Zero Initiate to tell the story behind my WoW alias, Blackfury.Exordium.

When I started playing the game I took the name “Blackhowling”, from Gau Ban of anime Shadow Skill. The name by itself  represents “darkness” and “wolf sounds”. Nothing, absolutely nothing can there be a more profound and cooler sounding name in my book. 

So there I go, drafting my character and selecting how I should look in-game. Typed in “Blackhowling” in the Name field, only to find out the name is already in use. 

Disappointed from not being able to use the name, I retained “Black” and started thinking of anything to compliment it. Violence, Anger, Passion, Hate  - leading to Fury. There you go. Black and Fury. Surprisingly enough, it’s also the name of an epic item in-game. 

I’d like to go back to using Blackhowing anytime, though. I can only LOL so much when guildies start calling me pinkfury.

Lunae

Other than cutting my weekends short, my Mondays defy the rules of modern physics by compressing all these things when they don’t fit in a workday: 

1. Raise any change for the entire week (till 7pm only, or my project activities are delayed for a week). 2. Generate change reports (and project reports, if month-end). 3. Clock my project man-efforts, update project costing 4. Report to my boss for status updates 5. Attend a series of project meetings 6.  Perform scheduled project activities (site survey, project design, implementation, etc.) 7. Respond to hundreds of emails and return phone calls. 

So little time and so much to do – I never notice that i’ve spent more than 10 hours at work.

Yes. That’s always a good thing. At least until I wake up the next day. 

I really need to unwind.

Projekt: Datacenter Relocation

 
I’ve never been so busy in my chosen career. The entire month of Feb has been hellish for me and has done wonders to deteriorate my social life. The reason – try relocating 10 units of HP 42U racks to another datacenter. Yes. You’re reading it right. 10 Racks.

Ten in itself is just a small number after all, so let’s complicate things a little bit shall we? Inside the racks we have at least 5 production servers that takes 3-5U’s each. In addition, each server has it’s own corresponding storage array comprises of 16 disks each and occupying 3U of space by itself. Around 50+ servers, not including any tape drive and media library mounted inside selected racks. To make matters more interesting, each server has 3 to 4 network points. I wouldn’t even begin with the IP addresses. 

The amount of coordination and planning involved in the project was insane. We were going through the design phase in months, even had the implementation date delayed several times due to high risk factors. At one point I honestly thought that the entire project will be cancelled, but eventually we had the Feb 21 and 28 as the final dates for implementation. 

The last 2 weeks pre-D day was more engaging than I could ever imagine. My brain was split into three sections: 1. Reserved for my most basic bodily functions – breathing, blinking, eating, muscular movements 2. Allocated for work related activities like meetings, planning, coordination, last minute checks and 3. High Availability (HA) for brain section number 2. Yes, for the past few weeks I was nothing short of being dubbed as a corporate zombie. 

To take things to the next level, another PM was pushing me for completion and sign-off of my design document for another project. There’s also another PM pushing for a sign-off and completion of an older project’s handover document. Times like these, I’d wish THIS was real.

Surprisingly, it wasn’t too difficult on the day itself. We moved a couple of messaging servers and domain controllers without a sweat. A pair of bluecoat servers was also moved with minimal effort. I’ve planned for 3 hours per rack to be moved to a different datacenter, and at the end of the day it took us way less than that complete the migration process. The entire move was completed in 2 weekends as planned. Casualties include 2 faulty disks resulting in a broken storage array, and failed backup which resulted from a simple cabling fault. I feel I should list down a few things that my fellow engineers should take note of, in case you find yourself in a similar project:

1. Get the details – Server serial numbers, warranty information, application green zone, power requirements of individual devices, existing cabling structure, device owner and support – just to name a few. I can’t stress how important it is to master the details. Get all the details of everything, and put them somewhere you can reach with ease. Come up with a spreadsheet with all the details, and sit down with your team to brainstorm anything you might have missed. Memorize the server names, existing location, new location, rack labels, and cabling details if you have to. You’ll thank yourself later if things get out of hand. 

2. Write down a solid task plan and stick to it – Once you have all the details sorted out, visualize the actual move and write it down in sequence. Play it over and over in your head until you can see it happening in front of your eyes. Think of it until you dream about it and you see things that may need to be improved or added to your plan. Use all the details you have in conjunction with visualizing the entire activity. Think of several scenarios that might occur and plan on it accordingly. 

3. Cover yourself – Always, always take into consideration the impact of whatever change that you’ll be performing. Make sure that you have a solid Backout plan (revert to original state pre-change) before the activity. Make sure that you’re not taking unnecessary risks. Downtime caused by bumping into another rack during production hours is a sure fire way to get upper management’s attention.  

4. Get everyone in sync - Make sure that everyone knows what they’re supposed to do. Talk to all the people involved and make sure they know their corresponding roles during the implementation. It’s pretty standard to get all the contact information of all people involved, but with all the information you’re being fed daily you might miss this one. Don’t make that mistake. Ensure that everyone is in sync of the the plan and the timings. Make sure to inform and update your stakeholders every now and then. Oh, and don’t forget the users affected. 

5. It’s not over till it’s over – Once everything seems complete and you’ve passed all the necessary tests, continue to be alert. Double check if you have to. There may be some issues that may pop up later and that point to the activity as the cause. Monitor the hardware and ensure that everything is in order, get a sign-off or some form of handover document to get the owner’s concurrance that everything has been done according to plan and  your equipment in working order. If you’re changing any ip addresses make sure that hostname resolution is working as well.

Overall, I’d say the entire project team did an excellent job to ensure the overall success of the project. There’s a certain satisfaction that I felt upon completing the project that I hadn’t felt before. It was a great experience and made me feel a little bit more mature in terms of my professional career. I’ll be doing another move in the succeeding weeks, albeit lesser in scope but the impact is just as much if not greater.

Strict security policy works within the datacenter as expected, so no visuals for this entry. I’ll save it for another post.