Wednesday, February 29, 2012

Power outage at work today - work outage tomorrow

Quite a show. A transformer said goodbye cruel world and burned blue and white hot. We could see it from the tenth floor.  This caused the first outage, which of course shut down all the computers in the office. A few minutes later, the power came back on, abruptly, for a few minutes. The next closest transformer, a half block away, was so grieved by the loss of its neighbor that it threw in the towel as well and the power went down for an extended period.
The UPS in the server room worked perfectly. I had plenty of time to start shutting down servers gracefully.  I was near done when the power came back on, so I waited a bit to see if we'd lose power again, and started rebooting servers. They all came back up just fine.  Except for the two most important to the network, the domain controllers. Oddly, these never went down, were not shut down, nor rebooted. As an analogy, imagine all the traffic lights in your entire town defaulting to blinking red at once. Traffic comes to a grinding halt. Such is the result of a network domain controller going down.
 The power surges, in spite of surge protectors and massive UPS batteries in line, spiked the VM Ware, upon which the domain controller virtual machines reside.
This all transpired near the end of the day. I realized where the problem lay pretty quickly and called my outsource IT folks stat.  A tech arrived reasonably quickly but that's pretty much where the quick service ended.  The abridged version is that he had no effing idea how to deal with the virtual servers that his company built.
I pulled the plug on him at 9pm.
So, it's off to work at extra dark thirty in the morning to meet a different tech that knows our system better.  I'm praying for a quick resolution, otherwise I'm going to have 50 people twiddling their thumbs. Then will come the hand wringing and gnashing of teeth because there is no internet or email.
After this little crisis is resolved, I'm going to have a little church chat with my outsource folks as to why they put both the primary and secondary domain controllers on the same host server.
I've always had a thing about putting all the eggs in one basket.  IT guys seem to just love virtual servers, stacking them up on a host machine to max out the processor and available RAM.  This is efficiency in their eyes I suppose. Even with full backup images of the servers, it's a recipe for disaster in my opinion.
Today is a case in point. All the physical servers are doing just dandy.  Two of the virtual servers are virtually....dead.
So, tomorrow, I have this melt down to deal with plus a full day of meetings prepping for a major presentation to make the cut for a major project.  Said presentation occurs at the end of the day.
  Sucks to be me right now.
When I started this post, I was pretty sure it was going to be an abject four letter word fest.
Perhaps blogging is a form a therapy. Or, perhaps I'm maturing...nah, that couldn't be it.
What I'd really like to do is ream out the owner of my outsource IT company at full vocal volume.  It won't be necessary however, I have a meeting with one of his competitors a week from Friday.



  2. lmao. That made me feel better for about 30 seconds.
    Thanks young Skunktrapper.

  3. Good luck.

    Hopefully all it will take is moving the vmdk's to a new host and your off and running.

    We are highly virtualized too.
    My favorite was when the admin shut down instead of logged off of the host before we were running the esx version of vm-ware taking all servers out.


Comments are not moderated. Disagreement is fine as long as you address the message, not the messenger. In other words, don't be an ass.