Through the Roof
In the first weeks of September, 2005, a small group of Northrop Grumman colleagues, and a small group of IBM customer advocates and engineers were sitting around a room and on a conference call. The subject: How to get twenty or so tons of computing hardware out of the soup-bowl known as New Orleans.
Just days earlier, the levees protecting New Orleans from water in all directions had failed in spectacular fashion, and the city was inundated, in some places with over twenty feet of water. My program had eight figures worth of IBM server and storage infrastructure on the top floor of a building on the lakefront of the aforementioned soup-bowl. On the gear, the first two years of work on the DIMHRS project, a quarter-billion-dollar effort to consolidate a number of personnel functions across the military. I took a stab at weighing out the value of those two years and figure somewhere north of $100 million in intellectual property, entirely out of reach.
Now, before we get all spun up about a disaster recovery plan and all that, I feel it's important to note that we were already executing our recovery, and we were, at that very moment, already restoring data from our offsite backups onto a degraded alternate architecture, and the two hundred fifty or more employees were just about ready to get back to it. However, much like most DR plans, ours didn't so much take into account the simple and quite obvious idea that we might not be able to return to our primary facility for a very long time. The answer? Go get the gear.
How? We had fourteen frames of service gear and storage, wired into the infrastructure at our New Orleans data center, the largest of which was four thousand pounds all by itself. And, as for conditions, we assumed the worst. Roads are flooded, so no trucks off the highway to the facility. No electricity, so the elevators don't work.
So, here we are in this blue-sky meeting. All ideas are worth reviewing, they say. Anything that gets us our gear.
Lots of ideas flew around, most of which simply wouldn't work due to one or the other of "no power" or "no roads."
Suddenly, up pipes Ronnie P. of IBM. Apparently, Ronnie took the "we'll consider anything," quite seriously.
"How about this," he says. "I can get a couple of buddies to get hold of a CH-47 helo." He's a tactical guy, complete with a gelled-up crewcut. Tactical guys love to say things like "helo." Just in case it's not clear what we're talking about here, this is a CH-47:
"I say we park the trucks out on the I-10 somewhere. Then we get plasma cutters, and cut a hole in the roof of the datacenter. Then we can just cable up the racks and lift them out. We can land them on the highway, where we just roll 'em onto a truck. No problem."
That's right. Plasma cutters, a hole in the roof, and a helo lift, no problem.
You could have heard a pin drop. Most of us thought that he was out of his mind. Of course, we'd known Ronnie for a while, and we all pretty well knew he was at least a little out of his mind. But, I honestly think my buddy John fell in love with Ronnie, just a little bit, at the exact moment that he started talking about peeling off the roof of a DoD datacenter like the lid of a sardine can.
Of course, despite the fact that "all ideas" were being considered, this one was deemed even too ludicrous for us. And, fortunately (or not), before we got too much farther, we were informed that the New Orleans facility did, in fact, have generator power, access by roadway, and a working elevator. So much for the IT A-Team.
Now, on Monday, Jon and I had just finished recounting this story to a host of Later-Than-Katrina employees on the program. So, imagine my surprise on Wednesday, when I see this story, outlining a daring plot by Swedish bank robbers to, essentially, hover a helicopter over a bank's roof, rappel into the building through a hole in the roof, and airlift out the millions in loot. Sound familiar? I thought so.
We discussed at length, how anyone could have come up with such a Hollywood style crime, and then had the balls to pull it all off. We aren't positive, but we believe if you look very closely at the surveillance pictures of the helicopter, you'll be able to see that the pilot, cackling wildly, has a tactical haircut, and a shirt that says "IBM."
How do you say,"I love it when a plan comes together," in Swedish?
Just days earlier, the levees protecting New Orleans from water in all directions had failed in spectacular fashion, and the city was inundated, in some places with over twenty feet of water. My program had eight figures worth of IBM server and storage infrastructure on the top floor of a building on the lakefront of the aforementioned soup-bowl. On the gear, the first two years of work on the DIMHRS project, a quarter-billion-dollar effort to consolidate a number of personnel functions across the military. I took a stab at weighing out the value of those two years and figure somewhere north of $100 million in intellectual property, entirely out of reach.
Now, before we get all spun up about a disaster recovery plan and all that, I feel it's important to note that we were already executing our recovery, and we were, at that very moment, already restoring data from our offsite backups onto a degraded alternate architecture, and the two hundred fifty or more employees were just about ready to get back to it. However, much like most DR plans, ours didn't so much take into account the simple and quite obvious idea that we might not be able to return to our primary facility for a very long time. The answer? Go get the gear.
How? We had fourteen frames of service gear and storage, wired into the infrastructure at our New Orleans data center, the largest of which was four thousand pounds all by itself. And, as for conditions, we assumed the worst. Roads are flooded, so no trucks off the highway to the facility. No electricity, so the elevators don't work.
So, here we are in this blue-sky meeting. All ideas are worth reviewing, they say. Anything that gets us our gear.
Lots of ideas flew around, most of which simply wouldn't work due to one or the other of "no power" or "no roads."
Suddenly, up pipes Ronnie P. of IBM. Apparently, Ronnie took the "we'll consider anything," quite seriously.
"How about this," he says. "I can get a couple of buddies to get hold of a CH-47 helo." He's a tactical guy, complete with a gelled-up crewcut. Tactical guys love to say things like "helo." Just in case it's not clear what we're talking about here, this is a CH-47:
"I say we park the trucks out on the I-10 somewhere. Then we get plasma cutters, and cut a hole in the roof of the datacenter. Then we can just cable up the racks and lift them out. We can land them on the highway, where we just roll 'em onto a truck. No problem."
That's right. Plasma cutters, a hole in the roof, and a helo lift, no problem.
You could have heard a pin drop. Most of us thought that he was out of his mind. Of course, we'd known Ronnie for a while, and we all pretty well knew he was at least a little out of his mind. But, I honestly think my buddy John fell in love with Ronnie, just a little bit, at the exact moment that he started talking about peeling off the roof of a DoD datacenter like the lid of a sardine can.
Of course, despite the fact that "all ideas" were being considered, this one was deemed even too ludicrous for us. And, fortunately (or not), before we got too much farther, we were informed that the New Orleans facility did, in fact, have generator power, access by roadway, and a working elevator. So much for the IT A-Team.
Now, on Monday, Jon and I had just finished recounting this story to a host of Later-Than-Katrina employees on the program. So, imagine my surprise on Wednesday, when I see this story, outlining a daring plot by Swedish bank robbers to, essentially, hover a helicopter over a bank's roof, rappel into the building through a hole in the roof, and airlift out the millions in loot. Sound familiar? I thought so.
We discussed at length, how anyone could have come up with such a Hollywood style crime, and then had the balls to pull it all off. We aren't positive, but we believe if you look very closely at the surveillance pictures of the helicopter, you'll be able to see that the pilot, cackling wildly, has a tactical haircut, and a shirt that says "IBM."
How do you say,"I love it when a plan comes together," in Swedish?
posted by Mark of NOLA at 8:00 AM
<< Home