|
|
|
| Non-WTF Job: C++ Developer at Good Grievance (Ronkonkoma, NY) |
| « HTTP 414: Way Too F#%&ing Long | Please Pay $0.00 » |
As a junior-level sysadmin at his university, Alcari had gotten used to frantic, middle-of-the-night support calls. Whether it was a mail proxy server freezing up, a replication process getting out of sync, or some application deadlocking, Alcari's solution was almost always the same: reboot the problem server. On a recent 4:00AM emergency call, however, the situation was a bit more dire.
"Ummm," the student working the graveyard shift at the helpdesk started, "I think the network is down. I'm getting calls from everyone, everywhere. Can you, like, check into this. Now?"
Alcari fired up his laptop and noticed that his connection to the university was not responding. He couldn't even ping the school's home page. Worst case scenario, he figured, the network equipment had taken a dive, and he'd have to replace a router or two. It certainly wasn't the best way to spend a Monday morning.
However, as he drove towards the university's utility building, he realized that he may have underestimated the source of the problem. His first clue was the small motorcade of police cruisers that whizzed by him. The second was when he almost got run off the road by a giant firetruck. But it all really sunk in when he noticed that they were all headed towards the same place he was: the smoldering remains of the utility building.
Alcari parked his car and navigated under a few lines of red and yellow police tape towards the center of the circus. By the time he made his way through, most of the excitement had died down, as the firemen were wrapping up their hoses and making sure no fiery embers remained. Alcari found his way towards an official.
"Well," the uniformed officer responded, "the good news is that no one was hurt. As for what happened... so far as we can tell, an acetylene cylinder must have tipped over, rolled across the floor, and blocked a heat register. That woulda caused it to decompose, explode, and, as you can guess, cause all the dozen or so acetylene cylinders to do the same. This is why we always chain up cylinders!"
As Alcari waited for the emergency crew to finish, he spotted a group of very sleepy coworkers walking towards the scene and met up with them to chat. "I'm thinking," one of the techs said, "it's gotta just be a power outage... you know, a few cut wires from the blast. There are like, four concrete walls between gas storage and the data center."
"Yeah," another tech responded with a gleam of hope, "explosions always take the easiest path, which woulda been the outside wall. We'll just have to babysit the power guys when then come in for repairs."
Yet another tech jumped in, "you know, we could probably even run the whole show if we can get another back-up generator."
Alcari was optimistic, too. They were all certain they'd have things up and running before day break. Within the hour, the fire marshal declared the building "structurally sound" and let Alcari and the team inside to look around. It wasn't a pretty sight:
As soon as they walked into The Server Room That Was, everyone's hopes immediately faded. The sight of molten cables, dripping plastic, and shards of concrete lodged into warped server racks was eclipsed only by the putrid smell of burnt electronics. It was kind of like that power supply we've all sniffed after burning out, but infinitely worse.
Sitting in the center of the former data center was the remains of The Beast: the university's core server that housed email, web, and several other critical functions. The small, metal plate reading "Core Server - Do Not Touch" was about all that was left intact. Everyone was completely bummed out.
"Wait a minute, guys," one of the techs lit up, "I know for a fact we set up a Secondary Server just in case... well, this happened."
"That's great!" Alcari remarked, once again hopeful that he might get some sleep today, "where's it located? The Administrative building?"
One grim-faced coworker cleared his throat, and walked over to the remains of a less-fortunate metal shelf. He brushed off some soot from a small, metal plate. It read "Secondary Server - Do Not Touch."
|
Hi, it's my story/event, so I'll try to answer a few questions:
No, unfortunately, there aren't any other pictures, that one was taken with my crappy VGA camera phone that night, the closeup pics are all black. The following day they blocked off the whole area, as someone decided that a gaping hole in a load-bearing wall probably wan't helping the building stay up.
Those aren't much good if you a lack a server to put them on. We did use them once we had jurry-rigged a network and replaced half a dozen routers to get it work a little. They were stored in the same building, but at least they were stored in "the vault" in the basement. I've never actually seen "the vault", so I can only hope it wasn't designed by the person who designed the server room.
Oh, that was nice. The installing of said racks in a temporary building, fixing/replacing the destroyed cables followed by round-the-clock work to get everything working, then doing it again when the new building was done, wasn't so much fun.
Well, they weren't stored in the same room. They were stored in the nearby gas storage building. (on the other side of the hallway, meaning one concrete wall, and a brick interior wall.) Except that it wasn't actually meant to store gas cilinders, it was designed to store sand, cement, gravel sheet metal and the like. It had, to quote one of the civil enginering students, "More in common with a lean-to garden shed then a gas bunker".
A wrecking ball and a bulldozer. The whole building had to go, seeing how there was a hole in the wall large enough to drive a small car through. |
| « HTTP 414: Way Too F#%&ing Long | Please Pay $0.00 » |