Comment On A Bit More Dire

As a junior-level sysadmin at his university, Alcari had gotten used to frantic, middle-of-the-night support calls. Whether it was a mail proxy server freezing up, a replication process getting out of sync, or some application deadlocking, Alcari's solution was almost always the same: reboot the problem server. On a recent 4:00AM emergency call, however, the situation was a bit more dire. [expand full text]
« PrevPage 1 | Page 2 | Page 3Next »

Re: A Bit More Dire

2008-06-10 10:04 • by Andreas Christodoulou (unregistered)
And I know of a few IT Professionals who would STILL just say "Ah, let's just reboot the affected servers then" when told of this problem.

Re: A Bit More Dire

2008-06-10 10:13 • by DOA
I'd like to hear the support calls...
"Hi, I can't login"
"Yeah, the server room blew up"
"What?"
"It blew up. Someone tried to log in during maintenance even though we told everyone not to do it. Say... was that you trying to log in a few minutes ago?"
*click*

Re: A Bit More Dire

2008-06-10 10:13 • by WhiskeyJack
So the server room burns down, and there's no offsite backup?

This is what counts as a "curious perversion in information technology" nowadays?

Re: A Bit More Dire

2008-06-10 10:14 • by Dutchy (unregistered)
"At least we have backups securely stored."
"You mean.. this?"

* Holds up a sign: "Backup shelf, do not touch."

Re: A Bit More Dire

2008-06-10 10:15 • by NaN
199639 in reply to 199629
Andreas Christodoulou:
And I know of a few IT Professionals who would STILL just say "Ah, let's just reboot the affected servers then" when told of this problem.


If you can't turn it off and on again, they won't even talk to you.

Re: A Bit More Dire

2008-06-10 10:26 • by Edd (unregistered)
How come the box of copier paper survived unscathed?

Re: A Bit More Dire

2008-06-10 10:26 • by alx5000 (unregistered)
*Core coment*
Do Not Touch

Re: A Bit More Dire

2008-06-10 10:30 • by Resolute (unregistered)
199642 in reply to 199641
alx5000:
*Core coment*
Do Not Touch


1. Read article
2. Parrot something said in article
3. ???????
4. Featured comment!

Re: A Bit More Dire

2008-06-10 10:31 • by JohnnyWalls (unregistered)
Fist?

Re: A Bit More Dire

2008-06-10 10:32 • by Edwin (unregistered)
Also the smartest place to keep your second server if you planned its for cases like this. Why people always think of things like that when its to late.

Re: A Bit More Dire

2008-06-10 10:37 • by Grovesy
199647 in reply to 199639
Reminds me of a story from a friend,

He was brought in towards the end of the procurement procedure for a disaster recovery contract his company was purchasing; it was pretty big they would install a complete duplicate of their live + staging systems in a DR centre.

The very senior managers had done all their stuff, selected the vendor that gave them the most freebies along with the cheapest price and it was up to him to get on with the actual ‘doing’ part of the deal.

After preliminary contract acceptance and various statements of intent, he was down at his company’s data centre, on the phone to his counterpart underlings from the DR company, when out of the corner of his eye, out through the window, across the way and noticed a large, squat building in the same business park with the DR companies logo on the front. He politely asked where their data centre was...

The real WTF was, management ignored his 'concerns' the contract went ahead as the DR vendor was the cheapest.. shame the DR centre was pretty much next door.

Re: A Bit More Dire

2008-06-10 10:39 • by Lars Vargas
TRWTF: No computer-safe fire suppression appears to have been in place (or at least not mentioned). Halon, anyone?

And yeah, what about those off-site backups?

Re: A Bit More Dire

2008-06-10 10:41 • by Cyrus
199650 in reply to 199643
JohnnyWalls:
Fist?


Congratulations on making a "fist" comment 25 minutes after the first comment?

Re: A Bit More Dire

2008-06-10 10:42 • by Shial
Well, on the bright side they get the chance to test out the procedures for worst case scenario.. assuming the procedure manual survived.

Re: A Bit More Dire

2008-06-10 10:44 • by Grovesy
199652 in reply to 199649
Lars Vargas:
TRWTF: No computer-safe fire suppression appears to have been in place (or at least not mentioned). Halon, anyone?

And yeah, what about those off-site backups?


umm, unless my sarcasm detector has failed, I don't see how Halon would have helped against an explosion which embedded bits of concrete into the servers...

Re: A Bit More Dire

2008-06-10 10:46 • by floorpie (unregistered)
That's nothing,

I worked at a place where the server room was under the bathroom from the floor above... until the neighbours shower sprung a leak...

Re: A Bit More Dire

2008-06-10 10:48 • by AccessGuru
What would Jesus do?

Re: A Bit More Dire

2008-06-10 10:48 • by WhiskeyJack
199655 in reply to 199651
Shial:
Well, on the bright side they get the chance to test out the procedures for worst case scenario.. assuming the procedure manual survived.


No worries, it was kept in PDF format on the server hard drive.

And in case something happened to it, you'll find another copy in the /backups/ directory of the same drive...

Re: A Bit More Dire

2008-06-10 10:57 • by Quango (unregistered)
Reminds me of an incident a few years ago when I was tech support for a "large international merchant bank" in the City of London, when the Bishopsgate bomb went off (see http://en.wikipedia.org/wiki/1993_Bishopsgate_bombing )

Fortunately our building only suffered a few cracked and broken windows - but was just inside the police cordon so all of the several thousand staff were evicted the whole site was off-limits for about a day (a LONG time if you're a market trader).

The nasty realization for the disaster recovery team was that the Disaster Recovery (DR) site, one of the bank's older unused offices, was right in the middle of the cordon. Worse still, the main fibre optics linking the two sites ran right down the Bishopsgate. Which now sported a "bomb crater about 4.3 m deep and 4.5 m in diameter"

Needless to say our DR site moved elsewhere later that year.

Re: A Bit More Dire

2008-06-10 10:58 • by Jake Grey
Well, at least nobody can blame the tech support team for that one, though I'm sure some network guru is having an apoplectic stroke over the primary and backup servers in physical proximity by now.

Re: A Bit More Dire

2008-06-10 11:01 • by Mike (unregistered)
Perhaps it was the nature and content of the article, but I swear when I read the last part about the secondary server, I thought it read "Secondary Server - Do Not ToRch".

Re: A Bit More Dire

2008-06-10 11:01 • by lonewolf
TRWTF is that shovel in the server room..

Re: A Bit More Dire

2008-06-10 11:02 • by Matt (unregistered)
199661 in reply to 199655
You can solve any computer problem by adding more memory.

Re: A Bit More Dire

2008-06-10 11:03 • by dkf
199662 in reply to 199654
AccessGuru:
What would Jesus do?
Keep off-Earth backups, of course. You can't be too careful.

Re: A Bit More Dire

2008-06-10 11:05 • by Hans (unregistered)
You know, I just went through this with about 7500 hundred other customers at the planet, when there was an electrical explosion that knocked the entire center off line for 2 days. 9000 servers went dark instantly.

Re: A Bit More Dire

2008-06-10 11:06 • by Hans (unregistered)
199664 in reply to 199654
What would Chunk Norris do?

Re: A Bit More Dire

2008-06-10 11:10 • by Nobody (unregistered)
I was waiting for the molten slag of a shelf with a tag reading "Backup Tapes - Do not Touch".

Re: A Bit More Dire

2008-06-10 11:11 • by akatherder
Why were there a dozen acetylene tanks in a server room? Do they keep some dynamite in the phone closet?

Re: A Bit More Dire

2008-06-10 11:12 • by SQB
199668 in reply to 199651
Shial:
Well, on the bright side they get the chance to test out the procedures for worst case scenario.. assuming the procedure manual survived.

You mean this? (Holding up charred remains of a cover sheet, on which you can still make out fragments of words "...nual - Do No T...")

Re: A Bit More Dire

2008-06-10 11:13 • by Ghost (unregistered)
199669 in reply to 199662
dkf:
AccessGuru:
What would Jesus do?
Keep off-Earth backups, of course. You can't be too careful.


In case of the Vogons arriving?

Re: A Bit More Dire

2008-06-10 11:15 • by Matt.C


So that's why it's a bomb.

Given the description of these canisters, looks like the Atari ST Row of Bombs might be more appropriate...

Re: A Bit More Dire

2008-06-10 11:15 • by Steve (unregistered)
This isn't a WTF?. It's an [b]Oh, sh*t[/i].

On the bright side, it means no more struggling with that crappy old server. . .

Re: A Bit More Dire

2008-06-10 11:18 • by f0dder (unregistered)
199672 in reply to 199654
AccessGuru:
What would Jesus do?
Jesus... wept.

acetylene tanks... what, you don't zap old hard drives before disposal?

2008-06-10 11:19 • by Hoser, Eh (unregistered)
certainly not best practices, then.

Re: A Bit More Dire

2008-06-10 11:21 • by drex_ej (unregistered)
199674 in reply to 199671
Seriously, I'd be happy about getting a couple racks full of new toys. Then again, we have off site storage.

Re: A Bit More Dire

2008-06-10 11:23 • by CoderHero (unregistered)
199675 in reply to 199671
Steve:
This isn't a WTF?. It's an [b]Oh, sh*t[/i].

On the bright side, it means no more struggling with that crappy old server. . .


Perhaps not THAT crappy old server, but I can see this scenario:

Manager 1: That fire cost us $120,000 in computer equipment and we don't have the budget to replace it
Manager 2: Well, I did hear that the Salvation Army is selling used 486's on the cheap!

Re: A Bit More Dire

2008-06-10 11:31 • by Mutant (unregistered)
199676 in reply to 199649
Lars Vargas:
And yeah, what about those off-site backups?


They may have had offsite backups... but without a server to restore them to, it's still going to be a lot of work...

It's also really hard for people to see ahead to what effects a *real* disaster will bring before it happens. Especially those signing the bills. Who's really going to predict a server room blowing up?

My company has suddenly bumped DR right to the top of the priority list after 3 incredible near misses (all unrelated) in 2 weeks. Before then enhancing our current (fairly reasonable) strategy wasn't deemed cost effective. Probably fair enough... I mean, how far do you take it? Nuclear War? Asteriod Strike? The Second Coming?

Re: A Bit More Dire

2008-06-10 11:34 • by some guy (unregistered)
If (User.Name == "Lyle")
{
Server.log = "Will I dream while I am sleep Dave?";
Server.SelfDestruct = true;
}

Re: A Bit More Dire

2008-06-10 11:40 • by Alan (unregistered)
199680 in reply to 199662
dkf:
AccessGuru:
What would Jesus do?
Keep off-Earth backups, of course. You can't be too careful.


That would be quite easy to do, all you have to do is transmit your data out into space.

Recovery would be more difficult as it would involve an FTL drive.

Re: A Bit More Dire

2008-06-10 11:41 • by Flash
199681 in reply to 199676
Mutant:
My company has suddenly bumped DR right to the top of the priority list after 3 incredible near misses (all unrelated) in 2 weeks.

Congratulations on making those "near misses" appear to be "unrelated." How did you do it?

Re: A Bit More Dire

2008-06-10 11:41 • by real_aardvark
199682 in reply to 199676
Mutant:
Lars Vargas:
And yeah, what about those off-site backups?


They may have had offsite backups... but without a server to restore them to, it's still going to be a lot of work...

It's also really hard for people to see ahead to what effects a *real* disaster will bring before it happens. Especially those signing the bills. Who's really going to predict a server room blowing up?

My company has suddenly bumped DR right to the top of the priority list after 3 incredible near misses (all unrelated) in 2 weeks. Before then enhancing our current (fairly reasonable) strategy wasn't deemed cost effective. Probably fair enough... I mean, how far do you take it? Nuclear War? Asteriod Strike? The Second Coming?
Oh, come now. The Second Coming isn't about DR -- it's a huge business opportunity.

Just imagine all those Scientologists and al Qaeda folk lined up to log in to www.dont-be-a-goat.com.

The bad news is, most of the money will go to a bunch of raving nutters deep in the Appalachians.

Re: A Bit More Dire

2008-06-10 11:44 • by Pat (unregistered)
199683 in reply to 199667
akatherder:
Why were there a dozen acetylene tanks in a server room? Do they keep some dynamite in the phone closet?


There weren't They were four concrete rooms away.

Re: A Bit More Dire

2008-06-10 11:45 • by snoofle
199684 in reply to 199676
Mutant:
Lars Vargas:
And yeah, what about those off-site backups?


They may have had offsite backups... but without a server to restore them to, it's still going to be a lot of work...

It's also really hard for people to see ahead to what effects a *real* disaster will bring before it happens. Especially those signing the bills. Who's really going to predict a server room blowing up?

My company has suddenly bumped DR right to the top of the priority list after 3 incredible near misses (all unrelated) in 2 weeks. Before then enhancing our current (fairly reasonable) strategy wasn't deemed cost effective. Probably fair enough... I mean, how far do you take it? Nuclear War? Asteriod Strike? The Second Coming?

Someone suggested this in another (unrelated) article, but it's apropos...

1. Buy enough deck planking to make a sufficiently large wooden table, and assemble
2. Buy enough arts-n-crafts paper in a variety of colors to cover said wooden table
3. Arrange colored paper on wooden table such that each sheet represents one pixel of the appropriate rgb value of the image you wish to back up
4. Await Google imaging satellite to fly over
5. Reuse paper and deck for subsequent imaging needs
6. Voila: cost effective offsite backup of even the biggest images!

Re: A Bit More Dire

2008-06-10 11:48 • by morry (unregistered)
Are there any more pictures? I can barely make out anything other than the studs in that one.

Re: A Bit More Dire

2008-06-10 11:52 • by stark (unregistered)
Haha, similar incident happened 2 times within a 2 year timespan in the beginning of this decade, in Kista, a suburb to Stockholm, Sweden. The "silicon valley" of Sweden.

Major power outage. Both citizens and big companies like Ericsson, Tele2, IBM where hit. It took them several days (yes days!) to fix it... Power to the whole area was supplied from a huge tunnel with both primary and secondary mains close together.

Re: A Bit More Dire

2008-06-10 11:53 • by Ben (unregistered)
199687 in reply to 199664
Hans:
What would Chunk Norris do?

Chuck Norris would use a dozen acetylene tanks to create an explosion which would rip through four concrete walls and reduce an entire data center to slag and ash. HTH

Re: A Bit More Dire

2008-06-10 12:00 • by Man 987876980 (unregistered)
199689 in reply to 199687
Ben:
Hans:
What would Chunk Norris do?

Chuck Norris would use a dozen acetylene tanks to create an explosion which would rip through four concrete walls and reduce an entire data center to slag and ash. HTH
No, not Chuck Norris, he said ChuNk Norris... who I presume is Chuck's obese brother.

Re: A Bit More Dire

2008-06-10 12:02 • by null (unregistered)
Q: How much more dire?
A: None. None more dire.

Re: A Bit More Dire

2008-06-10 12:14 • by FredSaw
Reminds me of my coworker's wife. He gave her a spare car key in case she locked herself out. She put it in the trunk.

Re: A Bit More Dire

2008-06-10 12:17 • by Man 987876980 (unregistered)
And of course, Chuck Norris wouldn't need tanks of explosive. He'd just do a roundhouse kick to get the same effect.
« PrevPage 1 | Page 2 | Page 3Next »

Add Comment