Comment On Slaves to The Process

At large, multinational companies, change is slow because of The Process. Not that Matt had any major problems with The Process — he knew what he was getting into when he started his job. A change begets meetings, which beget approvals, which beget forms that have to be signed in triple-triplicate, which beget more meetings, and maybe after a month or two you will have successfully added a column to a report.

All of Initrode Global's IT staff were salaried employees, except for network management, which was outsourced to a team of Highly Paid Consultants — and both groups took their allegiance to The Process very seriously.

One morning after an update, a key server performed a scheduled reboot. Since the KVM switch had been set to another server, some familiar and ironic words appeared on the screen.

Keyboard error or no keyboard present

Press F1 to continue, DEL to enter SETUP

What would you do in this situation? Probably either "find a keyboard, plug it in, hit F1" or "turn the KVM switch, hit F1." And that's exactly what one of the techs, "Clyde," was about to do before a senior tech saw him.

"Whoah whoah whoah! Wait! Whatareyoudoing?"

Clyde looked up from the screen. "There was a BIOS error, I'm just going to clear it to bring this server back online." He was second guessing himself at this point because of the senior tech's reaction. "I mean, right?" he added.

"Just give me a minute. Don't touch anything."

Clyde stood motionless while the senior tech retrieved a clipboard. As he returned, he flipped through a few pages and sighed. "This change request doesn't specify any keyboard activity, let alone any authorization for us to use the keyboard. And further..." he paused for a minute. "Wait here a sec. Don't touch anything."

Moments later, the senior tech returned with a printout. In big bold letters, the heading read "INTWEBSRV017." His eyes scanned the page briefly. When he found what he was looking for, he pointed it out to Clyde. "See, here it says this is a hands-off server. Any actions not specifically authorized in a change request requires a separate change request with approval from a Senior Manager."

The young tech was aware of all of the red tape and approvals that were required for most changes, but this was different. It was a BIOS error that required a single keystroke to clear, and he couldn't imagine anyone having a problem with it. He tried to cautiously argue his point without denigrating The Process. "Well, the change request does say that we're responsible for rebooting the computer. Surely pressing F1 falls under that... and regardless-"

"NO," the senior tech emphatically interrupted. "I just said that we're not authorized for this! You think that it's safe to just make The Process up as you go? Without The Process, we have nothing. The Process Be Praised!"

Defensively, the junior tech replied "No, you've got me wrong! I'm not saying that we do anything without the appropriate change request form signed in triple-triplicate, I'm just saying that it should be assumed that rebooting the computer means that we can clear this BIOS error."

"Look, I understand it sounds a little backwards, but we can not deviate from The Process."

Eventually the discussion grew to encompass everyone in the server room, and soon all of the techs were on an email chain to find out whose responsibility it was to clear the error.

Meanwhile, a whole department was sitting around twiddling their thumbs. Word had trickled down that there was some sort of server error and that the HPCs were on it. At this point, no one outside of the admins or the senior management knew exactly what the problem was, but assumed it must be big if the server was completely offline.

Hours passed and word eventually trickled down that it was a simple BIOS error, but not to worry, the network team wouldn't rest until it was fixed. Occasionally one of the admins could be spotted walking around collecting signatures.

The problem was collecting the right signatures — first, the dev manager refused to sign since their updates couldn't have caused the BIOS error, and that was outside of her jurisdiction. "Try the infrastructure team," she suggested.

"Yeah, that's uh, not really my, uh, domain," the infrastructure lead insisted. "Maybe Gary can help?"

Gary, the network lead, was more accommodating. "Yeah, I can sign off on this once I have a signed Network Request form."

An Emergency Change Request ticket was opened, and over the next few hours updates were provided and forms were signed. In the intervening time, most staff in the out-of-work department had gone home for the day, wondering whether they'd get to do any work the following day. The completed Network Request form was brought to Gary for his signature. And still, no one was sure whose responsibility it was.

"OK," Gary said with a smile. He signed his Herbie Hancock on the form and said "Great, we're all set now."

"That is, after we give the corporate VP a call."

After a long discussion with the corporate VP, an email indicating his approval, ticket resolutions, form filings, and last-minute meetings, the request to press F1 was finally approved.

The senior tech returned to talk to Clyde, who he now thought of as his mentee. "Good news. We got approval to clear the error. The Process works!" He looked at the screen again.

Keyboard error or no keyboard present

Press F1 to continue, DEL to enter SETUP

"I'll let you do the honors," he said to Clyde.

Clyde found himself at the same place he was nine hours earlier, and thought about how the day could've been different had the senior tech not seen him about to clear the error. He could've prevented the waste of hundreds of man-hours of putting a whole department out of work while the teams squabbled over whose responsibility it was, what signatures were required, and how many channels the change requests had passed through.

In an anticlimactic quarter of a second, Clyde pressed F1. He stayed to watch the system boot and to ensure that there were no other errors.

The good news is that The Process will be updated with more specific instructions should this scenario ever arise again. That is, after some Process Change meetings are held, approvals are received, forms are signed in triple-triplicate...