Who me? The right tool for the job is our motto.But in this week’s Who me? One Register The reader recalls what happens when the right tool uses the wrong tool.
Today’s story, from “Keith,” brings us back to the exciting days of the 1990s and the costly delivery of market data to customers eager to maintain an advantage.
When all the alarms in the world seem to sound on Keith’s console at the same time, the market is about to close on the day in question. Important customers suddenly lose connection and important data stops flowing. He glanced at the time—the maintenance window never opened before half an hour after the market closed—so something bad must have happened.
He rushed to the communication room and slammed the door open… and found that the communication team leader was standing shyly next to a bunch of modems and multiplexers that suddenly crashed and a harsh UPS. He is holding a screwdriver in his hand. Oh dear.
Target work? Replace the modem. Very simple, but due to the evolution of wiring over the years, wiring is not as simple as people hope. All power cords are bundled with cables.
In any case, the communications boss has decided to take the lead and untie the cable tie. Of course, the market is still open, but he is not trying to unplug anything, right? Just some preparatory work.
Of course, the cutter is on the other side of the room. But he has a screwdriver, so in order to save time, he chooses to break the tie by inserting the screwdriver, twisting and… oops.
“The tip of the drive slipped off,” Keith said, “and cut a power cord at random and sent a spike to the entire rack. UPS is complaining about overvoltage/undervoltage spikes and screaming to wake the dead.”
Usually, the circuit breaker in the power board can prevent too much damage, but the vibration is so severe that even the circuit breaker is damaged. Keith’s team is divided into two groups-one group looks for a spare strip and reconnects, the other group restores the hardware.
“Many of them are single powers,” he said sadly. “Not only that, almost every device has a blown fuse.”
Newer devices have their own circuit breakers, but most kits fall into the category of “replacement delayed due to budget”; technically, they are still within their useful life, but are already long in the teeth.
“No problem, right?” Ji thought. “We put a box full of fuses somewhere?”
No they don’t. They searched shops, closets and basements and found fuses of the wrong type and grade. Finally, the decommissioned devices were torn apart and their fuses were removed.
“I think we finally fully restored the rack at around 8pm, which is about four hours after the market closes,” Keith recalled. “This included upgrading two devices from newly installed hardware, which was originally scheduled to be used. For other purposes.
“Some devices need to be reconfigured manually, because of course we don’t have a backup of the configuration, except for some random printouts.”
A meeting was held the next morning, which included the largest cheese, including the CEO and president. As the head of operations, Keith attended the meeting. Like Sir Screwdriver himself, the communications director.
As expected, the action items came thick and fast. “Communications and operations must check whether each other’s lines meet the standards,” Keith recalled. “My job is to write down from the word of mouth knowledge we have been using.”
This work includes checking equipment and power supplies. For the 100 or so servers in the operating room and their neat wiring, this is not a problem. However, for Keith in the communication room and his less structured cable management method, this is a completely different challenge. He ran a dozen miswired equipment and racks on two power boards on the same UPS-luckily, all of these were fixed and documented before Sir Screwdriver restored his time-saving ways.
Some things did not get better in the past.
Have you stuck the screwdriver in the wrong place? Or try to restore a configuration that is only in the memory of the employee with the longest service time? Tell us your story via email, who am I? ®
