Stupid data center tricks Part II
One day, Bowers had taken some personal time and was out for a jog when his iPhone rang — the switch in the school of medicine’s server room was overloaded, causing denials to every service it hosted.
“The green lights go on and off when packets pass through,” he explained. “It had ramped up until the lights were more on than off.”
Bowers quickly began troubleshooting over the phone. He was able to determine that nothing on the school of medicine’s network had changed. Then he remembered that purple cable. He told his co-worker on the phone to unplug it, and activity on the switch went back to normal. Then he had his co-worker plug it back in and the switch overloaded again, proving that the problem was at the other end of the purple cable — in the university hospital building.
It turned out that an IT staffer who was normally based out of a satellite location came to the university hospital’s IT room to work on a project and needed extra connectivity. He inadvertently created a loop by plugging two network cables from the university switch into a hub he had added to the network so he could attach additional devices.
Get Daily News by Email
“So it kept trying to send data around in a circle, over and over,” says Bowers, which in turn caused the switch in the school of medicine building to overload.
Bowers says the network was cobbled together like that when he began working at the university, so he inherited the setup — which a better approach to network planning and design would have no doubt flagged as problematic. But at least now the IT department knows one scenario to avoid going forward: Jury-rigged cabling and traveling techies can be a bad mix Microsoft MCTS Training.
“We didn’t do an official lessons learned [exercise] after this, it was just more of a ‘don’t do that again,'” says Bowers. However, this event, combined with another incident where a user unwittingly established a rogue wireless access point on the school of medicine’s network and overloaded the switch, has convinced Bowers of one thing: “I hold to the concept that human errors account for more problems than technical errors,” he says.
Save $35, lose all your data
More often than not, data center mishaps are caused, directly or indirectly, by employers’ attempts to save money. In this case, it was all about saving $35 on a backup tape.
In 1999, Charles Barber worked as technical support manager at a health-instrument company (one that no longer exists) that made stand-alone, server-based equipment that connected to treadmills to collect the data resulting from patient stress tests. One of the company’s customers was a small medical practice in St. Louis where the administrative assistant also served as the IT person Microsoft MCITP Certification.
“She was pretty competent” — but not a trained IT professional, says Barber.
One Friday evening, she heard strange noises coming from the equipment’s server and realized that the hard drive had failed. That Saturday she purchased a new hard drive, installed it and reloaded Microsoft’s Windows Server and SQL Server, since she had saved the discs and documentation. Barber had provided written instructions for her on how to configure the server, in case such a thing ever happened, and the assistant did so successfully. (“I’ve had field engineers call me to get help with these things,” says Barber, but this woman managed it on her own.)