San Diego Power Outage Shows Importance Of Disaster Recovery


The massive electrical power outage which on Thursday night turned the lights off throughout Southern California, Arizona, and northern Mexico caught solution providers and their customers off-guard, but proper disaster recovery programs appears to have helped many recover quickly.

Even so, the outage once again pushes disaster recovery to the forefront, coming a week after a massive hurricane sent the U.S. Eastern seaboard into turmoil.

The electrical outage was caused by a maintenance worker error in Arizona, but unexpectedly spread to southern California and northern Mexico after safeguards which should have localized the problem failed, resulting in business and school closures and a huge traffic jam in the San Diego, Calif. area.

The severity of the outage also damaged solution providers' data centers and communications capabilities for several hours.

Rich Baldwin, CIO and chief strategy officer at Nth Generation Computing, a San Diego-based solution provider, said his customers had not reported any IT issues by about noon on Friday.

One customer, the University of California at San Diego, actually sent an e-mail to Nth to thank the company for having one of its engineers, Jason Walker, on site to reassure them that their disaster recovery infrastructure was working properly, Baldwin said. Nth had recently implemented the technology for the university, he said.

The power outage was not so kind to Nth's own data center, however.

"We host our web site at a collocation facility, and had no problems with it," he said. "But when the outage occurred, it seemed to spike the power several times, causing our data center's UPS to flip on and off several times. It finally locked into the off position. The batteries were fully charged, and we were running it at only a 50-percent to 60-percent load before the crash."

That UPS failure did cause Nth to suffer minor database corruption, Baldwin said. "But our IT people were on it early Friday morning," he said. "By the time we got into the office, all systems were online, and the databases were up. We didn't miss a beat."

For Bert Shure, account executive in the San Diego office of Denver-based Advanced Systems Group, the main problem was a temporary loss of communications.

Shure said he was not able to get Internet access from his Verizon mobile phone, or Internet or VoIP access from Time Warner Cable. However, he said, his neighbor's AT&T phone service was unaffected.

Shure also said his Google G-mail access disappeared for several hours. "I expected that when the power and Internet came back online, I would get the queued e-mail," he said. "But it took hours to get it."

However, Shure said, customers had not reported any major issues.

The power outage does mean that customers will be getting visits from their local solution providers to discuss beefing up their disaster recovery capabilities.

Baldwin said the outage was a big wakeup call for the importance of disaster recovery, especially after the much more serious problems caused by Hurricane Irene which last week lashed the East Coast, causing sever flooding and wind damage.

"I was talking to people on the East Coast who lost power for a week from Irene," he said. "We had only an 8-hour outage. This shows how dependent we are on power."

The power outage is a chance to talk to customers about upgrading their disaster recovery plans, Baldwin said.

Next: Power Outage A Good Time To Talk Disaster Recovery