Big Lessons From the Blackberry Blackout

A significant percentage of U.S. businesses and government offices lost an important element of their IT infrastructure Tuesday night and Wednesday morning when Research in Motion's (RIM) Blackberry network went offline, disrupting e-mail service for 10 to 14 hours. The outage's cause remains unclear, but its impact is undeniable: By some estimates, as many as 6 million users worldwide may have been launched into sudden withdrawal from their "Crackberry" addiction.

According to one Reuters report, even White House spokesman Tony Fratto opened a Wednesday morning briefing by apologizing for missed communications and noting the disruption to White House operations.

RIM's e-mail system represents one of the world's largest outsourced infrastructure operations, and as such, is a model for many in the channel to keep track of. It may yet be a bit early for a full after- action analysis of the incident, particularly as RIM has been very tight-lipped about the whole thing, but there are a few lessons to take away even now.

Lesson 1: When it comes to outsourcing, transparency is critical. One of RIM's biggest mistakes has been keeping its users in the dark about what actually happened, fueling frustration and speculation. It's difficult enough for clients to trust anyone else with critical aspects of their business. While it may seem easier to keep quiet when something goes wrong, users will assume the worst if they can't get information about a problem, and will take the resulting sense of helplessness and insecurity out on you. If you're managing outsourced infrastructure for clients, be prepared to give them even more information than they might ask for, even before they ask for it. If you're reselling someone else's outsourced infrastructure, demand that they provide you with the same.

Lesson 2: All systems fail, including critical ones. You can take all kinds of precautions and push failure rates very low, but you can't push them to zero. It's not possible to prepare for everything, and it would be incredibly expensive to try, but is also irresponsible to fail to identify those systems your clients consider critical and have contingency plans in place for their failure.

By the same token, it's up to you to make sure your clients have similar contingency plans, and are prepared to put them into action. As tempting as it may be to emphasize the resiliancy of your solutions to the exclusion of any possibility of failure, the last thing you want is a host of angry technology addicts blaming you because they can't get their next fix.

Tweet

   

More Networking