Death Of A Web 2.0 Service

Last week's power outages in San Francisco took down data center 365 Main, a Silicon Valley infrastructure nexus whose blackout in turn knocked offline some of the Web's most popular sites, including Craigslist, Netflix and Technorati. It was the kind of crash that haunts the nightmares of Web companies, stirring up hordes of cranky users and prompting a flood of abject apology letters and investigations into how a total crash, the kind data center operators swear could never, ever happen in their backed-up-to-the-redundant-rafters facilities, happened.

For most of the affected sites, the pain was acute but short-lived. 365 Main had power back on to its customers within 45 minutes, and almost all of its downed customers crawled back online within a few hours. By Wednesday morning, the crash was an unpleasant but dimming memory, a historical footnote serving as yet another reminder that even the "multiple nines" of uptime -- 99.99x to the nth degree -- that hosting providers love to tout leave enough wiggle room for disastrous outages.

But for one affected site, Rojo.com, 365 Main's collapse turned into something more than an annoying but routine service blip. It's become the cataclysmic disaster likely to serve as the final nail in the struggling RSS aggregator's coffin. More than a week after 365 Main's outage, Rojo remains offline, and a sizable group of irate users say they won't be going back to Rojo after its resurrection.

"RIP Rojo," one user summed up on Rojo's status blog, the only functioning part of the Web site.

Rojo's collapse is a cautionary tale for Web 2.0 companies: When your entire service and brand are wrapped up in "always on" availability, downtime has a tipping point, where it transforms from a customer service problem into an unrecoverable death spiral.

"One of the harshest realities of SaaS [software as a service] is that SaaS is actually quite a bit more reliable than traditional software," said Treb Ryan, CEO of OpSource, a Santa Clara, Calif., solution provider that focuses exclusively on SaaS technology and services. "I can almost guarantee you that there are quite a few people out there in the world right now that are not able to get to their Siebel implementations, for some reason or other, because their local networks are having problems. The problem for SaaS providers is that when there's an outage, all of their customers are affected at once. It's disastrous."

Rojo and its parent company, blog technology maker Six Apart, are vague about precisely what technical debacles have kept Rojo offline 10 days after its data center powered back up. A note posted to Rojo's blog on July 27, three days after 35 Main's outage, cited "database corruption" and ongoing efforts to repair damaged data tables.

"This datacenter situation was a perfect storm of failures. We do keep both master and slave database servers to guard against reasonably foreseeable events, but in this case both our master and replicated databases were corrupted," the note said. "Unfortunately, we do not have a usable third backup of the particular table that was affected. This is a situation that 'should' never happen on many levels."

365 Main's crash took down all of Six Apart's Web services, which include blog networks Typepad, Livejournal and Vox, but those sites recovered within the day. Rojo, which Six Apart acquired in Sept. 2006, is the only one to remain offline.

"I think it's going to take the next week before the service is back up," Six Apart spokeswoman Jane Anderson told ChannelWeb, after consulting with the company's engineering team about Rojo's status. No Six Apart executive was available to comment on Rojo's breakdown -- or on how the company will go about rebuilding Rojo users' faith in the service. Anderson emphasized Six Apart's commitment to bringing Rojo back online.

For many users, that's likely to be too little, too late. "I finally have the motivation to make a switch I'd been planning anyway," said Stephanie Johnson, who began using Rojo last year. Periodic short outages had her reconsidering her Rojo use, and she began experimenting with rival aggregator Google Reader. This week's prolonged Rojo crash has her ready to migrate for good.

"Service outages of any flavor are inevitable. Service outages of this duration deserve thorough explanations. Rojo has failed to communicate with users," Johnson said. "When they are back, I'll be logging in just long enough to get an OPML file of my subscriptions so I can take it to a different aggregator."

NEXT: Lack of communication makes matters worse