Google Sorry For Gmail Outage 'Scare,' Blames Software Bug


Google has apologized for a Gmail e-mail outage that wiped out thousands of Gmail inboxes Sunday into Monday, calling the downtime a "scare" and pointing the finger at a software bug introduced by a storage update.

And Google is still working to fix the issue. According to Google's Apps Status Dashboard, which illustrates performance information regarding Google Apps and offers updates, Gmail is still experiencing a "service disruption." Google said service will be completely restored soon.

"Imagine the sinking feeling of logging in to your Gmail account and finding it empty," Ben Treynor, Google vice president of engineering and site reliability wrote in a blog post apologizing for the Gmail outage. "That's what happened to 0.02 percent of Gmail users yesterday, and we're very sorry. The good news is that e-mail was never lost and we've restored access for many of those affected. Though it may take longer than we originally expected, we're making good progress and things should be back to normal for everyone soon."

Sunday night into Monday, dozens of Gmail users took to a user forum when they awoke to find messages in their Google Gmail inbox, folders and other data had vanished. Google estimated that the outage affected 0.29 percent of users initially. That number was reduced to 0.08 percent and now to 0.02 percent, Google said. At its peak, it was estimated that the Gmail outage affected the e-mail accounts of more than 150,000 Gmail users.

The thread on the Gmail user forum, where affected users voiced their outrage and concern, quickly grew to 15 pages as Gmail e-mail users said years' worth of data had been wiped out of their Gmail inboxes.

"I know what some of you are thinking: how could this happen if we have multiple copies of your data, in multiple data centers?," Treynor wrote. "Well, in some rare instances software bugs can affect several copies of the data. That's what happened here. Some copies of mail were deleted, and we've been hard at work over the last 30 hours getting it back for the people affected by this issue."

Treynor said Google also backs up to tape to protect Gmail data and e-mail from bugs. Tape, which remains offline, is an added layer of protection, but restoring data from tapes also takes longer than transferring requests to a different data center, Google said, explaining why the restore is taking "hours to get the e-mail back instead of milliseconds."

Google said the outage was created by an unexpected bug that was released as part of a storage software update, which caused affected users to temporary lose access to their Gmail e-mail accounts. Treynor said that when Google discovered the problem the deployment was halted and Google reverted back to the old version of the storage software.

Along with apologizing, Google warned affected users that e-mail sent to them between 9 p.m. eastern on Sunday February 27 and 5 p.m. eastern on February 28 was likely not delivered, and senders would have been notified that their messages were not received.

"Thanks for bearing with us as we fix this, and sorry again for the scare," Treynor wrote.