Storage VARs Perplexed By Missing White House E-mails

The Bush administration, responding to subpoenas, discovered in late 2005 that millions of e-mails in preceding years were missing, according to Congressional testimony and public statements by White House officials. In April, White House Press Secretary Dana Perino told reporters, "I think there are backup tapes."

At present, it is unknown whether the administration has made any progress in recovering the missing e-mails from those backup tapes. ChannelWeb spoke with e-mail storage and records compliance VARs to get their take on the situation.

Intermedia, a New York, N.Y.-based provider of hosted business e-mail services, has processed billions of e-mails and claims to have lost none.

"For Intermedia, it's 0 percent loss. Most other companies also have 0 percent. We've probably processed 5 billion emails and lost none of them. Every different layer of the system has backup and redundancy built into it. There's just no single point of failure in an enterprise-class e-mail system," said Intermedia VP of Strategy Rurik Bradbury.

"Knowing as we do the process of doing large-scale e-mail backup, there's no fall-off rate. You don't lose things. It's hard to imagine it wasn't someone who broke a policy and knew they were breaking a policy. It's just so unimaginable to remove something from a tape library or to lose a tape library, one of the two."

Bradbury cast doubt on a few theories that have cropped up about the cause of the missing e-mails, including problems that might have arisen with the 2002-03 migration at the Executive Office of the President from LotusNotes to Microsoft Outlook, or from simple data corruption.

"It's very unlikely that LotusNotes to Exchange is the reason for this. If you're migrating from Notes to Exchange, there's a migration procedure and your e-mail goes into the new system. The messages have to be loaded into the new Exchange server. And I can't imagine that data corruption is possible. If you had a giant Bond magnet, like from a 60's movie, maybe you could corrupt some backup tapes. But there's redundancy and backups built onto that. We've all thought of these things," he said.

Keith Norbie of Nexus Information Systems described the physical destruction of hardware that would be necessary to actually get rid of that many e-mails.

"It would be a fairly sophisticated process to get rid of them. To do it, you'd have to trash the laptop and PC of the user, the user's server, the other servers and the backup tapes. And if you are using WORM, you'd have to destroy the media," said Norbie, referring to "write once, read many" tape or disk media that is configured so that data can only be written to it once, and cannot be erased or overwritten.

And keep in mind that there are two parties in the e-mail process, inbound and outbound, said the director of the storage division at the Plymouth, Minn.-based solution provider.

"If there's no evidence of where the e-mails are, you'd have to back-track them [to the recipients]. That's FBI-level stuff," he said.

Norbie and Bradbury had some theories for how the e-mails might have disappeared without the sort of physical destruction of hardware described by the former. But neither storage expert thought they were very plausible.

Bradbury said "the only way to do it" would be to change IT processes, because "the tech is too good, so it's always human error or bad processes." Norbie could only imagine one way such a disappearance could happen without gross incompetence or maliciousness, though he noted that you'd only lose a few hours of data as opposed to large gaps over several years.

"There is one possible scenario. If there is no CDP [continuous data protection], and you do snapshots or clones every four hours, and e-mails get corrupted during that time, you could have to go back to the previous versions of the data. But you only lose a maximum of four hours," Norbie said.

Yet the White House is not the only high-profile example of head-scratching IT failures, said Bill Lyons of AXS-One. "There's no good excuse for losing e-mail at the White House. But whether it's the Republican National Committee, Intel or Morgan Stanley, we have some of the best known organizations making headlines because they may be using outdated technology to address current regulations and IT issues," said Lyons, CEO of the Rutherford, N.J.-based provider of records compliance management solutions.

"It's as if they wanted to mail punch cards from employee to employee as a substitute for text messaging. If they persist in playing the 'it's on my backup tape' card and applying archaic, paper-based processes to electronic records, we'll continue to see this type of headline."