All Backup Is Not Equal
AmeriVault, Waltham, Mass., worked with the EEOC to tackle the problems related to tape backups from multiple offices across the country and ensure that restore time could be measured in hours instead of days.
The main driver for the move to new and improved backup procedures was the government organization's geographical makeup. The EEOC houses about 700 employees in its D.C. headquarters, and has 15 district offices with about 90 employees, along with 35 local and field offices with between 10 to 80 employees each, said Everett Barnes, telecom and notebook division director for the EEOC. But three years ago, when the EEOC's security people began looking at the problems of backup tapes, the solutions seemed too expensive, Barnes said.
Fast forward to 2006. When AmeriVault looked at the EEOC's backup architecture, it saw each district, local office and field office as essentially a series of small businesses with stranded servers and no IT staff, said Adam Dameron, senior sales executive at the service provider.
"Each office copied data to tape, with maybe the office secretary or someone else swapping the tapes," Dameron said. "But even after spending all that money on their infrastructure, they still weren't confident that they could get their data restored. We see that in a lot of smaller offices."
The EEOC had been relying on a third-party company to collect and store the backup tapes from each office, most of which did not have their own IT staff, resulting in difficulties in determining whether data was backed up properly or not, and whether it could be restored, according to Barnes.
"We took our tapes off-site," he said. "But it didn't mean anything if the people doing the backups mislabeled the tapes or didn't do the backups properly. So we looked at online backups to reduce errors." The EEOC had two basic problems, Dameron said. The first was operational efficiency, which was compromised by having someone from each office busy with swapping tapes.
The second was the requirement of the government's Continuity Of Operations Planning, or COOP, which are preparations intended to keep the government running in case of a catastrophe.
"It's part of being able to recover operations and make sure the business of an agency can continue," Dameron said. "So if something happened, instead of going far to get the data, an agency can restore it quickly."
AmeriVault proposed to bring online backup technology to each office. Critical data would be backed up to two redundant sites in Massachusetts and to a third facility in Illinois, Dameron said. "There's about 1,000 miles between the sites, so there's a greater level of risk mitigation," he said.
The service provider proposed using Televaulting software from Asigra, a Toronto-based software developer. Televaulting is agentless software that allows data to be backed up from multiple sites.
"From a single client, we can manage and back up all the data in a network while increasing security and flexibility," Dameron said.
The backups themselves go to Network Appliance storage appliances. AmeriVault uses both NetApp and EMC for storage hardware, he said.
The contract was awarded on Sept. 17, 2006, with the unusual stipulation that the work be completed by Oct. 1, which was when the contract with the off-site tape management company expired, Barnes said. "And we did it," he said.
To make it work, each office needed a PC to serve as the backup device and to store a copy of the local databases separate from the Dell PowerEdge 2800 servers, one of which powers the IT infrastructure of each office.
Here, the primary challenge was the intense deadline, which caused the government agency to have the software installed on leased PCs. "So we had to make them do it again," Barnes said.
Eventually, a number of five-year-old Dell OptiFlex GX150 PCs were brought out to be used for the backup software, Barnes said.
"There was no risk," he said. "They were good-quality machines, and no one was using them. And if one went down, we'd still have a copy of the database."
As part of the implementation, AmeriVault worked with the EEOC to set backup policies and schedules, Dameron said. "After that, it was completely automated," he said.
That automation has made a huge difference for the EEOC, Barnes said. A big benefit was the elimination of a lot of different costs for the organization, starting with the purchase of new tapes and backup devices. But there were a lot of hidden costs that were also eliminated, Barnes said, including the cost of mislabeling tapes, of incorrectly backing up data and the cost of the off-site tape storage service.
"We took all that out of the works," he said. "Now the only thing that can go wrong is a server going down."
Compared to restoring data from a tape, which could take a week or longer, files can now be recovered in about an hour, Barnes said.