Selling Preparedness

However, the solution providers who help build customers' storage and other infrastructures are finding there are many lessons yet to be learned by those who have not yet implemented disaster-recovery programs.

Customers who live farther away from such disasters that have devastated entire cities or caused many fatalities, such as Hurricanes Ike and Katrina, and 9/11, are less likely to have learned the importance of disaster-recovery plans and infrastructures, solution providers agreed.

While everyone is interested in disaster recovery, customers' approaches depend in large part on geography, said Keith Norbie, director of the storage division of Nexus Information Systems, a Plymouth, Minn.-based solution provider.

"If you are in the path of a hurricane every three years, you get memory in the management team," Norbie said. "People buy based on pain."

However, he said, it gets harder to talk disaster recovery the farther north VARs go. "The East and West Coasts are seeing more aggressive adoption, the East Coast for financial companies, the West for high-tech," he said. "In the Midwest, people are starting to do it. But it's not as fast as if they are staring at hurricanes."

Insight Investments Corp.'s customers in the heartland of America often have put disaster-recovery plans in place that include running remote from the West Coast in order to meet regulatory and compliance requirements, said Kit Bowyer, a senior account executive at Insight, an Orange, Calif.-based solution provider.

"But I'm not sure they're as prepared as people on the Gulf Coast," Bowyer said. "They don't test their disaster-recovery plans with production data. They test with replicated data, but never do test with a workload on it."

Even after the IT director or CIO of a public company is shown the compliance and regulatory risks of not having a disaster-recovery program in place, their companies may not want to fund such a program, said Mitch Kleinman, president of Ryjac Computer Solutions Inc., an Irvine, Calif.-based solution provider.

"If they don't want to fund disaster recovery, at least they need to go down to the next level and vault their data," Kleinman said. "Solution providers have to at least put in some recommendations before the customer gets caught with his pants down in a disaster, or if a bad employee messes with data."

The lesson doesn't stop there, Kleinman said. "We like to give our customers a high-availability book, which documents what to do if there's a problem," he said. "But the book goes out of date the minute the customer makes a change."

Katrina, like 9-11 before it, produced a big uptick in interest in disaster-recovery at the corporate level, said Greg Nightingale, director of storage sales at Sirius Computer Solutions, a San Antonio-based solution provider.

"When you get into the details, they'll say, 'We're waiting for guidance from the DR committee,'" Nightingale said. "Or they'll say, 'We have a second site,' but then you find out it's a cold site, not a hot site. Then, after Katrina, things slowed down. People [have short memories]."

Customers also slow down implementation because they overestimate requirements, Nightingale said. "They'll initially say, 'Yeah, we need to be back up in two hours,'" he said. "But later, they think they want to be back up in one day."

One customer that learned its lesson from Katrina was Tidewater Inc., a New Orleans-based owner and operator of offshore oil-support vessels, which put in a disaster-recovery system after the hurricane with the help of Insight Investments and vendor partner Compellent Technologies.

It was a hard lesson, Bowyer said. "When Katrina hit, they loaded everything up in a pickup truck and headed to high ground," Bowyer said. "This time, with Gustav, they rolled their production to Dallas before Gustav struck, and rolled it back to New Orleans right afterwards."

Disaster recovery, however, is often not a skill that can be perfected in one try. Bowyer said that when Tidewater rolled over to the Dallas site, they found that they had not sized their storage requirements properly. The customer normally runs its data off of 32 spindles worth of hard drives, but there were only eight spindles available at the disaster-recovery site.

"It worked, but it was a bit slower," he said. "The lesson learned from Gustav was, you can test disaster recovery, but no one does testing with their production data. When running stressed systems, lower performance is OK. But it's not good if the stressed systems puke."

The old "it can't happen to me" mentality with regard to disaster recovery is still common, Ryjac's Kleinman said.

"One customer had two disasters," he said. "First, their RAID system was down. Then their UPS went down. But they still won't implement disaster recovery. Their mindset is, 'I've taken a hit, and it won't happen again.'"

There are ways to help implement disaster recovery without spending a lot of money, solution providers said.

Server virtualization is easy to use to create a highly available environment, Kleinman said. "For customers with virtual machines, if one [virtual machine] dies, the other takes over," he said. "But you need to geographize it," he said. "You need to make sure that a problem infrastructure can come up in a remote location."

That is true especially with customers who have x86-based servers that can easily be virtualized with technology such as VMware's ESX, Nightingale said. But for Unix-centric customers, or those with mixed environments, things are complicated by the multiple technologies.

"Virtualize Unix, virtualize servers, virtualize storage, and keep bandwidth costs down," he said. "That's a lot to bring together before the customer invests."

The trick to emphasizing the importance of disaster recovery for all customers is bring in technology that makes it easy for them, especially those that can make it part of a cloud infrastructure, Norbie said.

"In the future, as technology improves and things get less sticky on the operating architecture, you will see more clouds," he said. "People will move to the clouds for disaster recovery. VARs, you better get ready. The clouds are coming."

Next: Hard Lessons Learned From The Gulf Coast