Selling Preparedness

However, the solution providers who help build customers' storage and other infrastructures are finding there are many lessons yet to be learned by those who have not yet implemented disaster-recovery programs.

Customers who live farther away from such disasters that have devastated entire cities or caused many fatalities, such as Hurricanes Ike and Katrina, and 9/11, are less likely to have learned the importance of disaster-recovery plans and infrastructures, solution providers agreed.

While everyone is interested in disaster recovery, customers' approaches depend in large part on geography, said Keith Norbie, director of the storage division of Nexus Information Systems, a Plymouth, Minn.-based solution provider.

"If you are in the path of a hurricane every three years, you get memory in the management team," Norbie said. "People buy based on pain."

id
unit-1659132512259
type
Sponsored post

However, he said, it gets harder to talk disaster recovery the farther north VARs go. "The East and West Coasts are seeing more aggressive adoption, the East Coast for financial companies, the West for high-tech," he said. "In the Midwest, people are starting to do it. But it's not as fast as if they are staring at hurricanes."

Insight Investments Corp.'s customers in the heartland of America often have put disaster-recovery plans in place that include running remote from the West Coast in order to meet regulatory and compliance requirements, said Kit Bowyer, a senior account executive at Insight, an Orange, Calif.-based solution provider.

"But I'm not sure they're as prepared as people on the Gulf Coast," Bowyer said. "They don't test their disaster-recovery plans with production data. They test with replicated data, but never do test with a workload on it."

Even after the IT director or CIO of a public company is shown the compliance and regulatory risks of not having a disaster-recovery program in place, their companies may not want to fund such a program, said Mitch Kleinman, president of Ryjac Computer Solutions Inc., an Irvine, Calif.-based solution provider.

"If they don't want to fund disaster recovery, at least they need to go down to the next level and vault their data," Kleinman said. "Solution providers have to at least put in some recommendations before the customer gets caught with his pants down in a disaster, or if a bad employee messes with data."

The lesson doesn't stop there, Kleinman said. "We like to give our customers a high-availability book, which documents what to do if there's a problem," he said. "But the book goes out of date the minute the customer makes a change."

Katrina, like 9-11 before it, produced a big uptick in interest in disaster-recovery at the corporate level, said Greg Nightingale, director of storage sales at Sirius Computer Solutions, a San Antonio-based solution provider.

"When you get into the details, they'll say, 'We're waiting for guidance from the DR committee,'" Nightingale said. "Or they'll say, 'We have a second site,' but then you find out it's a cold site, not a hot site. Then, after Katrina, things slowed down. People [have short memories]."

Customers also slow down implementation because they overestimate requirements, Nightingale said. "They'll initially say, 'Yeah, we need to be back up in two hours,'" he said. "But later, they think they want to be back up in one day."

One customer that learned its lesson from Katrina was Tidewater Inc., a New Orleans-based owner and operator of offshore oil-support vessels, which put in a disaster-recovery system after the hurricane with the help of Insight Investments and vendor partner Compellent Technologies.

It was a hard lesson, Bowyer said. "When Katrina hit, they loaded everything up in a pickup truck and headed to high ground," Bowyer said. "This time, with Gustav, they rolled their production to Dallas before Gustav struck, and rolled it back to New Orleans right afterwards."

Disaster recovery, however, is often not a skill that can be perfected in one try. Bowyer said that when Tidewater rolled over to the Dallas site, they found that they had not sized their storage requirements properly. The customer normally runs its data off of 32 spindles worth of hard drives, but there were only eight spindles available at the disaster-recovery site.

"It worked, but it was a bit slower," he said. "The lesson learned from Gustav was, you can test disaster recovery, but no one does testing with their production data. When running stressed systems, lower performance is OK. But it's not good if the stressed systems puke."

The old "it can't happen to me" mentality with regard to disaster recovery is still common, Ryjac's Kleinman said.

"One customer had two disasters," he said. "First, their RAID system was down. Then their UPS went down. But they still won't implement disaster recovery. Their mindset is, 'I've taken a hit, and it won't happen again.'"

There are ways to help implement disaster recovery without spending a lot of money, solution providers said.

Server virtualization is easy to use to create a highly available environment, Kleinman said. "For customers with virtual machines, if one [virtual machine] dies, the other takes over," he said. "But you need to geographize it," he said. "You need to make sure that a problem infrastructure can come up in a remote location."

That is true especially with customers who have x86-based servers that can easily be virtualized with technology such as VMware's ESX, Nightingale said. But for Unix-centric customers, or those with mixed environments, things are complicated by the multiple technologies.

"Virtualize Unix, virtualize servers, virtualize storage, and keep bandwidth costs down," he said. "That's a lot to bring together before the customer invests."

The trick to emphasizing the importance of disaster recovery for all customers is bring in technology that makes it easy for them, especially those that can make it part of a cloud infrastructure, Norbie said.

"In the future, as technology improves and things get less sticky on the operating architecture, you will see more clouds," he said. "People will move to the clouds for disaster recovery. VARs, you better get ready. The clouds are coming."

Next: Hard Lessons Learned From The Gulf Coast HARD LESSONS LEARNED FROM THE GULF COAST

Solution providers along the Gulf Coast developed new strategies after Katrina, and earlier this fall, Hurricane Gustav was the first big test to see whether the changes would prove successful. Several VARs said all their hard work paid off.

One of the big lessons from Katrina, said Bill Long, president of Integrated Network Systems, a Metairie, La.-based solution provider, was that customers who backed up their data locally couldn't get to it or lost it after that 2005 storm. Customers kept tape drives in bank vaults or on local off-site servers, thinking that even if the power went out, they could get to the office to retrieve them. Of course, in many cases that couldn't happen as parts of the city were shut down for weeks, and even months.

This time, small-business customers took their servers or tapes with them or, even better, made sure their data was backed up in another part of the country, Long said.

"We went around to a half dozen or so customers and unplugged their servers after we did images of their OS and tape backup," Long said. "Every single [customer] is better prepared now."

Global Data Systems, a Lafayette, La.-based solution provider, took that a step further. After Katrina, the company purchased 10 mobile, 5-foot-by-8-foot trailers that have satellite capability and can run on generators. The trailers provide customers with their rerouted voice traffic, Internet and e-mail, just as they would in a regular office, said Chris Vincent, senior vice president of Global Data Systems.

"It's got a 1.2-meter dish on it. We pull it up anywhere, run a cable inside. It's been great. We also have some 26-foot mobile offices with generators," he said. Global Data Systems marketed the units as insurance policies for "a couple hundred dollars a month" in case of a disaster, Vincent said. After Gustav, he had more demand than trailers, and the solution provider also rented them to customers who need voice and data access in hard-to-reach places, such as off-shore oil rigs or in the Bayou, Vincent said. Prior toGustav, solution providers like Global Data Systems also had their own business-continuity plans in place.

"We're just more prepared now, in general, from communications with employees, to figuring out how to pull these guys together. We have different levels of evacuation," Vincent said.

Meanwhile, Universal Data Inc. helped clients by securing space in two Baton Rouge data centers for a month before Gustav arrived. Prior to the storm, the New Orleans-based solution provider helped back up several clients' mission-critical information and applications to the data center, said Jim Perrier, president of UDI.

While Baton Rouge got hit harder than New Orleans this time, and the data centers were running on generators, UDI's customers didn't miss a beat.

--Scott Campbell