How Vulnerable?

Many IT organizations are much more prepared to recover from a major disaster than they were a year ago, but a good number of others are still laying out or implementing plans to make sure they can resume operations if they lose a data center.

In the weeks and months following the attacks, many firms realized that they had no contingency plans if their data centers were wiped out, says Vinny DiSpigno, CEO of PWR Systems, based in Bohemia, N.Y. "Back then, business continuity was all the rage," says DiSpigno, whose customers requested proposals to build everything from remote storage systems to hot sites. However, "None of the companies that we were working with actually pulled the trigger on it," he says, recalling one recent prospect that walked away from a $170,000 bid to add a fully redundant data center. "All of these budgets have been shelved."

Even national integrators, such as Dimension Data, Reston, Va., found that clients who expressed interest in remote data- mirroring, where storage systems are backed up in near real-time, have left proposals in limbo.

"To be honest, there was a lot of talk and planning, but in terms of spending dollars, I've seen that to be somewhat limited," says Dimension Data senior consultant Russ Norman. "There has been a lot of lip service, but there haven't been a lot of implementations."

According to a recently released report by IDC, expenditures for backup-and-recovery services will grow only marginally, from $3 billion last year to $3.13 billion this year. By comparison, the same report found that total expenditures for business-continuity services, which includes security, will be $33 billion, up nearly 9 percent from $29.9 billion last year. Why the lag in spending on backup and recovery, given the magnitude of last year's losses? "I think it's at a stage where companies are still looking at it," says Mike Luebke, president of Verizon Information Technologies, noting many customers are trying to link backup and recovery to other IT initiatives so it doesn't appear purely as an added cost. Plus, there are numerous options with significant cost differentials.

"It's not just something companies can jump into," adds TowerGroup analyst Larry Tabb, who says many companies are still determining the best way to improve their disaster and recovery methods. "A lot of companies are fixing gaps, but there are some who still don't know exactly what is needed."

While most large companies and government agencies have had limited disaster-recovery and contingency plans in place, Sept. 11 taught them that they had insufficient disaster-recovery plans in place, says independent telecommunications consultant John Compitello, who spent much of his career working for Wall Street firms addressing business continuity.

"Many firms had quasi-disaster recovery scenarios but not to the extent that was necessary," Compitello says. "The assumption that the whole entity could be [eliminated was taken into consideration, but people always considered the probability was little and that's where they got hurt more."

The Prepared Ones
Compitello notes exceptions such as Lehman Brothers, the Bank of New York and the New York Board of Trade, as among those that were able to resume operations almost immediately, despite losing their main operational facilities. For example, because the New York Board of Trade, which operated out of the World Trade Center, had real-time mirroring of its Tandem systems and Windows servers with alternate carriers, the commodities exchange could have begun trading at its backup facility in Long Island City the next day, though in deference to the exchanges and member firms that were not ready, it waited.

"We were collecting data up until the time the disaster hit," says telecommunications manager Bob Gabar.

The New York Board of Trade is still operating out of its disaster-recovery location in Long Island City, which it has since expanded, as it builds a new trading facility in Manhattan. Like its old location, the new facility will mirror data between the exchange and its backup locations, using alternate network routes. The total cost: $30 million.

Overall, it will cost $3.2 billion to replace the systems destroyed in the World Trade Center attacks, according to TowerGroup. That includes 16,000 trading desks equipped with multiple flat-screen displays, 34,000 PCs, up to 20,000 positions that were in adjacent buildings, 8,000 Intel servers and 5,000 Unix servers.

In Washington, numerous servers running operational and workgroup data at the Pentagon were wiped out by the attacks, says Ray Bjorklund, vice president of consulting services at Federal Sources, McLean, Va. While the command and control systems of the national security forces, which are mirrored in real-time, were never affected by the attack on the Pentagon, administrative data and systems handling budget proposals lost a few days worth of data. But many on the team with knowledge of the budgeting process were killed that day, Bjorklund notes.

EDS, Plano, Texas, the contractor awarded the Navy Marine Core network last year, is credited with helping the Pentagon resume operations within 24 hours, Bjorklund says. Still, while government agencies are looking more closely at what needs to be mirrored in real or near real-time, few radical changes have been implemented, he notes. "There's been a lot more emphasis put on continuity of operations and improving backup systems, but I wouldn't say it was put on the fast track," he adds.

They Thought They Were Ready
Then there are those who thought they were protected from disaster, but were not. Many companies were backing up data, but didn't have easy ways to recover that data. "What they began to realize is they had the data, they just had no way to connect it to the people who needed it," says Jim Simmons, CEO of SunGard Availability Services in Wayne, Pa., one of the largest operators of hot sites and disaster-recovery services.

A growing number of customers are now looking at remote data replication, so that mission-critical data can be available within a minute of the loss of a data center. Michael Leib, director of marketing and strategic relations at TotalTec, a New York-based storage integrator, cites as an example Municipal Credit Union (MCU), which provides financial services to 300,000 city, state and federal employees and health-care workers.

With offices next to the World Trade Center, MCU had to evacuate and did not have real-time backups of its systems, according to TotalTec officials. In early August, the Manhattan district attorney charged 66 people with fraudulently withdrawing money they didn't have from their accounts in the hours after the attacks, leading to $15 million in stolen funds because the systems were down. MCU would not comment.

But Leib said TotalTec has since added HP's (formerly Compaq's) Data Remote Mirroring software. It runs on Compaq storage controllers linked to SANValley's Fibre Channel-to-IP bridges and a StorageWorks EMA 12000 with 1 TB capacity to make sure MCU's customer records would be available in near real-time (within one to two minutes), should its data center be dislocated again. Other systems would need to be made available within 24 hours.

"All too often, customers will design their protection scheme to accommodate only backup," says Todd Pekats, director of professional services at Neartek, a storage integrator and consulting firm in Lakeville, Mass., that had numerous clients in the World Trade Center. The notion of narrowing the backup window even eluded one of the world's largest banks, Pekats says. While he declined to name the client, he said the bank's Compaq servers sustained significant damage, and backup tapes in New Jersey would have taken many months to restore because of a poor internal implementation of its IBM backup software. Luckily, Compaq was able to restore the original disks, and the bank only recently completed the implementation of a secondary server backup solution from Veritas.

Even companies with departments devoted to business continuity found out after Sept. 11 that they need to do more. For example, American Skandia, a company that sells annuities, formed a business-continuity unit back in 1999 to prepare for Year 2000. Despite no Y2K disruptions, American Skandia kept the unit intact, assuming that the need to recover systems might strike at some point.

Although based in the New York suburb of Shelton, Conn., Skandia's data communications capability was disrupted on Sept. 11, even though it experienced no other systems interruption. The company, which has its own trading operation and mirrors its data center to another location in Colorado, actually conducted its own simulations post 9/11 to see what impact it would have had if it had a presence in the World Trade Center. The goal,to see how such a disaster might impact its business. "Each of our business areas took into account the situations that actually occurred during 9/11 and relived it as if they were in the WTC event," says Diane Nemphos, American Skandia's director of business continuity.

Their findings? Response times could be improved by mirroring their EMC storage systems between sites and a disaster-recovery location, says Jerry Cooperman, American Skandia's vice president of information systems and technologies.

While he's reviewing proposals from various EMC resellers, he's also open to less expensive solutions such as Tivoli's Storage Manager. But to truly share disk and tape resources from different vendors, Cooperman envisions using virtualization software.

"Hardware doesn't always do it, so we may have to explore some software solutions for certain systems as well," he says.

Meanwhile, the largest of Wall Street firms are stepping up their efforts to cooperate with each other should another disaster occur. The Securities Industry Association (SIA) has formed a business-continuity committee, which comprises representatives of key financial services committees. Among those leading the effort are Deutsche Bank, J.P. Morgan and Chase, Merrill Lynch and the New York Stock Exchange. Their efforts include building a shared-information network and a shared IP backbone, and operating an already operational command center should another disaster strike.

More than 150 of the largest firms have provided input, says Art Trager, SIA staff advisor. "I think everyone is better prepared than they were last September just because of the considerable efforts that have gone on since, and the lessons that have been learned," Trager says.

Indeed, although a year has passed, many customers are still looking for the right solution and trying to balance the need to not only back up data, but recover it with minimal impact on operations. The reality is, however, that not all data needs real-time access and IT managers are not just writing blank checks to make sure they can recover everything within a split second of a disaster. That said, Pekats notes that any IT manager who is aware of a major risk should be documenting it to upper management and should be prepared to present a solution to the problem.

"If you're aware of a condition that could be costly or catastrophic and you don't take steps to mitigate it, you are going to be held responsible," he says. However, solution providers shouldn't resort to scare tactics. "That's unethical," he says. "It's a complex interaction."