Jamie Shepard, regional vice president at Lumenate, agreed with Intelisys' Pryfogle and Clustrix's Purohit that Verizon should have put redundancies in place. He said that Verizon should not be able to remove one piece and cause the entire system to collapse. However, the outage also pointed to a bigger problem, he said.
"I'm assuming they had architecture built in a program. You have to. It's the government," Shepard said.
Government audits would make an architecture problem unlikely, Shepard said, because that is one major area that would have been checked. Instead, he said it was much more likely that the problem came from a process change instead of a technical change. Shepard compared the problem to electricity in a house, which can be maintained during a power outage with a generator.
"The problem was the generator didn't kick in," Shepard said. "It was architected right, but the process is broken."
What Verizon failed to do, Shepard said, was put in place processes such as risk assessment and mitigation that would have established plans of action in case of network problems. Verizon, he said, should have gone through the plan for the site and created contingency plans for possible problematic scenarios at each turn.
"You can't tell me that the tech is always at fault here; ... it's the risk mitigation. You have to openly state that if we miss a step here, here's the risk. I can guarantee that didn't happen here," Shepard said. He said he speculates someone didn't look at the process enough and looked only at the surface-layer architecture.
Regardless of whose fault it is and how easy the problem is to fix, the reputational damage to Verizon is done, the solution providers told CRN.
"What I don't know are the details around what was dictated to them, the requirements that were dictated to them, what they did around redundancy, data base replication -- I don't know those details. But, I can say that, although [Verizon] Terremark has a strong reputation, even Terremark is vulnerable to outages when the root cause is basically a poorly architected solution," Pryfogle said.
PUBLISHED OCT. 28, 2013