Human Error Behind Customer Horror Stories

Conqwest CEO Michelle Drolet, for example, said her firm was brought into an account to find out what was slowing the organization's network to a crawl. The Web security firm discovered a gaming Web site that was being run by internal IT staff at the customer. "They were running their own dot-com company," Drolet said. The customer fired the employees after Conqwest's discovery, and the assessment and quick fix cost the client only about $20,000, she said.

At least that customer's network was still running. Chris Ellerman, vice president of professional services at Meridian IT Solutions, said his firm for some time had been urging one of its clients,a Midwestern Web hosting company,to implement a firewall, but the customer didn't get the message until a security breach brought down all of its systems for more than 27 hours.

The hosting provider, which ran about 5,000 Web sites and was attempting to go public, saw its bandwidth plummet because of the breach. The solution that Meridian brought to the table, including the firewall and services, cost roughly $30,000, Ellerman said.

The hosting provider remains in business, but the company still hasn't reached the point of "totally locking down the servers," Ellerman said. "They're definitely not bulletproof."

A mortgage company client of FishNet Security also had some network leaks. Before signing on with the security solution provider, the mortgage firm demanded a security audit, said FishNet President and CEO Gary Fish. But minutes after starting the audit, FishNet uncovered a sizable security hole: It had access to about 450,000 mortgage applications, he said.

"We had gotten people's income and social security numbers, and the site had been up for a year," said Fish. "With a few code changes, it took five minutes to fix. It was a SQL injection problem."

Though that problem was nipped in the bud, Guardent was called in after the fact when the CEO of one customer, a $500 million company, logged into its e-mail system and found that its entire mail log had been deleted, said Dan McCall, executive vice president and co-founder of the managed security services firm. After some forensics work, Guardent discovered that the e-mail deletion was the work of a disgruntled IT employee who had been fired.

"They thought for sure that they were compromised from the outside, but they were attacked from the inside," McCall said. "I don't know what happened to that employee, but we found him pretty quick." The breach easily cost the customer hundreds of thousands of dollars, added McCall. "It was brutal for them," he said.

One customer of solution provider Icons also didn't know about something. Icons COO Paul Rohmeyer said that when the client called in his firm for a security assessment, it found that a server run by a third-party provider had been transformed to host a pornography Web site. "Someone had hacked into it, took over the box and locked it down. It was actually pretty clever," said Rohmeyer.

Icons shut down the server, added a host-based intrusion-detection system and made recommendations to improve vulnerability management and other processes, Rohmeyer said. The solution cost less than $10,000, but if the porn site server incident had been publicized, the cost to the client would have been immeasurable, he said. "This would have affected both the client and the managed service provider because the box was in their care," he said. "It's hard to quantify it if you wake up in the morning and find yourself in The New York Times."

Vigilinx also was hired to get to the bottom of a situation when a financial institution's network experienced intermittent outages, said Kenneth Cavanagh, vice president of professional services at the security solution provider. The problem: One of the financial firm's top executives gave his son access to the corporate network so they could share files, but the executive didn't know the college student was using the network to run a fantasy football league, he said.

"The son had created a home page on his dad's directory and then instituted a dormwide football league and gave out the VPN ID password to several hundred college students. They received all kinds of weird denial-of-service attacks and other things, and we traced it to this high-ranking vice president in the organization," Cavanagh said. "So it takes you back to the human factor. All of the technology in the world couldn't get you around this guy giving his ID and password to his son."