Investigators have ruled out a cyberattack or any attempts of market manipulation and say a software flaw forced the halt of the Nasdaq stock market Aug. 22, according to a preliminary incident report issued by the Nasdaq OMX Group, the financial services corporation that owns and operates the Nasdaq stock market.
The firm said the software vulnerability in the system that supplies the quote data to the industry was compounded by connection problems with the New York Stock Exchange's Arca system, which electronically trades more than 8,000 exchange-listed equity securities. The issues caused the backup system to fail, according to the preliminary Nasdaq investigation report (.PDF). The report found no evidence of an attempted intrusion into the systems or of an unusual burst of quotation or trading messages in connection with the incident.
"This latent flaw prevented the system's built-in redundancy capabilities from failing over cleanly and delayed the return of system messages to users," the report said. "The combination of large system inputs and delayed outputs ultimately degraded the ability of the SIP [Securities Information Processor] system to process quotes to an extent that a shutdown of the system was in the broader public interest, to prevent information asymmetry and ensure fair conditions for all market participants."
[Related: Industry Bellwethers Taking Hit As Smaller Companies Step Up]
Nasdaq apologized for the problems and said it was in the process of identifying ways to build in redundancies, making the SIP more resistant to system errors and trading fluctuations. The firm said it would present recommendations within 30 days.
"Nasdaq OMX is currently identifying potential design changes to further strengthen the SIP's resiliency, including architectural improvements, information security, disaster recovery plans and capacity parameters," the firm said in its report.
The system is designed to handle 10,000 messages per second. But the vulnerability, coupled with the connection issues with the NYSE Arca system, caused the system's performance to rapidly deteriorate, according to the report. Capacity of the quote supplying system was eroded when the NYSE Arca system attempted to reconnect more than 20 times. A stream of inaccurate stock symbols also generated rejection messages, causing problems, according to the report.
Security experts told CRN that they weren't surprised by the findings and said they wouldn't be surprised if more vulnerabilities were discovered in the future. Software coding errors are common and vary in scope and risk. That's why critical systems must have built-in processes for redundancy, failover and offline backup, said Pete Lindstrom, principal and vice president of research at Spire Security. Critical systems need to be tested regularly, he said.
"We've crossed the threshold of once-a-year testing because we've demonstrated that any mission-critical, complex system should be tested constantly to try and bring it down and document the incident response scenarios," Lindstrom told CRN. "If you are not thinking about doing real-time attempts to overload and break systems in your own environments, then these incidents are going to happen once in a while."
Other security experts agree and said the financial industry has constantly been ahead of other sectors in addressing cybersecurity issues and reducing fraud. They also point to video service Netflix, which has subscribed to the model of conducting real-time penetration testing. The firm uses Chaos Monkey, a service that runs in Amazon Web Services (AWS), and can be configured to run on other cloud providers to test system tolerance to instance failure. The system is designed to run extensively when an engineering team can immediately address issues that arise.
"We have found that the best defense against major unexpected failures is to fail often," Netflix said in an announcement when the source code was released to the public last summer. "By frequently causing failures, we force our services to be built in a way that is more resilient."
PUBLISHED AUG. 30, 2013
