NSS Labs Adjusts Testing To Address FireEye, AhnLab Criticism

NSS Labs will test the effectiveness of breach detection systems against zero-day exploits, altering its testing methodology to address some of the criticism from security vendors FireEye and AhnLab, which challenged the credibility of April test results.

The Austin, Texas-based independent testing firm issued a document outlining its adjusted testing practices June 6. NSS Labs CEO Vikram Phatak told CRN that the company stands by its comparative analysis report but made adjustments to directly address concerns and add clarity to its processes for future tests.

"I am sensitive to the representation that somehow we did something wrong or were not transparent in our methodology," Phatak said. "There is a huge amount of transparency both for the enterprises that request the tests and the vendors before we even conduct the testing."

[Related: FireEye, NSS Labs Continue To Trade Barbs Over Testing Report Credibility ]

Sponsored post

Phatak said NSS Labs partners with Exodus Intelligence, an Austin-area software bug hunting firm in the emerging vulnerability broker market. Exodus along with VUPEN, ReVuln, Netragard and Endgame Systems employ hackers to find software bugs and openly engage in selling their vulnerability findings to governments and corporations. It's a murky area often criticized by software security advocates because the bugs are not immediately reported to the software maker for patching. Instead, they may be used by intelligence agencies to support surveillance and cyberespionage attacks against adversaries.

NSS Labs has used zero-day exploits for previous tests when requested by its enterprise and government clients, Phatak said. The organization ensures that the tests are not public-facing, minimizing a leak of the zero-day vulnerability to the public, he said. When tests are concluded, the organization provides information to the software maker and computer emergency readiness teams, he said.

FireEye lashed out against the NSS Labs testing in April when it scored "below average" in the comparative group product test that pitted the FireEye Web and email Malware Protection System (MPS) appliances against products from AhnLab, Fidelis, Fortinet, Sourcefire (Cisco) and Trend Micro. Both FireEye and AhnLab earned a "caution" designation from NSS Labs due mainly to their below-average security effectiveness scores and cost of ownership, according to NSS Labs. FireEye said the tests used known malware samples rather than evaluating detection of a zero-day attack. AhnLab also denounced the test results.

NEXT: FireEye Declines To Respond To Changes

Other adjustments to the testing methodology include eliminating an evaluation of the breach detection system management consoles. NSS Labs added a check on the email traffic delay when the appliances scan a high number of plain text and HTML email messages and attachments. The trials also will include test detection of malware in Office, PDF files and software installers downloaded from an SMB file share. A test evaluating the effectiveness of threats in traffic protected by SSL encryption was made optional.

Solution providers toldl CRN that product effectiveness testing is a component in the evaluation process for some companies. Testing reports become progressively important to larger organizations with mature information security programs, they said. More than 90 percent of FireEye's business is driven through the channel.

FireEye, which is facing increased pressure from breach detection competitors, declined to comment on the updated testing methodology.

In April, FireEye CTO Dave Merkel issued a statement questioning the legitimacy of the results, pointing to the company's track record in providing protection in "real-world deployments."

"In 2013, FireEye found 11 of 13 exploitable zero-day vulnerabilities, tracked more than 40 million callbacks, nearly 300 separate APT campaigns and uncovered numerous new malware families and espionage campaigns," Merkel said in the statement. "Any lab test is fundamentally unable to replicate the targeted, advanced attacks launched by sophisticated criminal networks and nation-states. The best way to evaluate FireEye is for an organization to deploy our technology in their own environment and they will understand why we are the market leader in stopping advanced attacks."

In a separate blog post and an interview with CRN, Manish Gupta, FireEye's senior vice president of products, said NSS Labs failed to test the platforms against zero-day attacks that use custom malware designed to exploit previously unknown and unpatched vulnerabilities. All of the malware used in the testing had been known for months or years, Gupta said.

NSS Labs' Phatak told CRN that FireEye was opposed to testing the platforms using a zero-day exploit and only started raising concerns after the results were revealed and it was apparent it did not score well against the field of competitors. The company should be credited with opening up a new market with its technology, but now it must compete against a growing field of competitors, Phatak said.

"They hadn't really had the critical analysis of what they do and don't do before our last test round," Phatak said. "They wanted to be the only game in town for as long as they possibly can, but now the expectations have gone beyond just early technology."