NSS Labs Adjusts Testing To Address FireEye, AhnLab Criticism


NSS Labs will test the effectiveness of breach detection systems against zero-day exploits, altering its testing methodology to address some of the criticism from security vendors FireEye and AhnLab, which challenged the credibility of April test results.

The Austin, Texas-based independent testing firm issued a document outlining its adjusted testing practices June 6.  NSS Labs CEO Vikram Phatak told CRN that the company stands by its comparative analysis report but made adjustments to directly address concerns and add clarity to its processes for future tests.

"I am sensitive to the representation that somehow we did something wrong or were not transparent in our methodology," Phatak said. "There is a huge amount of transparency both for the enterprises that request the tests and the vendors before we even conduct the testing."

[Related: FireEye, NSS Labs Continue To Trade Barbs Over Testing Report Credibility]

Phatak said NSS Labs partners with Exodus Intelligence, an Austin-area software bug hunting firm in the emerging vulnerability broker market. Exodus along with VUPEN, ReVuln, Netragard and Endgame Systems employ hackers to find software bugs and openly engage in selling their vulnerability findings to governments and corporations. It's a murky area often criticized by software security advocates because the bugs are not immediately reported to the software maker for patching. Instead, they may be used by intelligence agencies to support surveillance and cyberespionage attacks against adversaries.

NSS Labs has used zero-day exploits for previous tests when requested by its enterprise and government clients, Phatak said. The organization ensures that the tests are not public-facing, minimizing a leak of the zero-day vulnerability to the public, he said. When tests are concluded, the organization provides information to the software maker and computer emergency readiness teams, he said.

FireEye lashed out against the NSS Labs testing in April when it scored "below average" in the comparative group product test that pitted the FireEye Web and email Malware Protection System (MPS) appliances against products from AhnLab, Fidelis, Fortinet, Sourcefire (Cisco) and Trend Micro. Both FireEye and AhnLab earned a "caution" designation from NSS Labs due mainly to their below-average security effectiveness scores and cost of ownership, according to NSS Labs. FireEye said the tests used known malware samples rather than evaluating detection of a zero-day attack. AhnLab also denounced the test results.

NEXT: FireEye Declines To Respond To Changes