Palo Alto Networks, FireEye Criticize NSS Labs; Testing Firm Defends Itself


Network security industry rivals Palo Alto Networks and FireEye are questioning the credibility of tests conducted by NSS Labs, following a new report that aimed to test the effectiveness and gauge the total cost of ownership of the industry's so-called breach detection appliances.

FireEye had declined to participate in the recent testing, but engineers at NSS Labs proceeded with tests on the company's Malware Protection System (MPS) appliance anyway. It scored "below average" in the NSS Labs comparative group product test. Palo Alto Networks was not invited to participate in the testing firm's latest report.

In a wide-ranging interview with CRN, Palo Alto Networks CTO Nir Zuk criticized the legitimacy of NSS Labs' testing methodology. The security networking vendor has declined to participate in NSS Labs testing in the past because of what Zuk calls a "flawed sales model."

[Related: NSS Labs Intrusion Prevention Tests: Did Your Vendor Partner Pass?]

In order to get as many vendors in the report as possible, according to Zuk, the company sets the testing methodology very low, enabling "mediocre vendors" to compete. The company issues licenses called "reprint rights" to vendors for publishing testing results, negotiating a fee with vendors that can exceed $100,000 before the testing results are made public. The fee gives vendors the right to publicize the report and distribute it to potential clients.

"You don't want to do a report on two vendors; you want to do a report on 10 vendors and charge each of them $100,000, and that's how you make money," Zuk said. "If you are going to have a very high bar for your test and only one or two vendors are going to succeed in your test, the vendors are going to stop paying for it."

NSS Labs can't generate enough revenue from customers so it sets the bar where the vendors want it, Zuk said.

"The vendors pay a lot of money; this is all vendor-paid and there's a degree of influence," Zuk said. "If you think you are much, much better than others and you think the bar should be much higher, then maybe you don't want to participate in some of these tests."  

The report in question, "Breach Detection Systems Comparative Analysis and Security Value Map," was issued April 2 by NSS Labs. It evaluated security appliances from AhnLab, Fidelis, FireEye, Fortinet, Sourcefire (Cisco) and Trend Micro, establishing metrics on a wide range of issues, from the time it takes to deploy and configure the devices to the effectiveness in detecting malware and exploits used by attackers. 

NEXT: NSS Labs Responds To Criticism