FireEye CTO Dave Merkel first issued a statement Wednesday questioning the legitimacy of the report. FireEye, which has been gaining much attention following its successful IPO last December, has been highlighting the number of zero-day threats it detected in 2013 as part of its marketing strategy.
"Any lab test is fundamentally unable to replicate the targeted, advanced attacks launched by sophisticated criminal networks and nation-states," Merkel said in the statement.
In a detailed blog post, FireEye's senior vice president of products, Manish Gupta, called the NSS methodology "severely flawed." Gupta said FireEye no longer participates in the tests, insisting that they should run in a production environment. .
"The FireEye product they used was not even fully functional, leveraged an old version of our software and didn’t have access to our threat intelligence [unlike our customers]," Gupta said.
The FireEye appliance also wasn't connected to the FireEye threat intelligence feed to receive blacklisting updates. Gupta criticized the malware samples used during the testing process. The FireEye appliance missed some common threats, but Gupta said all the vendor products should have been tested against "new and unknown" threats, such as a zero-day exploit.
NSS Labs is fully defending its methodology and is standing by the legitimacy of the results. Palo Alto Networks was going to be included in the next round of testing, said Vikram Phatak, who told CRN Wednesday that he was perplexed by the Palo Alto Networks criticism.
NSS Labs does not conduct a "pay-to-play" model of testing in which vendors must pay to participate, Phatak said, adding that the firm also got out of the certification business in 2009 to bolster the legitimacy of its tests. NSS Lab engineers conduct testing based on customer requests. If a vendor declines to participate in testing, NSS Labs will buy the appliance or software to conduct tests, he said.
"In any test and every test that is published and made public we do not take a single penny from the vendors that are examined," Phatak said, saying the firm attempts to mirror the Consumer Reports testing practices. "Most of the money we receive to pay our bills come from enterprise clients, many banks and oil companies, who require an evaluation based on testing data without the subjectivity."
Phatak addressed the FireEye criticism as well, saying that NSS Labs tested the appliance against live, real-time exploits and malware. The FireEye platform couldn't support detection on all platforms tested, resulting in subpar results.
"There are other products out there that are probably better based on what the data is saying," Phatak said. "The core issue is that they had a product that didn't work with 64-bit operating systems,"
The product also didn't get the highest detection rates because it uses an open-source antivirus engine (ClamAv), in addition to whitelists and other methods to detect common malware strains. The combined, standard detection methods are not 100 percent effective, Phatak said. To detect suspicious files, the vendor tests them in 10 virtual machine detonation chambers.
Solution providers reached by CRN declined to speak about the criticism against NSS Labs. They called the reports useful for some larger clients that conduct a systematic evaluation process. A thorough evaluation typically involves attempting to review tests from an independent source, according to security experts.
PUBLISHED APRIL 3, 2014