Apparently Microsoft isn’t too happy with the outcome of a recent test that labelled its antivirus software as one of the worst performing around.

The company has challenged the methodology used by Germany’s AV-Test, after its Security Essentials and Forefront Endpoint Protection programs failed to pass its tests and gain certification. Out of 25 such programs put through their paces by AV-Test, only three failed to meet the grade.

According to AV-Test, Security Essentials fails to come up to scratch with its ability to detect new malware samples and zero-day exploits when compared to rival antivirus software programs. However, AV-Test acknowledged that Microsoft’s product did block all prevalent malware in its tests. Overall, Security Essentials achieved a score of just 71% in November and 78% one month later, far below the industry average of 92%, and not enough for the software to gain certification.

Following these findings, Microsoft issued what can best be described as a ‘polite rebuttal’ of the German security lab’s claims via a blog post:

“Our review showed that 0.0033 percent of our Microsoft Security Essentials and Microsoft Forefront Endpoint Protection customers were impacted by malware samples not detected during the test,” wrote Joe Blackbird, Head of Microsoft’s Malware Protection Center.

“In addition, 94 percent of the malware samples not detected during the test didn’t impact our customers.”

Blackbird went on to outline three different reasons as to why AV-Test’s findings are unfair:

“AV-Test reports on samples hit/missed by category. We report (and prioritize our work) based on customer impact.”

“AV-Test’s test results indicate that our products detected 72 percent of all “0-day malware” using a sample size of 100 pieces of malware. We know from telemetry from hundreds of millions of systems around the world that 99.997 percent of our customers hit with any 0-day did not encounter the malware samples tested in this test.”

“AV-Test’s test results indicate that our products missed 9 percent of “recent malware” using a sample size of 216,000 pieces of malware. We know from telemetry that 94 percent of these missed malware samples were never encountered by any of our customers.”

Blackbird further called into question AV-Test’s claims in an ever so slightly patronizing manner, ‘acknowledging’ that It’s not easy for independent security firms to come up with tests that can mimic virus attacks as they happen in the real world.

Microsoft probably does sound a little bit arrogant here, but it’s worth pointing out that AV-Test itself has previously admitted that its tests are far from perfect. However, the security firm still insists that its tests are valid. In an interview with ZDNet earlier today, AV-Test CEO Andreas Marx said that the big concern nowadays was the use of “server-side polymorphism”, a technique by which malware can alter its appearance without having any impact on its ability to do the task it was designed for – thus enabling it to evade antivirus software.

“Today, every attack is somehow targeted. One example is server-side polymorphism which means that every visitor of a malicious website gets a different variation of the same malware,” Marx told ZDNet.

“This means the malware file looks different, but behaves the same. So the prevalence for this sample is very low, as just one user was affected, worldwide.”

Marx went on to describe AV-Test’s methodology, saying that what it does is to pluck ‘samples’ of the major known malware families, and test products against these.

“As of today, every two seconds we see three new malware samples, which are summing up to a few million samples per month. Instead of looking at millions of samples, our focus is on the unique families. Out of every family, we select recent samples in order to use them in our tests. So the impact of these samples is indeed low, however, the impact of the malware family is considerably high. We favor the family-based approach over the sample-based one because of today’s malware situation.”