How we test anti-virus software

Posted on 5 Jan 2010 at 16:44

 

Gallery

The important thing about reviewing anti-virus software is testing them using real viruses in the wild, downloaded to a computer in the same way that you'd find them. By doing this, the full security suite, including its firewall and any browser plug-ins, are tested to the full. This gives the clearest impression of how good a security suite is.

Too many magazines cheat when it comes to virus testing. The worst offenders simply use the Eicar test files. These are not viruses, but simply text files that all anti-virus products are designed to detect so that you can verify that your software is working.

The next step up is where other magazines use a collection of viruses they've sourced from an anti-virus vendor. These viruses are stored in a directory and scanned by each of the test products. This is unrealistic in many ways. First, it doesn't test each product's full capabilities. Secondly, by scanning for viruses, the software never has to defend against an attempted malware installation. Finally, the source of the viruses is a big problem, as vendors will only ever release samples to testers that their product detects. The only advantage that people who use this kind of testing talk about is the massive sample size, making grand claims about the thousands of tests they've used.

This misses the point. The sample size is not the important factor, it is the freshness of the malware that really counts. Any anti-virus vendor can spot thousands of weeks-old malware samples, but how do they deal with brand new threats? This is where our testing comes in.

We use fewer virus samples (20 for this test), but each one is live on the internet and found and verified by our virus researchers, not vendors. Computer Shopper’s specialised anti-virus testing lab is one of the most advanced in the world. In fact we're one of the few magazines that's a member of the Anti-Malware Testing Standards Organization (AMTSO), an official body designed to improve the standards of anti-virus testing.

No other computer magazine currently tests a security suite’s anti-virus protection capabilities to the same level of detail or accuracy. The primary aim of our anti-virus tests is to create a realistic environment that accurately reproduces the way a PC encounters malicious software in the real world. For example, in our tests we expose 'victim' systems to web-based threats by visiting an infected website on the internet using Internet Explorer, rather than by loading viruses and Trojans from a CD or over the internal network.