How We Test: Anti-virus software
A look at the real-world testing methodology that sets our anti-virus testing lab apart from the crowd
Threats
We expose each anti-virus program to a range of current threats as they appear in the wild. The threats we obtain are between a few minutes and a few days old, taken from a list of potentially malicious URLs that we compile ourselves. Our test systems include machines that are used exclusively to visit and monitor the behaviour of these sites before they’re used in our main tests. Those with an active virus payload are added to our threat exposure system.
False Positives
Anti-virus software must defend against real threats, but shouldn’t make false positive detections that interfere with your ability to use legitimate programs and safe websites. We download and install a range of software from popular websites, from IP scanners to games.
We haven’t gone out of our way to trip up the products by installing products with a potentially malicious use, such as password crackers. We observe the reactions of the anti-virus applications, taking particular note when programs are prevented from running or made to appear inordinately threatening.
The Testing Procedure
In each round of testing, we expose the systems to the same threat. We follow a strict monitoring process that involves both software and a human tester who takes notes on the anti-virus software’s response to the threats. We observe as the threat either progresses or is stopped by the anti-virus software. We analyse the system and our log files to establish whether the malware has been completely blocked or not.
If any malware is still present, we reboot and scan it with the anti-virus product. If it’s stopped the malware taking hold, there’s little to see at this stage. If the system has been infected, this scan gives the software a chance to remove malicious programs and system changes.
When we interact with both malware and the anti-virus programs, we act as a naïve user. We give malware an opportunity to establish itself, and we always select the default options presented by our anti-virus software or any malware. If we’re not presented with a default option, we wait 20 seconds for the program to select one automatically and then, if it doesn’t make its own choice, we choose the first option.
[IMG ID=”156769F”][/IMG]
Analysis
Our live monitoring usually makes it obvious as to whether or not the anti-virus software has defended the system. If there is less evidence of infection but the system is not definitely clean, we analyse our log files. Systems are deemed to have been defended if we can find no evidence of any malicious alterations to the Registry or the addition of any undesirable files.
We regard a computer as compromised if a virus continues to run or has visibly altered the system’s configuration in a manner likely to cause future threats after a reboot and manual virus scan. If the anti-virus program hasn’t deleted, contained, or prevented the system from running the infected files, or if the operating system is rendered unusable, then the software has failed.