How we test anti-virus software

Posted on 5 Jan 2010 at 16:44

 

Gallery

THE TEST SYSTEMS

To ensure a level playing field, we install each anti-virus program on one of our test PCs, which have identical hardware and run identical Windows XP Professional installations. These have been updated to Service Pack 2 (SP2), but no further patches or Windows updates have been applied. This is the most common software setup for many computers that access the internet today and, therefore, the most targeted by hackers. Each anti-virus product is installed and updated with the latest virus definitions. Although no further Windows updates are applied, the anti-virus software is updated to its latest definitions for each test. To ensure a level playing field, we expose each system to the same threat within a 24-hour period.

MONITORING

We use a range of tools to help us monitor and analyse our test results. The systems are pre-installed with software that allows us to record and watch events in real time as Windows processes start and stop. We can observe network traffic to and from the PC and make and compare Windows Registry and file-system records to check for undesirable modifications.
Malware downloaded from websites is often subject to rapid change and, although it may appear as if the same malicious software is being downloaded every time a victim visits a particular website, there are often subtle variations that can make it harder to detect. Because we want to expose each anti-virus suite to exactly the same threat, we use a special, custom-built system that ensures each of our test computers receives the same threats when they visit an infected site.

THREATS

We expose each anti-virus program to a representative range of current threats as they appear in the wild. The threats we obtain are between a few minutes and a few days old, taken from a list of potentially malicious URLs that we compile ourselves. Our test systems include machines that are used exclusively to visit and monitor the behaviour of these sites before they are used in our main tests. Those with an active virus payload are added to our threat exposure system.

FALSE POSITIVES

Anti-virus software must defend against real threats but shouldn’t interfere with your computer’s ability to use legitimate programs and access safe websites. We download and install a wide range of software from popular websites, from IP scanners to offline flash games. We haven't gone out of our way to trip up the anti-virus products by installing products with a potentially malicious use, like password crackers. We observe the reactions of the anti-virus applications, taking particular note when our false positives are either prevented from running or made to appear inordinately threatening.

TESTING PROCEDURE

We carry out our tests in rounds. In each round, we expose the systems to the same threat, and we follow a strict monitoring process that involves both software and a human tester who takes notes on the anti-virus software’s reactions to the threats. To begin the test we first expose a system to some malware and then observe as the threat either progresses or is stopped by the anti-virus
software. We analyse the system and our log files to establish whether the malicious software has been completely blocked or not.
If any malware is still on the system, we then reboot and scan it with the anti-virus product. If the software has stopped the malware taking hold, there is little to see at this stage. If the system has been infected, this scanning process gives the software an opportunity to remove any malicious programs and system changes it encounters. We then save the results and logs and restore the system to a clean state.
When we interact with both malware and the anti-virus programs, we act as a naïve user. We give malware an opportunity to establish itself, and we always select the default options presented by our anti-virus software or any malware. If we’re not presented with a default option, we wait 20 seconds for the program to select one automatically and then, if it doesn’t make its own choice, we choose the first option.

ANALYSIS

Our real-time monitoring system usually makes it obvious as to whether or not the anti-virus software has defended the system. In cases where there is less evidence of infection but the
system is not definitely clean, we analyse the computer’s log files. Systems are deemed to have been protected if we can find no evidence of any malicious alterations to the Registry or the addition of any undesirable files. We regard a computer as having been compromised if a virus continues to run after a reboot and manual virus scan. If the anti-virus program hasn’t deleted or contained the infected files, or if the operating system is rendered unusable, the software has failed.