|
VTC University of Hamburg PC Scanner Test 97-07 Virus Test Center (VTC) at Hamburg University`s Faculty for Informatics recently tested on-demand scanners for their ability to identify PC viruses. Tests were performed on VTCs virus databases, which were frozen on their April 30, 1997 status to give AV producers a fair chance to support updates within 7 weeks. The test goal was to determine detections rates, reliability (=consistency) of virus identification and reliability of detection of submitted or publicly available scanners. Moreover, development of detection quality was measured where more than one update was available during the test period (including updates up to June 22, 1997). Essential information about the virus databases: 12,826 File Viruses in 83,910 infected files, 938 System Viruses in 3,387 infected images, and 617 Macro Viruses in 2,036 infected documents.
The test contained versions the following scanners: One product (Power AntiVirus, PAV) of GData/Germany proved to be an almost identical version of another one (AVP). All other products seem to differ, though some products evidently share some basic detection engine. One more scanner (PCVP) was tested but didnot properly execute on any test, so it was withdrawn from the test. One other scanner (TNT) was requested to be tested by the producer (Carmel) but only under the explicit condition that its results were not published; this was inconsistent with VTCs rules, and the scanner was excluded from the test. Moreover, several attempts to contact several reputed AV producers was answered with electronic silence. An overview of the results is given in "6a-sumov.txt", and details of the respective tests under DOS, Windows 95 and Windows NT can be found in related files (see 1content.txt). Eval #0: Evaluation of Scanner Improvement between last tests: Concerning performance of DOS scanners, a comparison of virus detection results in test "1997-02" with the new test "1997-07" shows how scanners behave and how manufacturers work for adapting their products to the growing threat of new viruses. The following table lists the development of the detection rate of scanners (most actual versions in each test), and it calculates the change (+ indicating improvement) in detection rates. For reasons of fairness, it must be noted that improvement of those products which have yet reached a very high level of detection and quality (say: more than 90 or 95%) is much more difficult to acchive than for those products which reach lower detection rates. Moreover, changes in the order of about +-2% are not significant as this is about the growth rate per month, so detection depends strongly whether some virus is reported (and analysed and included) just before a new update is delivered.
=====================================================================
SCANNER Boot-V.Detection File-V.Detection Macro-V.Detection
97/2 97/7 CHANGE 97/2 97/7 CHANGE 97/2 97/7 CHANGE
=====================================================================
ALERT 93.6% 95.4% + 1.8% 98.8% 94.1% - 4.8% 96.5% 66.0% -30.5%
AVAST 94.1% 98.5% + 4.4% 98.9% 97.4% - 1.5% 99.3% 98.2% - 1.1%
AVG 70.9% 70.4% - 0.5% 79.2% 85.3% + 6.1% 25.2% 71.0% +45.8%
AVP 64.8% 99.3% +34.5% 98.5% 98.4% - 0.1% 99.3% 99.0% - 0.3%
AVScan 60.9% 71.0% +10.1% 73.4% 80.6% + 7.2% 58.0% 68.6% +10.6%
DRWEB 44.3% 74.0% +29.7% 93.2% 93.8% + 0.6% 90.2% 98.1% + 7.9%
DSAV 99.8% 99.5% - 0.3% 99.7% 99.6% - 0.1% 97.9% 98.9% + 1.0%
FMACRO n/a n/a n/a n/a n/a n/a 98.6% 98.2% - 0.4%
FPROT 85.0% 82.5% - 2.5% 90.7% 89.0% - 1.7% 43.4% 36.1% - 7.3%
FWIN n/a n/a n/a n/a n/a n/a 97.2% 96.4% - 0.8%
IBM n/a 94.5% n/a 93.6% 95.2% + 1.6% 65.0% 88.8% +13.8%
ITM 12.9% 38.0% +25.1% n/a 81.0% n/a 81.8% 58.2% -23.6%
NAV 66.9% 67.1% + 0.2% 80.7% 86.4% + 5.7% 84.6% 95.6% +11.0%
NVC 86.0% 91.4% + 5.4% 87.4% 89.7% + 2.3% 13.3% 96.6% +83.3%
SCN 82.5% 95.3% +12.8% 83.9% 93.5% + 9.6% 95.1% 97.6% + 2.5%
SWP 94.8% 92.6% - 2.2% 95.9% 94.5% - 1.4% 87.4% 89.1% + 1.7%
TBAV 78.6% 77.4% - 1.2% 95.5% 93.7% - 1.8% 72.0% 96.1% +24.1%
VBS n/a 8.0% n/a 43.1% 56.6% +13.5% n/a n/a n/a
VDS n/a 45.5% n/a n/a 44.0% n/a 16.1% 9.9% - 6.2%
=====================================================================
Eval #1: Evaluation for overall virus detection rates under DOS: The following grid is applied to classify scanners:
- detection rate above 90% : the scanner is graded "very good" - detection rate of 80-90% : the scanner is graded "good enough" - detection rate of 70-80% : the scanner is graded "not good enough" - detection rate of 60-70% : the scanner is graded "rather bad" - detection rate of 50-60% : the scanner is graded "very bad" - detection rate below 50% : the scanner is graded "useless" To assess an "overall grade" (including boot, file and macro virus detection), the lowest of the related results is used to classify the resp. scanner. If several scanners of the same producer has been tested, grading is applied to the most actual version (which is, on most cases, the version with highest detection rates). Only scanners where all tests were completed are considered; here, the most actual version with test completed was selected. The following list indicates those scanners graded into one of the upper three categories:
"Excellent" DOS scanners:
"Very Good" DOS scanners:
"Good Enough" DOS scanners: Remark: The following scanners fail a good classification by just one category: Alert AVG, DrWeb, Norton AV and TBAV. Concerning "In-The-Wild" viruses, a much more rigid grid must be applied to classify scanners, as the likelyhood is significant that a user may find such a virus on her/his machine. The following grid is applied:
- detection rate is >95% : scanner is "very good" - detection rate is >90% : scanner is "good" - detection rate is <90% : scanner is "risky"
"Excellent" DOS scanners (equal rating, alphabetical order):
"Very Good" DOS scanners:
"Good" DOS scanners: Eval #2: Evaluation for detection by virus classes under DOS: Some scanners are specialised on detecting some class of viruses (either in deliberately limiting themselves to one class, esp. macro viruses, or as that part is significantly better as other parts). It is therefore worth notifying which scanners perform best in detecting file, boot and macro viruses. Compared to the last test, the number of "excellent" macro virus detectors has significantly grown (as has the class of "good" ones which is not listed here); in contrast, "standard" file viruses (and even more) boot viruses seem to be less attractive in product upgrading. Those products with grade "excellent" are listed below. 2.1 Detection of file viruses:
"Excellent" DOS scanners:
"Excellent" DOS scanners:
"Excellent" DOS scanners: Eval #3: Evaluation of File and Macro Malware detection: Several scanners are able to detect also non-viral malware. The related test includes 49 macro-related specimen and 163 file-related "malware".The following grid is applied to classify detection of macro malware:
- detection rate of 80-90% : the scanner is graded "very good" - detection rate of 60-80% : the scanner is graded "good enough" - detection rate of < 60% : the scanner is graded "not good enough" "Excellent" DOS scanners: ----------
"Very Good" DOS scanners: Eval #4: Evaluation for overall virus detection rates under Windows 95 and Windows NT: The number of scanners running under Windows 95 and Windows NT is still small, though growing. Significantly less products were available for these tests, compared with the traditional DOS scene. The same grid as for the DOS classification is applied to classify scanners according to their ability to detect file and macro viruses under Windows 95 and Windows NT. The following list indicates those scanners under Windows 95 graded into one of the upper categories "excellent" and "very good" upon detecting file and macro viruses:
"Excellent" Windows 95 scanners:
"Very Good" Windows 95: The following list indicates those scanners under Windows 95 graded into one of the upper categories "excellent" and "very good" upon detecting file and macro viruses:
"Excellent" Windows NT scanners:
"Very Good" Windows NT: Conclusion: Scanners under Windows 95 and Windows NT need still some de- velopment to reach the same high level of detection and quality as present DOS scanners. Final remark: More detailed information about the test, its methods and viral databases, as well as detailed test results are available for anonymous FTP downloading from VTCs HomePage (VTC is part of the Working Group AGN): ftp://agn-www.exvtc.de//pub/texts/tests/pc-av/1997-07 Any comment and critical remark which helps VTC learning to improve our teste methods will be warmly welcomed. The next comparative test is planned for January 1998, with viral databases to frozen on November 30, 1997. Any AV producer wishing to participate in that test is invited to submit related products.
On behalf of the VTC Test Crew: |