|
Eval #1: Evaluation for overall virus detection rates
Eval #2: Evaluation for detection by virus classes
Eval #3: Evaluation of Macro Malware detection
Eval #1: Evaluation for overall virus detection rates:
The following grid is applied to classify scanners:
- detection rate above 95% : the scanner is graded "excellent"
- detection rate above 90% : the scanner is graded "very good"
- detection rate of 80-90% : the scanner is graded "good enough"
- detection rate of 70-80% : the scanner is graded "not good enough"
- detection rate of 60-70% : the scanner is graded "rather bad"
- detection rate of 50-60% : the scanner is graded "very bad"
- detection rate below 50% : the scanner is graded "useless"
To assess an "overall grade" (including boot, file and macro virus
detection), the lowest of the related results is used to classify the
resp. scanner. If several scanners of the same producer have been tested,
grading is applied to the most recent version (which is, on most cases,
the version with highest detection rates). Only scanners where all tests
were completed are considered; here, the most actual version with test
completed was selected.
The following list indicates those scanners graded into one of the
upper three categories:
|
"Excellent" scanners: | DSAV 768 (99,7% 99,8% 97,9%)
AVP 2.2 (98,8% 97,0% 96,5%) |
|
"Very Good" scanners: | AVAST!77/1 (98,9% 94,1% 99,3%)
Alert41/15 (98,8% 93,6% 96,5%)
Sweep 294 (95,9% 94,8% 95,1%) |
|
"Good Enough" scanners: | F-PROT2.25 (90,7% 85,0% 98,6%=F-MacroW 102)
Scan 2.53 (83,9% 82,5% 95,1%) |
Remark: The following scanners fail a good classification by just one
category:
AVPlite, DrWeb, TBAV 706 and NAV.
Concerning "In-The-Wild" viruses, a much more rigid grid must be applied
to classify scanners, as the likelyhood is significant that a user may
find such a virus on her/his machine. The following grid is applied:
- detection rate is 100% : scanner is "excellent"
- detection rate is >95% : scanner is "very good"
- detection rate is >90% : scanner is "good"
- detection rate is <90% : scanner is "risky"
|
"Excellent" scanners: | DSAV 768 (100% 100% 100%)
AVP 2.22 (100% 100% 100%)
|
|
"Very Good" scanners: | FPROT 2.25 (99,2% 98,9% 100%)
Scan 2.53 (100% 96,7% 100%)
NAV 3.0 (99,2% 96,7% 100%)
Sweep 2.94 (100% 100% 95,5%) |
| "Good" scanners: | AVAST! 77/1 (100% 94,5% 100%)
Alert 41/15 (100% 93,4% 100%)
TBAV 707 (100% 100% 90,9%) |
Eval #2: Evaluation for detection by virus classes:
Some scanners are specialised on detecting some class of viruses (either
in deliberately limiting themselves to one class, esp. macro viruses, or
as that part is significantly better as other parts). It is therefore worth
notifying which scanners perform best in detecting file, boot and macro
viruses. The same grades are applied as in the "overall" grading (see 1).
2.1 Detection of file viruses:
|
"Excellent" scanners: | DSAV 768 (99,7%)
AVAST! 97/1 (98,9%)
AVP 2.22 (98,8%)
Alert 41/15 (98,8%)
Sweep 294 (95,9%)
TBAV 706 (95,5%)
|
|
"Very Good" scanners: | IBM AV 2.51 (93,6%)
DrWeb 318 (93,2%)
F-PROT 2.25 (90,7%) |
|
"Good" scanners: | Norman VC 351 (87,4%)
Scan 2.5.3 (83,9%)
NAV 3.0 (80,7%)
|
2.2 Detection of boot viruses:
|
"Excellent" scanners: | DSAV 768 (99,8%)
AVP 2.2 (97,0%) |
|
"Very Good" scanners: | Sweep 2.94 (94,8%)
AVAST! 77/1 (94,1%)
Alert 41/15 (93,6%) |
|
"Good" scanners: | Norman VC 351 (86,0%)
F-Prot 2.25 (85,0%)
Scan 2.53 (82,5%)
|
2.3 Detection of macro viruses:
|
"Excellent"scanners: | AVAST!77/1 (99,3%)
AVP 3 lite (99,3%)
F-MacroW 1.02 (98,6%)
DSAV 768 (97,9%)
F/Win 4.03 (97,2%)
Alert 41/15 (96,5%)
Scan 2.5.3 (95,1%)
|
|
"Very Good" scanners: | DrWeb 316 (90,2%) |
|
"Good" scanners: | Sweep 2.94 (87,4%)
NAV 3.0 (84,6%)
ITM 3.11b (81,8%)
|
Eval #3: Evaluation of Macro Malware detection:
Several scanners are able to detect also non-viral malware. As existence
of macro malware is published, the (yet small) macro malware database was
used for an initial test for malware detection. The following grid is
applied to classify detection of macro malware:
- detection rate > 90% : the scanner is graded "excellent"
- detection rate of 80-90% : the scanner is graded "very good"
- detection rate of 60-80% : the scanner is graded "good enough"
- detection rate of < 60% : the scanner is graded "not good enough"
|
"Excellent" scanners: | ---------- |
|
"Very Good" scanners: | AVP 3 lite (86,7%)
DSAV 768 (86,7%) |
|
"Good Enough" scanners: | AVAST 77/1 (66,7%)
NAV 3.0 (66,7%)
Alert 41/15 (60,0%)
DrWeb 318 (60,0%)
Scan 2.5.3 (60,0%)
|
More detailed information about the test, its methods and viral databases,
as well as detailed test results are available for anonymous FTP downloading
from:
ftp://agn-www.exvtc.de/pub/texts/tests/pc-av/1997-02/
General information is also available from VTCs
HomePage (VTC is part of working group "AGN")
Any comment and critical remark which helps VTC learning to improve our
teste methods will be warmly welcomed. The next comparative test is planned for
May-June 1997, with viral databses frozen On April 30, 1997. Any AV producer
wishing to participate in that test is invited to submit related products.
On behalf of the VTC Test Crew:
Dr. Klaus Brunnstein (February 20, 1997)
Last updated: 26.02.97
|