CAVEAT: This is my humble opinion based on real-World results.

Just wanted to post a brief write-up on my experiences so far with running several antivirus products "in the wild" - in my malware honeypots.

The reason I am running these honeypots is twofold:
1) To catch new malware
2) To test antivirus products in the most "extreme" of conditions...running on a completely unpatched Windows 2000 box with their most vulnerable ports exposed to the Internet.

Initially I set it up for #1 only...but it soon became apparent that some products could really handle the task and some were really not up to par. Granted, this is an extreme situation...so it isn't realistic to actually tell anyone to run their real systems this way...but this is also why it makes a great testbed! How well does your AV protect if it is the *only* protection???

First off, the setup:

3 Honeypots -
Windows 2000 Professional Gold (no patches)
Running in VMWare GSX 3.1 on RHEL 3 ES (2.4.21-15.0.3.ELsmp)
Ports 135, 137, 138, 139, 445 open to the Internet

Only difference between them is that two of them have 4GB disks and one only has 2GB disk.

I chose 2 of the more popular "corporate" AV solutions:

- Symantec AntiVirus Corporate Edition (SAV) 9.0.2.1000
- McAfee VirusScan Enterprise 8.0.0

I then chose the hacker's choice (my favorite, and many others!) AV solution:

- Kaspersky Personal 5.0 (Actually, Defender Pro 2005 - which is an identical OEM rebadge of KAV 5.0 - available at Wal-Mart for $19.99 for 3 licenses, hehe)

I used default values for all apps *except* I changed Symantec's product so it would update daily instead of the ridiculous once per week on Friday setting it ships with by default.

Let me cut to the chase here and tell you who is winning so far: McAfee VirusScan Enterprise 8.0.0. Surprised? Let me break down some of the reasons (and features) that make this so, using a comparison against Symantec (I'll get to KAV later, don't worry!):

(IP addresses sanitized)


Access Protection

When the Symantec box did not have detection for a new malware variant the file was able to execute and grab its payload at will, plus flood the network while it tried to propagate. However, when the McAfee box also did not have detection for this variant...but there is one big difference, it uses port filtering, so there was plenty of this in my logs:

11/20/2004 7:03:02 PM Blocked by port blocking rule svdll32.exe Prevent IRC communication 00.00.00.00

Therefore, the malware was unable to spread, download it's payload, and receive further instructions. On top of that, McAfee will stop certain suspicious file manipulations, i.e.

System:Remote C:\WINNT\system32\Paccard.exe Prevent remote creation/modification/deletion of anything in the Windows folder and subfolders Action blocked :Create

Finally, there is one more additional layer of protection McAfee *would* provide, but it is not enabled by default:

11/21/2004 9:18:38 AM Would be blocked by behaviour blocking rule (rule is currently in warn mode) 2000WK02\User System:Remote C:\WINNT\system32\Ultraedit.exe Prevent remote creation/modification/deletion of anything in the Windows folder and subfolders Action blocked :Create

Buffer Overflow Protection

The Symantec box is constantly rebooting. This is simply a way of life for any Windows box that is unpatched and listening on port 445 to the Internet. However, the McAfee box features buffer overflow protection, so it's reboots are fewer and far between, and even if it does reboot due to a buffer overflow, it is logged:

11/22/2004 12:39:15 AM Blocked by Buffer Overflow Protection WORKGROUP\SYSTEM C:\WINNT\system32\lsass.exe::GetProcAddress bo:stack

Pretty neat, huh?

On Access Scanning

Then you have your "standard" antivirus features...the "realtime" or "on-access" scanning. However, with McAfee, we're treated to more verbose information along with - pause for amazement - the source IP!

11/21/2004 9:18:44 AM Deleted 2000WK02\User System:Remote C:\WINNT\system32\Ultraedit.exe W32/Sdbot.worm.gen (Virus) 00.00.00.00

Talk about making it easy for the deskside monkeys to clean up. I will add, however, that it is not 100% fool-proof in it's IP information. Sometimes it just doesn't have the information for some detections. I am not sure why...but even if it gives me an IP half of the time, I'm a happy camper.

So, anyway...back to KAV as promised - this app works, and it prevents almost everything. It did get infected a few times though. So those of you that consider it a magical silver bullet...well...nothing is perfect. However, my main issue with KAV is that it is buggy as hell! Compared to the other apps, it really was not as robust. I mean, true, SAV would allow more malware in, but at least it was stable while the box was infested! lol But seriously, it would really lock things up sometimes...causing me a lot of grief. I still use this personally on my systems, but I think stability wise, SAV is best. With McAfee second and KAV a distant third.

I would rate McAfee best overall, KAV second, and SAV third. I'll post more but I'd like to see what interest there is in this before I blather on for eternity. I'll also try to post more concrete numbers. I've been so busy catching new stuff and playing with the honeypots and practicing computer forensics that I've pretty much ignored gathering statistics!

The poor Trojan/AV section is pretty empty recently so I figured I'd try to spice things up!!!

Please keep posting, im interested in reading what you have to say.

If I can get the chance this weekend I would like to start having a go at this myself.

Did you post a tutorial sometime back explaining how to setup your layout?

Cheers for your posts, keep them coming.

Interesting, thanks for sharing your info, however according to Virus Bulletin 100 - who claim to have test diff AV s/w from over 27 vendors

SAV has passed 26/32 tests,
McAfee has passed 18/35 tests,
Kasp has passed 24/37 tests,
Trend has passed 11/18 tests.
NOD32 has never failed a test.

About the VB100 tests: see http://www.virusbtn.com/vb100/about/index.xml

SAV Rating http://www.virusbtn.com/vb100/archives/pro...ml?symantec.xml
McAfee/NAI http://www.virusbtn.com/vb100/archives/products.xml?nai.xml
Kasp: http://www.virusbtn.com/vb100/archives/pro...l?kaspersky.xml
Trend: http://www.virusbtn.com/vb100/archives/pro...s.xml?trend.xml

About updating the SAV virus defs daily- their Liveupdate is an issue - i don't see why they don't release daily microdefs, but if u check out their website & search for cegetter or rcegetter batch file, their is an alternative method of getting daily defs.

Thanks,

Thanks for the comments guys.

I am aware of the VB100 tests, heck, I went to their Virus Bulletin conference a couple months back..heh. However - although their tests are extremely valuable and pretty much the only of their kind - they are not real-world tests, unfortunately.

If you look up the products I used in my honeypots, you'll note they all received VB100 awards. Therefore, we really need to ask ourselves - what does receiving a VB100 award mean?

My opinion is that receiving a VB100 award is essentially like detecting the EICAR virus. It is a test virus, controlled and detected by all as a "safe" and "sane" way to test that your product is working as it should. Finding that virus shows your product works, and things are good. My test was to show how they performed under fire, as the only protector of an extremely vulnerable system on the Internet. As you can see, the results are slightly different than VB100.

Besides, as a security professional and/or hobbyist - do you really want to trust one of your vital layers of security to magazine reviews and lab tests?

Just take a look at their testing methodologies. They are valid and well-defined, but their tests cannot really be compared to a live test. First of all, they would probably consider me insane. Second of all, it is apples to oranges in my opinion.

Also, MindSmith - RE: SAV defs. I changed their update to daily and in fact, I manually updated their definitions daily. You could argue that this is "cheating" - however, Symantec's definitions are so lacking that I felt I needed to give them any "leg up" I could. Also, my intention with this testing was to detect new malware. So I wanted to use any publically available definitions that were out so I would stop getting the "old" stuff.

kbnet - thanks again for the comments. I did post some information on my honeypot setup as well as tools used. I was a trial member back then so it is in that forum. Feel free to contact me (and this goes for anyone) if you have questions on specifics. Perhaps I'll get a detailed write-up of my setup going if there is need/interest. My previous post on my honeypot setup was not extremely detailed.

really looking forward to see the statistics. really interesting to read.
you should consider to take all the Av's from Virus Bulletin 100 to the test, maybe send em a mail about what version they used. i would reall want to see NOD32 battle it out if its *that* good
plus then you can compare the tests

anywayz, keep it up, good stuff.

FLX

Cool Sk3tch. I think that the whole subject around malware has become blurred of late with vendors like Trend claiming the HTML.Citifraud - which is actually just an HTML with a URL to a fake Citibank Form are viruses only add to the confusion. , Viruses, Worms, Blended Threats, Spyware, Adware - to date I have yet to encounter a single AV product that protect against all of these threats.

I use NOD32, Symantec, Trend, & Virusscan on different systems & so far SAV & NOD32 are (IMHO) overall much better in detection & removal of malware. But then again the importance of tests like VB100 (not mag reviews) is that they give structure & clear definition to the whole issue around testing & malware - which is often very subjective rather than being objective. The data from VB100 only serves to reinforce that.

Ulitmately looking at the number of Exec Mode Vulnerabilities on the Win32 platforms which when exploited by worms, often bypass the detection abilities of AV , a stateful inspection firewall, with application control capabilities, plus & good AV (that includes spyware, & adware detection) seem the best way forward.

The number of Dos vulnerabilities in each of the AV products it also important, of late Trend, NAI, Sophos, NOD32 have have zipped file bypass vulnerabilities - which therfore affect the products detection & protection it is able to provide. (See BUGTRAQ). The resilience of the AV software to retro-viruses, and DoS attacks are just as much an issue for me to consider & review.

I still would use VB100 ratings as a benchmark, and then look at additional features such as heuristics, worm-blocking, & virus removal as key points in my eval.

All said & done, (though many may disagree) I'd still rate SAV & NOD32 at number 1. Looking at all the issues that would affect the reliability of my AV's detection & repair capability.

Just my 2 cents.

Thanks guys, good discussion. Will relook at McAfee's buffer overflow protection again.

Ciao.

Yes verying intresting mate, i love reading your posts and when I get sometime I will try some of these my selfs.. Sorry if you said before but how do you monitor what the worm does once it is on your system?

Cheers

Silent Bob - I use a combination of the VMware snapshot function and a MD5 hash of most files on the system (i.e. system directory, program files, docs and settings, etc). The MD5 file integrity checker I use is Afick...which is free (uses Perl). It can get a bit tedious because everytime I change anything on the system I need to re-snapshot and re-MD5 the system (i.e. a new definition update). With the exception of the KAV system, which I just let run (since it is set by default to update every 3 hours - there is no way I could MD5 it that often unless this was my full-time job!).

Once the box is infected, I run a few utilities that capture some forensic information and then run Afick to see what files were changed. I also snapshot the Task Manager and the Registry (the "Run" key). Then I wrap it all up and submit it to vendors. If they're interesting enough, I'll sometimes post them here (I posted one about a Warez/FTP SDBot variant in the exploit research and discussion area a few days ago if you want a good example of the data captured).

MindSmith - I have only looked a NOD32 a few times. I will consider setting up a honeypot with them. I have heard that their heuristics are second-to-none. The only issue with NOD32 is that it is not very prevasive throughout corporate enterprises. For what reason, I don't know..but it seems that only the "Big Three" (Trend, Symantec, McAfee) have really penetrated corporate environments. My colleagues still laugh at me when I talk about Kaspersky being the best...lol.

i personally think that nod32 is the best
iv been usin it for 3 months now and i havent had a virus once (at least know ones)
it also blocks certain spyware, certain java applets( like on crack or porn site), and the anoying activX installation thing on crack/serial sites
im very pleased with this program. the real-time scanning does not consume a lot of cpu memory: for me : in taskmanager 7,627K and another process 2,620
whereas KAV uses a ****load oc memory when usin it with real time scanning.
nod32 has many options : auto update, exemtions of folders, MS office document scanner, internet monitor (bloks spyware, activx, bad java applets on crack sites).
you can modify the settings of each. it logs stuf too
in can do the basic AV stuff too
hope it helps in a way
nod32 is the best: hxxp://www.nod32.com

Really interesting reading sk3tch

Sure you can keep on going with these tests, they are exactly the kind of tests I wanted to see when playing the (dumb) end user who thinks having just an AV is enough to be protected !

So far I was up with SAV and KAV and it was no doubt KAV definitions and identification process were better than Symantec's ones. But I didn't knew about this stability lack, and same I didn't know about nod32 before reading the answers to your post... it's definetly a constructive post that u made

Finally I would say that I'm really interested in knowing how this nod32 behaves in such a "wild environment" !

Hope to hear soon again from this subject

Alright everyone, quick update...I've purchased Norman Virus Control and NOD32 and I'm readying honeypots with those products as I type.

Unfortunately, to make room for these guys I bumped off Defender Pro 2005 a.k.a. Kaspersky Personal 5.0. I just can't recommend KAV for people (even if you only pay $20 at Target or Wal-Mart buying Defender Pro 2005 ). I'm even testing Kaspersky Personal Pro 5.0 and it is the same as the regular Personal version except it has more "tweakability" - I'd recommend a different Windows AV. Their definitions are top-notch...product stability is lacking. I do, however, love their other AV products (SMTP, HTTP proxy scanning on Linux is excellent!).

I'll keep you guys posted on how the "rookies" do!