|
We guarantee your privacy: 1. We will never sell, rent, or give away your address to any outside party, ever. 2. We will never send you any unrequested e-mail, besides newsletter updates. 3. All unsubscribe requests are honored immediately, period. Privacy policy |
|
TOP STORY Questions arise on PC World tests By Brian Livingston A sweeping review of 10 security suites published in a major computer magazine last month featured some very unlikely rankings for this crucial category of products. After examining the evidence, I've found that some material facts were omitted from the article, rendering its ratings useless. The cover of the July 2006 PC World Magazine promised a review of security suites that would give readers "total protection against spyware, hackers & spam." Inside the magazine, a lengthy article summarized extensive test results by AV-Test.org, a respected antivirus research group based in Magdeburg, Germany. The magazine's product rankings, however, seemed inexplicable. When good software ratings go bad I reported on July 27 that CNET had given its Editors' Choice award in a June 4 review of security suites to Zone Alarm Security Suite (ZASS). PC Magazine's Editors' Choice went to the same product in a June 13 article. But PC World's ratings, which were first posted online in May, dropped ZASS to 6th place out of 10 products reviewed. The magazine's top honors went to Symantec Norton Internet Security 2006. I promised in my previous article to find out why and report the answer to you today. I have no love of any hardware or software vendor. If a product drops from being top-rated to merely mediocre, I'll say so in my Security Baseline section, below, or my Reviews Overviews, which I update online. PC World's ratings, however, are so puzzling that I immediately suspected something was wrong. After looking at some of the raw data, I believe AV-Test did provide PC World with accurate figures on the security suites that the German lab tested. Essential tests, however, were left out. The errors fall into three broad categories: 1. The review ignored behavior-based protection. Behavior-based protection, which stops suspicious activity, was left out of the tests. Signature-based virus scanning is declining in effectiveness, but at this point only a few of today's security suites include behavior-based protection. This crucial feature, which could represent a huge difference in malware detection, was simply left out of PC World's scoring. 2. The review omitted complete leak-test results. Leak tests rate a security suite's abilities to prevent malware that somehow sneaks into your PC from successfully sending your personal data to a remote server. AV-Test's findings revealed widely divergent scores for the tested suites. But the results for most vendors were left out of PC World's ratings. 3. The review turned off some suite features. Integrated security suites should be tested with all features turned on. PC World, however, chose to disable some capabilities in order to run tests aimed at other capabilities. I'll describe each of these three problems below. (story continues below) |
Windows Secrets NewsletterIssue 81 2006-08-10 Contents (Scroll down to Index) TOP STORY Questions arise on PC World tests SECURITY BASELINE ZASS remains in Security Baseline HOT TIPS MS software leads to new headaches WOODY'S WINDOWS The best ways to surf anonymously OVER THE HORIZON PowerPoint is still a big security risk PATCH WATCH Install MS06-040 to avoid the Next Big One PERIMETER SCAN The report from Black Hat and Defcon WACKY WEB WEEK Wacky Web Week title YOUR PREFERENCES About your subscription Newsletter Control Panel Windows Secrets home page How to subscribe Change your delivery address Change your preferences Access past free issues Access past paid issues Upgrade to paid version Search for info (WinFind) Submit a Windows tip Get subscription help How to unsubscribe Circulation: over 140,000 |
|
ADS
|
|
(continued from above) Consumer Reports backs up CNET and PC Mag The well-regarded U.S. product-testing magazine, Consumer Reports, hit the newsstands last week with its own ratings of PC security programs. The lab's testing separately rated the antivirus, antispyware, and antispam programs available from each vendor. In addition, the magazine contracted with security experts to generate 5,500 original virus variants to test behavior-based protection. CR also monitored how quickly the companies released updated signatures in real time over a period of weeks as new threats emerged on the Net. Zone Alarm Security Suite received Consumer Reports' Quick Picks award — the magazine's version of Editors' Choice — for "the best all-around protection." Perhaps because it's well known that security suites haven't yet mastered the latest spyware, CR also gave Quick Picks awards to Webroot Spy Sweeper and PC Tools Spyware Doctor in the antispyware category (with the free Spybot as a complement). These ratings make sense. They dovetail with CNET and PC Magazine's latest findings, both in the rankings and the award winners. Besides PC Magazine's Editors' Choice for the Zone Alarm Security Suite, for example, Editors' Choice awards also went to Webroot and PC Tools in the magazine's latest, July 2006 reviews of antispyware apps. To be sure, it's not unusual for magazines to differ in their ratings of computer products. For one thing, PC World's tests were conducted in April using ZASS version 6.0 and the then-current versions of competing products. The other publications' latest awards are based on the newer ZASS 6.5. But when a category is as important as security suites, and when one magazine's rankings deviate so much with no logical basis, I look for a reason. I found the answer in personal interviews with principals at AV-Test, Symantec, McAfee, and Zone Labs. To solicit comments, I provided AV-Test and PC World with draft copies of this story. I then participated in a telephone conference call with PC World editor-in-chief Harry McCracken, test center director Ulrike Diehlmann, and senior associate editor Narasu Rebbapragada. The review ignored behavior-based protection Near the middle of PC World's July 2006 article, I found a few sentences that related to nothing else in PC World's review:
Behavior blocking isn't a panacea. But when combined with traditional signature scanning it's a major enhancement. It should hardly be ignored. (Behavior-based protection should not be confused with heuristics, a technique that looks for suspicious patterns in executable code. See TechTarget's Apr. 12 article on antivirus trends.)
I arranged an interview with Andreas Marx (photo, right),
co-manager of AV-Test.org. I was one of the first American journalists to write
about this university-based antivirus research group in my Executive Tech
column back on
Feb. 23, 2004. At that time, the
lab's ratings of antivirus
programs were being used by German publications, but its work wasn't yet widely
reported by U.S. magazines.Explaining the value of behavior blocking to stop new malware variations, Marx told me by telephone:
The review omitted complete leak-test results Another omission involves leak tests. Let's say that a Trojan horse somehow manages to install itself on your PC. A leak test determines how many little critters are able to defeat a security suite and slip your data out to a hacker's server. PC World's Rebbapragada, the author of the piece, mentioned the leak-test scores in just a single paragraph near the end of her article:
Table 1, below, shows the percentage of leak tests that each security suite passed, according to raw data sent to me by AV-Test. Most of the products passed only one or two of the 17 tests. Aside from the single paragraph cited above, none of this was mentioned in PC World. ![]() Table 1. Percentage of 17 leak tests passed by security software. Higher numbers are better. Source: AV-Test.org PC World's editors say:
Unfortunately, there's no way a reader could know, based on the information in PC World's article. The review turned off some suite features The third concern about PC World's ranking of security suites is the magazine's practice of turning off some features during testing. This is intended to allow the magazine to use existing tests that are specific to adware, spyware, virus detection, and the like. But does testing one security component while other components are turned off actually reflect the real-world performance of an integrated suite? Vendors are increasingly combining all of their individual security products into a single, integrated package. Representatives of Symantec, the company that won PC World's Best Buy award, explained to me how two separate software components can strengthen each other when brought together into a single product. "The firewall might detect some activity independently," said Kraig Lane, Symantec's group product manager of consumer Internet security products. "Then it can say that the antivirus [component] should quarantine some file." In other words, each component can use the strengths of the others. Providers of security suites say they want real-world testing. McAfee's suite did extremely well in PC World's ranking, receiving almost the same overall scoring as Symantec. (The two suites were rated 83 and 84 points, respectively, out of a possible 100.) Even having received such a high rating from PC World, McAfee's director of product management, Marc Solomon, expressed concerns about testing new products with older routines. "I'd really like to know how they tested this, to see if they turned off the antivirus in order to test the firewall," Solomon said in a telephone interview. PC World's editors tell me:
Moving toward 100% protection, all the time Today's worms and rootkits can be difficult or impossible for Windows users to remove. Once the devious little critters have snuck into a system, they can be devilishly hard to detect and eradicate. For this reason, it's important for security suites to be installed before a PC is set up and exposed to the Internet. Gateway computers, for example, now ship with a 90-day free version of McAfee Internet Security Suite automatically enabled. In my opinion, most such vendors' annual subscription fees to continue the protection are reasonable. The question is, How much protection does the best security suite provide? Users want to know how often a real-world threat can slip through the automatically updated armor of these suites. (Every six days? Every six months? Almost never?) For his part, AV-Test's Marx says he's satisfied with PC World's article. In an e-mail after reading a draft of this story, he noted:
PC World has a reputation for excellence in its technical material. Disclosure: I myself was once a contributing editor, writing a monthly column for the magazine for a couple of years in the 1990s. The publication's quality has steadily improved since then, in my opinion. But mistakes can hurt a publication, even if most of its work is solid. I request that PC World retract its ratings of security suites. This topic is important enough to warrant spending the money to write up a new set of real-world tests. In our conference call, PC World editor-in-chief McCracken told me, "We won't retract that. We feel we made the right decisions." He also said, however, "I think you will see us do behavior-based testing in the next few months." The online version of the security-suite review is posted at PC World's site. For details on Marx's antivirus testing group, visit AV-Test. Readers, I leave it up to you at this point. I welcome your expertise on how security suites should be tested — and whose tests you find to be the most dependable. Many subscribers have asked me whether installing separate programs to handle firewall, virus, spam, and spyware duties wouldn't be superior to installing an integrated security suite. That's certainly true for large enterprises. Corporations with IT staff capable of evaluating these programs will always put together their own layers of protection. Many home users and small businesses, however, don't have this luxury. They need to run one or two products that they can understand. Security vendors — and the test labs that review their products — will inevitably concentrate more and more on integrated suites to meet this demand. My hope is that all the competing suites will improve enough that their detection of malware becomes virtually foolproof. Then these products can compete over which one is easiest to use, has excellent customer support, and is affordably priced. We won't know when that day has come, however, unless the major test labs convince us that their methods reflect real-world protection. To send us more information about security suites, or to send us a tip on any other subject, visit the WindowsSecrets.com contact page. You'll receive a gift certificate for a book, CD, or DVD of your choice if you send us a comment that we print. The Windows Secrets Newsletter is usually published bimonthly, but not in the second half of August in order to give those of us who produce it a break. We'll print the best comments in our next issue, which will appear on Sept. 14. Thanks for your support. Brian Livingston is editor of the Windows Secrets Newsletter and the coauthor of Windows 2000 Secrets, Windows Me Secrets, and eight other books. |
|
THE SECURITY BASELINE ZASS remains in Security Baseline By Brian Livingston My top story, above, says it all. The recent review of security suites that was published in the July 2006 issue of PC World, which I drew attention to in the July 27 newsletter, doesn't appear to be reliable. It won't be considered when determining which suite is top-rated by respected test labs. As a result, the ZoneAlarm Internet Security Suite (ZASS) remains in this issue's Security Baseline as the product that has the most No. 1 ratings from major computer reviewers. This includes an exhaustive new ranking of PC security software in the September 2006 issue of Consumer Reports Magazine, which is now on newsstands in the U.S. The Security Baseline as it stands Based on the latest published tests, the best four products to give your PC comprehensive protection against hackers are (1) a Linksys hardware firewall, (2) ZoneAlarm Security Suite, (3) Webroot Spy Sweeper for antispyware protection, and (4) Shavlik NetChk Protect for update management. See details below.
|
|
HERE'S A TIP You're reading the free version of the Windows Secrets Newsletter Subscribers to the paid version receive additional information in each issue. Some of the extras this week are:
Brian Livingston / Hot Tips. The best information available on making Windows
work the way you want it to:• MS software leads to new headaches • Don't install the IE7 beta on crucial PCs • Norton, OneCare, et. al., force auto-updates on • Disable dumprep.exe to stop 100% CPU usage
Woody Leonhard / Woody's Windows. You get a pointed look at Microsoft's
operating system through our guru's flat screen:• The best ways to surf anonymously • The sad state of your privacy • What about you can be tracked • Selecting the best "anonymizing" service
Chris Mosby / Over the Horizon.
The steps you need to take NOW to protect yourself, because patches aren't yet available for some known threats:• PowerPoint is still a big security risk • PowerPoint still has big, exploitable flaw • Flaw in powerpnt.exe causes unknown impact • Closing PowerPoint files corrupts memory
Susan Bradley / Patch Watch. We tell you which official patches have problems and,
more importantly, how you can work around them:• Install MS06-040 to avoid the Next Big One • Top priority: install the 921883 Patch • How to deal with August's "Dirty Dozen" • The priorities for patching Apple OS
Ryan Russell / Perimeter Scan.
How you can use free or commercial software to automate patching and upgrading,
whether you're responsible for 5 PCs or 50,000:• The report from Black Hat and Defcon • You can get hacked via wireless drivers • More evidence of virtual machine rootkits • More cross-site scripting worms? Paid subscribers can access all old and new paid newsletter content Make a contribution to support our research into Windows and you'll immediately be able to read and search through scores of valuable articles. In addition, paid subscribers are entitled to download valuable content that we license for you at least once every calendar quarter. To upgrade, simply make a contribution of any amount you choose If you do this by Sept. 13, 2006, you'll instantly be sent the full, paid version of today's newsletter. To upgrade to the paid version of Windows Secrets, please visit WindowsSecrets.com/upgrade. Thanks in advance. |
|
ELECTRONIC BOOKSHELF — new e-books from the editors
Spam-Proof Your E-Mail Address, 2nd Ed.This 32-page e-book by Brian Livingston gives you step-by-step instructions that can eliminate 97% of the spam that would otherwise clog your e-mail account. You could call it "Livingston's Spam Secrets." The PDF-format e-book is the result of months of experiments and tests we conducted. We now receive little or no spam to the addresses we used as guinea pigs. These tests show that you can actually reduce your volume of spam to practically nothing, not just battle an unstoppable and ever-growing flood. The methods we describe work with Windows, Apple, and Linux and don't require any filters or block lists — but you can use those in addition to the book's techniques, if you wish. More info |
|
WACKY WEB WEEK
|
|
ABOUT YOUR SUBSCRIPTION The Windows Secrets Newsletter is published twice a month on alternating Thursdays. Issues appear 2 days and 16 days after Microsoft Patch Tuesday (the 2nd Tuesday of each month). Only the first issue of the month is published in August and December to allow vacation breaks. A short "news update" is sometimes published between regular newsletters. Publisher: WindowsSecrets.com LLC, 300 Queen Anne Ave. N. #456, Seattle, WA 98109 USA. Vendors, please send no unsolicited packages to this address (readers' letters are fine). Editor: Brian Livingston. Contributing Editors: Susan Bradley, Woody Leonhard, Chris Mosby, Ryan Russell. Research Director: Vickie Stevens. Program Director: Brent Scheffler. Trademarks: Windows is a registered trademark of Microsoft Corporation. The Windows Secrets series of books is published by Wiley Publishing Inc. The Windows Secrets Newsletter, WindowsSecrets.com, WinFind, Windows Gizmos, Index of Reviews, Security Baseline, Briefing Session, Windows Patch Watch, Perimeter Scan, Update Management, and Wacky Web Week are trademarks and service marks of WindowsSecrets.com LLC. All other marks are the trademarks or service marks of their respective owners. How to subscribe: Anyone may subscribe to this newsletter by visiting our free signup page. HOW TO UNSUBSCRIBE: To unsubscribe from the Windows Secrets Newsletter,
|