My catch-rate is better than yours! nah, nah!

Have you seen how some email security vendors seem to focus almost exclusively on their catch-rate and false-positive rates?

‘Junk Giant announces reaching 99.5680% catch-rate.’

‘Piranha Systems reaches 0.3507% false-positive at its customer base.’

Okay. So everyone in the industry does 99%+ catch-rate and less than 0.5% false positives, we do it, they do it. Woopdeedooo, let’s do the happy dance!

And that’s the only thing many vendors will push for.  Features? Blah.  Easy to use? Blah.  Support? Double blah (many outsource it offshore anyway).

Let’s see! using their own famous ‘Spam ROI Calculators,’ let’s do a small comparison with the following assumptions: a typical employee receives about 85 emails per day⁽¹⁾ ; 85% of which is actually spam. It takes 5 seconds to read and delete a spam message that inadvertently got through, and about 30 seconds to review the quarantine report and release a false positive (legit email that was quarantined by mistake). Remember: we are comparing two email security solutions here, not a ‘with vs. without’ scenario.

Okay, so these tables show the difference between a 99.1% catch-rate and a 99.5% catch-rate email security solution (you can do the math if you want to compare other performance figures): an employee will typically save 1.44 seconds a day with the 99.5680% catch rate system.  Wow! Hey, come to think of it, it takes more time to say ‘Hi’ to my cubicle neighbor.  Let’s apply this major productivity loss to an employee with a $40k salary, a 40-hour work week and a 2-week annual vacation, and it boils down to a mind blowing $2.00 yearly productivity loss difference per employee⁽²⁾ ! Oh! my! God!  Let’s declare bankruptcy right away guys.

You figured that one out, so now the pushy sales guy goes on with the false positive rate, and how you could lose business and all.  First, browsing the daily quarantine report ensures you don’t lose anything at all. Second, even with an exaggerated time to release a FP (30 seconds), the table above shows that the difference between a 0.50% FP and a 0.35% FP solution makes even less difference than the catch-rate: $0.79 yearly. Yes: that’s 79 cents!

Oh, and by the way, all these 4-decimal rates are measured on custom mail feeds.  They don’t even come close to your own mail feed. Your spam feed will vary a lot according to your organization’s geographical location, industry, age, and internet activity, to name a few. So, statistically, the error margin between their test/measure feed and your real-life mail feed is probably several orders of magnitude higher than the gain achieved on a 99.5680% vs. 99.1687% catch-rate.  So what’s the point with these 4-decimal catch and FP rates? Don’t vendors have anything else to say?

I do. Let’s talk about support, for example. First things first: a well designed, reliable and easy to install product will require little to no support at all.  As we know, this is unfortunately not always the case, we have to check into support plans and options.  A good service and support team will leave you happy with the issue resolved within anywhere from a few minutes to a couple of hours.  Crappy support will leave you with an unacceptable solution after hours – if not days – of frustration, talking to a level-1 filtering and offshore support staff who asks you to reboot your system 4 times and apply 38 unrelated security patches and service packs.  You know it won’t solve your issue but the support staff won’t talk to you until you perform these operations.  Just for the fun of it, let’s assume the good support took 1 hour from your IT admin to solve the issue and the bad support took 4 hours (and we all know it can be much, much more).  Apply that to a $70k yearly salary for an IT guy and you get a $105 difference for a single, relatively minor incident. Do I need to mention the cost of an unresolved issue blocking all company email productivity for an entire day? And they say the 4-decimal catch-rate is the most important thing to look at? Give me a break.

I think it’s more important to know if the solution integrates with Exchange and Active Directory. Can it do automatic user discovery and verification? It does take time to manually add new employees or remove employees who left the company.  How does the software handle the quarantine? What about user and domain delegation? Does it let users or domain administrators adjust anti-spam and virus scan aggressiveness? I can’t begin to imagine the time an IT admin loses by having to change rules, blacklist addresses or release mails for 100 employees (not to mention an even larger organization, or worse – an ISP). Does the email security solution run on Linux or on Windows? These two systems require totally different skill-sets for administration and maintenance. Does it support Virtual Machines, and is their support staff trained for VM?

What is your perspective on these new email security ‘spec wars?’ What criteria are most important to you?

• (1) Ferris Research, Industry Statistics
• (2) Before you ask, using a more polluted environment with 90% as the percentage of spam brings the figure to 2.12$ yearly.