Why would an anti-spam filter with tons of awards guaranteeing 99.9%+ catch rate and ultra-low false positive rates perform poorly in the real world?
Why would an email security software from another vendor do better despite having a lower score in the same certification/award program?
Here’s the scenario:
- Anti-Spam Product A, with a SuperAntispam Award demonstrates 99.80% catch-rate and 0.13% false positive rate
- Anti-Spam Product B has the same SuperAntispam Award and demonstrates 98.14% catch rate and 0.04% false positives.
Product B has converted ex-customers of Product A who had complained about poor catch rates. Does that make any sense? How can that be?
Spam streams are very different from one geolocation to another (Russia vs. North America), from one IP subnet to another (ARIN vs RIPE), from one business activity to another (Pharma Company vs. Software Vendor). Because an Anti-Spam performs well at one point does not mean it will work well for everyone on the planet.
Another criteria which influences the catch-rate is the language. Some anti-spam engines work best in English, and poorly or not at all in French, Russian or Chinese. What language is used most in your organisation? Is it unilingual or multilingual?
Time of the year is another consideration: spam waves are frequent during specific periods of the year (Christmas, New Year’s, Valentine’s, Black Friday, Superbowl, World Cup, you name it). All these events differ depending on the region and location. If a vendor passed the certification during a ‘low period’, does it mean it will fare as well during a ‘busy period’?
Let’s not forget configuration. An improperly configured anti-spam gateway can lead to tons of holes through which spam can get through: no DNSBL or Sender Reputation System, whitelists that open the door to spoofing, invalid DKIM or SPF setup. A properly designed anti-spam gateway can be effective the minute it is set up, while others require professional and lengthy installations to get up-and-running.
All these variables make it extremely hard for anti-spam certification companies to produce absolute numbers for their ratings.
Anti-Spam award and certification scores sure are an important measure of performance and reliability, but experience has shown that this should not be considered without some kind of additional real world test, for your organization, in your industry, using your mail and IP infrastructure, transporting mail to and from your employees, whose content is written in your languages.
Leave a Comment