As many of you know, tummy.com has been using greylisting extensively for about 3 months on our mail server as well as on the mail server running many of the community mailing lists we run. We run Spam Assassin and use other techniques to cut out a lot of the crap, but were still getting too many messages. The addition of greylisting and SPF to the rest of our techniques really helped. Before we were getting hundreds of messages a day, now it's down to a couple per day. Incredibly effective.
At ISPCon, I've spent some time speaking to Michael of Linux Magic about their new product, Blacklist Mastering System (BMS). He hadn't heard of greylisting, but when I described it and the success we were having with it, he was skeptical about it's performance. While the BMS does solve one of the big problems of blacklisting – the network latency, I was skeptical of the performance of just blacklisting. So I ran some tests.
I pulled data from our logs for the last 3 days of e-mail greylisting. We had 33,019 messages that were blocked by greylisting during that time. Of that 31,449 were on at least one of the 18 blacklists that BMS uses. Or, in other words, 1,570 deliveries would not have been blocked by just the blacklisting. In other, other words, false negatives.
Turning it around, as far as I can tell, during this time, there were 0 false positives due to greylisting during this time. In fact, none of our incoming, legitimate e-mail so far this week has been delayed by the greylist. However, the BMS would have blocked 55 messages from 5 legitimate senders so far this week. That's pretty good for a false positive rate, considering you could whitelist these senders.
The real key is a combination of techniques. If we were just using blacklisting, those 55 messages would have been lost in a quarantine of over 31,000 messages, making it impossible to find them to whitelist. Even with greylisting, it would have been searching for 55 messages out of around 400 (better, but still a lot of junk to sift through).
In our system, with the combination of techniques we are using, it requires almost no regular maintenance. Periodically, we will add something to the whitelist. If spam makes it through to my mail box, I'm kind of aggressive about adding it to the rules on our mail server to block it in the future. That takes maybe 15 minutes a week.
The real advantage of the BMS is it's performance. Based on my numbers, BMS lookups are around 2 to 3 times faster than my greylist database lookups. On a heavily loaded mail server, that could be very important. My greylist implementation operates at around 160 lookups per second, where the BMS I clocked (on the same machine) at 440 per second. Luckily, our company mail server isn't handling anything close to that rate.
We've found greylisting to be a very low-maintenance and highly accurate way of dealing with spam. The BMS could be an effective part of a solution, but I'm still worried about the false positives that seem to happen with blacklists.comments powered by Disqus