Sunday December 04, 2005 at 17:19
Subject: Stopping spam with sender address checks.
Keywords:
Spam, Technical
Posted by: Sean Reifschneider
Related entries:What we are doing for anti-spam. by Sean Reifschneider, Sunday December 03, 2006 at 14:33
I installed some capacity analysis software, which I'll probably write
about shortly, on our mail server yesterday. Part of that was that I found
that there were over 30 messages in the mail queue. All of these were
bounces related to spam, which couldn't be delivered because of the sender
address wasn't valid. So I decided to do a little survey to see how much
of our e-mail comes from invalid senders...
I wrote a quick Python program that, given an e-mail address, would
contact the domain's mail server(s) and then ask if it would accept e-mail
for that address. I pulled sender addresses from our mail logs over the
last week to feed the system and then fired it off to check the addresses.
I started it off running and went to bed. When I got up, it had only
made it 15% of the way through the 26,093 addresses. So, I converted from
sequential-processing to running 50 checks in parallel. While the checks
used up 40% of the CPU time on a fairly speedy box, it was able to complete
97% of them within an hour. The remaining few percent took 2 more hours.
Here's how the results came out:
(Post Reply)
-
Total sender addresses checked: 26,093
Mail servers which accepted the sender address with a 2xx or 4xx
response: 10,789
Sender address which had servers which accepted an SMTP connection:
22,562
Obvious addresses that were rejected with a 5xx response but I
wanted to get e-mail from anyway: 16
(Post Reply)
| Comment |
Alan McNeil Subject: monitoring the mail queue |
I have to make sure the boss' love notes get thru within 5 minutes SO
I monitor my mailq remotely with mrtg. On the mail server I have this script:
Which returns number of items in the mailq on line 1 and number of recipients on line 2. Next add a line in inetd.conf:
(I define smtp-qstat and some other management ports in /etc/services on all servers) Then on the mrtg machine, I have this qstat.pl script:
The script takes the qstat from the outbound and inbound mail servers and formats the number the way mrtg likes. Then add this in your /etc/mrtg.conf
Presto - instant mailq monitoring! One day I saw the queue heading for 2000, I tracked it down to one of the tech who had forwarded all his mail to yahoo, where he was forwarding it back.
#!/bin/sh
# smtp-qstats: show the number of things left in the queue
#
if [ -x "/usr/bin/mailq" ]
then
exec mailq -OMaxQueueRunSize=0 | tail -1 | cut -d ' ' -f 3
exec mailq | grep -c "[[:space:]]\{6\}<"
fi
Which returns number of items in the mailq on line 1 and number of recipients on line 2. Next add a line in inetd.conf:
smtp-qstat stream tcp nowait root /root/scripts/smtp-qstat
(I define smtp-qstat and some other management ports in /etc/services on all servers) Then on the mrtg machine, I have this qstat.pl script:
#!/usr/bin/perl
# this script relies on port 7777 of mailserver returning qlength
use strict;
use Socket;
my (@remotes, $port);
@remotes = ("in-server","out-server");
$port = "7777"; #smtp-qstats
# Straight out of the blue Camel - pp. 349
my ($iaddr, $paddr, $proto, $line);
my @num;
@num = (0,0);
my $index = 0;
my $mx;
foreach $mx (@remotes) {
$iaddr = inet_aton($mx) or die "no host: $mx";
$paddr = sockaddr_in($port, $iaddr);
$proto = getprotobyname ('tcp');
socket (SOCK, PF_INET, SOCK_STREAM, $proto) or die "socket: $!";
connect (SOCK, $paddr) or die "connect: $!";
# munge the data
while () {
if ($_ =~ /(\d+)$/) {
$num[$index] = $1;
}
}
close (SOCK) or die "close: $!";
$index++;
}
# print the data for suitable for mrtg
print "$num[0]\n$num[1]\n1\n$remotes[0]\n";
The script takes the qstat from the outbound and inbound mail servers and formats the number the way mrtg likes. Then add this in your /etc/mrtg.conf
Title[qmail]: Mail Options[qmail]: gauge,growright,noinfo YLegend[qmail]: emails LegendI[qmail]: inbound LegendO[qmail]: outbound Legend1[qmail]: inbound queue entries Legend2[qmail]: outbound queue entries Legend3[qmail]: max inbound queue 5m Legend4[qmail]: max outbound queue 5m Target[qmail]: `/root/qstats.pl` SetEnv[qmail]: MRTG_INT_IP="" MRTG_INT_DESCR="qmail" Directory[qmail]: qmail MaxBytes[qmail]: 50000 XSize[qmail]:340 YSize[qmail]:120 PageTop[qmail]: <H1>Mail Queues</H1> <TABLE> <TR^gt;<TD>System:</TD> <TD>Queues</TD></TR> <TR><TD>Description:</TD><TD>Mail Queue size</TD></TR> </TABLE>
Presto - instant mailq monitoring! One day I saw the queue heading for 2000, I tracked it down to one of the tech who had forwarded all his mail to yahoo, where he was forwarding it back.
| Comment |
Author:
Sean Reifschneider Subject: Better to be more active... |
Graphs are nice for historic and trending information, and they're pretty to look at, but if you want to react quickly and consistently to mail queue problems there's nothing like alerting. From a simple script that checks the queue and send an e-mail whenever it exceeds a certain threshold, to alerting only when it's been over the threshold for a particular length of time or based on 95th percentiles... Nagios is a pretty good tool for doing more complicated alerting, we use it for some pretty complicated monitoring and alerting.
So, instead of it only getting fixed when you happen to be around a browser and checking the graphs, you can get a message on your cell phone any time of the day or night, if you so desire. It allows you to be much more proactive to your boss' love-letter log-jam.
Sean
| Comment |
Alan McNeil Subject: Monitoring is my friend |
You're right, monitoring is my friend. Nagios in particular. I access the same mailq script remotely from the nagios monitoring machine. Most of the mrtg stuff is eye candy that gives the tech support people some idea of whats going on without calling network ops.
Long ago I learn to rate my todo list items with "effort" and "looks" factors. Do one high-looks item for the boss, one hard one to make real progress.
| Comment |
Nataraj Subject: Postfix address verification |
Hi Sean,
Have you looked at the postfix address verification support? It's described in http://www.postfix.org/ADDRESS_VERIFICATION_README.html
| Comment |
Author:
Sean Reifschneider Subject: Yes, we use address verification. |
We do use address verification, which is part of why our mail queue was so empty. If we didn't, it would probably be thousands of entries. Many spammers and viruses are using legitimate sender addresses, which pass validation just fine.
Sean