Your Linux Data Center Experts

We've run a public NTP server for quite some time, about 4 or 5 years now. However, recently we've been getting increased volumes of calls from people examining firewall logs, and coming to the conclusion that our NTP server is attacking their network. Recently, the average is one call per business day. Don't get me wrong, I think it's great that people are watching their networks closely, there are too many networks that aren't watched at all. However, I just don't scale. I have 8 hours a day to get my work done, 15 to 30 minutes spent dealing with other people's misconfigured firewalls is a huge fraction of that time.

As a firewall administrator, it is your job to do all these things before contacting the administrator of the remote network. If you do these things you can be reasonably sure that it is not your firewall which is misconfigured. Please don't expect the remote network administrator to help you fix your firewall. With all due respect, that's your job.

Investigate

All TCP/IP connections are made up of 5 things: the protocol, an IP address and a port number for both the source and destination. Ports above 32,000 are usually dynamically assigned. If you are seeing a huge collection of different port numbers in the higher range, it's probably because one end of the connection is binding to a new port frequently, and every new connection attempt results in a different port being assigned. This is just the way that TCP/IP works.

Some firewalls and some NTP clients will cause outgoing NTP connections to come from unique, high ports. This can, over time, make it look like the remote NTP server is scanning through the ports on the destination system. How do you tell the difference? Well, a regular port-scan would primarily be hitting lower-range ports, in particular privileged ports, below 1024. Also, a regular port scan would normally hit thousands of ports per minute, where normal NTP activity would normally only hit one port every minute or two.

The big give-away is that if you watch the traffic between your host and the host in question, is the remote host sending packets only after your system sends packets? In other words, is the remote system responding to a request from one of your systems? For example, NTP will typically have the client machine send one packet and the server will then respond by sending about 15 packets, each on 1 second intervals, to your server so that it can accurately determine the time.

Research the Attack

Once you've isolated the ports that are being used, find out what they are used for. In the case of NTP, the remote system will probably be “attacking” you from port 123 (the NTP server port). It may or may not be hitting port 123 on your end – that depends on your firewall and NAT client settings. Is this a legitimate service that you may be requesting or that the remote end may be requesting from you? For example, if it's NTP, check your internal interface on your firewall to see where NTP traffic is going.

In fact, it's probably a good idea to watch the internal interface on your firewall for packets going to or from the “attacking” IP address, just to see if anything on your internal network is talking to that host.

After you've identified the services, give some thought to whether it's plausible that the logged packets are due legitimate traffic getting blocked by a misconfigured firewall.

Analysis

Does the traffic actually look like an attack? Having your firewall log the packets isn't good enough. If you don't have enough experience to be able to tell the difference between an attack and a misconfigured firewall, calling the admin of the remote network should be the last thing you do. Instead, spend some time analyzing the packets and perhaps take no action until you've had more experience doing analysis.

A real attack would often not use “Well Known Service” ports to launch the attack. They often would tend to hit many different ports very quickly if they are doing port-scanning (that's what port-scanning is) and usually they would be touching well known service ports on your end (looking for vulnerable well-known services). This would be typical patterns for a port-scanning attack.

For a Denial of Service attack, it would typically be hundreds or thousands of packets per second. 50,000 packets in a day is fairly low volume, it's less than one packet per second average. This level is unlikely to be a DoS attack, even on a modem connection. It may be part of a probing attack, but those tend to be shorter in duration.

Google is your friend

Search for both the remote host IP address and also for the ports identified during the earlier phases on google. For example, if the remote host is “198.49.126.4” and you're getting packets from port 123 using protocol UDP, you might want to do the following searches:

The first search above will fairly quickly show you that port 123/udp is related to NTP. The (current) topmost link on the second query mentions firewall misconfiguration and shows that other people are using 198.49.126.4 as an NTP server. Of the 5 results from that search, 3 of them are related to people using that box as an NTP server, so it's pretty obvious that it's an NTP server.

Don't Jump to Conclusions

What is the most obvious answer? For example, if the traffic is coming from the NTP port on a remote server, you need to assume that it's in response to NTP requests you are making. It's not legitimate to assume, at that point, that it's part of an attack. Further investigation is definitely warranted before making that call.

When Making the Call

Remember that their time is at least as valuable as your time. This can take many forms, but definitely should include your having done adequate researching to be at least 90% sure that you are actually reporting a legitimate attack and not a misconfigured firewall on your end. Also, you should never ask the remote network administrator to help you fix your firewall, that's your job not theirs. They have no idea what sort of firewall you have, it's nearly impossible for them provide any substantial suggestions without spending significant time reviewing your firewall rules, finding out what kind of firewall you are using, and probably reading documentation on your firewall.

Have all the information at your finger-tips. Last week I received a call saying that our IP was attacking their system, and when I asked what ports were involved they had to go back and pull another report from the firewall. While I was waiting for this to happen I asked them for their IP address and set up a network trace looking for packets to their system. After a couple of minutes, I saw that once a minute their server was sending an NTP request to our server, and our server was then replying once per second for the next 15 seconds. This is work they really should have done, and information they should have had in front of them at when they called.

Don't Be Afraid to Ask for Help

If you can't tell for certain if an set of packets is really an attack, you need more experience. There are many places you can go to gain this experience, and probably should do so before you start reporting the attacks to the remote network admins. Some possibilities for this are: Books, Seminars or other Training, Consulting, and also Users Groups. Users Groups are very good free public support arenas.

However, it's very impolite to ask the administrator you are calling for help fixing your firewall. The Maytag man may be lonely, but I've never seen a network administrator that is looking for more work. Firewalls vary enough that it's nearly impossible for them to tell you how to resolve the problem. They have just as much right to spend some extra time with their friends and family as you do. Probably more, because, you know, their firewall isn't the one that's misconfigured. ;-)

In Conclusion

As a network administrator, I want to hear about problems our network is causing. However, as a network administrator, any time I spend on chasing other peoples misconfigured servers, firewalls, or services is time that I can't spend administering my own. This is particularly a bad deal for people running public services, because as more people use those services, more and more of their time may be taken up by responding to bogus inquiries. This is why it's extremely important to be sure that it's a legitimate attack before throwing it over the fence to the administrator of a remote network. You can't tell if they are offering a public service, so you really need to be very sure before involving the remote network admin.

comments powered by Disqus

Join our other satisfied clients. Contact us today.