Your Linux Data Center Experts

As I mentioned in my previous journal entry, traffic shaping under Linux is pretty sad. I finished the entry off saying that the FreeBSD shaping using “ipfw” looked like it might allow what we needed, but I hadn't gone through any testing of it. I have since set up a FreeBSD system in our test environment and tried to emulate something on par with what the client is looking for.

I set up several source machines on the outside of the FreeBSD system, and 4 more client machines on the inside network. I had set up shaping for 4096 addresses on the inside, each getting their own 10KB/sec shaping. I had the 4 clients then set up hundreds of IP aliases, and each download 10MB worth of data from many of these IP addresses. I also set up my laptop on one of these addresses and ran some test downloads while this was going on.

In the end I built no fewer than 5 kernels, but I was able to get the performance of the system up to the point where it was handling this load with no problem. It required some serious tuning, but the end result was shaping for over 400 simulated clients passing traffic equivalent to an ATM DS-3 line, and shaping it reasonably well. I was also able to override the default shaping on my laptop and adjust it dynamically with no problems.

I was extremely impressed with the shaping under FreeBSD. I made an error in the first set of rules I created, but it was obvious that there was a problem just by looking at the resulting tables and equally obvious to fix. After having used tc on Linux, this was a welcome change. Better, the rules involved in setting up individual shaping for 4096 hosts involved 4 commands. This set up the “default” shaping. Overriding it for a particular host was similarly easy and obvious.

One neat thing about ipfw under FreeBSD is that there are multiple rule-sets, and you can atomically switch between these rule-sets. So, you can have one set of rules, set up another set of rules which do not impact the traffic flow while setting them up, then you can atomically switch to the new set of rules with no interruption. Usually, changes to tc rules involve traffic anomalies for several seconds during and after rule changes.

ipfw doesn't seem to support anything as advanced as HTB under Linux, but it does provide some mechanisms for traffic sharing which could come close. It's also kind of weird that FreeBSD pushes packets back to user-space when doing NAT. However, that seems to work as long as it's appropriately configured. We once had a client running FreeBSD with the NAT daemon improperly configured and it was hammering on our network. However, it did seem to work fine, and (better) it could do shaping based on the internal private IP, even with NATing turned on. Under OpenBSD with “pf”, this wasn't possible because it could only shape on outbound interfaces, and by that time the source IP was gone.

Now, if we could just get ipfw and friends under Linux. :-)

However, as rise was mentioning on #hackingsociety today, the “trickle” program also allows some pretty neat traffic shaping. It works in user-space by having dynamic loading use it's own socket code. You can implement shaping based on process ID, and even (when running the trickled) implement cooperative shaping among multiple processes. It's a pretty neat trick, but only available on the host that's at one end-point of the traffic. In this case, we need a router that sits between the customer equipment and the Internet.

Trickle should work great for limiting my BitTorrent traffic, though. It offers a max upload rate, but not download limits.

comments powered by Disqus

Join our other satisfied clients. Contact us today.