Beta 1.2b5 is available http://m0n0.ch/wall . However right now all incoming traffic are broken. 1.2b3 freezes a lot, so if you want to try, try 1.2b2. Try searching on Google for download links.

Oh did I mention it's free?

Now the downside: ipfw is only a port based traffic shaper. It does not work for connections which use random port numbers. Even P2P software uses a lot of random port numbers. For example if a partitcular connection is "remote" rather than "local", then even if you've correctly setup port forwarding, port based traffic shapers will still miss this P2P connection.

To address this problem, there are layer 7 filtering solutions like l7-filter on Linux. Too bad that I don't know much about Linux and command line stuff, or it would be a great package. And I've not found any pre-built package that comes with a easy to use GUI interface.

ipfw under FreeBSD is not at all port-based. You set up "pipes", which implement the shaping, and then you forward packets to the appropriate pipe using any sort of rules you'd like including ports, IP addresses (or parts thereof), protocols, etc... So, you could limit TCP differently from UDP, or certain hosts differently from others.

As far as l7filter goes, as far as I know it's just another pretty front end on top of "tc", just like tcng, smoothwall, etc. This means that it inherits all sorts of the problems that tc has including inability to shape bittorrent traffic, only 16 "bands" in which you can shape, etc...

Thanks for the reply.

Sean

Humm... I use a pre-built m0n0wall image, which by default comes with a web GUI that allows configuring things like pipe, weight of pipes, which port(s) to use which pipe, etc.

Yeah ipfw is pretty good, for a port based traffic shaper.

You said it's also pipe-based as well. That's true. However this pipe/port based design has a big flaw: you need to know the port number of the problematic traffics.

Take your favourite ipfw for example, and Bit Torrent which didn't work for your tc setup (I haven't setup ipfw from command lines, but things in m0n0wall should apply):

1. Pipe setup: either setup a upload/download pipe for BT, or one upload pipe + one download pipe.

2. Then assign the weight on pipe(s).

3. Then assign ports to pipes. Usually

Upload pipe:
port 6881-6899 source   ANY port target
ANY port       source   6881-6899 target

Same goes for download pipe.

I implement this, and even with all 6 PCs on the LAN using BT in full, they can still do telnet/ssh/web surfing without much problem. Ping to Yahoo hoovers between 30ms to 150ms, with average 70ms. All this happens on a Pentium 233 PC with a 3.0M ADSL line.

Now why I said ipfw isn't perfect? It relies on port number to filter/shape traffic.

It's true that most services like BT or FTP will run on well-known ports. However "most" isn't "all".

Very often I see a client on LAN hooking up to a BT server. That BT server chose a non-standard port, e.g. port 29420. Client will be using arbitrary port by default, i.e. port 1024-5000. So this connection is completely missed by ipfw.

Yeah the pipe feature will somewhat limit damage can be done by that "missed" traffic, especially if you implement pipes to restrict bandwidth per PC. Unfortunately on my setup, it doesn't help a lot with reducing lagging in online games. From time to time I'll see ping to game servers hoover between 30ms to 150ms if BT is on.

So if you just want a smooth web surfing, m0n0wall or ipfw works really well for priceless. If you want a 100% smooth gaming/VoIP experience with all those P2P thing running in background, you need to inspect connection in layer 7 / application layer.

Sure, if you want to shape by application it can become somewhat difficult, depending on the circumstances. For that sort of application, you'd ideally want to watch the traffic and classify the flows based on the content, instead of just port numbers. As you say, "L7 inspection". I haven't seen that ipfw has much of this functionality.

However, the case where I was having BitTorrent problems, I was the server, I knew very well what port would be involved. Worse, I had allocated a specific IP address for the shaped traffic, and wanted to limit traffic on that IP. In another case, I was trying to limit a download I was doing, and ran into similar problems using the tc shapers. For FTP and HTTP, they work as expected...

For lag in online gaming, you want to use prioritization or QoS. Bandwidth and lag are two very different things. Depending on the queueing sizes on your shaper box, if you drop a packet in the queue, it may have to wait behind some bulk traffic, which will delay it. You can enable QoS on the gaming traffic, which puts that traffic into a different track within the kernel, and the kernel will send packets with higher priority before the bulk traffic -- effectively causing your gaming traffic to come to the head of the line and get sent right away. The down side to that is that in the case of downloads, your up-stream ISP would have to have it configured properly to take maximum advantage of it.

Sean

(again I could be wrong, but)

"However, the case where I was having BitTorrent problems, I was the server, I knew very well what port would be involved."

Right on. I was hoping you would say that.

Don't know about your particular BT client, but all BT client I've used, like BitComet, Azureus, etc... even if you setup a specific server port, the client will still use abitrary ports sometimes. Or you've a way to make all BT connection "Local"?

Experiment
++++++++++
If you don't trust my word, just start your BT client and try seeding a torrent. If you're behind NAT, Make sure port forwarding for your server port works (you can verify by using http://btfaq.com/natcheck.pl ) .

After having your BT client seeding a while, do a netstat . You will see both server port and arbitrary ports are being used for BT. I did that, with a PC running only BT for 30 minutes (no web browsing, IM, etc etc for that 30 minutes), and see server & arbitrary ports are used.

So in the end we won't know ALL ports that would be involved.
========================================================================
Unfortunately many if not most ISP doesn't really support QoS bits. Just imagine every user tune their software to use "Urgent" QoS for normal web surfing. That alone makes end-user QoS useless.
And true about the QoS with gaming. m0n0wall somewhat does what you suggested, but due to the arbitrary port problem I mentioned, a lot of Bit Torrent traffic is getting through the normal pipe instead of low priority pipe.

Oh and thanks for replying. I can tell you're very experienced in Linux/networking/other stuff. Hopefully I will learn Linux and can chat with you more about it. (don't really want to leave an email here, afraid of spambots search for email address on websites)

And thanks for the blog on Postfix. Looks like I better not suggesting switching to Postfix in my workplace for now.

BitTorrent is not unique in the use of many different ports on the client side. The server side runs on whatever port you specify when setting up the tracker, and runs on a single well-known port per tracker. In my case, I was running the server, so that's what I really cared about -- I knew the 4 ports that were being used to serve the 4 different ISOs I was publishing.

Again, as I said, this was on a dedicated IP address, so I really didn't care about ports at all on my end, but even if I did, I knew exactly what 4 were of interest.

The BitTorrent client uses two classes of ports. It uses a single port for remote trackers to make incoming connections to, which is by default between 6881 and 6889. It also uses ports for the connections it makes to other trackers. These ports are allocated from a pool and each new connection uses a different one.

BitTorrent is not at all unique in this. Almost every protocol used on the Internet today will use a unique port per connection on the local side: HTTP, SMTP, FTP, SSH. In fact, it's probably easier to list the ones that don't do this. Off the top of my head, I can only come up with BOOTP/DHCP, and sometimes NTP, which regularly use Well Known Service ports on both ends.

So, if a shaping system falls down just because the port on one end of the connection randomizes, it's not going to be of very much use at all.

As far as QoS goes, you don't need QoS to be honored at every hop for it to be useful. Simply having QoS on your shaping box can help a lot, because it's a choke point. Say you have 1mbps symmetric service (to make the math easy). If, in a given second your game is pushing out 5KB and your friend's machine is pushing out 100KB worth of bulk data, the ordering of that data is very important.

Your 1mbps line takes 1 second to send 125KB/sec of data. So, on average it could take around 500ms for your 5KB worth of game information to actually make it up to your ISP's network, if it's interleaved within the 100KB worth of bulk data. With QoS, the game packets will be picked out from the queue to be sent first, then the bulk data will be handled after that.

This is most important at your the choke point at your shaper because once it's out on the faster networks of your provider and above, they're on a much faster network that you can't really control.

Not to overuse the "superhighway" metaphore, which we'd all hoped was dead long ago, but imagine it's like the light-controlled on-ramps to a freeway. Your 1mbps connection is like the controlled on-ramp which only lets on one car every 15 seconds or so. The queue of cars is like the packets backing up on your network ready to send to your ISP. With QoS, it's like the carpool lane at that controlled on-ramp, allowing some packets to go immediately to the head of the line.

Unless there's congestion, once the cars get on the freeway, they all are traveling at a fairly uniform high speed, and your higher priority QoS packets made it on ahead of the bulk, which will help a bit, but as long as there's no congestion on the links upstream, they're going to continue flowing fairly smoothly.

So, even if it's only honored at your choke point, QoS can help. The problem I mentioned earlier is that you don't control the receiving side of that choke point, so packets coming back in will probably be handled on a first come, first served basis.

True, but what if the port on both ends is randomized? When making an outgoing connection none of the clients I know bind the connection to a specific port. Nearly every client when initating a connection to another ip/port uses a randomized local port. So as long as the other clients port could be anything and the local port is random, you can't really match it based on port.

"As far as l7filter goes, as far as I know it's just another pretty front end on top of "tc", just like tcng, smoothwall, etc. This means that it inherits all sorts of the problems that tc has including inability to shape bittorrent traffic, only 16 "bands" in which you can shape, etc..."

You're horribly mistaken on 2 accounts. l7filter is not a pretty front end for tc, in fact it has nothing to do with tc. Any use of it will be done through iptables, and if you want to use it along with packet shaping with tc, you'll just be setting marks on the packets in an iptables chain.

Where did you come up with the "only 16 "bands" in which you can shape", are you under the mistaken beliefe you can only filter based on the TOS bits? There's been cases where people have split bandwidth between thousands of pc's, certainly more than 16.

Using l7filter one would mark all p2p traffic in the iptables chains, then they could seperate out traffic with that mark in a htb class. Regardless of the ports used, the traffic could still be identified and traffic controlled.

I spent some time with the Interphase guys at ISPCon, and they were saying that there shouldn't be any problem with tc handling up to 64 thousand shapers. They have a tool for their routers which makes it much easier to set up tc shapers. It just creates tc rules, but does so in a more sane way. They also seem to be using "iptables" to classify the traffic into the qdisc shapers, which may be a more sane way of dealing with tc. Use tc only for shapers, iptables for getting traffic to those shapers.

So, it seems like the majority of my problems with tc have been related to the state of the documentation. Don't get me wrong, that's a huge problem, but simply knowing that it's possible may allow me to make some more progress.

As far as port randomization goes, it doesn't really matter to me. My uses are, at this point, entirely IP based. I want to shape traffic based on either a source or destination IP address, no matter how many ports are used on that IP. Using iptables to classify traffic may allow some more rich matching, however, using things like connection tracking and the like.

The guy from Interphase mentioned that one problem a lot of people have with tc is that of default shapers. If some traffic doesn't match a specific cbq, it will fall to the default. That may be why BitTorrent was running unmetered. Using iptables may make it easier to ensure that this traffic is getting to the appropriate qdisc. Again, in my case I'm entirely caring about the IPs, not any port hopping or the like. I should be matching entirely based on the IP address, because my BitTorrent client was set to bind to a particular IP.