Also see my related articles on networking
set up for PyCon 2012, 2010,
This year the networking at PyCon went fairly smoothly, though not
without some issues. Many things went very well: users of 802.11a (5.2GHz)
basically just worked, the upstream connection (from a company we hadn't
used before) had absolutely no problem, and we had wired ports for people
to use if they couldn't connect otherwise. We also tested out some 802.11n
gear and had a surprising number of people using 802.11n.
The primary issue was, again, 2.4GHz. I'm starting to think that 2.4GHz
is just not going to cut it. Though I made some connections at PyCon that
are interested in further trying to "crack this nut".
The first few hours of the tutorials and the main conference we just
weren't ready. The problem with the tutorials was that the router went to
the warehouse (and my backup plan was suffering from terrible packet loss),
and the main conference hall didn't have power until moments before the
conference started, so I couldn't do the final test of the networking.
After these hiccups, things went very well.
I tried a few new things this year including putting out quite a few
wired network ports, and using 802.11n gear. Plus, 802.11b was not
available on our new APs. The new APs worked well in 5.2 GHz, but 2.4 was
really not happy.
Crimping your own RJ45 should be avoided.
Terrestrial wireless continues to work great.
PyCon volunteers are awesome!
Don't forget to load all the conntrack modules...
2.4GHz should be avoided.
We really need our router somewhere we can access it.
I should have named the extra ESSIDs more clearly.
We really needed a few minutes spoken intro to the network.
We needed more APs in the main hall.
Read on for more details.
We spiked up to 55mbps, but that was only very briefly. We could have
done pretty well with 35mbps of bandwidth. However, with the terrestrial
wireless (as we've used previous years), the cost of 100mbps was really
quite reasonable. Compared to what the hotel network provider wanted to
charge us, anyway. They wanted $25k before they would even provide a
quote, we ended up paying $5k for the terrestrial wireless (about half what
we were paying in Chicago).
Our uplink provider was One
Ring Networks and they were just great. They really had their stuff
together and they just worked. Absolutely no problems, everything went
very smoothly from the contracts to the networking. They were recommended
via a question I posted on LinkedIn, and I can highly recommend them
for anyone needing terrestrial backhaul in Atlanta.
The trend continued from last year with more 5.2GHz users, fewer
2.4GHz users and less of those being 802.11b.
In addition to the APs that we used the last several years, I also got
some 802.11n gear to try out. I was primarily interested in them for the
MIMO antenna stuff, hoping that it would improve performance.
The 2.4GHz band continued to give us fits. During the main conference
in the main rooms, it was basically unusable with heavy packet loss and
multi-second latency, despite having good signal. My netbook performed
better than last year, but it was still basically unusable. I'm just
coming to the conclusion that 2.4GHz is just not going to work with 150
users, despite being spread across a dozen APs and trying to keep the power
down. Only having 3 non-overlapping channels is just killing it, and it's
also competing with the hotel wireless gear.
5.2GHz performed very well though. There were some brief spikes in
latency, presumably as people were doing large downloads, but mostly
latency was in the 45ms range with less than 1% packet loss.
In short, if you want wireless to work at a conference: use 5.2GHz.
The new APs we were using had a hard limit of 63 wireless clients per
radio. We hit that very hard during the first few hours of the main part
of the conference because half the APs weren't working. This was caused by
us doing our own RJ45 crimping, which I really wanted to avoid.
This is the first year we've tried 802.11n. About half of our APs had
802.11n capability, on both 2.4 and 5.2GHz. The above graph splits out the
N users from the graph from the previous section. In other words, we had
around 324 out of 419 users of 5.2GHz running 802.11n.
I was very surprised to find that over half our total users were
connecting via 802.11n. Particularly after finding that my laptop,
which I was fairly sure would, did not connect. I thought I had gotten
an ABGN capable card, but I guess it's only ABG. Sigh.
The real trick was getting APs that would do 802.11n on both radios at
the same time. Most APs are not able to do N on both radios. So we ended
up using the Netgear WNDAP-350, a $300 AP but it does support lots of
features including gigabit Ethernet, N on both frequencies at the same
We also got plenty of high speed connections: of the samples saved off
we had the majority connecting at 130mbps, followed by a third fewer at
117mbps and just slightly fewer at 54mbps and 78mbps and 104mbps.
This year we tried getting more wired ports out there for people who
were having issues. While this was documented in both the Wiki page on
networking, and the e-mail sent out to attendees before the show, people
just really didn't know about it. I wanted a couple of minutes during the
initial show session to tell people about this, but that was vetoed.
Later the suggestion was passed along to me that we have Ethernet
ports available. We had nearly 200 available (for around 1,000 attendees).
Unfortunately, we were using inexpensive switches which don't have any
stats we can get to, so I don't know how many were actually used. However,
I rarely saw anyone plugged into them.
The suggestion was made to have Ethernet available on every table
instead of to about one out of every 10 tables like we did this year. I'm
reluctant to do that because it would have dramatically increased time to
set up (we had a team of 8 working around 4 hours for what we had), but
also because we didn't have people using what we did have.
Next year I'll probably double our number of switches, but I think the
real issue here was that we didn't have a few minutes to educate people on
The first day of the tutorials, we didn't have networking for the
first couple of hours. There were a few issues that contributed to this.
One was that our router was at the warehouse and hadn't been brought in the
set of boxes that came the day before. So we didn't have it until 9am the
first day of the tutorials.
But then the router had to go into a locked room, which apparently
nobody at the hotel had a key to, and the guy who did have the key didn't
arrive until 11am. At that point I plugged the router in and we were live.
This lead to a significant risk that if the router had issues, I
couldn't get to it to fix it, and it possibly would have involved several
hours to get someone to unlock the door. Next year I either want a key to
that room, or to have the Ethernet drops and power pulled outside that room
to where I can get to it.
Similarly, the first day of the main conference, the first few hours
the network was having serious issues. There were two issues contributing
to this. One is that through a mis-communication with Carl he got a
crimper and ends rather than the Leviton press-on port ends that I would
have preferred using (and then a small jumper, which I had plenty of). The
other was that the crimp on ends had the tabs in fairly close, so unless
you bent them up before plugging them in, they wouldn't click in place.
Because of this, we had several of the Ethernet connections come
loose, which caused the wireless APs to shut down their radios (I had
configured them this way). Which meant that the other APs were
overwhelmed. Of the 700 AP-minutes where the APs were at their limit, a
quarter of those occurred during the first 2 hours of the main conference.
We also didn't have power to the APs until literally a few minutes
before the conference started. The electrician that was supposed to show
up at 5am to wire them all completely failed us. So while we had the APs
out and individually tested the night before, we couldn't do a full system
check until the first break of the conference.
The other issue with crimping our own ends on was the time it took.
We only had one crimper, so we had no parallelism going on there.
This year I tried to get public IPs again, but we just weren't able to
get any meaningful number of them. So, we had to do NAT. This worked
well, and I had no complaints beyond the first one: one of the guys
couldn't establish a PPTP connection. I had forgotten to load the
NAT protocol modules... I loaded those up and it went smoothly after that.
I also did remember to up the number of connections it would allow, so
we didn't run into any problems. I tried to set up munin to track the
number of connections, but that failed to work. I'll have to fix that
plugin for next year.
Number one: We need more APs, particularly if there are going to be
50% more users next year.
I'd also like to try some directional antennas to cover part of the
main room and see if that resolves the issues with 2.4GHz there.
We really need to tell people what's happening as part of the
conference. For having the majority of our attendees using the network, I
think it's proven that it's important enough to warrant a few minutes about
availability of Ethernet ports and cables.
Finally, I'd like to put out more Ethernet ports. They're fairly
cheap to do.
A little more volunteer time, possibly that saved by not crimping our
own cables, would have allowed us to get ethernet cables on the tables by
the switches, if not plugged in and ready to go.
Using many small independent APs continues to work well for our needs.
The response I've gotten has generally been "There were issues the first
few hours, but after that things worked well."
tummy.com has smart people who can bring a diverse set of knowledge to
augment your Linux system administration and managed hosting needs. See
the menu on the upper left of this page for more information about our
comments powered by Disqus