Sean Reifschneider's Journal Entries

We did an upgrade of our CLUSTERIP based load-balanced DNS servers, and it started spewing the message "CLUSTERIP: no conntrack error". It took some digging, but I eventually tracked it down. To fix it you need to insert a rule with "-m state --state INVALID -j DROP".

The message is an indication that a packet that doesn't match an existing conntrack connection is received. So, just add a DROP rule.
(go to article | 0 Comments)

PyCon is run by volunteers, like myself. If you are planning on going, and can help the night before the tutorials or the main conference, and are interested in networking, please consider signing up to help with the network setup. A lot of the work is simply laying out cables and gear, taping down runs into the audience, and network testing and debugging.

If you are interested, please sign up on the PyCon 2011 Volunteer Sign-up Page (for networking and/or other sections). Thanks!
(go to article | 0 Comments)

If you are planning to attend PyCon 2011 and use the network, please read this. And if you are thinking of going and haven't signed up, please do so soon, it is expected to sell out. In a good way.

Executive Summary: Please bring 5.2GHz gear if at all possible (A.K.A. 802.11a). There will be a few hundred wired ports but they are highly concentrated. Please leave your 100+mW cards at home. Read this full message for all the details.
(read more | 2 Comments)

After some great comments related to my previous release, I've completely re-worked the movingaverage module and released it on PyPi (with the development still going on over at github.

I ended up switching all the results to use floating-points, because that really is what makes sense. I also found that my original algorithm, if I didn't do the full sum of the subset every time, was quite a lot faster than the deque version, but at the cost of not working with iterables, only with lists.

So in the end I have both algorithms, and also the ability to do a full re-sum, or use a rolling sum (the default is to re-sum because I know it's going to be as accurate as the computer can make it, with no chance for floating point drift). I may be being over-careful here, but I know that to some people this matters a lot and I know that I don't know enough about floating-point math to know how big a problem it may be.

So my one-line function is now over 200 lines, and around 5 hours of development, tuning and testing. :-) Admittedly, most of that is tests, but it's still grown quite a lot. I sure was glad to have the tests today when I re-wrote it.
(go to article | 3 Comments)

On Friday Evelyn was asking for a library that did moving averages. I took a look around for things like "moving average", "circular buffer" and "ring buffer", but couldn't find anything that seemed to do the job. Possibly because it's "too trivial" to implement on your own. So, I put together some code and released movingaverage.py on pypi and hosted the code on github.

However, the unit tests fail under Python 3, where division returns floats always. I'm pretty sure that what I want to do is change the test so that it understands when it's running on 3 and 2 and expects different values there, I don't think I want it to act the same on both platforms. Or, I guess maybe I'll need to add a "float divide" option to 2.6, or a "// divide" option to 3? Any suggestions?

It's literally a line long, at it's core. It's implemented as a list comprehension. It's taken me nearly 2 hours to do, of course. I did a simple test and found that there was a fence-post error. So I added unit tests, and released it. Then the bug-reports started rolling in; Evelyn has Python 2.6 on Mandrake, and the unit tests were using a 2.7 (Fedora 14) feature of the unittest module. Then I had to fix a license inconsistency (it's in the public domain)...

I'm reminded of when I did string.rsplit()... Initially it was refused because it was "too trivial to reimplement on your own". The demonstration implementation had a bug in it though. :-) Obviously, it eventually went into the standard library, but not without a year worth of discussion...

Enjoy.
(go to article | 7 Comments)

Earlier today I came across a very cute picture of a bunny in Alexander Artemenko's blog post about Cony.

Cony is a Python web server using the bottle micro-framework that I've been loving so much lately. It's goal is to make it so you can easily write "bookmarks" or "commands" that you can access through the URL bar in your browser. You configure your browser to send keyword requests to the Cony server, and Cony parses it and runs your own custom Python code based on that. It includes commands for "g <string>" to do a google search, "p" to search the Python documentation, and "pypi" to search the Python Package Index.

Continue reading for more details of Cony and my hacking on it.
(read more | 0 Comments)

I was making a screencast, as I mentioned a few days ago, but the Ogg Theora output that Istanbul wrote turned into a psychedelic experience after Youtube man-handled it. After a bunch of failed attempts with things like mencoder, Arista, Transmageddon, some help from the kind folks on #theora@freenode pointed me at ffmpeg.

The screencast is an Introduction to the Proxmox Virtualization Environment with DRBD replicated disc and is up on youtube if you'd like to check out the results.

Read on for what I was able to get to work.
(read more | 0 Comments)

A post by Brett Walker on the Interesting People list brings up a very good point about the Egyptian government shutting down the Internet in their country: There's currently a bill being reviewed to enact legislation requiring an Internet Kill Switch controlled by the DHS.

Time will tell how well it works out for Egypt, but now is probably a good time for you to contact your representative via this handy web form and let them know your thoughts on it.
(go to article | 2 Comments)

I had wanted to record a screencast, but I really wanted audio on it. Unfortunately, the state of Linux audio is (still) fairly sad. So, while I was able to grab a nice USB headset and plug it in and use Audacity with it, screencasting just wasn't happening.

Read the full post for details on how I got this working.
(read more | 0 Comments)

I've recently been implementing a set of Python routines for generating strong password hashes. The easy one, that's been around forever but isn't really that strong includes only 2 characters of salt. There are several stronger formats that include up to 16 characters of salt, and allow for longer passwords than 8 characters. This format is called Modular Crypt Format, but searching for documentation on the web does not produce much useful information.

Hopefully this post will help others who run into similar problems. This is what I've been able to find out about Modular Crypt Format.

Read on for details of this format.
(read more | 0 Comments)

The Python "crypt" module, which wraps the C library crypt(3) function, requires that you provide it with a "salt" argument, which should be a random string of certain characters in most cases. However, the Python library doesn't provide a helper for building such a salt. Not a big deal to write code to generate in a world of PDP-11s where you only need 2 salt characters. But in today's world, you probably want to be using SHA512-based password hashing in Modular Crypt Format, which is hard to find documentation on.

So I just submitted a patch to the Python bug tracker, and asked for comments, on a patch to add a "mksalt()" helper to the crypt library, and make a strong salt be selected if no salt argument is passed to "crypt.crypt()". Read on for more information.
(read more | 2 Comments)

I've been trying to find a way to do this since Martin Blais demonstrated the namedtuple he uses in his antiorm. I tried to directly use namedtuple, but it's a bit unusual in how it works and I couldn't adapt it to my existing code. Tonight I took another stab at getting this functionality, and ended up with something that seems to be working.

I haven't yet pushed this out to pypi, because I'm hoping to get some comments on my approach here. I'm almost certain that the way I'm doing this is cheating and may lead to problems, but my tests all succeed! Code is currently available as HEAD from github.

Read on for more details.
(read more | 2 Comments)

This weekend I added a new session manager to my bottlesession library, this one stores the session information in a "secure cookie" in the browser. So now bottlesession includes two easy ways of managing sessions. Using a database is going to be trickier, because no one schema is going to fit all users.

To read more about my continued experimentation with the bottle web application micro-framework, read the remainder of this post.
(read more | 0 Comments)

I had a heck of a time getting SSH and GPG agents working how I wanted on my Ubuntu 10.10 box running XFCE. The primary behavior I was trying to fix is that my GPG passphrase was asked for every time, which is annoying when I sign all outgoing e-mail. The other issue is that gnome-keyring does not support the "confirmation" (-c) option to "ssh-add".

Not sure why the gpg passphrase wasn't being stored properly, the dialog asking for it had radio-buttons to select how long the passphrase was remembered for, but that just wasn't being honored due to a "Certain fields have invalid values" error. Searching provided no fix for that.

I made some progress by going into Applications -> Settings -> Xfce4 Settings -> Session and Startup, click the Application Autostart tab, and unchecking the "SSH Key Agent (GNOME Keyring: SSH Agent)", "Secret Storage Service (GNOME Keyring: Secret Service)", and "Certificate and Key Storage (GNOME Keyring: PKCS#11 Component)". However, that was still running gnome-keyring-daemon which was taking over the SSH_AUTH_SOCK environment variable.

After a lot of searching, I couldn't find the right way to disable gnome-keyring-daemon, so I just did a "sudo chmod 000 /usr/bin/gnome-keyring-daemon". Then I added an startup entry (in the settings application above) to run "gpg-agent".

Once this was done, I logged out and back in, and had ssh-agent and gpg-agent running. Now I can "ssh-add -c" and it works properly, and my e-mail isn't asking for my long GPG pass-phrase every time I send mail.
(go to article | 5 Comments)

I've released a new python-memcached version 1.47, which has a small fix I didn't get in for the last release: the functionality to switch back to the old serverhash function was broken. I had noticed that I missed it right after the last release, but I decided to wait a few days before rolling a new one, in case any more changes rolled in.
(go to article | 0 Comments)

A month or so ago I found the Bottle Micro-framework. I'd never even heard of it, then I ran across it linked from another site and I really liked the look of it. And it really means it when it says it's a micro framework: it's a single file of around 2,200 lines.

Read the rest of this post for an example and my experimentation with user authentication.
(read more | 0 Comments)

I've released a new version of the Python memcache module, version 1.46 with links to information on pypi.

The biggest set of changes is related to tracebacks that happen if a server goes down. It also allows specifying host names without the port number for servers to connect to.

Please report bugs to the Launchpad page.
(go to article | 0 Comments)

Evelyn's laptop has been sending out a cron e-mail from the cron.daily "apt" job reporting a segfault. I've done some digging and debugging of the "apt" script and have been able to track down the issue. Please read on if you are running Maverick, particularly 32-bit, with automatic updates turned on.
(read more | 1 Comment)

I have a need for doing password strength testing. I had reviewed a few options, such as google's password strength checking service, which involves requesting https://www.google.com/accounts/RatePassword?Passwd=<password> but that returns "4" (strong) for passwords like "PaSsWoRd", and "3" (fairly strong) for "Secret"... I really wanted cracklib, but the CentOS system I was on doesn't have python-cracklib and the package for it is included in the base cracklib package.

So I decided to try using ctypes to wrap cracklib. It worked out great, though it does require Python 2.6 (which is available on the system as "python26"). So I've released the python-ctypescracklib package on github. I plan on putting this and the psycopg wrapper I wrote about recently in pypi when I have a few moments.

ctypes is very good stuff!
(go to article | 2 Comments)

I recently wrote about a new home information display I set up. The information itself is just HTML+CSS with a little bit of Python for the dynamic content. So the bulk of the information display involves running a full-screen browser at system boot time. I ended up selecting Google Chrome after some experimentation with Firefox and various full-screen plugins.

It took a bit of searching, but I found that Chrome has as "--kiosk" mode. That mostly did what I needed it to, after I set up the system to automatically log-in at boot, and configured Chrome to go to my sign page at startup.

The other issue I ran into is that if the system shuts down (say, because of a reboot or a power failure), it will come up with a "Do you want to restore your last session" banner. I worked around this by having the X session call a script instead of the browser directly. I created a tar file of the ".config" directory with the browser shut down. Then I set the script to remove the existing ".config" directory, untar the pristine one, and then start "google-chrome --kiosk". This has been running a few weeks and has been very reliable.
(go to article | 0 Comments)

I have a cunning plan to make a discless machine that boots via gPXE and iSCSI off the network from a machine in the basement. Preferably, accelerated with flashcache and an SSD. :-) So today I've been playing with setting up an iSCSI server, installing a distro to it, and then getting it to boot. Read below for some details about the configuration.
(read more | 0 Comments)

I've been wanting to play with FlashCache since I heard about it a month ago. It's a kernel module that allows you to set up an SSD to act as a cache for a another block device. Day-to-day you probably don't need 32GB of cache, but it'd be hard to whittle my laptop down to fit in that much space... Flashcache seems like the best of both worlds: the speed you want with the space you need. Read on for the results of my testing.
(read more | 2 Comments)

I've pushed up my Python Psycopg2 database wrapper to github after making some changes over lunch. I added "insert" and "dictinsert" methods that will automatically do insert statements based in either keyword arguments, or a dictionary of values passed. For example:

insert('users', name = 'Sean', password = 'secret')
dictinsert('users', { 'name' : 'Sean', 'password' : 'secret' })

If you're interested, grab it at Git Hub's python-psycopgwrap page.
(go to article | 0 Comments)

Sunday night I stayed up late and did a fresh install of Fedora 14 on my Thinkpad T61 -- my primary machine. I had F12 on it previously, and was actually quite happy with it. All my buttons like display brightness and volume worked fine, and I was going basically a month between reboots (with typically several suspends a day). I'll admit that part of the reason for the re-install was so I could switch to using btrfs on it. Part of the reason I did it was because Kristen was running F11 and really needed an upgrade, so mine was a dry-run for hers...

I've been using it heavily this week. Read on for my min-review of it.
(read more | 0 Comments)

To help with the "what files have changed" work-flow I mentioned in my last message, I have submitted a patch to btrfs-progs that implements a "subvolume last-gen" command that returns the latest generation represented by that subvolume.

The idea being that you could take a snapshot (or simply run it against your main file-system), and get a number suitable to give to "subvolume find-new" to tell what has changed since then or between then and a more recent snapshot.

NOTE: While it would be great to feed rsync off this information, currently no information about file deletions is included so you would still need to periodically run a full rsync to detect those sorts of changes. The same also seems to be true for renaming files.
(go to article | 0 Comments)