Sean Reifschneider's Journal Recent Entries

swarren and I were just chatting and he mentioned putting something into rc.local. I said that I tended to prefer using "@reboot with cron" over rc.local these days. Partly because I don't like rc.local, partly because it keeps most of my system maintenance stuff in one place on my laptop. Stephen hadn't heard about @reboot though...

The cron daemon supports several "@ nicknames" for use instead of the normal set of time values: @reboot, and @hourly (daily, weekly, monthly, yearly/annually).

I use this in my personal crontab for a job I want run once at boot time:

   @reboot ~/bin/archivevimswap >/dev/null 2>&1

This is a small shell script that moves my ~/.vim-tmp out of the way and creates a new ~/.vim-tmp. I have vim configured to put all the tmp files in there, so that I don't end up with them littered all over the disc.

Just to be canonical, I'll mention that you can also be lists, ranges, and steps, so things like "*/5 * * * *" is every 5 minutes, "0 9-17 * * *" is the top of the hour from 9am to 5pm, and "15,20 * * * *" runs at 15 and 20 minutes past the hour.

Lately though, I've been thinking about what a next-generation cron would look like. It would be nice to say "I have these 10 jobs that need to be run between midnight and 4am, but I only want 3 of them running at once." Rather than having to try to stagger them. I could imagine a use for also having it "kill -STOP" jobs if the load goes above a certain value, or "kill -CONT" when it drops. Or even coordination among machines (I have 10 machines, they all need to run CPU-intensive jobs after midnight, but I don't want to cause a spike in power consumption or drop in responsiveness among all of them).

So many possibilities...
(go to article | 0 Comments)

During the Language Summit we were speaking about packaging, and Guido said that he usually doesn't create packages. And I felt his pain, because I usually put it off a long as possible too. But from that I decided that I wanted to build a helper to make setting up the package files a no-brainer.

A couple of days into the sprint, I had something that was a good start. Continue reading for more details.
(read more | 2 Comments)

I've completed my wrap-up of the networking at PyCon 2010. I hope you enjoy reading about it as much as I enjoyed working on it.
(go to article | 0 Comments)

The hotel engineer comes over and is looking for someone in charge... "What can I help you with?" I ask him. He explains that they have to set these rooms they've given away from us to another conference and there are power cords taped down all over.

I call out "Python Developers Activate! Form of a righteous swarm!" 5 minutes later the rooms are picked clean like the bones of some (particularly tasty) carrion.
(go to article | 1 Comment)

I've hunted down some of the story behind the DNS server that runs at 4.2.2.2. I've taken some discussions I've found around the web, and some responses I got on NANOG, into a story on the history behind the popular 4.2.2.2 DNS server.
(go to article | 0 Comments)

Everything is, apparently, encrypted for transit and storage, with a key that you select. So there shouldn't be a security concern. You can select what you want it to synchronize, including bookmarks, passwords, preferences, history, and tabs.

One thing that kind of threw me is that the tabs show up under "History -> Tabs from Other Computers".

Weave seems to work fine so far. You just install the weave plugin, create an account, set the encryption password, and off you go. It's real easy, man.
(go to article | 0 Comments)

I just posted a Django snippet that (ab)uses a decorator to change how you call templates in Dango views. For example, it makes my view code something like this:

####################################
@with_template('friends/index.html')
def friends(request, context, username):
   context['user'] = User.objects.get(username = username)

And the view:

{% extends "base.html" %}
{% block content %}
<h1>{{ user.username }}'s Friends</h1>

I've continued to do a lot of testing and other poking at ZFS-FUSE under Linux, and things are looking pretty good. I had run into a problem with the stable 0.6.0 release when used with lots (13+) of devices with very long names (50-ish character /dev names). However, that seems to be resolved with a very short patch as mentioned in this mailing list thread.

Additionally, zfs-fuse has just been placed into Debian for the Squeeze release (including the patch above). So, before long we should be able to "apt-get zfs-fuse" and have solid access to the joys of ZFS. Though note that this does not include dedup functionality -- there are still some bugs in the base ZFS code to be worked out in that.

I've been running ZFS on my storage server for over a year, but I had found a version that mostly worked (though there were issues with heavy snapshot use). It wasn't until a couple of weeks ago that I finally decided to upgrade. It hasn't been trouble-free -- because of the bug above I ended up having to reload my pool. But considering that I have good backups, I decided I might as well put the recent stable release to the test.

Unfortunately, at the moment one of my test machines has had some sort of hardware failure, and the other I had to swap to different hardware because we needed the chassis that it was on for other testing, and it's not booting in the new chassis. So for the moment, my stress testing is on hold.
(go to article | 2 Comments)

A month or so ago I finally got a system that would correctly install Proxmox VE -- my earlier tests didn't produce much progress. That initial test was done on a virtual machine on my laptop. To make matters worse, my laptop runs 32-bit, so I had to run the test under full QEMU virtualization, which made performance terrible. The amazing thing I found then was that a OpenVZ virtual environment actually worked quite well in those challenging situations.

It did definitely whet my appetite to do more experimentation with Proxmox VE. It has loads of interesting features and I was dieing to try it. It wasn't until recently that I had some spare hardware sitting around that would support KVM virtualization (CPU virtual support). I happened to get several systems that I could spare for some testing.

Read on for my impressions of the new 1.5 release of Proxmox VE and how it compares to VMWare ESXi.
(read more | 0 Comments)

On some systems I have long-running screen sessions, like on one system where I run folding under it, and I've also started running the zfs-fuse daemon under screen. The normal naming causes me to end up with a bunch of screens that I have to hunt through to find the one I want:

guin:~$ screen -x
You may wish for a screen, what do you want?
        13658.pts-13.guin       (Detached)
        13639.pts-13.guin       (Detached)
        13678.pts-13.guin       (Attached)
Type "screen [-d] -r [pid.]tty.host" to resume one of them.
zsh: exit 1     screen -x
guin:~$ 

I started hunting for a way to get some more information about what was on each screen, and found the "-S [sessionname]" option. Just what I need:

root@stow:~# screen -x
There are several suitable screens on:
        20873.pts-4.stow        (Detached)
        20303.zfs       (Detached)
        20224.folding   (Detached)
Type "screen [-d] -r [pid.]tty.host" to resume one of them.
root@stow:~# 

So now I can see which one s the one I started for just whatever (the "pts" one), and I can resume screen sessions with "screen -x folding" or "screen -x pts" to get to the generic one.
(go to article | 0 Comments)

1. What's the coolest Python application, framework or library you have discovered in 2009 ?
   Hard to pick... Sprox I'm dying to try (for putting databases on the web), and Pinax looks very promising for making social sites. Django is really the only new thing I've spent significant time in though, and it is working fine.
2. What new programming technique did you learn in 2009?
   I've been messing around with decorators, and I've really become comfortable with list comprehensions. Because I'm a system administrator, I'm always years behond the curve deploying new features, because they need to work on the last few enterprise releases.
3. What's the name of the open source project you contributed the most in 2009? What did you do?
   I flitter around between proejcts, but the Python memcached module is the big, consistent one. I'm doing maintenance on it, so accepting patches and making new releases.
4. What was the Python blog or website you read the most in 2009?
   I usually skim Coder Who Says Py, it's great content but usually too involved for my typical lunch-time reads. Richard Jones' Log and pyOraGeek usually have small snippets and I always read them when they're up.
5. What are the three top things you want to learn in 2010 ?
   I'd love to write some code in Python 3.0 and get comfortable with it, but obviously still need to really target 2.x -- so 3to2 will be important. I'd like to play with CouchDB and it's replication.

I've also done a fair bit of e-book reading recently. Read below for my thoughts on the Kindle (which I don't have), Gutenberg, and more...
(read more | 0 Comments)

I really like the idea of rdiff-backup, but the drawbacks kept stopping me from deploying it more widely. The nicest thing is that it stores deltas as files change, so if you have a large file that changes a little bit every day, rdiff-backup only stores the little bit that changes. If you use the rsync hard-link trick to keep historic data around, it duplicates the whole file every day which can quickly add up on a slowly changing multi-gigabyte database file.

Problems with rdiff-backup include:

• Server and client versions need to be the same. In a mixed environment, this means you're going to have to maintain your own packages for many of CentOS 4, CentOS 5, Hardy, Karmic, Fedora 12... My personal backup server has clients that are most of those...
• rdiff-backup doesn't deal very well with intermittently connected systems. If you have a big set of changes that takes several days to push up at a throttled rate, and you are disconnected part way through, it needs to start over from scratch.
• Worse, this failure may require running the next rdiff-backup with a special option to clean up the broken backup directory.
• No throttling like the "--bwlimit" option in rsync.

After giving this all some thought, I came up with the idea of using rsync to pull the data over the network, and using rdiff-backup to maintain the historic backup information. Read below for my experiences with this.
(read more | 8 Comments)

I've been increasingly using the PowerDNS Recursor (pdns_recursor), and I've been pretty happy with it. I've been testing it for something close to 2 years now, and have really had no problems with it. Initially using it as a caching DNS server on my laptop, then putting it in place as a backup recursor for our hosting. A week or so ago we completed setting up it up on a HA load-balancing cluster as our primary recursor.

As part of that setup, I set up munin to graph some usage and performance stats. Read below for more information and links to these plugins.
(read more | 0 Comments)

My "Ultimate Storage Box" has been in service 18 months now, so I thought I'd do a status report. Particularly as ZFS-FUSE still kind of has a "bad rap" as being a kludge. Which, you know, it kind of is... But it's the only option Linux users have for a modern file system (ext4 is just, IMHO, basically a polishing of the Berkeley FFS from 1983). I wish we could have ZFS in the kernel, but the powers that be have decided that will never happen. And btrfs is still years away (in the last month it did get the ability to remove snapshots finally, yay!).

Read on for my experiences over the last 18 months with ZFS-FUSE.
(read more | 5 Comments)

Not to subvert the recent 0.6.0 release of ZFS-FUSE, which I think is great... But the thing I'm really interested in is the dedupication code that just recently got released, and isn't even in the current OpenSolaris release yet. Read below the fold for more information on dedup in ZFS-FUSE.
(read more | 0 Comments)

A few weeks ago at our favorite coffee shop, the net went just right into the toilet... Jamie had recently put Tomato on their WRT54G router, so I was able to get in and see what the problem was. Someone was hammering the network, saturating both the incoming and outgoing, and had been for an hour. But how to tell who it was?

The solution I ended up with was to read /proc/net/ip_conntrack and look at the local machines with lots of remote connections, indicating file-sharing. Seconds later, the network was usable again and the owner was thanking us because they could run credit cards again.

But newer kernels don't have this file any more. They do have similar functionality though, available through the "iptstate" tool. Read below for more information.
(read more | 1 Comment)

I'm running some stress-testing of ZFS-FUSE, in particular the new "dedup" version. It seems to be working fine, but as part of my stress testing I'm writing a bunch of random data. /dev/random and /dev/urandom are very good sources of randomness, but they sure use a lot of CPU time.

After a (very) brief search I found the frandom driver. A few minutes later I had it built and installed in my Karmic test system. My stress test program has gone down from using 90% CPU time to 20% (with ZFS using the remaining 180% :-).
(go to article | 0 Comments)

IPVS load-balancers work great, but typically they are deployed as one or more usually two additional machines. For people wanting to deploy a 2 machine cluster and grow it, or even start with one machine but configure it to grow, this additional set of machines is a large overhead.

A much less frequently used option, but one that has been around forever, is the CLUSTERIP iptables module. This module works by allowing multiple machines to cooperatively have the same IP address, but only one will handle a specific connection or client machine.

Read on for more details about this little used, but very useful clustering mechanism.
(read more | 0 Comments)

I just wanted to note that in the 1.45 release of the python-memcache module I have switched the hash algorithm to be compatible with the other libraries, thanks to code by Andre Cru and Ludvig Ericson. To switch back to the old-style hashing, call:

memcached.useOldServerHashFunction()

Read on for more information about the release.
(read more | 0 Comments)

I've been using Django for some personal projects recently, and mostly I've been liking it just fine. However, I've had a bit of trouble with figuring out how to do some of the things I want to do. The documentation seems to lead to things like class documentation rather than examples of use, and the class documentation is really only a part of the story.

For example, one of the first things I wanted to implement on my photoblog was pagination. However, the documentation I could find was of fairly limited use, without creating a bunch of supplemental code myself. Which is what I did.

However, I then found a snippet on DjangoSnippets which lead me down the right path for doing the pagination much easier.

I've written up what I've found as an article on setting up pagination in Django.
(go to article | 0 Comments)

As part of getting ready for PyCon 2010, I have (finally, sorry :-) written up my report on the network at PyCon 2009 in Chicago. It includes the details of the wireless network in Chicago, and my thoughts on why Netbooks had so many problems.
(go to article | 1 Comment)

After digging around some and finding that my Sandisk Cruzer 32GB was failing to work in my car stereo because it was showing up as a CD-ROM drive because of "U3" crud (a Windows convenience), Jim DeWitt found a Linux utility that I was able to use to get rid of that CD-ROM partition without requiring me to find a Windows machine.

I wanted to make this post so that others in a similar situation could find it, since I wasn't able to find anything by search for things like "linux u3 uninstall".

There's a program called "u3-tool" at Sourceforge (http://sourceforge.net/projects/u3-tool/). This tool built easily enough, but I ran into issues when trying to use it via libusb.

What I ended up doing was running "./src/u3-tool -p0 /dev/sdb". That caused it to create a 0-length U3 partition. At that point, ejecting and re-inserting the USB drive caused it to only show the USB storage device, and not the CD-ROM device.
(go to article | 0 Comments)

Evelyn just got signed up with Cricket, because at $40/month she couldn't resist. Our Sprint WWAN cards are $60/month, and requires a 2 year commitment, which was over her threshold of pain. The Cricket plan is pretty tempting, considering it still offers unlimited data, includes a free modem, and apparently has no contract. It was a little tricky to get working, but only a little. Read on for more information.
(read more | 2 Comments)

Last night I couldn't sleep and so I finally attacked the problem of getting it so that my Sierra Wireless USB 598 dongle will automatically come up when it's connected. This includes poking at NetworkManager to tell it to disable wireless, and make the connection via the "Auto Mobile Broadband (CDMA) connection". Read on for more details of how I got this working.
(read more | 1 Comment)