Sean Reifschneider's Journal Recent Entries

Slashdot has a story about using competing development teams to find the best solution. This reminded me of the HP-BASIC or "Rocky Mountain Basic" project.

The story I heard, in the late '80s when I was working on testing in the Loveland Instrument Division of HP on their port of RMB to Unix, was that HP was looking to create a BASIC variant, and they set up two teams, one in Colorado and one on the east coast. These variants were Rocky Mountain Basic and East Coast Basic.

The developers were given some time to work on their visions, and then presentations were set up to allow a choice between them to be made. Once released, RMB was generally considered to be a pretty great dialect. I programmed in RMB early on, while I was also learning Pascal, and found that RMB was much less panful than the other BASICs I had dealt with, like Microsoft's.

I also remember a meeting where our manager asked a bunch of us for suggestions on what to call the port to HP-UX, and we were pretty much all in favor of "RMB/UX" or "BASIC/UX". But our manager dropped the bomb "It can't have a slash in it." I remember everyone being pretty annoyed at that. But, in looking at the RMB wikipedia page, it looks like the slash won out in the long run.
(go to article | 0 Comments)

A recent thread on the zfs-fuse mailing list has announced that the long-awaited Lustre project to make a native ZFS module for Linux has made good progress. This was announced as being the future for Lustre probably a year ago, but I haven't heard anything about it until this post on the list.

Things are still pretty early it sounds like -- zfs-fuse is likely to be the best choice for probably the next 6 months at least, but it is a significant step toward getting ZFS native under Linux.

The ZFS license is still CDDL, which means that it won't be included in the kernel.org kernel, instead it'll be an out-of-kernel module like DRBD (until recently) or Xen, etc...

This comes at the same time as the ZFS-FUSE 0.6.9 release, which includes deduplication and many other great features. In my testing of 0.6.9b3, it's been working really great. I've been hammering on it with both "zfsstress" and also running it on a test backup server, and it's been running very solidly.

The deduplication has been working well, though you really do need a lot of memory in the ARC cache if you want it to perform well. For this system with 8x2Tb drives, I figure I'll need to put at least 8, and possibly 16GB in the ARC cache. I currently have 8GB RAM, and a 2GB ARC, which is about as much as I can do in an 8GB system. The host will take up to 32GB RAM though, so I have room to grow. My plan is to upgrade it to 8GB and push the ARC up to 8GB, then see how it works. I blew out the original 800MB ARC with deduplication at around 900GB stored in the pool.

It looks like with compression plus deduplication I'm getting a 1.9:1 space savings. Not sure how this compares to the deduplication+compression in BackupPC, but I'm expecting it to do much better simply because I can do block-level changes (large files that just have small appends/updates to them, like databases or log-files).

Anyway, that's the ZFS news for today. :-)
(go to article | 0 Comments)

I'm working on replacing our BackupPC backup infrastructure (because BackupPC just takes too long), and one of the things I needed to do was schedule backup jobs. In BackupPC you can tell it to run 4 jobs in parallel, and whenever it wakes up if there are slots free and backups to run, it will start some more.

I wanted similar capabilities, but without writing my own scheduler; it's not rocket science, but it's still a complicated bit of code. Ideally, to improve on BackupPC, I'd like to have one job start as soon as another ends, rather than waiting for the next scheduler wake-up.

As I've mentioned before, xargs can manage running multiple jobs. You can specify how many to run in parallel, and it gets the list of arguments to run from stdin. So, what I came up with is a crontab which looks like this:

00 22 * * * echo 1.example.com 2.example.com [...] \
      15.example.com | xargs --max-args=1 --max-procs=4 /path/to/harness
00 09 * * * echo a.example.org b.example.org c.example.org \
      | xargs --max-args=1 --max-procs=1 /path/to/harness

The first line starts at 10pm and runs the harness with the system name to back up as the argument. It runs it for 15 hosts, running 4 in parallel. The second cron entry starts at 9am and runs the 3 example.org backups one at a time (they are hosted off-site and no need to hit their network or ours harder than necessary).

In the past I would manually add the cron entries for each host at specific times, but sometimes jobs would run long and load would go way up, or sometimes there were idle periods where nothing happened... This is definitely an improvement over that, with minimal additional coding.

Wherever possible: Avoid writing code.
(go to article | 0 Comments)

This morning I figured out how to get rrdtool to report the 95%ile utilization to stdout. It's kind of convoluted how you do it, you have to use the "graph" subcommand, but write the graph to /dev/null, and use "PRINT" instead of "GPRINT". For example:

eval `$RRDTOOL graph -f '' -s "$1" /dev/null \
   DEF:in="$2":in:AVERAGE \
   DEF:out="$2":out:AVERAGE \
   CDEF:inbits=in,8,* \
   CDEF:outbits=out,8,* \
   VDEF:95pct_in=inbits,95,PERCENT
   VDEF:95pct_out=outbits,95,PERCENT \
   PRINT:95pct_in:"IN='%.2lf %Sb'" \
   PRINT:95pct_out:"OUT='%.2lf %Sb'"

Where "$1" is the period start time (like "-1d" for showing the 95%ile of today), and "$2" is the .rrd file name. I do an "eval" to parse the output (making $IN and $OUT shell variables). The "-f ''" tells is not to write an image size string.

It may also be useful to change the last two lines to use a format something like "IN='%.2lf; IN_MAGNITUDE=%S" (to get something like IN=2.50 and IN_MAGNITUDE=M) or just "IN=%.0lf" (to get the full bits like IN=2500000).

Again, rrdtool proves to be amazingly flexible, given enough time to wrap your mind around it.
(go to article | 0 Comments)

I've just completed a patch to the Python syslog module to add the method enable_exception_logging(). It sets up a sys.excepthook so that unhandled exceptions are logged to syslog. By default, it chains to the existing excepthook.

So, once this code gets accepted, you will be able to have exceptions logged by doing: "import syslog; syslog.enable_exception_logging()".

For Python software that runs from cron or init or Apache, it can be very useful to capture the exceptions in a persistent location.

I'd appreciate some reviews of the code, it's in Issue8214.
(go to article | 2 Comments)

Just a FYI follow-up on my previous post about having started running some stress testing of btrfs: it seems to have run into a bug in btrfs after around 8 days. I had done a couple of stops of btrstress, removals of the test data, and restarts. I had also enabled compression, which I'm wondering if that was the cause of the problem. The issue seems to be a NULL pointer dereference (which has been reported to the btrfs mailing list).

I'll look at starting another run, once I've made sure nobody needs any information off the system as it is now.
(go to article | 0 Comments)

Computer hardware is pretty reliable these days. However, even with good procedures and hardware in place, there is still the possibility of data-loss. As we found out on Monday night... Despite having a well documented and tested workflow, RAID data redundancy, monitoring, top-notch personnel, and operating in a very conservative manner, we had a data-loss event that impacted 6 of our virtual hosting customers.

This is, to the best of my recollection, the first major data loss event we've had related to our hosting, since we began the hosting service over 11 years ago. I wanted to document what happened, both to provide information to the clients that were impacted and also as a lesson to the other readers.

Read on if you are interested in all the gory details. In big-enterprise circles, this is called a Service Outage Analysis (SOA).
(read more | 0 Comments)

xargs is a great command-line tool for parceling huge lists of files to not exceed command-line limits on length or numbers of arguments. However, it also has some arguments that cause it to manage running multiple, parallel jobs. Read on for how I used this to cut one of my jobs execution time by 75%.
(read more | 0 Comments)

I've created a patch for the Python syslog module which:

• Makes openlog arguments keyword args.
• Makes openlog ident argument optional.
• If ident is not passed to ident, basename(sys.argv[0]) is used.
• The first call to syslog.syslog() calls ident() with no options (if it hasn't previously been called).
• Variously related documentation changes.

The patch is in the issue tracker as Issue 8451. If anyone out there has the inclination to review it, I'd appreciate it.
(go to article | 0 Comments)

Several months ago I wrote a "zfsstress" program. This program emulates the use-case of our old backup servers, which would regularly cause the OpenSolaris systems we were running them on to reboot. As I've mentioned before, zfsstress has shown that the stable release of zfs-fuse is quite good.

Now that btrfs has the ability to delete snapshots, I decided to port zfsstress to btrfs, and the result is available at ftp://ftp.tummy.com/pub/tummy/zfsstress/ along-side the zfsstress program.

I've been running btrstress on a test Fedora 13 Beta system for the last week, and it's been working very well. I am getting "unlinked 1 orphans" in dmesg periodically, I haven't been able to find anything saying what that is about. So far though, btrfs is looking pretty good.

One kind of curious thing is that I let btrstress fill up a 200GB partition, then I deleted the btrstress subvolume. The delete of 200GB of data returned in around 20 seconds, but df showed the file-system still full. It was cleaning up in the background, which is just great -- I don't have to wait for it before the command returns. It took around 10 minutes for the data to be completely removed.
(go to article | 0 Comments)

I've been running some tests with real, live data on the ZFS-FUSE devel branch that I mentioned previously I am testing.

My initial tests were performing rather poorly, in the area of 1MB/sec for 250GB. In other words, it took 3 days to rsync 250GB. However, a conveniently-timed thread on the ZFS-FUSE mailing list saved the day. The remainder of this message includes some tools and techniques for determining how big your ARC needs to be to get good performance.
(read more | 0 Comments)

I've been continuing to do a variety of testing with ZFS-FUSE, concentrating largely on the development branches which implement deduplication. I'm using Emmanuel Anne's branch, the 94767eeb512704d673e301eb6c837ee108739bd4 branch. With a change to kernel parameters, this has been running without any problems. Continue reading if you would like more details.
(read more | 0 Comments)

For about the last 6 months I've been trying to decide what advice I would have wanted when I was 18 (for me, that was 1988). A recent XKCD combined with a "homework assignment" from my about-to-graduate nephew's English teacher finally combined forces to get me to sit down and write a letter from the past. What advice would you want to have heard as you were about to take another step into the "real world"? I mean besides "Don't Do It (tm)!".

Read my letter from the past here.
(go to article | 1 Comment)

swarren and I were just chatting and he mentioned putting something into rc.local. I said that I tended to prefer using "@reboot with cron" over rc.local these days. Partly because I don't like rc.local, partly because it keeps most of my system maintenance stuff in one place on my laptop. Stephen hadn't heard about @reboot though...

The cron daemon supports several "@ nicknames" for use instead of the normal set of time values: @reboot, and @hourly (daily, weekly, monthly, yearly/annually).

I use this in my personal crontab for a job I want run once at boot time:

   @reboot ~/bin/archivevimswap >/dev/null 2>&1

This is a small shell script that moves my ~/.vim-tmp out of the way and creates a new ~/.vim-tmp. I have vim configured to put all the tmp files in there, so that I don't end up with them littered all over the disc.

Just to be canonical, I'll mention that you can also be lists, ranges, and steps, so things like "*/5 * * * *" is every 5 minutes, "0 9-17 * * *" is the top of the hour from 9am to 5pm, and "15,20 * * * *" runs at 15 and 20 minutes past the hour.

Lately though, I've been thinking about what a next-generation cron would look like. It would be nice to say "I have these 10 jobs that need to be run between midnight and 4am, but I only want 3 of them running at once." Rather than having to try to stagger them. I could imagine a use for also having it "kill -STOP" jobs if the load goes above a certain value, or "kill -CONT" when it drops. Or even coordination among machines (I have 10 machines, they all need to run CPU-intensive jobs after midnight, but I don't want to cause a spike in power consumption or drop in responsiveness among all of them).

So many possibilities...
(go to article | 1 Comment)

During the Language Summit we were speaking about packaging, and Guido said that he usually doesn't create packages. And I felt his pain, because I usually put it off a long as possible too. But from that I decided that I wanted to build a helper to make setting up the package files a no-brainer.

A couple of days into the sprint, I had something that was a good start. Continue reading for more details.
(read more | 2 Comments)

I've completed my wrap-up of the networking at PyCon 2010. I hope you enjoy reading about it as much as I enjoyed working on it.
(go to article | 0 Comments)

The hotel engineer comes over and is looking for someone in charge... "What can I help you with?" I ask him. He explains that they have to set these rooms they've given away from us to another conference and there are power cords taped down all over.

I call out "Python Developers Activate! Form of a righteous swarm!" 5 minutes later the rooms are picked clean like the bones of some (particularly tasty) carrion.
(go to article | 1 Comment)

I've hunted down some of the story behind the DNS server that runs at 4.2.2.2. I've taken some discussions I've found around the web, and some responses I got on NANOG, into a story on the history behind the popular 4.2.2.2 DNS server.
(go to article | 0 Comments)

Everything is, apparently, encrypted for transit and storage, with a key that you select. So there shouldn't be a security concern. You can select what you want it to synchronize, including bookmarks, passwords, preferences, history, and tabs.

One thing that kind of threw me is that the tabs show up under "History -> Tabs from Other Computers".

Weave seems to work fine so far. You just install the weave plugin, create an account, set the encryption password, and off you go. It's real easy, man.
(go to article | 0 Comments)

I just posted a Django snippet that (ab)uses a decorator to change how you call templates in Dango views. For example, it makes my view code something like this:

####################################
@with_template('friends/index.html')
def friends(request, context, username):
   context['user'] = User.objects.get(username = username)

And the view:

{% extends "base.html" %}
{% block content %}
<h1>{{ user.username }}'s Friends</h1>

I've continued to do a lot of testing and other poking at ZFS-FUSE under Linux, and things are looking pretty good. I had run into a problem with the stable 0.6.0 release when used with lots (13+) of devices with very long names (50-ish character /dev names). However, that seems to be resolved with a very short patch as mentioned in this mailing list thread.

Additionally, zfs-fuse has just been placed into Debian for the Squeeze release (including the patch above). So, before long we should be able to "apt-get zfs-fuse" and have solid access to the joys of ZFS. Though note that this does not include dedup functionality -- there are still some bugs in the base ZFS code to be worked out in that.

I've been running ZFS on my storage server for over a year, but I had found a version that mostly worked (though there were issues with heavy snapshot use). It wasn't until a couple of weeks ago that I finally decided to upgrade. It hasn't been trouble-free -- because of the bug above I ended up having to reload my pool. But considering that I have good backups, I decided I might as well put the recent stable release to the test.

Unfortunately, at the moment one of my test machines has had some sort of hardware failure, and the other I had to swap to different hardware because we needed the chassis that it was on for other testing, and it's not booting in the new chassis. So for the moment, my stress testing is on hold.
(go to article | 2 Comments)

A month or so ago I finally got a system that would correctly install Proxmox VE -- my earlier tests didn't produce much progress. That initial test was done on a virtual machine on my laptop. To make matters worse, my laptop runs 32-bit, so I had to run the test under full QEMU virtualization, which made performance terrible. The amazing thing I found then was that a OpenVZ virtual environment actually worked quite well in those challenging situations.

It did definitely whet my appetite to do more experimentation with Proxmox VE. It has loads of interesting features and I was dieing to try it. It wasn't until recently that I had some spare hardware sitting around that would support KVM virtualization (CPU virtual support). I happened to get several systems that I could spare for some testing.

Read on for my impressions of the new 1.5 release of Proxmox VE and how it compares to VMWare ESXi.
(read more | 0 Comments)

On some systems I have long-running screen sessions, like on one system where I run folding under it, and I've also started running the zfs-fuse daemon under screen. The normal naming causes me to end up with a bunch of screens that I have to hunt through to find the one I want:

guin:~$ screen -x
You may wish for a screen, what do you want?
        13658.pts-13.guin       (Detached)
        13639.pts-13.guin       (Detached)
        13678.pts-13.guin       (Attached)
Type "screen [-d] -r [pid.]tty.host" to resume one of them.
zsh: exit 1     screen -x
guin:~$ 

I started hunting for a way to get some more information about what was on each screen, and found the "-S [sessionname]" option. Just what I need:

root@stow:~# screen -x
There are several suitable screens on:
        20873.pts-4.stow        (Detached)
        20303.zfs       (Detached)
        20224.folding   (Detached)
Type "screen [-d] -r [pid.]tty.host" to resume one of them.
root@stow:~# 

So now I can see which one s the one I started for just whatever (the "pts" one), and I can resume screen sessions with "screen -x folding" or "screen -x pts" to get to the generic one.
(go to article | 1 Comment)

1. What's the coolest Python application, framework or library you have discovered in 2009 ?
   Hard to pick... Sprox I'm dying to try (for putting databases on the web), and Pinax looks very promising for making social sites. Django is really the only new thing I've spent significant time in though, and it is working fine.
2. What new programming technique did you learn in 2009?
   I've been messing around with decorators, and I've really become comfortable with list comprehensions. Because I'm a system administrator, I'm always years behond the curve deploying new features, because they need to work on the last few enterprise releases.
3. What's the name of the open source project you contributed the most in 2009? What did you do?
   I flitter around between proejcts, but the Python memcached module is the big, consistent one. I'm doing maintenance on it, so accepting patches and making new releases.
4. What was the Python blog or website you read the most in 2009?
   I usually skim Coder Who Says Py, it's great content but usually too involved for my typical lunch-time reads. Richard Jones' Log and pyOraGeek usually have small snippets and I always read them when they're up.
5. What are the three top things you want to learn in 2010 ?
   I'd love to write some code in Python 3.0 and get comfortable with it, but obviously still need to really target 2.x -- so 3to2 will be important. I'd like to play with CouchDB and it's replication.

I've also done a fair bit of e-book reading recently. Read below for my thoughts on the Kindle (which I don't have), Gutenberg, and more...
(read more | 0 Comments)