Sean Reifschneider's Journal Entries

Earlier this month we switched our bandwidth accounting system to use new software. Previously we used a kernel module on a system, which was conceptually very cool. But with CentOS 4 going end-of-life, that kernel module doesn't work on CentOS 6 and I wanted to go a different route rather than fix the module.

So earlier this month, after extensive testing in parallel with our normal traffic stats collection software, I flipped the switch and started using the new software. Except that I forgot to re-enable the database update code, so it was only updating the graphs. I had disabled it to prevent double-accounting when running the old and new code in parallel during testing.

But, I log all those updates to syslog. I was able to pull out the logs of all those updates and replay them into the database. Logging of updates is a great safety net.

Which leads me to my next point. The simple code I had written to replay the logs, just was going to do one big commit at the end. I didn't think about how many updates we were talking about, over 2.5 million... I also wasn't writing out status, so I wasn't sure how far along it was. So I eventually killed it after an hour of running, and added code to commit every 1,000 updates and then write out a progress message.

With these changes, it ended up finishing after 18 minutes, giving me progress all the way along. Of course, I could have consolidated the data down in the program and done only a few updates, but 18 minutes was totally acceptable.
(go to article | 0 Comments)

Via Richard who got it from Tarek...

1. What's the coolest Python application, framework or library you have discovered in 2011?

I haven't been using it much lately, but looking back the library I think I was most excited about was bottle a Python micro-framework for building web applications. The thing I like is that it's extremely small, just a single file. For simple things that I might have used a CGI for in the past, bottle is great.

As far as the thing that I use the most on a day-to-day basis, that would probably be cony. Cony is a personal URL-bar command-line, so you can write custom shortcuts to different locations. For example, I have shortcuts like "backup [HOSTNAME]" which searches for the host and takes me to the page on the backup server, or "nagios [CUSTOMER]" which takes me to their nagios page.

2. What new programming technique did you learn in 2011?

Via a recent code retreat (written about previously), I really got into test driven development. Which is great, because our system administration work I like to call "test driven system administration", via Nagios, so it's very comfortable to me. But I hadn't had time to really get comfortable with testing my code until I got this opportunity at code retreat.

3. What's the name of the open source project you contributed the most in 2011? What did you do?

That's a tough one, because I've mostly done a lot of small contributions rather than any large ones other than ones I maintain. So probably the python memcache module or vPostMaster would be at the top. Until, that is, we release the source to our backup system, which I spent a lot of time contributing to this year. Aside: We've finally settled on a name for it: Network Attached Backup.

4. What was the Python blog or website you read the most in 2011?

I don't really follow any Python blogs... Most of my news comes from my friends feeds in facebook or plus. But the number one website I read in 2011 is almost certainly the library documentation. :-)

5. What are the three top things you want to learn in 2012?

I don't really set out goals for what to learn, certainly not a year in advance, because things change too fast. In general I want to continue to improve my business chops.

I'm expecting that in 2012 I'll be able to start programming seriously in Python 3.

6. What are the top software, app or lib you wish someone would write in 2012?

I'd love to have something that did a really good job of bringing together and making searchable all the data that comes at me. Thunderbird can often be pretty good for e-mail, but sometimes it seems like it just misses huge parts of my correspondence for reasons I don't fully understand. But I'd also like web pages I visit in there, photographs, and more...

Thanks Tarek and Richard! So what are your answers?
(go to article | 0 Comments)

Several of us have been running Gnome 3 lately, and have been happy enough with it. One thing I was thinking was that it would be nice if the date/time bar app showed my google calendar. Then Mike found this project that does just that: gnome-shell-google-calendar.

But, I subscribe to 8 other calendars. So whenever we had a company meeting, it would show a bunch of duplicated events, and it was hard to tell my events from others. So I hacked on it this weekend and have pushed a new version up to github: Sean's fork of gnome-shell-google-calendar.

This version de-duplicates events with the same title that start at the same time, and also displays what calendar the event came from. So far it's been working really well for me, but the upstream author is reporting an error so I'm working with them on that at the moment.
(go to article | 0 Comments)

Yesterday Bill Tucker and Matt Rose organized a Code Retreat in Fort Collins. It was "Global Day of Code Retreat", and there were events happening all around the world. The basic idea was that you paired up and for 45 minutes used best practices (as if you had all the time in the world to do it) to work on a programming problem. Then everyone deleted their code, stood up and discussed the previous round. Then you paired up with someone new and started over on the same problem.

It was a great event, extremely rewarding on many levels. Read on for more of my thoughts on the event.
(read more | 2 Comments)

Vim Omnicomplete is extremely useful. We use vim for an internal project and have some cases where I wanted to do a custom Omnicomplete, however I'm not really that comfortable with vim's own scripting language. Looking at the existing completion scripts really wasn't helping much either.

What I really wanted to do was build the "omnifunc" completion function in Python. After a few hours of playing, I was able to come up with exactly that and get my completion code working. Now when I press Control-X Control-O I get a menu of my custom completion items, based on project names. It's wonderful!

So I spent a few more hours cleaning it up and making it into a template that hides all the magic and presents (what I hope is) a dead simple structure that a Python programmer can use to make omnicompletion plugins. I've released this template on github as: vim-omnipy-template
(go to article | 0 Comments)

This weekend I wrote a lot of documentation. I spent around 12 hours on it in total. I knew there was no way I could stand being in LibreOffice for 12 hours, but I also knew that the consumers would prefer a prettier format than just plain text. I've been writing READMEs lately using markdown format, which converts to pretty HTML on github, so I decided to go that direction. Boy did it work well, once I found the magic tool: pandoc! Read on for more details...
(read more | 2 Comments)

One problem that production system administrators have is running systems with older Python versions installed as /usr/bin/python. Worse, you probably have some systems with a newer Python and some with an older Python. For example, in our environment we're managing some systems with Python 2.1 and some with Python 2.6...

And you can't just upgrade /usr/bin/python, because that will likely break many other things on the system, mean that packages that provide modules either have to be rebuilt or are no longer reachable by the Python interpreter, etc...

We've been struggling with this for one of our system maintenance packages, where there is newer Python available on the system, but just doing "#!/usr/bin/env python" will pick up a very old version.

The "ineedpy2" module will either try to find a newer Python version and re-run the currently-running program with that, or you can specify a minimum minor Python 2 version that you need, and only if you are running something older than that will it swap in a newer one. For example, you may have both Python 2.6 and Python 2.4 on a system, but 2.4 is sufficient, so you don't need to do the extra work, but if you are running under 2.3 it will.

For example, you can say "ineedpy2.rerunonlatest()" to get the latest available Python on the system, or "ineedpy2.requireminor(4)" to only re-run the program if you are on 2.3 or older.

I've released this module on github at https://github.com/linsomniac/ineedpy2 and will put it in pypi soon.
(go to article | 2 Comments)

A few weeks ago I had a 1U system that had a RAID card, but we needed to add more network interfaces to. I had the idea of switching it to software RAID, freeing up the only slot for the network card, and it went amazingly smoothly.

The bulk of the system was installed on an LVM, with just a small /boot partition. So what I did was add two drives to the system, and partition them with a small partition for /boot, and the remainder for LVM. These partitions were set up with software RAID-1.

I then added the LVM partition to the existing LVM, and did a "pvmove".

"pvmove" is a great tool that tells LVM to move the blocks from one device to another. It achieves this by first creating a mirror of a block from the source device on the destination, waiting for that mirror to sync up, and then breaking the mirror by removing the source block. After an hour or two, with the system still operating normally, the source RAID array just contained /boot.

Finally, I copied the /boot partition over to the software RAID, rebuilt the initrd to include the software RAID tools, re-ran grub-install on the two new drives.

So with fairly little muss and fuss, and little down-time, I had migrated from a hardware RAID array to software, on 2 new drivers. LVM is a very wonderful thing!
(go to article | 1 Comment)

As part of system automation work, the other day we were talking about changing the postfix configurations. Mention was made of using sed, but I brought out "postconf -e", a tool that postfix provides to allow modifying the configuration. Particularly useful for more complicated entries which can span multiple lines and a simple sed script wouldn't do the right thing.

You run it as: "postconf -e name=value"

This will modify the configuration to change the setting "name" so that is assigned the value "value". Of course, proper quoting is necessary if you are using special characters.

I also frequently use "postconf" to have it tell me what the current configuration is (because sometimes looking at the file it isn't obvious) and "postconf -d" to tell me what the default configuration values are.

Definitely useful tools to have in the tool-box.
(go to article | 0 Comments)

I recently had a kernel bug that caused a process to hang in D (disc wait) status. I decided to do a reboot, but thought I'd try upgrading the kernel before I did that. The kernel upgrade also got stuck in "D" while trying to do a depmod, so I finally did a reboot, and had to hard power down.

During the reboot, it would get to where it would say "Begin: Running /scripts/init-bottom", and hang there. I searched around and didn't find much, so I figured I'd document the solution here.

I booted into rescue mode (off the initial media, I wasn't able to boot with "init=/bin/sh"), and then got a shell in the root file-system. I did a "dpkg --configure -a", and then rebooted and that solved the problem.

I found this issue by trying to run "apt-get update" in the rescue environment and it helpfully told me that it had been interrupted and I needed to run the above command. :-)
(go to article | 2 Comments)

Last month I had wanted to try to get some real monitoring of a bunch of my personal systems. I spent weeks dabbling with setting up a serious monitoring system (tried Zabbix, Zenoss, nagios, and Opsview), but I was making little if any progress with those. Largely that was related to running a RHEL 6 derivative and running into compatibility problems...

After spending around a day of effort on them, I decided to roll my own. My needs were modest, I wanted to be able to run some nagios check scripts and get an e-mail if checks failed continually for 15 minutes. But not get e-mails every 15 minutes if they were down, and only alert if they failed several times.

Read after the fold if you're interested in my solution...
(read more | 4 Comments)

The error: "pycurl.error: (26, 'failed creating formpost data')" was a bit tricky to look up, but I was able to find a report on the curl mailing list that this error indicates that the file up are trying to upload via a form upload does not exist.

Sure enough, when I checked the value I was passing as the name of the file to upload, I was using the wrong name -- a name that didn't exist. So if you run into this error, verify that the filename you are passing does exist.

If this answer helped you, please post a link to this page so that it shows up high in the results. I had to do a lot of digging to get the answer.

In general I've been pretty happy with pycurl. I had thought urllib would let me upload a file, but it required a lot of extra code to do. pycurl isn't that well documented, but I was able to find examples that helped me get it done, and I've been happy with curl from the command-line, so I figured I'd try it. Other than having to build pycurl on one CentOS 4 box, and this information-free error message, it's worked quite well.
(go to article | 0 Comments)

I've been following this Intel forum thread on failures of the 320 series of SSD drives for a while now after having the Tom's Hardware article on it pointed out to me by our friends at Pattern Review.

So far it's been fairly full of speculation with little concrete information available... A week ago Intel responded that firmware fixes were being worked on.

Read on for more details about this, including the failure I've experienced.
(read more | 0 Comments)

Today I was trying to figure out how to check the expiration time of an SMTP certificate, to verify that after installing a new certificate the mail server was picking up the right certificate. What I came up with is:

SERVERNAME=mail.example.com
printf 'quit\n' | openssl s_client -connect $SERVERNAME:25 -starttls smtp | openssl x509 -enddate -noout

Just for completeness sake, checking it on a web server can be done with:

SERVERNAME=www.example.com
printf 'GET /\n\n' | openssl s_client -connect $SERVERNAME:443 | openssl x509 -enddate -noout

Stephen Warren tonight noticed that one of his backups of his storage server was transferring a file that it shouldn't have been. He was running with the "-c" option to rsync to do a full checksum check, so apparently this file had changed.

Stephen provided these instructions on how to compare the two copies of a file on a RAID array, to verify that they are the same, or in his case different. He wanted me to post them so he could find them again when he has to search google for them in the future. :-)

Read after the fold for the details of how to read from a single drive on a RAID-1 array.
(read more | 2 Comments)

Lately, Evelyn has come up with a very quotable, well, quote:

"Data is heavy." -- Evelyn Mitchell, 2011

We are frequently dealing with the storage, organization, and access of all the data we are accumulating. Part of this is that we can just store so much data, 3TB in a single 3.5" drive is plenty. But, we get less than 100 opportunities to access that per second (10+ms average access time).

Then we have to back it up, preferably off-site, and regularly re-read it to verify that it's good. And if the need to recover from backups arises, are you prepared for how long it takes to recover billions of files?

We're currently working with one client who wants the ability to roll back the entire system to a previous backup within an hour. However, one directory they have takes several hours just to run a "du" on...

I really wish Linux had some sort of hierarchical storage sub-system, but that just doesn't seem to be on anyone's radar. Something that could manage a pool of huge but slow hard drives, smaller but much faster hard drives, SSDs and non-volatile DRAM, automatically migrating chunks of data between the most appropriate storage technology, would be great.

Adaptec is taking a step in that direction with "MaxCache" and "Hybrid' RAID controllers, but both of these are strictly read-caching mechanisms. We also have the "flashcache" kernel driver, which can offer write-back caching to an SSD, but cannot survive a power failure or kernel crash without data corruption...

Meanwhile we just keep generating and storing more and more data.

So, yeah, data is heavy.
(go to article | 2 Comments)

On my desktop, where I have plenty of screen real-estate, whenever I start writing an e-mail, the window comes up all the way on the right side of my right display. Which is not at all where I want it, that ends up being at like my "two o'clock" position. This is not an issue on my laptop, because of the small screen.

Continue reading for more information on how I solved this.
(read more | 0 Comments)

I'd like to propose that we come up with some way of standardizing a way to make excluding chrooted "/proc" mount-points easier in backups. I believe that backing up more is better than backing up less, and so I prefer to not stop on a mount-point (because that often causes surprise when a new file-system is added and backups don't pick it up). But systems are just littered with "/proc" mounts for chrooted environments (postfix, ntp, named, etc...).

It would be nice if /proc were named something that made it a bit more unique, like "/proc-virtual-namespace", so I could feel like I can exclude descending into that directory by name. Excluding "proc" seems too likely to collide with legitimate files.

Sure would be nice if "/proc" included something like an ".rsync-exclude" file taht spit out an exclude rule for that directory. Or if rsync had exclude rules based on file-system type. Beyond that, I the best option I have come up with is to enable "one-filesystem", but then dynamically generate the list of directories based on the file-system types.
(go to article | 0 Comments)

I got one of those CD-ROM bay batteries, and it confuses the power LED. When charging, it will flash orange until the second battery is more than 15% (OMG, one battery is at 0%). And it doesn't start charging that battery until the first is at 100%.

It charges to 90% quickly, but the last 10% takes quite a long time. Kevin thinks I can probably swap which battery is being charged, maybe a little script that flips the batteries when it's charging and the primary hits 80%, then swap to top both of them up the last little bit.

So currently it takes an hour to charge #0 to 85%, another hour to charge it to 100%, an hour to charge #1 to 85%, another hour to charge it to 100%...

My last battery didn't even last a year before it was only holding around 90 minutes charge. So it's quite the refreshing change to go from 90 minutes of battery life to, well, I'm not exactly sure... I'm guessing with them totally fresh, something close to 6 hours. But I haven't even come close to running them both completely down yet.
(go to article | 0 Comments)

I never looked into it, until now I haven't specifically bought a USB hub. But recently I've been using the USB hub that's built into my monitor, but it powers off when the monitor does, so I decided to add some dedicated USB hubs. After doing some research, I came across the designation "Multi-TT", which seems to be the thing to get.

The difference, as I understand it, is basically like the difference between a network hub and a switch (though I'm sure that stretches the analogy). They do push the price up from $10 to $30, but that was within my price range.

The trick is that this feature is either not well advertised or something. Where I bought it there was only one USB hub listing "multi-tt" when I searched. I've been using them for a couple of days (I got 2, partly to have more bandwidth if I hook up multiple devices, mostly to have ports for more than one device (my laptop, at one point I had my game box by the computer)...
(go to article | 0 Comments)

Earlier today I ran into an issue where a software package was failing with "too many open files". I made several attempts at changing this using various settings in "/etc/security/limits.conf", but they just weren't taking hold.

After some digging, I found a Debian and Ubuntu set of bugs such as "'nofile' limit set in limits.conf not applied". However, those reports indicated that they could decrease the limit just not increase it. I couldn't change it up or down, despite verifying that PAM was configured to pull in the limits module for ssh, su, and cron.

I ended up doing a work-around, which was to start the process from a root shell after doing "ulimit -Sn 10240; ulimit -Hn 10240", and calling the process as "su -c 'bash -c script' username". This caused it to inherit the root shell limits. Since the end goal is to start up the process via an init script, this should work out fine.
(go to article | 2 Comments)

I've been having a heck of a time trying to make use of various audio devices on my Linux system (in this case, Ubuntu Maverick). The biggest problem seems to be that there are many devices (video card, audio card, bluetooth, webcam, USB headset) and I haven't been able to pick the device I want.

After a bunch of digging, I've finally been able to figure it out. Read on for information about how to select the input and output devices you want.
(read more | 2 Comments)

We did an upgrade of our CLUSTERIP based load-balanced DNS servers, and it started spewing the message "CLUSTERIP: no conntrack error". It took some digging, but I eventually tracked it down. To fix it you need to insert a rule with "-m state --state INVALID -j DROP".

The message is an indication that a packet that doesn't match an existing conntrack connection is received. So, just add a DROP rule.
(go to article | 0 Comments)

PyCon is run by volunteers, like myself. If you are planning on going, and can help the night before the tutorials or the main conference, and are interested in networking, please consider signing up to help with the network setup. A lot of the work is simply laying out cables and gear, taping down runs into the audience, and network testing and debugging.

If you are interested, please sign up on the PyCon 2011 Volunteer Sign-up Page (for networking and/or other sections). Thanks!
(go to article | 0 Comments)

If you are planning to attend PyCon 2011 and use the network, please read this. And if you are thinking of going and haven't signed up, please do so soon, it is expected to sell out. In a good way.

Executive Summary: Please bring 5.2GHz gear if at all possible (A.K.A. 802.11a). There will be a few hundred wired ports but they are highly concentrated. Please leave your 100+mW cards at home. Read this full message for all the details.
(read more | 2 Comments)