Sean Reifschneider's Journal Recent Entries

A decade ago I built a clock. It's a great clock, it uses a Beta Brite LED sign with a Python program running on a laptop with RS-232 interface to provide NTP-synchronized time, outside temperature read from the Internet, and the ability to telnet to a TCP/IP port and display custom messages.

It's been running quite nicely since I put the OS on flash after the last hard drive died. It's running on an ancient Thinkpad (so ancient that it has an on-board RS-232 port :-), and while it works quite well, it seems like a waste to be using all this space and power to run the sign.

Things like those wall-wart Linux systems got me to thinking about replacing the Thinkpad with a smaller, lower-power device. Read on for more information about what solutions I've found.
(read more | 0 Comments)

One of the tools I always like to have to double-check the network settings when I enter them is something that will take information like "192.168.1.2/26" and provide the other information you will need like the network range, broadcast, etc... In my spare time I've been working on some applications for Android phones including one called "ipcalc" which does this. If you have an Android phone, you can find it in the marketplace, just search for "ipcalc".
(go to article | 0 Comments)

One thing that progress bars don't tell you very well is when something has changed and by how much. You can see the overall progress, but if it's stalled and you look at it, you just can't tell if it's made any progress recently... Unless you watch it VERY closely.

I have an idea on this front... Make the progress-bar be in two colors and when the progress changes draw it in one of the colors. Then as time goes on fade the one color into the other. The overall effect would be similar to those progress-bars that are just a gradient, but with the gradient actually meaning something.

So you could look at it and if it was all the "old" color you'd know it has been stalled for a while with no progress. If it's all the "old" color with a chunk of the "new" color at the end you'd know that it was stalled but got some more information recently. If there are three different chunks of progressively faded color at the end, you'd know the relative age and size of the last 3 updates, etc...

Now somebody just has to implement it. Unless it's already implemented, in which case I've never seen it...
(go to article | 4 Comments)

Just a quick FYI for other people in a similar situation... We were recently testing Multipath under OpenSuSE via an LSI SAS controller that uses the mptsas driver. It would fail over to an alternate path, but would give a kernel oops and then would have to be hard booted before it would fail back. It looks to be a bug in the mptsas driver in the OpenSuSE kernel, and I was able to resolve it by building 2.6.29.4.

I hadn't found mention of this specific issue on google, but I did find that there have been a lot of changes to the mptsas driver. The oopses are gone after the upgrade to this new kernel.
(go to article | 0 Comments)

Last weekend Paul Hummer and I had a sprint on building packages of Python software from PyPI. Paul made good progress on the patch for building Debian packages, it's sorely out of date. I refactored my existing build code to more safely build untrusted packages. Read on for more about it.
(read more | 0 Comments)

Brett Cannon has called PyCon 2009 the "best PyCon ever". I totally agree. Though, as usual, I had a hard time making it to many of the talks (preferring instead the hallway track and one a few occasions getting called in to do networking stuff), the talks are now all up and available for download and watching. I have a bunch of them queued upto check out. Read on if you're interested in more of my impressions of PyCon 2009.
(read more | 2 Comments)

Over dinner tonight a group of us were discussing packaging of Python applications and packages. Many of these discussions happen among developers and users, but you often don't hear about it from the system administration side.

One of the things we brought up was that Python is kind of challenging for System Administrators, particularly in production systems, to handle. This is because Python moves pretty fast. A production-oriented LTS OS release will typically include system packages for an older Python. But developers want to build their programs with a newer version of Python -- usually the newest. This isn't the developers fault, or the system admins, or Python...

Read on for some ideas that were discussed...
(read more | 0 Comments)

Keeping a secondary name server up to date with respect to the list of zones that it needs to slave can be a challenge. Our primary name server has over 500 zones that it handles, and changes are reasonably frequent, say weekly. We currently use cron jobs between all the name servers we deal with that uses a locked-down SSH public key to periodically get the list of zones that the slave servers need to pull, then we rely on AXFR to transfer them.

This system has worked fairly well over the years. Recently I was studying PowerDNS, and saw it's "supermaster" ability in which you tell it what DNS servers to trust and if it receives a NOTIFY message for a zone it doesn't already know about from this server, it will automatically configure itself to slave for this zone and then use AXFR to transfer it.

In other words, it uses the NOTIFY messages your servers are already sending out to configure itself as a slave. This should eliminate the primary issue we've had with the SSH replication, which has been in the infrequent cases where SSH host keys change on various servers, the replication stops until we manually fix it. Read on for more of my thoughts about this configuration.
(read more | 0 Comments)

I've now been running ZFS with FUSE under Linux for around 6 months. I still have my storage server running ZFS, and while it hasn't had too many problems (ZFS has always caused me some level of stability problems), I will say that my storage box is now the only system I have running ZFS. In other words, I've gotten rid of the Solaris-based ZFS machines but did not replace them with FUSE-based Linux systems as I was hoping to. Read on for more of this saga.
(read more | 0 Comments)

Over the last month I've been using a Android Dev Phone -- the unlocked version of the G1. I've written up a fairly detailed review of the Android Dev Phone which covers my impressions of it. Over all, I think it's promising, and usable, but not without problems. See the link here for the full review.
(go to article | 0 Comments)

After an unclean system shutdown I always have these vim tmp files laying around that cause annoying messages to come up when I edit files that I had previously edited while my system went down. These things are spread out all over the file-system. I had a cron job set up that would use locate to hunt them down, but this caused two problems: it required more than a day for the files to be removed, because updatedb had to be updated, then I usually gave it 7 days of grace time in case I was editing something for multiple days...

The worst problem was a subtle bug in this code which caused me to have to recover my home directory from backups. This is why RAID is not a backup plan. :-)

I've recently switched to a new mechanism though, which works much, much better. Read on for more...
(read more | 0 Comments)

It sure would be nice if any of the installers had a "clone partition layout from another drive" option. We often end up setting up multiple drives with similar layouts, either because of RAID or LVM, and using the partitioning GUI can be painful during the install.

We were doing this the other day on a system with 4 drives, and I used something that I discovered in some other automated partitioning work we had recently done to automate the setup. Read on for more details.
(read more | 4 Comments)

Just an update on the Hitachi drive issues (see links to previous related posts if you need more information). Kevin tested a test kernel for CentOS, and that went well so the next CentOS 5 kernel should have that fixed. I also tested the current Hardy kernel, and it is fine.

So, the installers still have issues, but if you install either in the first 120GB or the last space beyond 140GB, then do updates and allocate the remaining space, you should be fine as far as working around it. Of course, if you boot from the CDs for rescue, you may run into this problem. Hopefully, the CentOS 5.3 media when they release it should resolve this. No idea if Hardy is going to roll new install media.
(go to article | 0 Comments)

Tonight at our LUG we had a great presentation about building a large scale LTSP network. As part of this they needed to spread the load out across a number of different machines. But they didn't have the ability to deploy a traditional stand-alone load-balancer.

I mentioned options of using CLUSTERIP or unifying the load-balancer with the application machines, so that a dedicated load-balancer isn't needed. I wanted to give some more information about these options because they aren't as well known as the more traditional methods. Read on for more information.
(read more | 0 Comments)

The other day I had two different clients asking about processing e-mail from a python program. In particular, each e-mail message that comes in gets handed off to be processed by this program. Setting up the mail server to call the program is fairly easy, by configuring local delivery and using the .forward file or similar. However, the program which processes the message needs to do a number of things to do it's job reliably.

I pulled together some pieces of existing code I had to handle these issues. Read on for an example of my code for processing e-mail.
(read more | 0 Comments)

After a month of working with Hitachi on this, and digging through some kernel changes, it looks like the issue I previously reported (part number 0A35415) is a kernel bug.

This seems to be a change in how the firmware handles addressing one specific block on the drive, which other Hitachi drives were more tolerant of. However, this does conform to the ATA specification, so the problem is definitely in the Linux ATA driver.

A fix has been committed to kernel 2.6.27-rc7. However, until this change is rolled into the distributions install media there will be the opportunity for this change to cause grief, particularly for Red Hat Enterprise, CentOS, and Ubuntu LTS users who may not get a re-roll of the install media for quite some time...

Read on for more details and work-arounds.
(read more | 3 Comments)

All the cool kids are doing it.

Instructions: Take a picture of yourself right now. Don't change your clothes, don't fix your hair - just take a picture. Post that picture with NO editing. Post these instructions with the picture.
(go to article | 0 Comments)

Supermicro doesn't seem to have one of those whizzy BIOSes that will update directly from a non-bootable media or Internet or Linux (even with the remote management/KVM cards). In the past, most BIOS updates I've been able to just squeeze into a DOS bootable floppy, but it's been getting increasingly hard. However, the latest systems BIOS file alone is 2.2MB, so I had no hope of getting it on the 1.44MB disc I've been using the past. So, I spent some time and figured out how to make a FreeDOS bootable image. Read on for the details.
(read more | 0 Comments)

I haven't seen this exact situation before, so I wanted to write it down for helping others via google. I was installing Hardy on a system with 9 drives, and the installer was detecting the drive I wanted to have the root partition on as /dev/sdi rather than /dev/sda. The boot drive was properly set in the BIOS. Read on for information on how to fix it.
(read more | 0 Comments)

In short, we started seeing dramatically fewer drive errors after running read/write testing of drives before putting them into production.

As I mentioned in my previous post (linked above), we run fairly long-running tests (around a week with 500GB drives) before using them. However, in my experience this is fairly unusual -- it's rare that I run into others that do similar levels of testing. Even for RAID systems (all the systems we deploy these days are using RAID), reduced errors in production save time and attention.

We started doing these tests because we were finding that maybe one in 5 or 10 drives would report sporadic read errors, but when we replaced the drive and ran a badblocks on the removed drive it was showing no problems.

I theorized that there might be some marginal areas of the discs, perhaps from production, perhaps from shipping, I don't know. These areas of the disc were good enough to be written to, but then after some fairly short time the data there could not be read again because of bit-rot.

Once I started doing the read/write badblocks testing, writing and then verifying the full disc 20 to 40 times, we found that these issues basically stopped showing up in production.
(go to article | 3 Comments)

As many of you who know me are aware, I've been a pretty big advocate or IBM/Hitachi drives for a very long time. I've probably deployed around 500 of them over the years, and had a very low failure rate, and when I have had problems IBM and more recently Hitachi have been extremely helpful. Even the drives that brought about the "Death Star" nick-name I had almost no problems.

However, over the last 45 days we've been trying to work with Hitachi to try to resolve some problems and have had basically zero luck making any progress. We've given Hitachi quite a bit of time to provide some information on this, but so far have basically made no progress with them. I wanted to open up the discussion to other to see if others were seeing similar results, or perhaps it's just the level of testing we do, and nobody else is doing this testing.

In short, we test all drives before we put them into production by using a read/write badblocks test. We've done this for probably 5 to 7 years as part of our standard hardware burn-in. No hard drive goes into our managed hosting facility without going through 4 to 7 days of full read/write testing.

With our most recent batch of 500GB drives (part number 0A35415), the labels say "May 2008", we started seeing errors during the burn in. And I'm not talking about one drive, I'm talking about 22 tested drives out of 2 different 20-packs of drives, tested on 4 different systems with different controllers, all of which did not report errors when testing drives from older batches.

Have you seen similar problems with recent Hitachi drives? What sort of testing do you do of new drives, would you have even noticed? Read on for more details of what we've been seeing, including information on how to do your own testing.
(read more | 3 Comments)

Usually when I insert my Sprint cellular wireless card it shows up on /dev/ttyUSB3, but the other day it showed up on ttyUSB6. When you insert the card it creates 10 ttyUSB files, and I just want my card to work. So, I wrote a small python program (based off code I have previously written for talking to serial devices). Read on for how I solved it.
(read more | 3 Comments)

I've been using enigmail inside Thunderbird for mail for a while now. However, the default setting was such that mutt wasn't auto-detecting the messages as encrypted when I was sending them, because the content-type wasn't set -- programs had to look for the "BEGIN PGP MESSAGE" line.

It took a bit of hunting, but I was finally able to figure out how to change this. Since it's non-obvious, here's what you need to do:

• Click on "Write" to start writing a new message.
• Click on the OpenPGP menu.
• Select the "Default Composition Options" item.
• Select the "Signing/Encryption Options" sub-item.
• Check the "Always use PGP/MIME" box.
• Click "OK".

You can't change this option from the main "Preferences" window for Enigmail, you have to do it from the Composition window. That's what tripped me up.

Encrypted-ly yrs,
Sean
(go to article | 0 Comments)

One of the things I really miss about bash is the help for the "test" command (for shell conditionals). There's the test man page, but for some reason I just always want to type "help test" to see what I can use in shell conditionals. I finally decided to fix this and came up with the following line which can be added to ~/.zshrc:

function help { bash -c "help $@" }