Recent Entries

A decade ago I built a clock. It's a great clock, it uses a Beta Brite LED sign with a Python program running on a laptop with RS-232 interface to provide NTP-synchronized time, outside temperature read from the Internet, and the ability to telnet to a TCP/IP port and display custom messages.

It's been running quite nicely since I put the OS on flash after the last hard drive died. It's running on an ancient Thinkpad (so ancient that it has an on-board RS-232 port :-), and while it works quite well, it seems like a waste to be using all this space and power to run the sign.

Things like those wall-wart Linux systems got me to thinking about replacing the Thinkpad with a smaller, lower-power device. Read on for more information about what solutions I've found.
(read more | 0 Comments)

I've been busy and then out on vacation and then busy again, but there are a bunch of short Fedora notes I thought I would mention for those interested. Read on for more...

One of the tools I always like to have to double-check the network settings when I enter them is something that will take information like "192.168.1.2/26" and provide the other information you will need like the network range, broadcast, etc... In my spare time I've been working on some applications for Android phones including one called "ipcalc" which does this. If you have an Android phone, you can find it in the marketplace, just search for "ipcalc".
(go to article | 0 Comments)

One thing that progress bars don't tell you very well is when something has changed and by how much. You can see the overall progress, but if it's stalled and you look at it, you just can't tell if it's made any progress recently... Unless you watch it VERY closely.

I have an idea on this front... Make the progress-bar be in two colors and when the progress changes draw it in one of the colors. Then as time goes on fade the one color into the other. The overall effect would be similar to those progress-bars that are just a gradient, but with the gradient actually meaning something.

So you could look at it and if it was all the "old" color you'd know it has been stalled for a while with no progress. If it's all the "old" color with a chunk of the "new" color at the end you'd know that it was stalled but got some more information recently. If there are three different chunks of progressively faded color at the end, you'd know the relative age and size of the last 3 updates, etc...

Now somebody just has to implement it. Unless it's already implemented, in which case I've never seen it...
(go to article | 4 Comments)

Just a quick FYI for other people in a similar situation... We were recently testing Multipath under OpenSuSE via an LSI SAS controller that uses the mptsas driver. It would fail over to an alternate path, but would give a kernel oops and then would have to be hard booted before it would fail back. It looks to be a bug in the mptsas driver in the OpenSuSE kernel, and I was able to resolve it by building 2.6.29.4.

I hadn't found mention of this specific issue on google, but I did find that there have been a lot of changes to the mptsas driver. The oopses are gone after the upgrade to this new kernel.
(go to article | 0 Comments)

Last weekend Paul Hummer and I had a sprint on building packages of Python software from PyPI. Paul made good progress on the patch for building Debian packages, it's sorely out of date. I refactored my existing build code to more safely build untrusted packages. Read on for more about it.
(read more | 0 Comments)

Just a quick note that I will be giving a class on libvirt in the Fedora IRC Classroom tomorrow at 19:00 (UTC time). Lots of good classes will be offered. See the above link for more info.

Brett Cannon has called PyCon 2009 the "best PyCon ever". I totally agree. Though, as usual, I had a hard time making it to many of the talks (preferring instead the hallway track and one a few occasions getting called in to do networking stuff), the talks are now all up and available for download and watching. I have a bunch of them queued upto check out. Read on if you're interested in more of my impressions of PyCon 2009.
(read more | 2 Comments)

Over dinner tonight a group of us were discussing packaging of Python applications and packages. Many of these discussions happen among developers and users, but you often don't hear about it from the system administration side.

One of the things we brought up was that Python is kind of challenging for System Administrators, particularly in production systems, to handle. This is because Python moves pretty fast. A production-oriented LTS OS release will typically include system packages for an older Python. But developers want to build their programs with a newer version of Python -- usually the newest. This isn't the developers fault, or the system admins, or Python...

Read on for some ideas that were discussed...
(read more | 0 Comments)

Keeping a secondary name server up to date with respect to the list of zones that it needs to slave can be a challenge. Our primary name server has over 500 zones that it handles, and changes are reasonably frequent, say weekly. We currently use cron jobs between all the name servers we deal with that uses a locked-down SSH public key to periodically get the list of zones that the slave servers need to pull, then we rely on AXFR to transfer them.

This system has worked fairly well over the years. Recently I was studying PowerDNS, and saw it's "supermaster" ability in which you tell it what DNS servers to trust and if it receives a NOTIFY message for a zone it doesn't already know about from this server, it will automatically configure itself to slave for this zone and then use AXFR to transfer it.

In other words, it uses the NOTIFY messages your servers are already sending out to configure itself as a slave. This should eliminate the primary issue we've had with the SSH replication, which has been in the infrequent cases where SSH host keys change on various servers, the replication stops until we manually fix it. Read on for more of my thoughts about this configuration.
(read more | 0 Comments)

Having been running BackupPC for the last several months, we wanted to hand off some of the periodic auditing that we do to a non-admin user. BackupPC allows for host-specific users that have full access to a specific host or hosts but there's currently no facility to allow for a user who can see data on each host in a read-only fashion. Read on for how we implemented this interface.

We use several Dell PowerConnect 5324 switches in our network infrastructure and for the most part, they've worked fantastically. As part of trouble-shooting a network issue earlier this year, we updated the firmware version to 2.0.1.3. Since then, we noticed an issue where our network monitoring system would become sporadically unreachable for polling the management interface on several, but not all of the upgraded switches.
(read more | 0 Comments)

Late last week I decided to take a closer look at midori , The Xfce web browser.

Short summary: I am now using midori as my day to day web browser. Read on for more detailed ups and downs.

Looks like I have managed to get talked into speaking to two of the local Linux Users groups next week on Fedora: nclug and blug.

I thought I would go over the upcoming Fedora 11 Features list and then talk a bit about some of the more interesting discussions that have been happening in Fedora development, and then finally talk about where people can join the community.

Hopefully there will be good turnouts on both meetings. If anyone has any suggestions for things they would like me to talk about, feel free to leave a comment here, email me, catch me on IRC, or start a thread on the blug or nclug mailing lists.

I've been really meaning to write up some thoughts from the (now not so) recent Fudcon 11. Read on for some random thoughts about this delightful event.

I've now been running ZFS with FUSE under Linux for around 6 months. I still have my storage server running ZFS, and while it hasn't had too many problems (ZFS has always caused me some level of stability problems), I will say that my storage box is now the only system I have running ZFS. In other words, I've gotten rid of the Solaris-based ZFS machines but did not replace them with FUSE-based Linux systems as I was hoping to. Read on for more of this saga.
(read more | 0 Comments)

Over the last month I've been using a Android Dev Phone -- the unlocked version of the G1. I've written up a fairly detailed review of the Android Dev Phone which covers my impressions of it. Over all, I think it's promising, and usable, but not without problems. See the link here for the full review.
(go to article | 0 Comments)

After an unclean system shutdown I always have these vim tmp files laying around that cause annoying messages to come up when I edit files that I had previously edited while my system went down. These things are spread out all over the file-system. I had a cron job set up that would use locate to hunt them down, but this caused two problems: it required more than a day for the files to be removed, because updatedb had to be updated, then I usually gave it 7 days of grace time in case I was editing something for multiple days...

The worst problem was a subtle bug in this code which caused me to have to recover my home directory from backups. This is why RAID is not a backup plan. :-)

I've recently switched to a new mechanism though, which works much, much better. Read on for more...
(read more | 0 Comments)

It sure would be nice if any of the installers had a "clone partition layout from another drive" option. We often end up setting up multiple drives with similar layouts, either because of RAID or LVM, and using the partitioning GUI can be painful during the install.

We were doing this the other day on a system with 4 drives, and I used something that I discovered in some other automated partitioning work we had recently done to automate the setup. Read on for more details.
(read more | 4 Comments)

Just an update on the Hitachi drive issues (see links to previous related posts if you need more information). Kevin tested a test kernel for CentOS, and that went well so the next CentOS 5 kernel should have that fixed. I also tested the current Hardy kernel, and it is fine.

So, the installers still have issues, but if you install either in the first 120GB or the last space beyond 140GB, then do updates and allocate the remaining space, you should be fine as far as working around it. Of course, if you boot from the CDs for rescue, you may run into this problem. Hopefully, the CentOS 5.3 media when they release it should resolve this. No idea if Hardy is going to roll new install media.
(go to article | 0 Comments)

Tonight at our LUG we had a great presentation about building a large scale LTSP network. As part of this they needed to spread the load out across a number of different machines. But they didn't have the ability to deploy a traditional stand-alone load-balancer.

I mentioned options of using CLUSTERIP or unifying the load-balancer with the application machines, so that a dedicated load-balancer isn't needed. I wanted to give some more information about these options because they aren't as well known as the more traditional methods. Read on for more information.
(read more | 0 Comments)

Just a quick post to note a few happenings in Fedora land for those interested:

This weekend will be the first set of IRC Classroom sessions in #fedora-classroom on irc.freenode.net. Take a look at https://fedoraproject.org/wiki/Communicate/IRC/Classroom for a list of classes and times. I'm teaching one on Firewall Basics on Sunday. Hopefully we will get a good turnout. Students (and Teachers) welcome.

It's also election season in Fedora land. There are 4 groups holding elections in December. See: http://fedoraproject.org/wiki/Elections for more info. In particular I would like to urge those folks that have disliked recent FESCo decisions to run for FESCo or perhaps see about putting forward someone who shares their views.

Finally, Fedora 10 is just around the corner. The Preview Release just went out this week. Testing is very welcome. I've upgraded my laptop from Fedora 9 with preupgrade and it went very smoothly. Congrats on all the great work for this release everyone!

The other day I had two different clients asking about processing e-mail from a python program. In particular, each e-mail message that comes in gets handed off to be processed by this program. Setting up the mail server to call the program is fairly easy, by configuring local delivery and using the .forward file or similar. However, the program which processes the message needs to do a number of things to do it's job reliably.

I pulled together some pieces of existing code I had to handle these issues. Read on for an example of my code for processing e-mail.
(read more | 0 Comments)

After a month of working with Hitachi on this, and digging through some kernel changes, it looks like the issue I previously reported (part number 0A35415) is a kernel bug.

This seems to be a change in how the firmware handles addressing one specific block on the drive, which other Hitachi drives were more tolerant of. However, this does conform to the ATA specification, so the problem is definitely in the Linux ATA driver.

A fix has been committed to kernel 2.6.27-rc7. However, until this change is rolled into the distributions install media there will be the opportunity for this change to cause grief, particularly for Red Hat Enterprise, CentOS, and Ubuntu LTS users who may not get a re-roll of the install media for quite some time...

Read on for more details and work-arounds.
(read more | 3 Comments)