Sean Reifschneider's Journal Recent Entries
Below is a summary of the most recent journal entries by this user. A full index of all entries is also available.Also available as: RSS
Sunday June 28, at 15:37
Subject: Serial to Ethernet adapters.
Keywords:
Ethernet, RS-232, Technical
A decade ago I built a clock. It's a great clock, it uses a Beta
Brite LED sign with a Python program running on a laptop with RS-232
interface to provide NTP-synchronized time, outside temperature read from
the Internet, and the ability to telnet to a TCP/IP port and display custom
messages.
It's been running quite nicely since I put the OS on flash after the
last hard drive died. It's running on an ancient Thinkpad (so ancient that
it has an on-board RS-232 port :-), and while it works quite well, it seems
like a waste to be using all this space and power to run the sign.
Things like those wall-wart Linux systems got me to thinking about
replacing the Thinkpad with a smaller, lower-power device. Read on for
more information about what solutions I've found.
(read more | 0 Comments)
(read more | 0 Comments)
Friday June 05, at 20:43
Subject: IP Address Calculator for Android Phones
Keywords:
Android, IP
One of the tools I always like to have to double-check the network
settings when I enter them is something that will take information like
"192.168.1.2/26" and provide the other information you will need like the
network range, broadcast, etc... In my spare time I've been working on
some applications for Android phones including one called "ipcalc" which
does this. If you have an Android phone, you can find it in the
marketplace, just search for "ipcalc".
(go to article | 0 Comments)
(go to article | 0 Comments)
Monday June 01, at 19:00
Subject: More information about change in a progress bar.
Keywords:
GUI, Ideas
One thing that progress bars don't tell you very well is when
something has changed and by how much. You can see the overall progress,
but if it's stalled and you look at it, you just can't tell if it's made
any progress recently... Unless you watch it VERY closely.
I have an idea on this front... Make the progress-bar be in two
colors and when the progress changes draw it in one of the colors. Then
as time goes on fade the one color into the other. The overall effect
would be similar to those progress-bars that are just a gradient, but with
the gradient actually meaning something.
So you could look at it and if it was all the "old" color you'd know
it has been stalled for a while with no progress. If it's all the "old"
color with a chunk of the "new" color at the end you'd know that it
was stalled but got some more information recently. If there are three
different chunks of progressively faded color at the end, you'd know the
relative age and size of the last 3 updates, etc...
Now somebody just has to implement it. Unless it's already
implemented, in which case I've never seen it...
(go to article | 4 Comments)
(go to article | 4 Comments)
Sunday May 31, at 13:55
Subject: Note: OpenSuSE 11.1 with mptsas and Multipath issues.
Keywords:
LSI, Multipath, OpenSuSE, SAS
Just a quick FYI for other people in a similar situation... We were
recently testing Multipath under OpenSuSE via an LSI SAS controller that
uses the mptsas driver. It would fail over to an alternate path, but would
give a kernel oops and then would have to be hard booted before it would
fail back. It looks to be a bug in the mptsas driver in the OpenSuSE
kernel, and I was able to resolve it by building 2.6.29.4.
I hadn't found mention of this specific issue on google, but I did
find that there have been a lot of changes to the mptsas driver. The
oopses are gone after the upgrade to this new kernel.
(go to article | 0 Comments)
(go to article | 0 Comments)
Saturday May 09, at 15:22
Subject: Using LVM snapshots for Python Package build chroots.
Keywords:
LVM, Python, Snapshots
Last weekend Paul Hummer and I had a sprint on building packages of
Python software from PyPI. Paul made good progress on the patch for
building Debian packages, it's sorely out of date. I refactored my
existing build code to more safely build untrusted packages. Read on for
more about it.
(read more | 0 Comments)
(read more | 0 Comments)
Sunday April 05, at 14:26
Subject: Review of PyCon 2009
Keywords:
PyCon
Brett Cannon has called PyCon 2009 the "best PyCon ever". I totally
agree. Though, as usual, I had a hard time making it to many of the talks
(preferring instead the hallway track and one a few occasions getting
called in to do networking stuff), the talks are now all up and
available for download and watching. I have a bunch of them queued upto
check out. Read on if you're interested in more of my impressions of
PyCon 2009.
(read more | 2 Comments)
(read more | 2 Comments)
Wednesday March 25, at 23:54
Subject: Python Packaging for Developers and System Admins
Keywords:
Packaging, Python
Over dinner tonight a group of us were discussing packaging of Python
applications and packages. Many of these discussions happen among
developers and users, but you often don't hear about it from the system
administration side.
One of the things we brought up was that Python is kind of challenging
for System Administrators, particularly in production systems, to handle.
This is because Python moves pretty fast. A production-oriented LTS OS
release will typically include system packages for an older Python. But
developers want to build their programs with a newer version of Python --
usually the newest. This isn't the developers fault, or the system admins,
or Python...
Read on for some ideas that were discussed...
(read more | 0 Comments)
(read more | 0 Comments)
Saturday March 21, at 14:48
Subject: Automatic name server replication with PowerDNS.
Keywords:
DNS, Technical
Keeping a secondary name server up to date with respect to the list of
zones that it needs to slave can be a challenge. Our primary name server
has over 500 zones that it handles, and changes are reasonably frequent,
say weekly. We currently use cron jobs between all the name servers we
deal with that uses a locked-down SSH public key to periodically get the
list of zones that the slave servers need to pull, then we rely on AXFR to
transfer them.
This system has worked fairly well over the years. Recently I was
studying PowerDNS, and saw it's "supermaster" ability in which you tell it
what DNS servers to trust and if it receives a NOTIFY message for a zone it
doesn't already know about from this server, it will automatically
configure itself to slave for this zone and then use AXFR to transfer it.
In other words, it uses the NOTIFY messages your servers are already
sending out to configure itself as a slave. This should eliminate the
primary issue we've had with the SSH replication, which has been in the
infrequent cases where SSH host keys change on various servers, the
replication stops until we manually fix it. Read on for more of my
thoughts about this configuration.
(read more | 0 Comments)
(read more | 0 Comments)
Wednesday March 18, at 13:49
Subject: See you at PyCon 2009?
Keywords:
Just wanted to let everyone know that I'm going to be at PyCon and feel
free to track me down and say "Hi". I'll be arriving the Monday before the
tutorials, and seeing up the wireless networks on Tuesday so they're ready
for Wednesday. If anyone wants to help with that and will be around
early... Hint hint. :-) Hope to see you there.
(go to article | 0 Comments)
(go to article | 0 Comments)
Saturday January 24, at 19:38
Subject: 6 Months of ZFS in Linux
Keywords:
Technical, ZFS
I've now been running ZFS with FUSE under Linux for around 6 months.
I still have my storage server running ZFS, and while it hasn't had too
many problems (ZFS has always caused me some level of stability problems),
I will say that my storage box is now the only system I have running
ZFS. In other words, I've gotten rid of the Solaris-based ZFS machines but
did not replace them with FUSE-based Linux systems as I was hoping to.
Read on for more of this saga.
(read more | 0 Comments)
(read more | 0 Comments)
Wednesday January 21, at 05:16
Subject: Google Android Dev Phone Review
Keywords:
Android, Review
Over the last month I've been using a Android Dev Phone -- the
unlocked version of the G1. I've written up a fairly detailed review of the
Android Dev Phone which covers my impressions of it. Over all, I think
it's promising, and usable, but not without problems. See the link here
for the full review.
(go to article | 0 Comments)
(go to article | 0 Comments)
Tuesday December 23, 2008 at 17:42
Subject: Cleaning up vim tmp files on boot.
Keywords:
Technical, vim
After an unclean system shutdown I always have these vim tmp files
laying around that cause annoying messages to come up when I edit files
that I had previously edited while my system went down. These things are
spread out all over the file-system. I had a cron job set up that would
use locate to hunt them down, but this caused two problems: it required
more than a day for the files to be removed, because updatedb had to be
updated, then I usually gave it 7 days of grace time in case I was editing
something for multiple days...
The worst problem was a subtle bug in this code which caused me to
have to recover my home directory from backups. This is why RAID is not a
backup plan. :-)
I've recently switched to a new mechanism though, which works much,
much better. Read on for more...
(read more | 0 Comments)
(read more | 0 Comments)
Friday December 05, 2008 at 11:10
Subject: Automating hard drive partitioning.
Keywords:
fdisk, Technical
It sure would be nice if any of the installers had a "clone partition
layout from another drive" option. We often end up setting up multiple
drives with similar layouts, either because of RAID or LVM, and using the
partitioning GUI can be painful during the install.
We were doing this the other day on a system with 4 drives, and I used
something that I discovered in some other automated partitioning work we
had recently done to automate the setup. Read on for more details.
(read more | 4 Comments)
(read more | 4 Comments)
Friday December 05, 2008 at 10:56
Subject: Current state of the Hitachi Hard Drives
Keywords:
Hard Drives, Hitachi, Kernel
Just an update on the Hitachi drive issues (see links to previous
related posts if you need more information). Kevin tested a test kernel
for CentOS, and that went well so the next CentOS 5 kernel should have that
fixed. I also tested the current Hardy kernel, and it is fine.
So, the installers still have issues, but if you install either in the
first 120GB or the last space beyond 140GB, then do updates and allocate
the remaining space, you should be fine as far as working around it. Of
course, if you boot from the CDs for rescue, you may run into this problem.
Hopefully, the CentOS 5.3 media when they release it should resolve this.
No idea if Hardy is going to roll new install media.
(go to article | 0 Comments)
(go to article | 0 Comments)
Tuesday November 11, 2008 at 22:25
Subject: Other load-balancing options...
Keywords:
Load-balancing, Technical
Tonight at our LUG we had
a great presentation about building a large scale LTSP network. As part of this they
needed to spread the load out across a number of different machines. But
they didn't have the ability to deploy a traditional stand-alone
load-balancer.
I mentioned options of using CLUSTERIP or unifying the load-balancer
with the application machines, so that a dedicated load-balancer isn't
needed. I wanted to give some more information about these options because
they aren't as well known as the more traditional methods. Read on for
more information.
(read more | 0 Comments)
(read more | 0 Comments)
Thursday October 23, 2008 at 22:49
Subject: So you want to process e-mail from Python?
Keywords:
E-mail, Python, Technical
The other day I had two different clients asking about processing e-mail
from a python program. In particular, each e-mail message that comes in
gets handed off to be processed by this program. Setting up the mail
server to call the program is fairly easy, by configuring local delivery
and using the .forward file or similar. However, the program which
processes the message needs to do a number of things to do it's job
reliably.
I pulled together some pieces of existing code I had to handle these
issues. Read on for an example of my code for processing e-mail.
(read more | 0 Comments)
(read more | 0 Comments)
Wednesday October 08, 2008 at 18:57
Subject: Follow up to the Hitachi issues.
Keywords:
Hard Drives, Hitachi, Kernel
After a month of working with Hitachi on this, and digging through
some kernel changes, it looks like the issue I previously reported (part
number 0A35415) is a kernel bug.
This seems to be a change in how the firmware handles addressing one
specific block on the drive, which other Hitachi drives were more tolerant
of. However, this does conform to the ATA specification, so the problem is
definitely in the Linux ATA driver.
A fix has been committed to kernel 2.6.27-rc7. However, until this
change is rolled into the distributions install media there will be the
opportunity for this change to cause grief, particularly for Red Hat
Enterprise, CentOS, and Ubuntu LTS users who may not get a re-roll of the
install media for quite some time...
Read on for more details and work-arounds.
(read more | 3 Comments)
(read more | 3 Comments)
Sunday September 21, 2008 at 23:07
Subject: Photo meme continued.
Keywords:
Photos
All the
cool
kids
are doing it.
Instructions: Take a picture of yourself right now. Don't change your
clothes, don't fix your hair - just take a picture. Post that picture with
NO editing. Post these instructions with the picture.
(go to article | 0 Comments)
Instructions: Take a picture of yourself right now. Don't change your
clothes, don't fix your hair - just take a picture. Post that picture with
NO editing. Post these instructions with the picture.(go to article | 0 Comments)
Sunday September 21, 2008 at 00:01
Subject: Using FreeDOS CD for BIOS updates.
Keywords:
BIOS, FreeDOS, Technical
Supermicro doesn't seem to have one of those whizzy BIOSes that will
update directly from a non-bootable media or Internet or Linux (even with
the remote management/KVM cards). In the past, most BIOS updates I've been
able to just squeeze into a DOS bootable floppy, but it's been getting
increasingly hard. However, the latest systems BIOS file alone is 2.2MB,
so I had no hope of getting it on the 1.44MB disc I've been using the past.
So, I spent some time and figured out how to make a FreeDOS bootable image.
Read on for the details.
(read more | 0 Comments)
(read more | 0 Comments)
Thursday September 18, 2008 at 02:38
Subject: Installing Hardy on many hard drives.
Keywords:
GRUB, Hardy, Technical
I haven't seen this exact situation before, so I wanted to write it
down for helping others via google. I was installing Hardy on a system
with 9 drives, and the installer was detecting the drive I wanted to have
the root partition on as /dev/sdi rather than /dev/sda. The boot drive was
properly set in the BIOS. Read on for information on how to fix it.
(read more | 0 Comments)
(read more | 0 Comments)
Wednesday September 17, 2008 at 00:00
Subject: Why do we test all our drives?
Keywords:
Hard drives, Technical, Testing
In short, we started seeing dramatically fewer drive errors after
running read/write testing of drives before putting them into production.
As I mentioned in my previous post (linked above), we run fairly
long-running tests (around a week with 500GB drives) before using them.
However, in my experience this is fairly unusual -- it's rare that I
run into others that do similar levels of testing. Even for RAID systems
(all the systems we deploy these days are using RAID), reduced errors in
production save time and attention.
We started doing these tests because we were finding that maybe one in
5 or 10 drives would report sporadic read errors, but when we replaced the
drive and ran a badblocks on the removed drive it was showing no problems.
I theorized that there might be some marginal areas of the discs,
perhaps from production, perhaps from shipping, I don't know. These areas
of the disc were good enough to be written to, but then after some fairly
short time the data there could not be read again because of bit-rot.
Once I started doing the read/write badblocks testing, writing and
then verifying the full disc 20 to 40 times, we found that these issues
basically stopped showing up in production.
(go to article | 3 Comments)
(go to article | 3 Comments)
Tuesday September 16, 2008 at 23:22
Subject: What's the deal with Hitachi drives lately?
Keywords:
Hard drives, Hitachi, Technical
As many of you who know me are aware, I've been a pretty big advocate
or IBM/Hitachi drives for a very long time. I've probably deployed around
500 of them over the years, and had a very low failure rate, and when I
have had problems IBM and more recently Hitachi have been extremely
helpful. Even the drives that brought about the "Death Star" nick-name I
had almost no problems.
However, over the last 45 days we've been trying to work with Hitachi
to try to resolve some problems and have had basically zero luck making any
progress. We've given Hitachi quite a bit of time to provide some
information on this, but so far have basically made no progress with them.
I wanted to open up the discussion to other to see if others were seeing
similar results, or perhaps it's just the level of testing we do, and
nobody else is doing this testing.
In short, we test all drives before we put them into production by
using a read/write badblocks test. We've done this for probably 5 to 7
years as part of our standard hardware burn-in. No hard drive goes into
our managed hosting facility without going through 4 to 7 days of full
read/write testing.
With our most recent batch of 500GB drives (part number
0A35415), the labels say "May 2008", we started seeing errors
during the burn in. And I'm not talking about one drive, I'm talking about
22 tested drives out of 2 different 20-packs of drives, tested on 4
different systems with different controllers, all of which did not report
errors when testing drives from older batches.
Have you seen similar problems with recent Hitachi drives? What sort
of testing do you do of new drives, would you have even noticed? Read on
for more details of what we've been seeing, including information on how to
do your own testing.
(read more | 3 Comments)
(read more | 3 Comments)
Saturday September 06, 2008 at 17:16
Subject: Program to hunt for Sprint (or similar) wireless device.
Keywords:
Technical, Wireless
Usually when I insert my Sprint cellular wireless card it shows up on
/dev/ttyUSB3, but the other day it showed up on ttyUSB6. When you insert
the card it creates 10 ttyUSB files, and I just want my card to work. So,
I wrote a small python program (based off code I have previously written
for talking to serial devices). Read on for how I solved it.
(read more | 3 Comments)
(read more | 3 Comments)
Friday September 05, 2008 at 19:56
Subject: Enigmail message composition and mutt.
Keywords:
Encryption, Technical
I've been using enigmail inside Thunderbird for mail for a while now.
However, the default setting was such that mutt wasn't auto-detecting the
messages as encrypted when I was sending them, because the content-type
wasn't set -- programs had to look for the "BEGIN PGP MESSAGE" line.
It took a bit of hunting, but I was finally able to figure out how to
change this. Since it's non-obvious, here's what you need to do:
Sean
(go to article | 0 Comments)
-
Click on "Write" to start writing a new message.
Click on the OpenPGP menu.
Select the "Default Composition Options" item.
Select the "Signing/Encryption Options" sub-item.
Check the "Always use PGP/MIME" box.
Click "OK".
Sean
(go to article | 0 Comments)
Monday August 25, 2008 at 17:13
Subject: "help" for zsh
Keywords:
Technical, zsh
One of the things I really miss about bash is the help for the "test"
command (for shell conditionals). There's the test man page, but for some
reason I just always want to type "help test" to see what I can use in
shell conditionals. I finally decided to fix this and came up with the
following line which can be added to ~/.zshrc:
(go to article | 0 Comments)
function help { bash -c "help $@" }(go to article | 0 Comments)