Sean Reifschneider's Journal Entries
Below is a summary of the most recent journal entries by this user. A full index of all entries is also available.Also available as: RSS
Next>>>
Saturday January 21, at 16:43
Thursday December 22, 2011 at 00:26
Sunday December 11, 2011 at 15:08
Sunday December 04, 2011 at 16:28
Tuesday November 29, 2011 at 02:13
Monday November 28, 2011 at 16:50
Sunday November 13, 2011 at 17:00
Tuesday November 08, 2011 at 23:11
Tuesday November 08, 2011 at 21:22
Monday October 03, 2011 at 16:09
Sunday September 25, 2011 at 16:32
Friday August 12, 2011 at 00:56
Tuesday August 02, 2011 at 13:46
Monday June 27, 2011 at 16:08
Thursday June 02, 2011 at 01:12
Wednesday June 01, 2011 at 02:50
Tuesday May 17, 2011 at 10:38
Friday May 13, 2011 at 19:57
Sunday April 17, 2011 at 18:07
Tuesday March 29, 2011 at 01:01
Tuesday March 15, 2011 at 17:06
Monday March 14, 2011 at 12:55
Friday February 25, 2011 at 03:43
Saturday February 19, 2011 at 21:27
Saturday February 19, 2011 at 20:26
Next>>>
Saturday January 21, at 16:43
Subject: A word on logging and database commits.
Keywords:
Logging, PostgreSQL
Earlier this month we switched our bandwidth accounting system to use
new software. Previously we used a kernel module on a system, which was
conceptually very cool. But with CentOS 4 going end-of-life, that kernel
module doesn't work on CentOS 6 and I wanted to go a different route rather
than fix the module.
So earlier this month, after extensive testing in parallel with our
normal traffic stats collection software, I flipped the switch and started
using the new software. Except that I forgot to re-enable the database
update code, so it was only updating the graphs. I had disabled it to
prevent double-accounting when running the old and new code in parallel
during testing.
But, I log all those updates to syslog. I was able to pull out the
logs of all those updates and replay them into the database. Logging of
updates is a great safety net.
Which leads me to my next point. The simple code I had written to
replay the logs, just was going to do one big commit at the end. I didn't
think about how many updates we were talking about, over 2.5 million... I
also wasn't writing out status, so I wasn't sure how far along it was. So
I eventually killed it after an hour of running, and added code to commit
every 1,000 updates and then write out a progress message.
With these changes, it ended up finishing after 18 minutes, giving me
progress all the way along. Of course, I could have consolidated
the data down in the program and done only a few updates, but 18 minutes
was totally acceptable.
(go to article | 0 Comments)
(go to article | 0 Comments)
Thursday December 22, 2011 at 00:26
Subject: New Year Python Meme
Keywords:
Programming, Python
Via Richard
who got it from Tarek...
1. What's the coolest Python application, framework or library you
have discovered in 2011?
I haven't been using it much lately, but looking back the library I
think I was most excited about was bottle a Python micro-framework for
building web applications. The thing I like is that it's extremely small,
just a single file. For simple things that I might have used a CGI for
in the past, bottle is great.
As far as the thing that I use the most on a day-to-day basis, that
would probably be cony.
Cony is a personal URL-bar command-line, so you can write custom shortcuts
to different locations. For example, I have shortcuts like "backup
[HOSTNAME]" which searches for the host and takes me to the page on the
backup server, or "nagios [CUSTOMER]" which takes me to their nagios page.
2. What new programming technique did you learn in 2011?
Via a recent code retreat (written about previously), I really got
into test driven development. Which is great, because our system
administration work I like to call "test driven system administration", via
Nagios, so it's very comfortable to me. But I hadn't had time to really
get comfortable with testing my code until I got this opportunity at code
retreat.
3. What's the name of the open source project you contributed the
most in 2011? What did you do?
That's a tough one, because I've mostly done a lot of small
contributions rather than any large ones other than ones I maintain. So
probably the python memcache module or vPostMaster would be at the top.
Until, that is, we release the source to our backup system, which I spent a
lot of time contributing to this year. Aside: We've finally settled on a
name for it: Network Attached Backup.
4. What was the Python blog or website you read the most in 2011?
I don't really follow any Python blogs... Most of my news comes from
my friends feeds in facebook or plus. But the number one website I read in
2011 is almost certainly the library documentation. :-)
5. What are the three top things you want to learn in 2012?
I don't really set out goals for what to learn, certainly not a year
in advance, because things change too fast. In general I want to continue
to improve my business chops.
I'm expecting that in 2012 I'll be able to start programming seriously
in Python 3.
6. What are the top software, app or lib you wish someone would
write in 2012?
I'd love to have something that did a really good job of bringing
together and making searchable all the data that comes at me. Thunderbird
can often be pretty good for e-mail, but sometimes it seems like it just
misses huge parts of my correspondence for reasons I don't fully
understand. But I'd also like web pages I visit in there, photographs, and
more...
Thanks Tarek and Richard! So what are your answers?
(go to article | 0 Comments)
(go to article | 0 Comments)
Sunday December 11, 2011 at 15:08
Subject: Google Calendar in Gnome 3
Keywords:
Calendar, Gnome3
Several of us have been running Gnome 3 lately, and have been happy
enough with it. One thing I was thinking was that it would be nice if the
date/time bar app showed my google calendar. Then Mike found this project
that does just that: gnome-shell-google-calendar.
But, I subscribe to 8 other calendars. So whenever we had a company
meeting, it would show a bunch of duplicated events, and it was hard to
tell my events from others. So I hacked on it this weekend and have pushed
a new version up to github: Sean's
fork of gnome-shell-google-calendar.
This version de-duplicates events with the same title that start at
the same time, and also displays what calendar the event came from. So far
it's been working really well for me, but the upstream author is reporting
an error so I'm working with them on that at the moment.
(go to article | 0 Comments)
(go to article | 0 Comments)
Sunday December 04, 2011 at 16:28
Subject: Report on 2011 Code Retreat
Keywords:
Programming, Python
Yesterday Bill Tucker and Matt Rose organized a Code Retreat in Fort
Collins. It was "Global Day of Code
Retreat", and there were events happening all around the world. The
basic idea was that you paired up and for 45 minutes used best practices
(as if you had all the time in the world to do it) to work on a programming
problem. Then everyone deleted their code, stood up and discussed the previous
round. Then you paired up with someone new and started over on the same
problem.
It was a great event, extremely rewarding on many levels. Read on for
more of my thoughts on the event.
(read more | 2 Comments)
(read more | 2 Comments)
Tuesday November 29, 2011 at 02:13
Subject: Python Template for Omnicompletion in Vim
Keywords:
Omnicomplete, Python, Vim
Vim Omnicomplete is extremely useful. We use vim for an internal
project and have some cases where I wanted to do a custom Omnicomplete,
however I'm not really that comfortable with vim's own scripting language.
Looking at the existing completion scripts really wasn't helping much
either.
What I really wanted to do was build the "omnifunc" completion
function in Python. After a few hours of playing, I was able to come up
with exactly that and get my completion code working. Now when I press
Control-X Control-O I get a menu of my custom completion items, based on
project names. It's wonderful!
So I spent a few more hours cleaning it up and making it into a
template that hides all the magic and presents (what I hope is) a dead
simple structure that a Python programmer can use to make omnicompletion
plugins. I've released this template on github as: vim-omnipy-template
(go to article | 0 Comments)
(go to article | 0 Comments)
Monday November 28, 2011 at 16:50
Subject: "Word" Doc Authoring with pandoc
Keywords:
Documents, Linux, Text
This weekend I wrote a lot of documentation. I spent around 12
hours on it in total. I knew there was no way I could stand being in
LibreOffice for 12 hours, but I also knew that the consumers would prefer a
prettier format than just plain text. I've been writing READMEs lately
using markdown format, which converts to pretty HTML on github, so I
decided to go that direction. Boy did it work well, once I found the magic
tool: pandoc! Read on for more details...
(read more | 2 Comments)
(read more | 2 Comments)
Sunday November 13, 2011 at 17:00
Subject: ineedpy2: Library to run newer Python from a system-installed Python.
Keywords:
Python
One problem that production system administrators have is running
systems with older Python versions installed as /usr/bin/python. Worse,
you probably have some systems with a newer Python and some with an older
Python. For example, in our environment we're managing some systems with
Python 2.1 and some with Python 2.6...
And you can't just upgrade /usr/bin/python, because that will likely
break many other things on the system, mean that packages that provide
modules either have to be rebuilt or are no longer reachable by the Python
interpreter, etc...
We've been struggling with this for one of our system maintenance
packages, where there is newer Python available on the system, but just
doing "#!/usr/bin/env python" will pick up a very old version.
The "ineedpy2" module will either try to find a newer Python version
and re-run the currently-running program with that, or you can specify a
minimum minor Python 2 version that you need, and only if you are running
something older than that will it swap in a newer one. For example, you
may have both Python 2.6 and Python 2.4 on a system, but 2.4 is sufficient,
so you don't need to do the extra work, but if you are running under 2.3
it will.
For example, you can say "ineedpy2.rerunonlatest()" to get the latest
available Python on the system, or "ineedpy2.requireminor(4)" to only
re-run the program if you are on 2.3 or older.
I've released this module on github at https://github.com/linsomniac/ineedpy2
and will put it in pypi soon.
(go to article | 2 Comments)
(go to article | 2 Comments)
Tuesday November 08, 2011 at 23:11
Subject: Switching from hardware to software RAID.
Keywords:
Linux, RAID
A few weeks ago I had a 1U system that had a RAID card, but we needed
to add more network interfaces to. I had the idea of switching it to
software RAID, freeing up the only slot for the network card, and it went
amazingly smoothly.
The bulk of the system was installed on an LVM, with just a small
/boot partition. So what I did was add two drives to the system, and
partition them with a small partition for /boot, and the remainder for LVM.
These partitions were set up with software RAID-1.
I then added the LVM partition to the existing LVM, and did a "pvmove".
"pvmove" is a great tool that tells LVM to move the blocks from one
device to another. It achieves this by first creating a mirror of a block
from the source device on the destination, waiting for that mirror to sync
up, and then breaking the mirror by removing the source block. After an
hour or two, with the system still operating normally, the source RAID
array just contained /boot.
Finally, I copied the /boot partition over to the software RAID,
rebuilt the initrd to include the software RAID tools, re-ran grub-install
on the two new drives.
So with fairly little muss and fuss, and little down-time, I had
migrated from a hardware RAID array to software, on 2 new drivers. LVM is
a very wonderful thing!
(go to article | 1 Comment)
(go to article | 1 Comment)
Tuesday November 08, 2011 at 21:22
Subject: Automated changing of Postfix configuration.
Keywords:
Linux, Postfix, Scripting
As part of system automation work, the other day we were talking about
changing the postfix configurations. Mention was made of using sed, but I
brought out "postconf -e", a tool that postfix provides to allow modifying
the configuration. Particularly useful for more complicated entries which
can span multiple lines and a simple sed script wouldn't do the right
thing.
You run it as: "postconf -e name=value"
This will modify the configuration to change the setting "name" so
that is assigned the value "value". Of course, proper quoting is necessary
if you are using special characters.
I also frequently use "postconf" to have it tell me what the current
configuration is (because sometimes looking at the file it isn't obvious)
and "postconf -d" to tell me what the default configuration values are.
Definitely useful tools to have in the tool-box.
(go to article | 0 Comments)
(go to article | 0 Comments)
Monday October 03, 2011 at 16:09
Subject: Lucid hangs after "Begin: Running /scripts/init-bottom".
Keywords:
Hang, Linux, Ubuntu
I recently had a kernel bug that caused a process to hang in D (disc
wait) status. I decided to do a reboot, but thought I'd try upgrading the
kernel before I did that. The kernel upgrade also got stuck in "D" while
trying to do a depmod, so I finally did a reboot, and had to hard power
down.
During the reboot, it would get to where it would say "Begin: Running
/scripts/init-bottom", and hang there. I searched around and didn't find
much, so I figured I'd document the solution here.
I booted into rescue mode (off the initial media, I wasn't able to
boot with "init=/bin/sh"), and then got a shell in the root file-system. I
did a "dpkg --configure -a", and then rebooted and that solved the problem.
I found this issue by trying to run "apt-get update" in the rescue
environment and it helpfully told me that it had been interrupted and I
needed to run the above command. :-)
(go to article | 2 Comments)
(go to article | 2 Comments)
Sunday September 25, 2011 at 16:32
Subject: Introducing nanomon: Extremely light-weight monitoring.
Keywords:
Linux, Monitoring, Python
Last month I had wanted to try to get some real monitoring of a bunch
of my personal systems. I spent weeks dabbling with setting up a serious
monitoring system (tried Zabbix, Zenoss, nagios, and Opsview), but I was
making little if any progress with those. Largely that was related to
running a RHEL 6 derivative and running into compatibility problems...
After spending around a day of effort on them, I decided to roll my own.
My needs were modest, I wanted to be able to run some nagios check scripts
and get an e-mail if checks failed continually for 15 minutes. But not get
e-mails every 15 minutes if they were down, and only alert if they failed
several times.
Read after the fold if you're interested in my solution...
(read more | 4 Comments)
(read more | 4 Comments)
Friday August 12, 2011 at 00:56
Subject: pycurl.error: (26, 'failed creating formpost data')
Keywords:
CURL, Error, Python
The error: "pycurl.error: (26, 'failed creating formpost data')" was a
bit tricky to look up, but I was able to find a report on the curl mailing
list that this error indicates that the file up are trying to upload via a
form upload does not exist.
Sure enough, when I checked the value I was passing as the name of the
file to upload, I was using the wrong name -- a name that didn't exist. So
if you run into this error, verify that the filename you are passing does
exist.
If this answer helped you, please post a link to this page so that it
shows up high in the results. I had to do a lot of digging to get the
answer.
In general I've been pretty happy with pycurl. I had thought urllib
would let me upload a file, but it required a lot of extra code to do.
pycurl isn't that well documented, but I was able to find examples that
helped me get it done, and I've been happy with curl from the command-line,
so I figured I'd try it. Other than having to build pycurl on one CentOS 4
box, and this information-free error message, it's worked quite well.
(go to article | 0 Comments)
(go to article | 0 Comments)
Tuesday August 02, 2011 at 13:46
Subject: Intel X-25E SSD Showing Up As 8MB
Keywords:
Failure, Intel, NCLUG, SSD
I've been following this Intel forum
thread on failures of the 320 series of SSD drives for a while now
after having the Tom's
Hardware article on it pointed out to me by our friends at Pattern Review.
So far it's been fairly full of speculation with little concrete
information available... A week ago Intel responded that
firmware fixes were being worked on.
Read on for more details about this, including the failure I've
experienced.
(read more | 0 Comments)
(read more | 0 Comments)
Monday June 27, 2011 at 16:08
Subject: Getting the expiration time of an SMTP certificate.
Keywords:
Linux, NCLUG, TLS
Today I was trying to figure out how to check the expiration time of
an SMTP certificate, to verify that after installing a new certificate the
mail server was picking up the right certificate. What I came up with is:
(go to article | 1 Comment)
SERVERNAME=mail.example.com printf 'quit\n' | openssl s_client -connect $SERVERNAME:25 -starttls smtp | openssl x509 -enddate -nooutJust for completeness sake, checking it on a web server can be done with:
SERVERNAME=www.example.com printf 'GET /\n\n' | openssl s_client -connect $SERVERNAME:443 | openssl x509 -enddate -noout
(go to article | 1 Comment)
Thursday June 02, 2011 at 01:12
Subject: Manually checking a RAID array for consistency.
Keywords:
Linux, RAID, Storage
Stephen Warren tonight noticed that one of his backups of his storage
server was transferring a file that it shouldn't have been. He was running
with the "-c" option to rsync to do a full checksum check, so apparently
this file had changed.
Stephen provided these instructions on how to compare the two copies
of a file on a RAID array, to verify that they are the same, or in his case
different. He wanted me to post them so he could find them again when he
has to search google for them in the future. :-)
Read after the fold for the details of how to read from a single drive
on a RAID-1 array.
(read more | 2 Comments)
(read more | 2 Comments)
Wednesday June 01, 2011 at 02:50
Subject: The burden of data.
Keywords:
Linux, Storage
Lately, Evelyn has come up with a very quotable, well, quote:
"Data is heavy." -- Evelyn Mitchell, 2011
We are frequently dealing with the storage, organization, and access
of all the data we are accumulating. Part of this is that we can just
store so much data, 3TB in a single 3.5" drive is plenty. But, we get less
than 100 opportunities to access that per second (10+ms average access
time).
Then we have to back it up, preferably off-site, and regularly re-read
it to verify that it's good. And if the need to recover from backups
arises, are you prepared for how long it takes to recover billions of
files?
We're currently working with one client who wants the ability to roll
back the entire system to a previous backup within an hour. However, one
directory they have takes several hours just to run a "du" on...
I really wish Linux had some sort of hierarchical storage sub-system,
but that just doesn't seem to be on anyone's radar. Something that could
manage a pool of huge but slow hard drives, smaller but much faster hard
drives, SSDs and non-volatile DRAM, automatically migrating chunks of data
between the most appropriate storage technology, would be great.
Adaptec is taking a step in that direction with "MaxCache" and
"Hybrid' RAID controllers, but both of these are strictly read-caching
mechanisms. We also have the "flashcache" kernel driver, which can offer
write-back caching to an SSD, but cannot survive a power failure or kernel
crash without data corruption...
Meanwhile we just keep generating and storing more and more data.
So, yeah, data is heavy.
(go to article | 2 Comments)
(go to article | 2 Comments)
Tuesday May 17, 2011 at 10:38
Subject: Adjusting window creation location.
Keywords:
KDE, Linux, Thunderbird
On my desktop, where I have plenty of screen real-estate, whenever I
start writing an e-mail, the window comes up all the way on the right side
of my right display. Which is not at all where I want it, that ends up
being at like my "two o'clock" position. This is not an issue on my laptop,
because of the small screen.
Continue reading for more information on how I solved this.
(read more | 0 Comments)
(read more | 0 Comments)
Friday May 13, 2011 at 19:57
Subject: I propose we standardize chrooted /procs.
Keywords:
Chroot, Linux, Standard
I'd like to propose that we come up with some way of standardizing a
way to make excluding chrooted "/proc" mount-points easier in backups. I
believe that backing up more is better than backing up less, and so I
prefer to not stop on a mount-point (because that often causes surprise
when a new file-system is added and backups don't pick it up). But systems
are just littered with "/proc" mounts for chrooted environments (postfix,
ntp, named, etc...).
It would be nice if /proc were named something that made it a bit more
unique, like "/proc-virtual-namespace", so I could feel like I can exclude
descending into that directory by name. Excluding "proc" seems too likely
to collide with legitimate files.
Sure would be nice if "/proc" included something like an
".rsync-exclude" file taht spit out an exclude rule for that directory. Or
if rsync had exclude rules based on file-system type. Beyond that, I the
best option I have come up with is to enable "one-filesystem", but then
dynamically generate the list of directories based on the file-system types.
(go to article | 0 Comments)
(go to article | 0 Comments)
Sunday April 17, 2011 at 18:07
Subject: Dual batteries in the T61.
Keywords:
Battires, Laptop, Thinkpad
I got one of those CD-ROM bay batteries, and it confuses the power
LED. When charging, it will flash orange until the second battery is more
than 15% (OMG, one battery is at 0%). And it doesn't start charging that
battery until the first is at 100%.
It charges to 90% quickly, but the last 10% takes quite a long time.
Kevin thinks I can probably swap which battery is being charged, maybe
a little script that flips the batteries when it's charging and the
primary hits 80%, then swap to top both of them up the last little bit.
So currently it takes an hour to charge #0 to 85%, another hour to
charge it to 100%, an hour to charge #1 to 85%, another hour to charge it
to 100%...
My last battery didn't even last a year before it was only holding
around 90 minutes charge. So it's quite the refreshing change to go
from 90 minutes of battery life to, well, I'm not exactly sure... I'm
guessing with them totally fresh, something close to 6 hours. But I
haven't even come close to running them both completely down yet.
(go to article | 0 Comments)
(go to article | 0 Comments)
Tuesday March 29, 2011 at 01:01
Subject: Not all USB hubs are created equal...
Keywords:
Linux, USB
I never looked into it, until now I haven't specifically bought a USB
hub. But recently I've been using the USB hub that's built into my
monitor, but it powers off when the monitor does, so I decided to add some
dedicated USB hubs. After doing some research, I came across the
designation "Multi-TT", which seems to be the thing to get.
The difference, as I understand it, is basically like the difference
between a network hub and a switch (though I'm sure that stretches the
analogy). They do push the price up from $10 to $30, but that was within
my price range.
The trick is that this feature is either not well advertised or
something. Where I bought it there was only one USB hub listing "multi-tt"
when I searched. I've been using them for a couple of days (I got 2,
partly to have more bandwidth if I hook up multiple devices, mostly to have
ports for more than one device (my laptop, at one point I had my game box
by the computer)...
(go to article | 0 Comments)
(go to article | 0 Comments)
Tuesday March 15, 2011 at 17:06
Subject: nofile limits not taking effect on Debian 5.
Keywords:
Debian, Limits, Linux, PAM
Earlier today I ran into an issue where a software package was failing
with "too many open files". I made several attempts at changing this using
various settings in "/etc/security/limits.conf", but they just weren't
taking hold.
After some digging, I found a Debian and Ubuntu set of bugs such as "'nofile'
limit set in limits.conf not applied". However, those reports
indicated that they could decrease the limit just not increase it. I
couldn't change it up or down, despite verifying that PAM was configured to
pull in the limits module for ssh, su, and cron.
I ended up doing a work-around, which was to start the process
from a root shell after doing "ulimit -Sn 10240; ulimit -Hn 10240", and
calling the process as "su -c 'bash -c script' username". This caused it
to inherit the root shell limits. Since the end goal is to start up the
process via an init script, this should work out fine.
(go to article | 2 Comments)
(go to article | 2 Comments)
Monday March 14, 2011 at 12:55
Subject: Selecting input/output devices in PulseAudio
Keywords:
Linux, PulseAudio
I've been having a heck of a time trying to make use of various audio
devices on my Linux system (in this case, Ubuntu Maverick). The biggest
problem seems to be that there are many devices (video card, audio card,
bluetooth, webcam, USB headset) and I haven't been able to pick the device
I want.
After a bunch of digging, I've finally been able to figure it out.
Read on for information about how to select the input and output devices
you want.
(read more | 2 Comments)
(read more | 2 Comments)
Friday February 25, 2011 at 03:43
Subject: Fixing "CLUSTERIP: no conntrack error".
Keywords:
CLUSTERIP, Conntrack, Linux, NCLUG
We did an upgrade of our CLUSTERIP based load-balanced DNS servers,
and it started spewing the message "CLUSTERIP: no conntrack error". It
took some digging, but I eventually tracked it down. To fix it you need to
insert a rule with "-m state --state INVALID -j DROP".
The message is an indication that a packet that doesn't match an
existing conntrack connection is received. So, just add a DROP rule.
(go to article | 0 Comments)
(go to article | 0 Comments)
Saturday February 19, 2011 at 21:27
Subject: Looking for Networking Help at PyCon 2011.
Keywords:
PyCon, PyCon-2011, Python
PyCon is run by volunteers, like myself. If you are planning on
going, and can help the night before the tutorials or the main conference,
and are interested in networking, please consider signing up to help with
the network setup. A lot of the work is simply laying out cables and gear,
taping down runs into the audience, and network testing and debugging.
If you are interested, please sign up on the PyCon 2011 Volunteer
Sign-up Page (for networking and/or other sections). Thanks!
(go to article | 0 Comments)
(go to article | 0 Comments)
Saturday February 19, 2011 at 20:26
Subject: PyCon 2011 Networking Preliminary Information
Keywords:
2011, Networking, PyCon, Python
If you are planning to attend PyCon 2011 and use the network, please
read this. And if you are thinking of going and haven't signed up, please
do so soon, it is expected to sell out. In a good way.
Executive Summary: Please bring 5.2GHz gear if at all possible
(A.K.A. 802.11a). There will be a few hundred wired ports but they are
highly concentrated. Please leave your 100+mW cards at home. Read this
full message for all the details.
(read more | 2 Comments)
(read more | 2 Comments)