tummy.com Main Blog Feed

40-day Streak at Github

2013-05-15T16:02:00Z

I don't get to develop software much, so mostly I do it recreationally. Recently, I got an urge to develop a "30 day streak" on github. It tracks the streak for you -- that's how I noticed it, it said my streak was maybe a couple of days...

I had a bit of a false start: 9 days in I misunderstood something and missed a day... Now I've built up 43 days.

It's been amazingly powerful, this trying to do something every day on github. I've made more progress on a number of pet projects than I have in quite a long time. I also started a new project, and have kept with it for over a month consistently. It's not uncommon for me to start a new project and then lose interest after a relatively short period of time.

There have been some days where my participation has been very small. For example, when I don't have much time or energy during the day, I will usually review the TODO list and do triage: adding items, removing items I have completed but forgot to remove, adding more details, or just ordering it based on priority. Valuable, but not exactly rocket science...

The new project I have been working the most on is a re-write of the Python memcache client, making sure it's fully tested and using a few new APIs (so far). The code is much cleaner than the previous code (which I didn't write, I've just been maintaining). The big benefit is that the same codebase works with both Python 2.7 and 3.3.

You can check out my progress on my github profile. I'm not sure how long I want to keep going with the streak... I definitely wanted to do 30 days, but didn't want to feel like I had to keep going once I hit that. I might consider breaking the streak once I get this new project released.

Nagios NRPE Exploit

2013-05-15T15:26:00Z

Note that there is an exploit for NRPE, the network daemon for running Nagios monitoring checks, which may allow unauthenticated users on the public Internet to run arbitrary commands on impacted systems. In particular, if you don't either firewall off NRPE, or enable the "allowed host" setting in NRPE, arbitrary commands can be run as the user the NRPE daemon runs as.

It looks like there is at least one very active person scanning for this exploit and using automated tools to compromise hosts via it. They seem to be deploying Bitcoin mining clients, and we have seen two machines in very different parts of the public Internet space compromised by this, using the same bitcoin mining account name.

If your machine has been compromised by this person, it will be running the "minerd" process, using all available CPU time.

It appears to impact NRPE version 2.13 and below. Here is an Opsview blog post about the exploit

Python unittest of Misbehaving TCP Servers

2013-05-11T15:18:00Z

I've been working on the next generation memcached module for Python, and it includes extensive tests. One of the things I wanted to test was what happened when the Memcache server unexpectedly dies.

Rather than using Mocks, I wanted to have my code connect to a real server that was misbehaving. So I created a class that you can give a simple expect/send program to, and it forks off and runs a server for a few seconds, then exits. The program is a list, with None indicating that it should receive data from the client, and a string indicating something to send.

This allowed me to track down a number of exceptions that I wasn't catching in my high-level wrapper code that treats exceptions as cache misses. Fantastically useful!

I've published this code to my unittest skeleton code repository. Here's a simple example:

from mctestsupp import RECEIVE, CommandServer

immediately_disconnect_server = CommandServer([])
sc = memcached2.ServerConnection('memcached://127.0.0.1:{0}/'
        .format(immediately_disconnect_server.port))
with self.assertRaises(memcached2.ServerDisconnect):
            memcache.set('foo', 'bar')

That also has a skeleton for testing Bottle web apps super easily. I've been using Bottle a lot lately for simple little web control panel type apps.

Correct Password Hashing in Python

2013-04-08T22:40:00Z

With Python 3.3, a set of my patches were accepted to enhance the crypt library and make strong password hashing easy. Just call "crypt.crypt()" with the password and the result will be a salted hash. You can then use that salted hash with "crypt.crypt()" to check the password again in the future:

>>> import crypt
>>> salted_hash = crypt.crypt('foo')
>>> len(salted_hash) 
106
>>> crypt.crypt('foo', salted_hash) == salted_hash
True
>>> crypt.crypt('bar', salted_hash) == salted_hash
False

More information is available in the crypt module documentation

New BackBlaze Storage Pod, 180TB for $11K.

2013-04-08T22:22:00Z

Back in 2009, BackBlaze came out of nowhere and announced the open sourced plans for a 4U system capable of storing over 60TB for $8K. Just a few weeks ago they released the Version 3.0 Storage Pod Plans, capable of storing 180TB for under $11K. Good to see they've been keeping up with new releases, looks like a lot of good enhancements like better vibration dampening.

First 5 Minutes Troubleshooting A Server

2013-04-03T12:48:00Z

Evelyn forwarded this blog post on the First 5 Minutes Troubleshooting An Unknown Server. It's pretty good, as a general guide for an unknown server.

Our focus is a little different: We try not to have unknown servers, which is why we try to get our customers to commit to an ongoing relationship rather than just getting called when there's an emergency. We have had success at preventing problems rather than reacting to them. The diagnosis process is quite different when you are more familiar with a box and you know that munin and nagios monitoring are set up, etc...

In general, I will say that when there is a problem on a customer system, especially if it is performance-related, I will usually look at "uptime" first. This tells you if the system was recently rebooted, and the system load. If the load is higher than 0.5 or so, I will usually run "vmstat 1" and watch that output, which will tell me if the system is spending a lot of "cpu wa" time (usually I/O related), has no "cpu id" (idle CPU resources), if there are many "b" (blocked processes), and if it's swapping.

However, as the beginning of the above post says, knowing what you are troubleshooting is best. I always prefer to know how to reproduce the problem that I am troubleshooting, and if any changes have been made since it was last working. Of course, some of this information you have to take with a grain of salt. If I had a nickel for every time I've heard "No, nothing has changed" when something had changed... :-)

Article on Budget Data-Center Storage

2013-03-25T13:59:00Z

Inexpensive storage tends to lack robustness, but robust storage tends to be spendy. I've spent time over the last year brainstorming and decided to write down some of my ideas, hoping that it either helps out someone else or that it sparks discussion to provide some more ideas. Ideas on Budget Storage for the Data Center. Any clever ideas?

SSH Network Tunneling

2013-03-04T23:37:00Z

In the past I've used PPP over SSH to do simple network tunnels. This is not recommended for any serious networking, but last week I was doing some VoIP testing and wanted to isolate NAT on both ends from being the problem. You typically don't want a TCP tunnel for a VPN, but I just needed something simple for an hour of testing.

I found that SSH now has a "-w" option which will set up "tun" devices on either end and transport the traffic between them. If you say "ssh -w 0:0 hostname", it will set up a "tun0" on both ends. Then you just need to ifconfig the two tunnel endpoints. You can use the "LocalCommand" setting to do one ifconfig and the remote command to do the other, for example:

ssh -o PermitLocalCommand=yes \
    -o LocalCommand="ifconfig tun0 192.168.0.1 netmask 255.255.255.252" \
    -w 0:0 $HOSTNAME \
    'ifconfig tun0 192.168.0.2 netmask 255.255.255.252; sleep 900000'

Note that on the server you will need to set "PermitTunnel yes" in /etc/ssh/sshd_config and restart SSH. This needs to run as root on both ends to build the tunnel interface.

It looks like this functionality is not exactly new, but I hadn't heard about it before. I tend to avoid tunneling over SSH, because TCP over TCP tends to react badly, so I haven't looked at it in probably a decade. But in this case I just needed something simple, and GRE tunneling would have been problematic because of NAT on both ends.

Git cheat-sheet

2013-03-02T13:33:00Z

I am a huge fan of version control. However, I do very little development, and even less Linux kernel development. So my interactions with git are usually limited to "git pull", "git push" and "git commit -a". Sometimes I run into having to do unusual things, like last night where I accidentally ran a "git rm" on the wrong file and needed to get it back.

I'm starting a Git Cheatsheet to record examples of commands that I use infrequently but want to remember. So that I don't have to keep bugging my git go-to-guy Stephen Warren about the same thing. :-)

What you need to know about ARP, in article form.

2013-03-02T13:19:00Z

A few weeks ago we ran into a situation where two machines ended up on the same IP address, on machines an hour away, despite testing which was done before leaving. Worse, the duplicated IP was on the system management console, so this was the only way into the machine, an OS had not yet been installed. Knowing how ARP works can allow you to work around a situation like this, so I wrote an article about it: Networking Basics: How ARP Works.

New nanomon with built-in daemon mode scheduler

2013-02-23T13:33:00Z

I've released a new nanomon which includes an internal scheuler. The primary benefits of this scheduler are that it can run checks more frequently than every minute, and multiple copies will not be started if the checks take longer than the check frequency. You can now run checks every 15 second, for example.

BIND+RRL patched RPMs available

2013-02-20T15:12:00Z

As I mentioned previously about BIND response rate limiting, we recently had some problems with our DNS server being hammered, probably as part of a denial of service attack on some remote entities.

DNS is a fairly troublesome protocol because typically it uses UDP, and UDP makes it trivial to cause the server to send its response to an innocent third party.

The correct fix for this seems to be Response Rate Limiting, which there are some patches available for BIND. Request rate limiting, they suggest, happens at the wrong level, and can't tell the difference between someone asking for a bunch of distinct queries rather than the same query over and over like what our attack was about.

But, getting those patches applied to the RPMs for CentOS was a bit of a pain. Quite a bit, actually, took the better part of a day to get the new RPMs built and tested.

I've made availbale the packages I've built for 32-bit CentOS 6 at http://yum1.tummy.com/bind-rrl/centos/6/. You should be able to list this location in your "/etc/yum.repos.d/bindrrl.repo" file like this:

[bindrrl]
name=BIND RRL for Enterprise Linux 6 - $basearch
baseurl=http://yum1.tummy.com/bind-rrl/centos/$releasever/$basearch
enabled=1
gpgcheck=1

That should allow you to use yum to install and update to the RRL patched versions.

New release of nanomon.

2013-02-19T15:25:00Z

I've released a new version of nanomon which make the "UP" e-mail report what services have recovered. Previously it simply reported that all services were up, because at the time the "UP" comes through it doesn't remember what was down.

nanomon is extremely simple monitoring, but with enough features that it will prevent false positives and prevent e-mails from being generated for every check, alerts only get sent on service transition.

Wrote a Quickstart for Static Sites with Mynt

2013-02-15T13:49:00Z

I recently presented to NCLUG about building a static site using the "Mynt" tool. As part of that I have written an article about using Mynt to build a static site. This is based on the work I did with building the new tummy.com site, and combines all the tricks I found related to making a full site, rather than using Mynt to make a blog, which is what the Mynt tutorial goes over.

Preventing Power Outages, Super Bowl Edition

2013-02-11T19:37:00Z

There has been a lot of discussion about the power outage at the Super Bowl. Power failure is a subject near and dear to most computer users, especially those in the Data Center. A lot has been written about the outage, including finger pointing in all different directions. I especially like the statements that the "faulty device was manufactured in Chicago". As if the readers will conclude "Oh, that explains everything!"

Backup Power FTW?

One piece of commentary I've particularly found interesting is the personal blog post of Amazon Distinguished Engineer James Hamilton. I'll admit that my first reaction on hearing about this blog post was "I'm not sure Amazon is one to be talking about preventing blackouts". Coming shortly after the Christmas Eve 17-hour outage of Netflix (which uses Amazon's infrastructure) and an hour outage of Amazon's own site. Particularly as it seems unlikely that an Amazon engineer has any direct knowledge of the specific inside workings of the venue where the outage occurred.

The blog entry is worth a read and does offer some good suggestions. However, it also offers some more problematic suggestions.

The good suggestions are:

Replacing the lighting with equipment that doesn't take 20 minutes to come online. Half the outage was after power was restored.
Splitting power up into smaller zones and interleaving the lights. Instead of half the Superdome going black, maybe it could have been designed so that every 4th or 6th light went out.
Automated Recovery could have shortened the power outage from 15 minutes to seconds. Though the lights still would have taken 20 minutes to come back on.
Improved testing procedures seem likely to have found this fault, if testing at maximum expected utilization and at overload had been done.

However, I would call into question the recommendation of installing backup generators. Admittedly, I'm making this call a week after the blog posting, and we have more information about the cause of the failure. I did make this call a week ago, I just haven't had time to write about it until now. That's my story and I'm sticking to it.

Why Not Backup Power?

My primary concern here is that you have to be extremely careful when adding complexity to a system. As the saying goes, "For every difficult problem, there's a simple, obvious, solution that is completely wrong." Adding UPSs and generators to a system that has insufficient testing, is almost certainly a bad thing.

At this point it looks like the fault was caused by an incorrectly configured breaker designed to protect equipment at the stadium. It sounds likely that the breaker was configured with too low a set-point, which caused it to trip prematurely.

My Conclusions

The biggest gain here in reliability would be better testing procedures. That likely would have found the configuration issue with the breaker before it impacted the event. However, improved testing isn't "sexy". The sound-bite we are hearing is "Amazon engineer says spending $10M (60 seconds of advertising worth) on generators would have prevented blackout".

The next priority I would look at is decreasing the lighting startup time. Any power outage causing 20 minutes of darkness seems like a big red flag.

One thing the blog doesn't go into is how you select the results of the brainstorming on possible countermeasures. That is the more important part of the Service Outage Analysis process: you brainstorm solutions and then you analyze them and pick the appropriate ones, usually in terms of cost/benefit.

Our Experience with Power Outages

In a Data Center environment, they tend to be pretty rare. In our facility, we haven't had a single power outage to any of our cabinets since 2004 when we moved in there.

Most outages I know of in the data center are human caused, typically smaller scale issues. For example, cabinets accidentally or intentionally being run too close to the limit. Or whole-room incidents like the livejournal outages years ago where an EPO (Emergency Power Off) button was pushed by someone working in the room.
Twice over a couple of years.

I can only imagine that pressing the EPO button generates a RG interrupt. You know: Resume Generation...

Many of these issues can be addressed by high availability clusters running with no shared single point of failure. For example, several years ago we had a transfer switch which, instead of being 30A was two separate 15A sides. But all monitoring for it was on the 30A side... The total load was well under rated capacity, but one half was just a bit too high, so that half triggered a circuit breaker.

In our case, redundant machines are always in other cabinets, so those gracefully took over after a few seconds.

In that case, it was decided that the appropriate countermeasure was to replace this gear with gear that had monitoring at the individual breaker level. That's been the only significant power outage we've had at our facility in 7 years.

And that's the name of the game: taking appropriate countermeasures.

A Quick Review of Python Static Site Generators

2013-02-09T23:26:00Z

We've been working to completely revamp the tummy.com website and as part of that have reviewed a number of different options.

Trying the Big Boys

The old site was built using a tool-chain I built a decade ago, and there are much more modern choices that take care of a lot of things I had to build in that system. So I started looking at fairly off-the-shelf systems including:

Wordpress is really a best of breed blog system, I've been using it for my personal blog and it's really fantastic. But I spent some time playing with it for a site publisher and it really felt more oriented towards a blog than a site.
Mezzanine is a Python-based content management system, so it would be something I could dig into the code if need be. But I really wanted something I wouldn't have to dig into. The reason I kept looking here was that it has a separate front page and content pages, and they use different markup, and I wanted several pages the same, like the front page, and didn't want to have to manage them in different markup. Plus, it seemed like a very big, complex system.

Static Site Generators

After looking at these I started thinking I might be able to get away with something extremely simple. For some of our dynamic content, we can do Javascript rather than server-side, and everything else is basically static, except for blog feeds and comments.

Our site has a lot of blog entries (~450) but also has articles (30), software pages (13), presentations (40), and our company information (30 pages). The company pages are the primary reason for this site, so a solution that put the blogs in #1 spot was not an option.

So I started looking at static site generators, including some like Jekyll, but mostly I looked seriously at these:

Nikola

Nikola generates a wonderful looking blog, but it is really targeted at making a blog first. The main page always seems to be the blogroll, where I wanted my site top page and many other pages, and a blog as well. I did install and run some trials with it, and loved the look of the blogs.

Nikola had a lot of configuration and tools available for it, which was good. But it really felt targeted towards a blog site or a blog site with some stories.

It uses the Mako template engine which I've used in the past and been happy with. I've always been able to do what I needed to do with Mako where Jinja2 I sometimes run into issues with if I wan to include a template that includes another template, for example.

In the end, I decided Nikola probably wasn't for me. For a straight up blog site, it is probably really good. I could always fall back to using Nokola, or even Wordpress, for just the blogs component of the site.

Pelican

Pelican has quite a lot of activity on github. More than Nikola by quite a bit. It also has a lot of features, but again is targeted towards primarily a blog. While you can supply your own template pages to get installed (and you have to list every such page you want to install), if you install a top-level index.html, it is going to overwrite the main blog page and you lose it.

Compared to Mynt, which publishes all the files you give it in your documents directory, the listing of the pages (they call them templates) in the config file seemed obtuse. I had always figured I'd just write a snipped of Python code, since the config file is Python code, that would dynamically generate it.

I was basically able to get the site working entirely in Pelican, except that the blog pages would get overwritten.

Another thing I ran into is that in the templates, Pelican exposes fairly little functionality to you. The templates seem to just be meant for theme creators and the blogs, and it doesn't expose things like date formatters which Mynt did.

I probably could have fixed these issues, but in the end I made a decision that of the three Mynt just fit better for publishing a site rather than a blog.

Mynt

Mynt has very little github activity, less than either of the above. I almost gave up on it right at the beginning because the documentation for getting started is pretty thin. It shows you how to install, create posts, and build the results. However, as I started digging deeper there wasn't much deeper you could go.

While I can't find it now, I ran into this part of the documentation that said something to the effect of "You shouldn't need documentation, if you do I haven't done my job." I almost wrote the whole thing off, but I decided "Ok, let's put this claim to the test..." I looked in the directory that was created as part of the quickstart guide, and it all seemed pretty obvoius.

So I started modifying it and adding pages. The top level index.html is a Jinja2 template, which accesses a layout defined in the "templates" directory (all "special" directories start with a leading underscore). Posts are just a data structure handed to the template, for example.

I will say the layout feels a little more rudimentary than Pelican or Nikola. The abilities are more limited, for example it doesn't understand blogs with multiple authors (we are just having the author be a tag for the posts). It also can't generate feeds for arbitrary tags without this patch to apply feeds for every tag.

I implemented our customer testimonials using Javascript, and the comments we've outsourced to Disqus, which is what all of the static site generators seem to use. Though Mynt doesn't directly support Disqus like the others do, it was trivial to add to the templates.

Customizing and theming were all done manually, which in some ways is better than Pelican where the theme is not something you can really touch. For my custom theme, this worked out great. If you want to have someone else in charge of theming, or use only an existing theme, one of the others might be better. Though Pelican and Nikola didn't have that many themes.

While there are some places where it feels limited, in general if you are building a site, rather than just a blog, I think Mynt is the best choice of the three.

New tummy.com Website

2013-02-09T23:17:00Z

The tummy.com website has been revamped with a more
modern look. Sorry for the RSS feeds re-delivering the last few entries,
the new system apparently pushed them out with slightly different
information so the RSS readers picked them up as new items.

The new site was done using the Mynt static site
generator, which worked out really well.
I'll be writing up more about it and the other alternatives I looked at
shortly.

Interesting read on request versus response rate limiting.

2013-01-11T12:28:00Z

We've recently been the reflection point in a DNS-based reflection+amplification attack. We implemented some rate limiting to prevent it, and as part of my research on this topic I found this discussion to be fascinating. In particular, the trade-offs between request rate limiting and response rate limiting... It's about half way down in this dns-operations thread on "DNS ANY from Amazon".

SVN directories

2012-12-21T14:22:00Z

Just a note on something I've noticed in Ubuntu Quantal... It includes Subversion version 1.7.5, which has this compelling feature: The .svn directories have been merged into a single directory at the top level of your checkout. So no longer are the subdirectories littered with .svn directories. Makes script and find commands of sub-sections of the repository easier.

Skeleton for testing Python code.

2012-12-14T23:18:00Z

Thanks to Bill Tucker and 2011's Code Retreat that he put on, I've become quite the convert to testing. I knew I should be doing it, but I never found the time to become really comfortable with them until Code Retreat.

However, especially when I was starting, I found I really needed some examples and wanted some boilerplate I could just drop in place. At the Code Retreat I had just recently come back across an entry I made on my personal Wiki that had some examples. Since then I've expanded on them and today I included some examples of testing Bottle web applications (I still totally love Bottle by the way).

I've create a github account that has these examples and related code such as a "Makefile" which automatically runs the test. You can find it at https://github.com/linsomniac/python-unittest-skeleton. I hope you find it as useful as I have.