On security and service.
Q. What do all PHP content management systems have in common?
A. They will all have vulnerabilities if left unkept.
Recently, we experienced a mild denial of service attack against several of our hosted servers. The attack primarily targeted SSH on one of our subnets. The origin of the attack was a single IP in Hong Kong.
I don't get to develop software much, so mostly I do it recreationally. Recently, I got an urge to develop a “30 day streak” on github. It tracks the streak for you – that's how I noticed it, it said my streak was maybe a couple of days…
Note that there is an exploit for NRPE, the network daemon for running Nagios monitoring checks, which may allow unauthenticated users on the public Internet to run arbitrary commands on impacted systems. In particular, if you don't either firewall off NRPE, or enable the “allowed host” setting in NRPE, arbitrary commands can be run as the user the NRPE daemon runs as.
I've been working on the next generation memcached module for Python, and it includes extensive tests. One of the things I wanted to test was what happened when the Memcache server unexpectedly dies.
With Python 3.3, a set of my patches were accepted to enhance the crypt library and make strong password hashing easy. Just call “crypt.crypt()” with the password and the result will be a salted hash. You can then use that salted hash with “crypt.crypt()” to check the password again in the future:
Back in 2009, BackBlaze came out of nowhere and announced the open sourced plans for a 4U system capable of storing over 60TB for $8K. Just a few weeks ago they released the Version 3.0 Storage Pod Plans, capable of storing 180TB for under $11K. Good to see they've been keeping up with new releases, looks like a lot of good enhancements like better vibration dampening.
Evelyn forwarded this blog post on the First 5 Minutes Troubleshooting An Unknown Server. It's pretty good, as a general guide for an unknown server.
Inexpensive storage tends to lack robustness, but robust storage tends to be spendy. I've spent time over the last year brainstorming and decided to write down some of my ideas, hoping that it either helps out someone else or that it sparks discussion to provide some more ideas. Ideas on Budget Storage for the Data Center. Any clever ideas?
In the past I've used PPP over SSH to do simple network tunnels. This is not recommended for any serious networking, but last week I was doing some VoIP testing and wanted to isolate NAT on both ends from being the problem. You typically don't want a TCP tunnel for a VPN, but I just needed something simple for an hour of testing.
I am a huge fan of version control. However, I do very little development, and even less Linux kernel development. So my interactions with git are usually limited to “git pull”, “git push” and “git commit -a”. Sometimes I run into having to do unusual things, like last night where I accidentally ran a “git rm” on the wrong file and needed to get it back.
A few weeks ago we ran into a situation where two machines ended up on the same IP address, on machines an hour away, despite testing which was done before leaving. Worse, the duplicated IP was on the system management console, so this was the only way into the machine, an OS had not yet been installed. Knowing how ARP works can allow you to work around a situation like this, so I wrote an article about it: Networking Basics: How ARP Works.
I've released a new nanomon which includes an internal scheuler. The primary benefits of this scheduler are that it can run checks more frequently than every minute, and multiple copies will not be started if the checks take longer than the check frequency. You can now run checks every 15 second, for example.
As I mentioned previously about BIND response rate limiting, we recently had some problems with our DNS server being hammered, probably as part of a denial of service attack on some remote entities.
I've released a new version of nanomon which make the “UP” e-mail report what services have recovered. Previously it simply reported that all services were up, because at the time the “UP” comes through it doesn't remember what was down.
I recently presented to NCLUG about building a static site using the “Mynt” tool. As part of that I have written an article about using Mynt to build a static site. This is based on the work I did with building the new tummy.com site, and combines all the tricks I found related to making a full site, rather than using Mynt to make a blog, which is what the Mynt tutorial goes over.
There has been a lot of discussion about the power outage at the Super Bowl. Power failure is a subject near and dear to most computer users, especially those in the Data Center. A lot has been written about the outage, including finger pointing in all different directions. I especially like the statements that the “faulty device was manufactured in Chicago”. As if the readers will conclude “Oh, that explains everything!”
We've been working to completely revamp the tummy.com website and as part of that have reviewed a number of different options.
The tummy.com website has been revamped with a more
modern look. Sorry for the RSS feeds re-delivering the last few entries,
the new system apparently pushed them out with slightly different
information so the RSS readers picked them up as new items.
We've recently been the reflection point in a DNS-based reflection+amplification attack. We implemented some rate limiting to prevent it, and as part of my research on this topic I found this discussion to be fascinating. In particular, the trade-offs between request rate limiting and response rate limiting… It's about half way down in this dns-operations thread on “DNS ANY from Amazon”.
Just a note on something I've noticed in Ubuntu Quantal… It includes Subversion version 1.7.5, which has this compelling feature: The .svn directories have been merged into a single directory at the top level of your checkout. So no longer are the subdirectories littered with .svn directories. Makes script and find commands of sub-sections of the repository easier.
Thanks to Bill Tucker and 2011's Code Retreat that he put on, I've become quite the convert to testing. I knew I should be doing it, but I never found the time to become really comfortable with them until Code Retreat.
Google has an excellent tool to allow for command-line access to parts of their API. I'm especially fond of the ability to edit documents using vim. I recently bought a Nexus 7 tablet and wanted to cleanly view many of these documents upon it but the default width of about 65 characters varied greatly from my default vim setup and made for some less than smooth reading when viewed on the Nexus 7. Read on for how I was able to modify both my environment and
~/.vimrc to make this process very smooth.
We recently ran into an issue with Opsview notification emails from a fresh install on a CentOS 6 system. The delivered message had some headers misplaced in the body of the message rather than grouped with the rest of the message headers. These three headers, specifically: