On security and service.
The tummy.com team responded professionally to the Heartbleed exploit.
As you have heard, there was a massive, multi-year security breach of the
certificate handling system of the most popular way of securing services on
websites and other Internet-facing ports, OpenSSL and libssl.
This security breach allowed anyone who could connect to an https
connection (or other SSL connection such as a vpn) connection, to send
a carefully crafted message to request 64K of the contents of RAM of
the targeted server. This message did not result in a successful
connection, even though a significant amount of data was returned in the
response. As it is extremely uncommon practice to log responses
to failed requests, no one knows how common this attack was. What we do
know, is that approximately half a million websites, including Yahoo,
twitter and tumblr, were vulnerable to this exploit. And these sites were
vulnerable for likely the past 2 years or so.
On Monday night, I first heard of an OpenSSL exploit, which is always
significant, and told the tummy team on our internal IRC channel. Tuesday
morning, all affected Linux distributions, which are recent versions of
CentOS, RHEL, Ubuntu, Debian and Fedora, had released updated packages,
which rolled back to the prior, good version of OpenSSL and libssl.
The tummy team spent Tuesday and much of Wednesday, triaging affected
systems, applying updates to the most significant https websites first, and
all other systems after that. Kristen, Mike, Alex, Jesse and Brandon all
quietly and diligently worked to make sure our clients are safe.
Sean, who is in Montreal this week at PyCon, gave me a valuable overview
of the internal tummy system risk, and likely client impact.
I'm proud of my team this week. They handled the largest security breach in
my multi-decade professional career with cool heads and skilled hands.
As one of our clients said “I saw that the fix was pushed on our system
yesterday after it became available. We just want to thank Tummy for being
on top of and pro-active with this issue. We greatly appreciate it.”