Recently our home backup server (which is used for backups of many of
our laptops) had a RAID issue and because of a slightly quirky setup I got
confused and ended up losing all the data on it. As I mentioned recently
in some of my other recent posts, I went through some experimentation and
the end result is an almost entirely new system and configuration.
Below I'll discuss some of my choices for this system. Let's start
from the end.
The Final Result
Here is what the end result of the system is:
6TB usable, with double parity
2GB (probably will upgrade it to 8GB soon)
Core 2 Quad core 2.4GHz
I previously had a tummy.com backup server, which we all backed up our
laptops to as well as other things like the pristine disc copies from when
we got the laptops, and also a personal storage server for Evelyn and I. In
order to save power and space, I decided to combine these into a single system.
I hadn't done this previously simply because of the number of discs it
would take to hold it all.
I want ZFS because it goes to extraordinary lengths to ensure that
what is written to the file-system is not corrupted months or years down
the line. Since many of the things we are storing are infrequently used,
like digital photos, copies of scanned bills, invoices, and other business
and personal documents, for example, having checksums and the ability to
detect and correct corruption is useful.
However, as this is a central point for storing pretty much every bit
of information we have, including sensitive business information and
private and confidential personal information, it has to be
encrypted. I don't want to have to worry about it getting stolen, but this
also protects in case we were to, say, accidentally sell the hard drives on
ebay before wiping them. There's just no reason not to store the data
I ended up using Ubuntu Hardy 8.04 64-bit for the system. Largely
because it is the most recent free LTS release, so it has a newer kernel
and supported the SATA Port Multiplier and Encryption in the installer. I
had to use the "Alternate" installer to get the encryption during the
For the system disc, I just let the Ubuntu Alternate installer do it's
work and set up the encrypted LVM.
For the individual encrypted partitions, I set them up using
"cryptsetup luksFormat" to format each of the partitions. I had set up all
discs with a /dev/sd*4 partition, and the system disc only had partitions 1
and 2, so I know any disc currently hooked up with a 4th partition is one
of the encrypted set.
When I "luksOpen" them, I give them a named based on the LUKS UUID, so
this name will never change unless I reformat the partition. So I have a
for file in /dev/sd*4; do
cryptsetup luksOpen $file `cryptsetup luksUUID $file`
if [ "`ls /dev/mapper | grep -e '.*-.*-.*-.*' | wc -l`" != 14 ]; then
echo "Didn't find all 14 devices in /dev/mapper. Exiting."
The "Didn't find all devices" condition is in there because the eSATA
controller/Port Multiplier sometimes doesn't see all 5 of the external
drives at boot time. This seems to only happen at boot after a power
cycle, and clears up after a reboot, but I want to get a warning before I
start the ZFS.
mentioned in a previous journal post on ZFS under Linux, I did some
researching on the ZFS-on-FUSE mailing list to see how things were
progressing. It looked like it was getting regular updates, but only into
the development version. I ran some testing before deciding to go down
this path. If it fails I have a fall-back plan of trying a virtual or even
dedicated machine running OpenSolaris accessing the crypto volumes via
So far it's been almost a month using ZFS under Linux and it's been
going very well.
For ZFS I used the latest checkout from "mercurial" on the ZFS FUSE Wiki. There is
still fairly regularly updates being done, and the latest release tar-file
is over a year old. On the mailing list there was regular bug-fixes going
in, so the latest checkout seems to be the only way to go.
I ran the "configure" and installed any of the packages it needed. In
particular, I remember needing libfuse-dev and fuse-utils. Once
"configure" completed without problems, I ran "make" and "make install".
Then you have to start the zfs daemon, which I did by going into the
"src/zfs-fuse" directory and running "nohup ./run.sh </dev/null
>/dev/null 2>&1 &". This has to be done every time you
reboot, but after you decrypt the partitions.
Then you can create the ZFS pool with "zpool create data raidz2
/dev/mapper/*-*-*-*", if you have named your partitions like I have, and
don't have any other /dev/mapper partitions matching a file-name with 4
hyphens in it. WARNING: Everything existing on those drives will be
This creates a "/data" directory which is the root of the ZFS
file-system. You can then use "zfs" to create new sub-file-systems,
snapshots, and more...
The primary use of this system is daily backups of a bunch of other
systems. One of the drawbacks of Ubuntu 8.04 is that it shipped with rsync
version 2, even though rsync 3 was out when it released. In fact, Fedora
9, which shipped a month before Hardy, included rsync 3...
There's a huge memory and speed improvement using rsync v3, so I
custom built the Debian package of rsync-3, and installed it on this
The case I used was one of the few pieces of the existing backup box
that I kept. I had purchased it around a year ago, the current version of
it seems to be the R5605-BK
by Rosewill. I selected this case because it will take 5 drives
internally, with a nice fan in front of them, but is relatively compact.
There are cases that will take 10 or more drives in a tower configuration,
but they tend to be huge. This case also has 4 5.25" bays, so I can put a
5-in-3 SATA enclosure in there and have the case hold 10 drives with good
When I initially got this case I started with the 5 internal drives
and then about 6 months later added the SATA enclosure (more on that below)
to double the capacity. With the current configuration, I'm at 15 drives.
So this case worked out quite well as far as expandability.
The Power Supply I used was a high efficiency unit from Seasonic. This
wasn't an exact fit for my new motherboard, and this exact power supply
isn't available any more it seems. The motherboard I used requires an
8-pin CPU power header, but the Seasonic I had has a 4 and 6 pin. I admit
that I kludged it together using both of these, but if I were buying it new
I'd go with something that had an 8-pin CPU power cable.
For example, the SS-500GM
seems like a good choice.
Note that I don't have a beefy graphics card in this system. That's
why I can get away with a 330W power supply to drive a 90W CPU and 10 hard
drives. I went with an "80+ certified" power supply to reduce wasted
energy and heat.
Internal 5-bay Enclosure
In addition to the 5 internal bays, I added a Supermicro
Mobile Rack CSE-M35T-1. This fits in 3 5.25" bays and adds 5 3.5" bays
including a nice fan and front removable sliding trays plus activity
The down side of this is that it's pretty long. With the original
motherboard I had in the system it would just barely fit. With the current
board, there are memory slights that make it so this enclosure doesn't fit
completely flush in the case.
External 5-bay Enclosure
I also added a 5-bay external enclosure, the AMS
DS-2350S. These are kind of pricy, but make it so I can add a lot more
drives without worrying about the internal power supply or space to put
them. If I were starting from scratch I'd consider a smaller case and
putting all drives external, but then we're talking over $700 just for the
enclosures. However, the current setup allows me to add 3 more enclosures
without doing any other system changes.
This enclosure has a 5-port SATA Port Multiplier in it. I posted
in my journal previously about SATA Port Multipliers, so read that if
you would like more details. The short form is that it requires a recent
kernel, but allows 5 drives to be connected to a single SATA port.
Internal SATA Card
For connecting the 10 internal hard drives, I use 4 of the 6 internal
ports, and then a Supermicro AOC-SAT2-MV8
8-port PCI-X SATA card. Note that this can be used in a normal PCI slot
with speed limited to around 100MB/sec, as long as the slot doesn't have
tall components mounted behind it.
External SATA Card
This card connects the external 5 bay enclosure, I selected the Addonics ADS3GX4R5-E.
This card has 4 external SATA ports, a PCI-X interface, and is reported to
work under Linux. Most of our servers still have PCI-X interfaces in them,
so I figured I'd try this card instead of testing out any of the
PCI-Express cards (which tend to be even newer and I've run into more
issues there in Linux.
Going with a PCI-Express card is tempting, since most motherboards
these days have them rather than PCI-X or extra PCI slots. My choice of
PCI-X moved me into a more expensive motherboard, but I also wanted to get
more experience with this specific remote control card so I was kind of
already prepared to go with a more expensive motherboard.
For the motherboard, I decided to get something with at least 2 PCI-X
slots because the SATA cards are PCI-X. I probably don't really need the
performance, but the ability to do a "scrub" of the data to verify
integrity in a reasonable amount of time is a good feature.
I selected the Supermicro MBD-X7SBE
which is a Core 2 or Xeon 3000 series capable board with 2 PCI-X 100 slots
and 2 PCI-X 133 slots, plus 4x and 8x PCI-Express slots.
Part of the reason I selected this board was also that it can do full
remote management, and I want to get more experience with that.
The above motherboard supports a add-in card called the
AOC-SIM1U+. This board allows you to power on/off/reset via IPMI, a
web interface, or SSH, and also allows remote KVM and emulated media via a
Java client. I've played with it before and it worked well, but we will
probably be making much more extensive use of these in the future and I
wanted more experience with them.
The CPU I got is the Intel Q6600 Quad core 2.4GHz. It supports
hardware virtualization, 32 and 64 bit OS, and has a lot of horsepower.
With my planned encryption of the discs, plus compression of backups via
ZFS, I wanted to have plenty of horsepower. This was almost a perfect fit,
I found out later, all 4 cores run between 85 and 90% utilized when I'm
running a full ZFS "scrub".
I forgot to order RAM initially, so I used the old RAM out of the
existing box, which is 2GB of DDR2 memory. The system will take up to 8GB,
and with the RAM being as cheap as it is, I'll probably go that direction
before long. Mainly to try to keep the disc cache as big as possible, to
improve backup performance. The 2GB has performed quite well though.
Cost: 8GB around $170, 2GB around $40.
I used an existing 250GB Hitachi Deskstar SATA drive for the system
disc, and 14 Hitachi Deskstar 500GB drives. I've been very happy with the
Hitachi drives I've used, and have had very few failures (out of hundreds
I've used). I probably would have sprung for the Ultrastar version, but we
had these drives in inventory, already tested, and for our normal use
should probably be using the Ultrastars.
The drives are where you have to do the hard math. 500GBs are far
less than half as expensive as the 1TB drives right now. Plus, when doing
RAID you're wasting drives. So, I could have gotten 6 1TB drives at $200 or
12 500GB drives at $75 for the data storage. The need for the $240
external enclosure and $80 controller card pretty much makes them even in
price. Add on the 2 drives worth of redundancy I want though, and I'm only
wasting $150 with the 500GB drives instead of $400 with the 1TB drives...
Cost: $75 each ($1050 for 500GBx14). The 250 was just a spare drive,
but new cost would be around $40.
It's been great that Sun open-sourced the ZFS code. It's a great
technology and it's nice that we haven't had to duplicate effort by
building a similar set of functionality under Linux. Obviously, the
flexibility of Linux has been a huge help here as well, since even though
it couldn't be brought into the kernel directly, we were still able to use
The above images of the individual components are from the
manufacturers web sites or product literature.
tummy.com has smart people who can bring a diverse set of knowledge to
augment your Linux system administration and managed hosting needs. See
the menu on the upper left of this page for more information about our
comments powered by Disqus