By Sean Reifschneider Date March 6, 2013
Discrete storage, discs in server chassis, can be very wonderful in many ways: low latency, simple, robust... Todays data center environment is more and more often demanding features available from stand-alone storage.
Virtualization provides many benefits to the server environment, but (with the exception the smallest deployments) all but require shared storage such as a SAN. Over the years, our experience has shown time and time again that the solid SANs tend to run well into the 5-figures to get started. And the 4-figure solutions tend to be brittle if not unusable.
Additionally, many storage needs in the data-center quickly outpace the storage and IOPS available in a single 1U or 2U chassis (typically 4-8 drives). There are 2U and 3U chassis that can take 12 to 24 drives, but often at the time you are buying the chassis you can't predict the requirements 6-12 months down the line. Over-buying all your chassis can be quite wasteful...
What's a company with 3-5 virtual hosts (say, $3K-$7K of hardware) to do if you don't have $30K to invest in the entry-level Equallogic?
I've spent a lot of time thinking about this over the last several years, and while I don't have a drop-in solution for this, I thought others might find the options I've explored to be useful.
Sheepdog is a distributed storage system targeted specifically at virtual disc images. It has been around for years, but still seems to be in an experimental stage. However, you can easily add and remove storage nodes to it for additional capacity and redundancy. The Proxmox virtualization system includes limited support for Sheepdog in v2.1, and so we are setting up a test cluster to try it out. It looks promising and performance is good, but we are just beginning to try it out.
Ceph is similar to Sheepdog, but it provides a more general-purpose feature-set. It can provide a straight block device interface, as well as a general "object storage" system. This is still experimental, though the developers report having production deployments. Proxmox v2.3 includes support for provisioning to Ceph storage in the web UI, but the storage cluster must be provisioned manually. In my test of the block interface, performance was dramatically slower than raw disc access (around an order of magnitude), so that would have to be resolved before it would be acceptable in most deployments.
Back in 2009, a company I had never heard of announced the open sourcing of plans for a "storage pod": an $8,000 machine with 67TB of storage. Just recently, they have revved that plan to version 3.0, which ups it to 180TB for $11,000. Unfortunately, it's not just something you can buy for $2K for the base system, you can only get that price if you source everything yourself, which I have not found to be easy. Protocase offers ready-to-go chassis for $5,400, or $7,600 with redundant power supplies. Probably a good option if you need 45 drives and can do with minimal if any redundancy. Backblaze designs redundancy into their applications to survive failures of this hardware.
Gluster is a Linux-based file-system that can be deployed across systems to provide access from multiple nodes, replication across nodes, etc... Performance can vary, and there have been reported problems with the "NFS" access mode (which boosts performance), but for simple uses like distributing configuration files across systems or low performance situations it can be great. Growing and shrinking the cluster can require completely stopping and re-starting the storage system and associated downtime. This probably won't solve your virtual disc storage needs, but it has proven useful to us.
FreeNAS is freely available and is a custom FreeBSD distro that deploys to bare metal and results in a SAN/NAS using ZFS. The most recent version also includes encryption support. The web interface can be obtuse for dealing with iSCSI, but in general is a welcome benefit over using the command-line tools. The company behind FreeNAS provides commercial support and also full systems. ZFS can provide a read cache (as opposed to tiering) and write journals to SSD. We've deployed a few of FreeNAS systems and while it's a bit early to say for sure, in general they are looking promising.
This is a commercial bundling of Solaris+ZFS as a storage server, so the $2K price is for software-only. You provide the hardware. They charge based on raw storage capacity, count on $100-$200 per TB. Add-ons include high availability, Fibre Channel, Virtualization integration and migration. ZFS can be used to cache "hot" reads (as opposed to tiering) and journal writes to SSD.
While above the budget I've targeted for this article, the available features on this has fascinated me. However, I've never had an opportunity to test one out. In particular, this is one of the only options I've seen in this price range that has automatic storage tiering. It will automatically detect SSDs and place hot, random data on them, while storing bulk (or colder?) data on spinning discs. It has redundant controllers. Their strength has always been that you can expand capacity by pulling a drive and putting in a bigger one, and the unit figures out what to do to maintain redundancy but expand capacity. While Drobo has a strong name in SO/HO environments, I've heard conflicting reports in larger business environments. Would love to put one through its paces, if it works as they say I suspect I'd find the budget for it. However, I've since found NetApp FAS2220s may be in the $8K price point, which gives me pause at spending $12K on a Drobo.
For a brief period, NetApp decided to create a "little brother" called the StoreVault. Many of the same features, but at a lower price. Missing were redundant controllers, and some software features, but otherwise similar to the NetApp gear. Now you can only find them on ebay, rarely, I'd imagine with little to no support available.
When NetApp discontinued the StoreVault line, they recommended the FAS2020 as a replacement. That line is now in the FAS2200 series, with pricing not readily listed, but sounds like it could be had for around $8K. Used FAS2020 run more like $2K, but those usually are without discs. I'd imagine it's unlikely you can stick any old drive in there.
We've had clients with these and they've been wonderful to work with. Incredibly robust, redundant everything, smart (it tries to figure things out rather than require you to configure specific RAID configurations)... I would have several of these if the price weren't $30K and up. These often show up on Ebay, but I've always been concerned about the availability of firmware to be able to "stack" units (where the same version of firmware is needed). Buyer beware, but at this price you can get two full chassis for a hot spare.
This is a 4U server that Sun used to sell, with the ability to hold 42 drives. You would typically deploy this with Solaris as a storage server, but you could also deploy this running another OS such as FreeNAS or Linux with tgtd. No support, redundant power supplies but not controllers (it's just a PC). If you need lots of spindles, it may be a form-factor worth looking at.
This is a 4U chassis with 42 drive bays in it and redundant controllers. It supports both FibreChannel and iSCSI, and after a few initial stability problems we were able to get a firmware version that was quite stable. The down side is that this is a "drawer of drives" sort of format, so you need to be able to pull the chassis out for drive replacement. New these run $13K+, but they sometimes come up on ebay and may be worth considering. But: Buyer Beware...
I have just recently run across this, and it looks fascinating, though it's hard to tell what exactly the benefit is over using FreeNAS on a server chassis. Certainly this box is better if you are looking for a more "drop in" solution rather than something more custom from building a FreeNAS box. In short, it looks mostly like a commercial packaging of a ZFS-based SAN. It does have redundant power supplies, but no controller redundancy. At over $5K without any drives, the FreeNAS route looks worth considering.
While the feature-set of this seems, no pun intended, promising, the reality was problematic. It has redundant controllers, redundant communications paths, and the ability to expand the unit with external JBOD units, 16 drives at a time. A client purchased one of these units to augment an existing Equallogic, but at a lower cost. Despite fairly light use, repeated discussions with support, and numerous configuration changes and firmware upgrades, the controllers would reboot frequently (both controllers at the same time). Admittedly, this was several years ago, perhaps they have the problems resolved but after a year of them promising that there was a fix in the firmware that was just around the corner, and the new firmware continuing to have the same issue, it was retired.