I've been using ZFS for a backup server via the Nexenta distribution for around 3 months now. In addition, I've been testing the new Nexenta. While ZFS is insanely cool, the road has been far from smooth. Here are some of my impressions.
The concepts behind ZFS are, of course, exceedingly cool. All data written to disc is checksummed, so the file-system can detect corruption at any stage, including a disc or bus silently corrupting data you write to it. The transparent compression is also great, it's saving us around 25% on storage space.
Nexenta, if you aren't familiar with it, is an OpenSolaris distribution with many parts of the OpenSolaris codebase (kernel, SMF, etc) combined with apt-get and GNU tools from Ubuntu Dapper. It's currently in it's 5th Alpha release, with the first version I tested being the second alpha.
The cool thing about Nexenta is that it includes apt-get to easily install packages. One of the guys at the NCLUG meeting last week is a Solaris guy, and he was griping about not being able to just install packages for Postfix and Mailman. This should help that. Except that with Alpha 5 I haven't been able to get a single package to successfully install via “apt-get”. It always errors out. Luckily, the install CD includes many packages, and others I can download the Debian package directly from the Nexenta web site. I'm installing pretty minimal sets of packages on these boxes, so that's not a big problem.
The bigger problem is that the kernel update packages come pretty slowly. This wouldn't be as much of a problem except that I'm having serious problems with ZFS, and would like the ability to try newer kernels. The OpenSolaris developers seem to be releasing new kernels every 2 weeks like clockwork, but I have yet to see a kernel update package available.
In the early days of Nexenta, kernel updates via packages weren't possible, but as of Alpha 5 they are supposed to be. “apt-get dist-upgrade” doesn't show anything to update from what the CD installed though. Alpha 5 has been out just over a month, so there should be some level of updates.
As I mentioned above, ZFS is having some issues. The initial install I did was running fairly well, but once the amount of data on the system got to be around 1TB, the “scrub” process (where it goes through and verifies checksums) will hang the ZFS storage pool after running about 12 hours. This “scrub” process is also how you recover from a disc failure.
I'm happy to report that my testing of Alpha 5 shows that this is resolved. I was running some tests with a simple set of 1TB of data and running repeated scrubs. It would do the scrub in around 3 hours, and I kept running them for 4 days without the pool hanging.
So, my plan was to migrate the data from the existing ZFS to the new one on the new box I'm setting up. ZFS includes this nice backup utility that will dump the raw file-system snapshots, so I could preserve all the data and snapshots on the original system, given a bit of scripting to wrap around it.
Sadly, in my testing of copying a file-system with it's 10 or so snapshots, the initial file-system copied over fine, but the migration of the incremental snapshots caused a kernel panic and the system to reboot. I guess I need to set up a test system with Alpha 2 on it, create a ZFS, and then try doing an upgrade to Alpha 5 and see if that will not freak out.
In general, I have found the Nexenta user-space to be pretty solid. The only real problems I've had have been related to ZFS. It still seems that ZFS is not ready for prime time. Interestingly enough, it looks like ZFS is being used in Solaris 10 now, because Sun announced this new server that has 24 discs in a 4U chassis, and it's using ZFS for the file-system. I imagine that the attention that gets will translate pretty quickly to more stability in OpenSolaris.
Finally, there is a google summer of code project to port ZFS to Linux. I imagined that would be a pretty tough port, because of the way ZFS unifies RAID, LVM, and the file-system. However, at BLUG, Kevin looked up the project and found that it's being ported as a user-space file-system. Not really usable for many things, but I could definitely make use of that for some archive storage needs. The checksummed data would be really nice to have on Linux.comments powered by Disqus