Your Linux Data Center Experts

I've been wanting to play with FlashCache since I heard about it a month ago. It's a kernel module that allows you to set up an SSD to act as a cache for a another block device. Day-to-day you probably don't need 32GB of cache, but it'd be hard to whittle my laptop down to fit in that much space… Flashcache seems like the best of both worlds: the speed you want with the space you need. Read on for the results of my testing.

Overview

Flashcache ships with two modules, named “flashcache” and “flashcache-wt”. The first is a write-back persistent cache and the second is a separately-developed write-through version. Write-through means that your writes have to make it onto the back-end storage before they are committed, so you don't get the performance gains on writes. However, it doesn't have the issue that the write-back does currently of possibly storing incomplete data in the case of a power outage or kernel lock-up.

My test system is a 1U server with 4GB of RAM, an Intel 32GB SLC Enterprise SSD, and a Hitachi 500GB 7200RPM SATA hard drive. I use 7GB of the SATA hard drive for the OS, and the remainder is for the file-system. The software I used is the latest out of the Github repository.

Note that unlike the “consumer” level MLC drives, this drive does not have the issues with degraded performance that were well reported when SSDs started coming out.

The first good news is that flashcache seems to have active development going on and discussions in the mailing list.

NOTE: My benchmarking here is not exhaustive, but is just meant to get some impression of the performance and overhead of flashcache as well as some experience using it.

Building

My first major problem was apparently with my initial install being 32-bit. The modules won't load due to unresolved symbols in the kernel modules. So I did a re-install with Lucid 64-bit and then the modules were able to build no problem.

To build the modules I just had to install the “linux-headers-2.6.28-19-generic” package and then run: make KERNEL_TREE=/usr/src/linux-headers-2.6.28-19-generic

After loading the resulting module I attempted to create a flashcache with the default parameters, and the machine paniced… I later tested out the “1.0” release instead of the latest out of github, so read below for more information on that.

Write-Through

So next I tried the flashcache-wt module. It also built fine and I was able to create the cache and begin using it with no problem. I put an ext3 file-system on it and then created an 8GB file to test with.

The writing of this 8GB file completed at a rate of 77MB/sec. I then did a “dd” read of it and got 120MB/sec. The drive is rated at “up to 250MB/sec”, but to be honest I'm really more interested in random access.

I had been looking around for a benchmark program that seemed to be good at doing random I/O, and didn't find one that I liked. I ended up writing a simple one in Python that detects the size of the file, and then does 30-seconds worth of random 512-byte reads followed by another 30 seconds of random writes. My results are around 4500 reads/sec (or around 0.2ms), and 500 writes/sec (or around 2ms).

One thing I didn't really think about when I was doing this test is that I have 4GB of RAM and am testing with an 8GB file, so probably half those reads are coming from RAM. In my write-back test below I blow out that RAM cache for better numbers.

Note I also ran a longer test for 300 seconds with similar results.

It's also worth noting that “mke2fs” and “fsck” both run in such a way that flashcache will not cache that I/O. So your initial file-system creation or checking will not be improved by them.

Write-Back with 1.0

So now I built the 1.0 release and tested out the write-back module. This was able to successfully load and my dd of the 8GB file of zeros ran at a slightly faster rate of 93MB/sec (compared to 77MB/sec with wt).

One interesting thing here is that while I was writing to the flashcache, it was writing dirty pages off to the hard drive fairly quickly. But once my 8GB write finished, the hard drive write rate has dropped dramatically. It's taken over 30 minutes to write 4GB. Maybe that rate would go up if the cache were over 25% utilized…

For the write-back random I/O test I get around 2700 reads/sec (0.4ms), and 1800 writes/sec (0.6ms). Very respectable numbers.

Of course, one thing to note is that there is an overhead in managing the flash as a cache device rather than just using it as raw storage… It has to manage data structures and traverse them so there may be several accesses required for what would be just a single access to a raw SSD.

Raw SSD Throughput

So what sort of overhead are we talking about? I made a raw file-system on the device and wrote the same 8GB test file to it. This ran at 180MB/sec (instead of the ~120GB seen with flashcache). This is right around what the drive is rated for writing. Reading that back in (after blowing away the RAM cache) ran at 211MB/sec.

For random I/O I got 3100 reads/sec (0.3ms) and 2300 writes/sec (0.4ms). So it seems like something around a 33% hit for using flashcache instead of using the raw SSD directly.

Raw Spinning Throughput

To round out the tests, I also ran the random I/Os on the spinning disc and got 74 read/sec (14ms) and 60 writes/sec (17ms). Those numbers seem a little high, but not very far off. I was expecting closer to 12ms.

Conclusion

So how much does the flashcache help over just a regular spinning disc? Quite a lot; close to but under 100x for random I/O.

So, it feels like the state of it is that you can get very fast reads, safely, but at the cost of having writes still be at spinning-media speeds. The write-back module has a known data-corruption bug, which I think is the blocker preventing it from being worth considering even if I can get the cache to be successfully created.

My experience has been that the performance of normal consumer “MLC” SSDs is good, but not great. Particularly with writes, every few days I find my machine becomes non-responsive for 10 to 20 seconds for no good reason I can tell. But the pricing of the SLC enterprise drives is way out there if I have to fit everything on it. An 80GB (which I can just BARELY fit on) Intel MLC drive is under $300, but a 64GB Intel SLC is $800. For server purposes it probably makes sense, but I'm not going to be using these on my workstation or at home.

But, if I can get the performance of an SLC SSD for $400, without having to fit everything in 32GB, that becomes a possibility. Once they fix the potential data corruption issue, that is.

comments powered by Disqus

Join our other satisfied clients. Contact us today.