I'd never run a "grow" on a Linux software RAID array before, but my storage server needed some more space. The manual page for mdadm is not really obvious about how exactly you add drives to a RAID-5 array, but everything went smoothly once I figured out that you first have to add the drives as a hot-spare. Read below for more details.

I'm running a 10-drive RAID-5 array of 250GB drives. These were the sweet spot for price/performance when I did the system setup. I went back and forth about upgrading to newer drives, but finally decided to just see if I could add 5 more 250GB drives to limp along for another year.

My setup is a bit peculiar, I'm running a RAID-5 on /dev/md2, which is a LUKS encrypted volume with ext3 on it. Everything went smoothly though. Note that I'm only adding two drives initially, because I need a PCI-Express video card to free up a PCI slot to add more SATA ports to get all 5 new drives.

First I needed to add the additional drives to the array as hot spares with the commands: "mdadm --add /dev/md2 /dev/sdc3; mdadm --add /dev/md2 /dev/sdd4".

Then I told the array to grow from the current 10 drives up to 12 (using the two hot-spares added above) with: "mdadm --grow -n 12 /dev/md2".

Now I waited around 16 hours for the array to restructure itself to the new layout. It has to basically copy things in place from the 10 drives out across the 12, so this just takes a while. Check "cat /proc/mdstat" to watch the progress.

Then I needed to tell the encrypted partition it was bigger. I first had to enter my passphrase with "cryptsetup luksOpen /dev/md2 storage", then resize it with "cryptsetup resize storage". If this was a LVM partition, I'd have needed to do a "pvresize" instead of the above.

Finally, I had to increase the size of the file-system. I could have done an online resize by mounting it and then using ext2resize, but I hadn't done an fsck on that partition in over a year so instead I just went ahead and did a full fsck then ran "resize2fs /dev/mapper/storage".

To answer a few questions I've seen asked about the resizing...

Is it safe to do the RAID-5 resize while the device is in use? Yes, it should be. It's designed to keep track of where it is so you can even reboot while it's growing the array, and it will pick up where it left off. It basically does a lock of a slice of the data and reads it from the current layout, then writes it back out to the full new set of drives as a slice, then unlocks it. So if you write anywhere else during the operation, it isn't impacted, and if you write the slice that's being relocated, the write is just deferred until it's been relocated.

You can't grow the file-system until the RAID resizing is completed. This is because while the resize is going on, the size of the RAID array is still limited at the old size. Once it completes, it becomes the full size. So it's not a progressive growth, like you'd get with an online resize2fs. You're stuck waiting for the array to grow before you can make use of any of the space.

Do you lose any data on the array while doing this? Nope, it preserves all the data. I had run a "dd if=/dev/md2 of=/dev/zero" before doing the reshape on it, to make sure that there weren't any bad blocks on any of the drives before doing the reshape. Running this sort of verify can find marginal sectors that aren't normally used, and it's an important part of running a RAID array. Also, make sure that you don't have a failed drive before running the reshape. Many people deploy RAID arrays and then don't check them for failure, because they consider them to be robust to failure. Then they only notice the second drive failure... :-( Of course, you should have a backup of your data in any case, and definitely when doing something like reshaping, but in the normal case you shouldn't lose data -- I definitely didn't.
(Post Reply)