Recent Journal Comments

/me likes

It would be nice if we could easy_install wcfqfu without having to remember long FTP URLs and without having to wait for distutils2 to appear in some future release of Python.
(go to entry which this is a comment to)

I'd have liked to have seen that!
(go to entry which this is a comment to)

I'm using a variety of drive sizes for ZFS, mostly 250GB and 500GB, mostly because these are what I already have available. If I were buying drives for a new storage box today I'd probably use 5 2TB 5400RPM drives to give me 6TB of storage with RAID Z2, slower but also using less power.

Of course, RAID Z2 makes smaller drives more attractive, because of less overhead (10 1TB drives would have 8TB data 2TB parity, 5 2TB drives gives 6TB data 4TB parity), so it would mostly be a matter of the math.

10 drives would fit internally on my current layout, so anything under 11 would be my best bet.

Sean
(go to entry which this is a comment to)

What size of drives do you use ZFS?
(go to entry which this is a comment to)

Yes, I keep several rsync copies: one for the daily rdiff incremental source and one for the monthly rsync source. I don't have the bandwidth to the backup server to do a full rsync every day -- it's limited by the 768kbps on my home network, and I have 4 or 5 machines to do every day.

So far it has been working extremely well. Once I got the initial setup issues ironed out, it hasn't had any problems.
(go to entry which this is a comment to)

This looks intriguing to me, but I'm curious if you are keeping the rsyncs around (so you only need to xmit the rsync diffs before taking rdiffs) and incurring a ~100% storage requirement increase, or are you removing the rsync copy every time?

Thanks
(go to entry which this is a comment to)

The recipe looks good. except I could never get the cdrom drive to work off the bat. Would have been nice to see how you did that in more detail
(go to entry which this is a comment to)

I took a look at rdiff-backup and made the Move to it. This makes a lot more sense then rsync with Hardlinks.

Thanx for turning me on to rdiff-backup.
(go to entry which this is a comment to)

This was very useful, thank you.
(go to entry which this is a comment to)

That sort of article is still a ways off. I had attempted to do some serious use and testing of btrfs a few months ago, but at that time the ability to delete snapshots was not implemented.

They've just implemented this and it'll be in 2.6.33 (Probably Fedora 13 and Ubuntu 10.10, I've been told not in the upcoming 10.04).

At this point I'd say the comparison I'm using is that btrfs has just caught up with ZFS circa 2005. It's got very cool features and sounds very promising, but it's very young. The latter is not really something you want in a file-system. ;-)

File-systems are just hard things to get right. I'm reminded of ext3, which was just a set of changes to ext2, and that took around 4 years to get out and another couple to stabilize.

Sean
(go to entry which this is a comment to)

The 'x' feature is handy as well. I used this to delete a NAT table entry for a machine that had changed IP address. I couldn't reconnect with a VPN through my linux router until the NAT entry had expired. Instead of waiting for the TTL to expire I was able to erase the connection entry with iptstate.
(go to entry which this is a comment to)

Any chance of an article comparing btrfs and zfs-fuse?
(go to entry which this is a comment to)

Sean, thanks for your replies. Well, I have basically 2 other problems with rdiff-backup (besides the ones you already noted); it doesn't compress and it can't currently back up ACLs and EAs. And on several setups we need to use them. OTOH I'm somewhat more familiar with tools that create monolithic archives + increments (yes, they're more vulnerable to disk errors), like dar which I started experimenting with lately. And, if possible I'm for a free, open source solution. Anyway, your insight and experience discussed here shed some more light to some other aspects too, thanks again.
(go to entry which this is a comment to)

BackupPC is a very nice tool for some situations. I was planning on deploying it for our laptop backups, but the workflow I developed for our normal backups is for Ubuntu and Ubuntu was having issues on the system I deployed for our laptop backups. CentOS 5, which I ended up putting on that host, uses packages that required quite a lot more work above what the existing workflow was.

So, that was the motivation that got me to spend some time trying this other backup mechanism. rdiff-backup has some very nice features.

As you say, some database files do need to be dumped or flushed to get a consistent backup. I was speaking primarily of database files that are append-only, or similar files like log files.

Yes, we have thought of using tapes, but they do not really fit our needs. We have also considered agent-based backup systems, but have dismissed them for similar reasons.

For backing up virtual systems we basically just treat them as if they were normal machines. That has served us well.

Sean
(go to entry which this is a comment to)

Hi. I'm reading your blogs sometimes to always find something interesting. With my startup, I'm planning out backup strategies for several boxes we hope to have more of in the future. One important point is what backup strategy/medium is to use. I think I gather my thought in the form of some questions I hope you find some time to answer in short.

- I've read earlier you are using BackupPC. For a few hosts it's working well for us. Did you not find this tool sufficient, having to look for alternatives (besides the ability of partial updates with other mentioned BU tools)?

- Have you ever thought of using tapes? The equipment is relatively expensive, but tapes are relatively cheap. And no worries of many files and hardlinks. There are worries of other issues, of course. But with that many hosts you're dealing with daily, I'd consider using them...

- Have you thought of using agent-based backups? Many of them can solve the ever-increasing monolithic file backup problems too (well, some of it, most particular being the backup of various database files on MS hosts and Outlook PST files, etc). There are free tools to choose from, too, like A(Z?)manda, Bacula, etc.

- I'm also wondering why you like to backup database files as-is (being one of your primary reasons looking for diffing backup tools). A database datafile or tablespace file might not be in sync with disk contents (TBH on even a moderately intensive workload, it never is), so it's practically impossible to create a consistent, working copy. A mechanism should be used that either syncs/locks database files during backup (which in turn causes a stall in DB access), or other methods should be considered, IMO, like dumping consistent backups of the DB in question, and backing up that dump. Or use facilities that are meant for consistent offline backups, like the binary logs of MySQL, etc.

- I've noticed you have changed your VPS infrastructure to VMWare some time ago (what were your company using before that btw?), do you have any recommendations to look for when doing incremental backups of VMs? We too plan on providing some VPS solution, the most probable solution we'll provide is OpenVZ and/or KVM-based virtualization. The latter uses images similar to VMWare.

- chris
(go to entry which this is a comment to)

I've been pretty happy with it. The service seems to be the same as Sprint, we haven't seemed to run into the 5GB/month throttling. The card is a little weird with the mode switch under F12, the original Sprint modem that didn't require that was a bit more reliable, sometimes I have issues getting it to switch mode. Under Karmic this seems to be less of an issue.

Sean
(go to entry which this is a comment to)

What's your opinion of Cricket Wireless' service and their data plans after ~4 months of use?
(go to entry which this is a comment to)

I really hope they succeed at this, but so far it looks to only be a wordpress blog with a post in October and a few replies to comments over the next week or two, and then complete silence. "Check out our twitter feed for the latest developments" -- twitter seems to only have the initial announcement as well.

So, I'm not jumping for joy about this yet. ;-/
(go to entry which this is a comment to)

"I wish we could have ZFS in the kernel, but the powers that be have decided that will never happen."

Maybe you don't have to wish too hard, as that might come true soon enough: http://kqinfotech.wordpress.com/
(go to entry which this is a comment to)

check it out! Performance has increased, it supports more current features etc
(go to entry which this is a comment to)

Ok, I'm even more years late... *G*

I liked your settings, but the quad-click also selects spaces in front of the line content like this:

>     This is a test       \n
 ^^^^^^^^^^^^^^^^^^^    <- would be marked

using: xterm*on4Clicks: regex [^ \t][^#$]+

>     This is a test       \n
      ^^^^^^^^^^^^^^    <- would be marked

(go to entry which this is a comment to)

If you like Science Fiction, check add this website to the list of sellers that offer non DRMed books in most formats: http://www.webscription.net/

There are some available for free and others usually cost around 5$ each. There are some good books and series there.
(go to entry which this is a comment to)

--link-dest is exactly the sort of "linking trick" that I reference in my original post. It works fine for fairly small sets of data, but it's quite expensive to remove these archived backups (requiring hundreds of thousands or millions or more file delete operations, directory traversal, etc).

The biggest issue is, say you have a 20GB file that every night has 100KB appended to it. Or 100KB within it updated. You know, a fairly typical database file. And you are keeping 30 incremental copies...

With the hard links, this requires 600GB of storage.

Using rdiff-backup or ZFS snapshots, you end up using more like 20.03GB for the same dataset. Or a space savings of 97%.

Disc space is cheap, but it's not that cheap when you're talking about 5 TB versus 150TB...

As far as LVM snapshots go, you have to know beforehand how much space one of these snapshots is going to require, or you have to overcommit and make the snapshot volume larger than you ever expect it to reach, or you have to set up the snapshots to automatically extend when they run short (but not out) of space.

As far as overallocating, I'm looking at having a thousand backup copies on one of our larger systems. Over-allocating by even just 1GB results in a wasted terabyte right there. And I probably can't guess that close to right. Or I have to snapshot the whole backup file-system, and count on each snapshot being rather large, but also that if I move a backup from one host to another, it's still going to have all of that data reserved (probably through the old snapshot copies) for the next year.

Oh, and if you have 30 snapshots of a piece of data and it changes, you have to write and keep 30 copies of that changed volume (one for each snapshot), or you have to have rolling snapshots (one snapshot snapshotting another snapshot, can you do that) and then only be able to trim off the ends, so no trimming from the middle -- you need rolling snapshots for each backup type).

If you have had good luck with setting up and managing 1,000 LVM based snapshots, I'd love to hear about it. However, it seems like it would be a maintenance and performance nightmare. I've toyed with trying it out, since I think LVM snapshots are more robust than ZFS or btrfs are right now, but I just haven't gotten up the urge to try it.

So, the mechanism I wrote about is similar in ideas to many of these, but I believe it's dramatically simpler than LVM snapshots while saving more space than hard links.

The target I'm shooting for is like ZFS snapshots. They automatically manage their space, so you don't have to guess at how much space is going to be used by a snapshot, and it gets allocated out of the main file-system. This is because as blocks are written, they are copied elsewhere (copy on write). And you can create many "light weight" sub-file-systems within the ZFS file-system.

So, I'd create a "backups" ZFS, and then within that make one file-system for each system. For each of those, every night I would take a snapshot. I would delete the snapshots as time went on such that I ended up keeping monthly interval snapshots beyond 6 weeks, and weekly beyond 14 days.

ZFS managed all the complexity behind the scenes.

Sean
(go to entry which this is a comment to)

You could you rsync with --link-dest "link destination"

I have rsync setup with lvm snapshots and it works very well. I have been running this for about 6 months without any issue.

check out the following URL for more info. http://www.mikerubel.org/computers/rsync_snapshots/

Paul...
(go to entry which this is a comment to)