I'm running ZFS on OpenSolaris kernel (Nexenta, for GNU userland) for home storage, with my main zpool at 9.06TB spread over 8 hard drives, for almost 7TB worth of ZFS storage.

ZFS isn't quite what's billed to be, but it's (mostly) better than every other current choice. You can read the positives everywhere else, but let me point out some of the negatives.

* Space wastage: block sizes for files grow up to 128K, but they never shrink, and the last block in a file is currently always the same as the overall file block size. This means that files e.g. 129K in size will have 127K of wasted space on disk, and files that were once 128K and truncated back down to 1 byte will still have 128K block size. Depending on the kinds of files, you can see space wastage rates in the region of 25-45%. This isn't theoretical: I had to reduce max block size to 8K on one filesystem to avoid huge wastage.

* Streaming latency: as a media server, I often play movies from data stored on this server, but there's a problem somewhere in the stack (network, OS file cache, filesystem implementation). If I'm playing a movie or TV episode from a ripped DVD (i.e. VIDEO_TS directory) I can guarantee that there'll be at least one glitch in every 40 minutes of playback, where network transfer has dropped to zero for 3 or 4 seconds.

* Device removal: you can add more storage to a ZFS pool, and one can replace any given device with a drive equal or larger in size, but it's not possible to remove a device. If you start out with 8 disks, you're stuck with 8 disks unless you want to backup and restore the entire FS - but at least FS streaming is easy with e.g. zfs send <snapshot> | ssh <remote> zfs receive <filesystem>.

* Fragmentation: there is no defragmentation solution currently. Sure, people say it's "not an issue", that "ZFS is not pathalogical", but the same was said about NTFS, and NTFS fragments like crazy when small incremental writes are used. ZFS performance is known to fall off a cliff (e.g. less than 1% of prior perf) as you approach capacity (symptomatic of fragmented free space) and, more importantly, stay abysmal until you back off quite a bit and preferably drop a whole file system or two. Recommendation is to stay under 80%, which isn't very full.

* Performance: performance is a bit of a red herring; you can make ZFS as fast as you want by adding mirroring and striping. The more interesting case is when you don't have mirrors and stripes out the wazoo, and are relying on RAIDZ or RAIDZ2, and here the performance depends on access patterns. The way the parity and checksumming works, you need to burn a lot of cache to get decent small random read/write performance, because entire blocks need to get read and written even for small updates. So performance looks great for a while, then falls off.

I still think ZFS is one of the best choices for this kind of application - the software RAID, multiple-volume handling is terrific - but it's far from the last word, and not 100% baked (needs defragmentation and data recovery tool ecosystem). BTRFS looks like it could be a very worthy contender, but from what I've read, I haven't seen that it values a storage-unit/pool/filesystem stack, where storage-unit can be a disk, a file or a partition, and still seems stuck on the old device/filesystem approach.