> As you mention, FreeNAS isn't optimized for that but a plain BSD install with ZFS didn't fare much better either and required a rather inane amount of work compared to other solutions that worked OOTB.

That's pure conjecture. Plus I actually have ran ZFS on a Pentium 4 on FreeBSD. For a while that was my personal backup server.

> If you use mdadm and ext4 you can do it online.

Not according to any of the docs I've read in the past. Happy to be proven wrong if you have a citation though.

> There is a lot of home users. People will use RAID5s and 6s at home, they'd rather have no-though-just-works solutions.

> The home desktop is possibly the largest filesystem market next to "disposable virtual machine".

I don't disagree with that but how many of those users are running LVM + mdadm? And how many of those people are just using unraid (or similar purpose built solution for home users)? How many users are just using an off-the-shelf NAS like QNAP et al?

The issue I have with your complaints is that you're flip-flopping across a multitude of industries from desktop to enterprise, which seems grossly unfair for making a balanced opinion. I'm not going to deny that ZFS will lack some desirable features compared to some speciality tools but like I said before, it's about picking the right tool for the job rather than arguing that ZFS is garbage because it doesn't win every single argument in every specific speciality.

I mean you're not even willing to take on board any different configurations of ZFS nor alternative platforms it ships on - instead only basing your biases on one failed attempt at running FreeNAS (something which openly states they need beefy hardware to run) on equipment that is several generations old while then comparing that one specific ZFS instance against every alternative tool you've read about. That hardly seems fair, don't you think?

> They don't behave the same. In these solutions if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage.

No raid would give you 3TB of extra storage when you add a 2TB HDD. That would be breaking the laws of physics because you're now asking your server to create 1TB from thin air.

But to assume you meant 2TB of extra storage when you add a 2TB HDD: in your situation you don't get redundancy so your data becomes at risk if you have a failure. You can actually expand your ZFS tank the same way if you really wanted (but no-one who cares about their data would want to) by adding a HDD vdev (unraided) to an existing tank.

The issue is when you want to expand an existing raidz while retaining redundancy. To do that on any of the existing FOSS tools I've used, you'd have to either buy more than one HDD or do an offline restripe (as is the case with mdadm). I guess at least mdadm allows you to do it - even if it takes the storage pool offline for several days. Personally I've always opted to buy multiple drives and expand the pool (tank) by adding an additional raidz vdev because that not only expands the storage array but also then gives me another failure option (eg I can then lose 2 disks simultaneously without losing data). As you start to expand the number of disks you have, you equally increase the likelihood that more than 1 disk might fail. So having that extra redundancy disk is desirable to me.

> On ZFS with a RAIDZ1 you'd get 5TB of storage since you'd have to stand up two RAIDZ1 vdevs.

Actually no. You have to stand up two vdevs but a vdev can be a raidz, mirror, a solitary HDD or even a file on an existing file system. So you can just add an additional HDD to a ZFS tank but you wouldn't get redundancy on that additional HDD (which is why nobody suggests you should do it).

> I've been running a few hypervisors using ZFS as storage backend, though at the moment I'm migrating to a LVM thinpool based backend. I've had a few share of problems on that as well, notably ZFS at some point consume almost all of the 32GB of memory and running the OOM killer in the linux kernel until everything had crashed and the host required a hard reboot. Though this hasn't happened since.

Yeah, I don't much like how ZFS's cache isn't shared with the kernel as freeable RAM. I get why they did it that way and generally it's not an issue (aside monitoring / reporting tools) because the ZFS kernel module should deallocate memory itself when it detects system memory is getting low but clearly that didn't happen in your case (which is strange - sounds like you found a bug that was worth reporting). Frankly speaking, I'm not a massive fan of ZoL anyway. I've got it working before but honestly I much prefer running ZFS on FreeBSD or Solaris / OpenSolaris,

With regards to your particular use case: I wouldn't have recommended running ZFS for your hypervisor backend anyway. You don't really want CoW for write-heavy processes - particularly ones that have their own journals and/or snapshotting systems such as databases and hypervisors would. Like I said before, it's about using the right tool for the job and XFS would be my preferred Linux file system in those particular instances.