>where once the units entered into production they were never touched because physical maintenance had a tendency to cause more failures than it fixed.

Well, to make this happen, you can't use conventional RAID (well, you could use conventional raid, and just set, say, five spares, but once you are out of spares, the array would be binned)

The thing is? it's very rare to take out an array when swapping a drive. (I've... actually done it quite recently, but that was a combination of me making an extraordinarily unwise choice (Oh, I can get by with a used chassis/backplane, even though it's a brand I don't normally use, and a design I'm probably not qualified to re-qualify, no problem!) and me being an idiot when it came time to force-reassemble the RAID (I still don't have good docs on this, but one of my people now understands the problem of what data is on what drive, and what order to list drives during the force very well. I just need to get them to document. Or, really, I should sit down with them, really understand it, simulate a similar failure, and then document it myself. For now we've instituted a policy that two people need to sign off before using force with any mdadm command.)

I mean, if you are using conventional RAID, you need to keep spares. heck, you could run a 36 bay raid, and just designate, say, 4 spares, and 90% of the time, you'd want to replace the whole chassis before you ran out of spares. But now we're paying what, another 10% for disks? (the disks completely dominate the cost of cheap storage arrays; really nice 36 bay chassis[1] with room for a motherboard cost well under $1500. for a few bones more, you can get a similar chassis with 45 disks and no slot for a motherboard... And, of course, you can go way cheaper than that if you are willing to resort to disks that can't be swapped... but my point is that the cost of the chassis, even those nice supermicro chassis where all disks can be swapped without de-racking the thing, is already way below the cost of the disks you are throwing in it, so buying more disks in exchange for getting a cheaper chassis is likely a false economy.

I mean, you do have a point when it comes to labour... it is a big deal to get someone to swap a drive (and if you don't have a spare, the swap needs to happen right quick)

and yeah, swapping drives isn't without danger.

Now, the economics of this changes if you have some CEPH like system where you can fail an arbitrary number of drives in one chassis and still have the good drives function. But those systems are all relatively new, and have quite a lot of complexity overhead. I mean, in theory, if you have an array half full of good disks, with such a system you could migrate that data to a new array completely full of good disks, then remove and refurb the array that is half bad, but want to talk about complexity and chances to screw it up? yeah.

Also note, most drives I buy come with a warranty, meaning a bad disk is a token that is good for one completely free disk. 'enterprise' disks don't fall in price nearly as fast as consumer grade disks (yeah, go find me a 500gb 'enterprise' 3.5" 7200rpm disk for under $70 that isn't used or refurbished. Yeah, that's what I thought.) And often, if you warranty a really obsolete drive, you get back one that is fairly newish. I've warrantied a bunch of WD re3 drives and gotten back re4 kit (difference is that re4 has larger cache and fewer platters, meaning fewer/lighter r/w heads, meaning better seeks and, of course, fewer platters means less spinning weight and thus less power consumption.)

Of course, that's not a factor if you buy your drives without warranty. I don't know any way to extract even the shipping cost out of bad drives that have no warranty. (If you do, lemme know; I'm giving 'em away right now. Hell, half of 'em could be resold by unscrupulous folks; I discard drives once they start showing uncorrectable read errors, even if there is enough space to remap the bad sector, while some people don't replace the drive until they run out of space to see bad sectors (e.g. you will then start seeing /consistent/ bad sectors across badblocks runs.)

[1]http://www.supermicro.com/products/chassis/4U/847/SC847A-R14...