"That reads almost like the marketing blurb on a can of "magic enterprise pixie dust". ;-)"

Lol. So one of the things that Web 2.0 can teach is where to layer. Its something we do aggressively at Blekko and if you read the OpenCompute stuff that Facebook has done, or the papers on AWS or S3 as well the fundamental theory is all very similar.

Basically you have a 'system' which is nominally some CPUs, some disk, some memory, maybe some Flash, and a network connection. You create a pretty flat layer 2 'fog' out of them. Basically you name them algorithmically so the ops guys can find it when its broken, you assign everything fixed IP addresses by where it sits in the rack and for some level of 'nominal' you run all of the base set of software on each 'droplet' in this fog.

Now you go a bit further and you separate things by which ethernet switch they are on, which powerline circuit, maybe even which 'phase' of the incoming power, etc. And you let all your droplets count modulo the number of failure domains so if its power/net/cooling that would be three failure domains 0, 1, and 2.

Now as a scheduling exercise, lets say your application uses a soft storage layer to replicate by 3 or by N depending on your IO/sec requirement and you need Y amount of computer power and Z amount of RAM to hold the working set so that you can deliver the right latency. You figure out how many 'drops' (you might call them machines) you think that is, you install that many plus say 10% more. So to put real numbers lets say you have 100 drops (machines) and 10 spares. You spread those droplets through your three failure domains 37 per.

Now when all 100 machines are healthy, you run along just fine, everyone humming. Plague monkey decides that machine #82 is 'unhealthy' (maybe their disk is spewing errors, maybe their network keeps dropping pings) the key is that its kinda there and kinda not. For Map/Reduce or Hadoop type systems perhaps it is consistently the straggler in the reduce loop. Plague monkey comes along, shoots #82 in the head, sends an email to Ops that says machine R3-S4-N18 (rack 3, shelf 4, switch-port 18)is sick, its been shot in the head, spare machine R5-S1-N2 was tagged as being 'it'.

Now your system still has 100 machines running the application, but its down to 9 spares and there is one machine on the sick list.

Now if you've designed your system to be resilient to failures, and you have because you are using cheap whitebox servers which you know are going to fail randomly. You know that you can deliver against your service level SLA with up to 15 machines down (85 out of a nominal 100 running).

Operations can pick a 'badness' number which is '20' which is to say once 20 machines are identified as needing to be 'fixed' the margin for error is down to 5 machines (which is to say of your 110, 20 are dead so 90 are still kicking and all part of the 'active' pool). At that point even an unhealthy machine that is still providing some service is better than dropping below 85 which would take you from degraded (remember at 90 your are off 10 machines) to dead.

So in my shop at least, long before you get to the 20 machines out of 110 dead you're installing replacement machines. But if you were having a particularly bad long weekend (and the pagers had all mysteriously failed) then you can make things 'less worse' by not shooting any more machines in the head.

"The nasty stuff for complex applications is in unexpected/non-linear bottlenecks (RDBMS-monkey?), intermittent failures (packet-loss-monkey?), cascading failures (monkey tag-team?) and... human error (beer monkey, bug monkey?)."

There is an architectural invariant which avoids these, which is that you don't build systems which can't tolerate the sudden loss of the machine they are running on. Its something Google does masterfully and Oracle does not at all (AFAICT, its been a while since I was near an Oracle implementation). Which is why I think of it as the Web 2.0 way of doing things.

The AWS and S3 papers allude to some of these strategies as well, and of course NetFlix has pretty much gone 'all in' as they say in the poker tournaments. And no, not much of this stuff is 'off the shelf' (yet? who knows) But from an operational investment standpoint its nice to know that you can take a number of hits before you stumble. It really makes the economics work out much better.

Its not a free lunch however, you really are running those 10% of spare machines and not generating any revenue from them. So when utilization determines profitability you start balancing running on the edge vs not having any on-call ops guys around the data center during the weekend. There are ways to mitigate that pain, you can use those machines for development or test, throwing them into the 'production' cluster on an as-needed basis but at the end of the day if you just have to run them for resiliency you need to compare that cost to the cost of engineering more resilient equipment. But hey if it was easy anyone could do it :-)