This is a wonderful article, but I feel like it is kind of missing a certain depth. Read Donella Meadows’ classic Thinking in Systems for instance.

So like, yes, of necessity the largeness of a system causes the existence of feedback loops such that it maintains an equilibrium state, a homeostasis. This is kind of just a restatement of what a system is, and to characterize it as “the system is constantly failing,” while that is totally true, seems a little bit sensationalist. Like, you are one of these complex systems, do I say that you are constantly failing to draw breath? No, but I imagine some of your alveolar sacs routinely fail to fully inflate and others are too covered with mucous to absorb oxygen and whatever else, and often your breath runs ragged. Similarly with “you cannot identify a root cause,” a kid dies in a pool and we routinely have a medical examiner pronounce that the cause of death was drowning, which in the respiratory sense can be fairly well isolated to a root cause, “could not breathe air because no air was supplied by the system upstream.” The failure to find a root cause essentially only becomes hard because there is another aspect of complex systems which Meadows discusses, which is that they bleed into each other and into the World At Large, they do not have clearly defined boundaries until we impose them for analysis. Draw the boundaries larger than the respiratory system, say the social system, and now the reason that the kid died has something to do with parents never having been trained on what drowning actually looks like, so they heard some light splashing and looked over and said “aww isn’t he enjoying the water” not realizing that this was actually a “holy fuck my son is struggling to keep his mouth above water” moment—this perhaps speaks to a larger problem of not having good water education in society, maybe that is the root cause. Stuff like that. In that sense yes there are failures at all levels, but at any given level of explanation byou can more or less say that this is the thing that at that level explains the problem.

So my servers go down, there is a valid root cause at the level of the individual server; there is also one at the level of the cluster as a whole and what that was doing; there is a valid cause at our developer and operations practices; there is a different cause at the level of company culture too. I don’t think that it makes sense to say that there is no root cause. The root cause of the Challenger explosion was that it was too cold to fly that day and they still flew the shuttle anyway. One can zoom in (“the fuel tanks were supposed to be sealed by rubber rings, but at the cold temperature of launch day the rubber no longer sealed the tanks”) or out (“NASA leadership built a false confidence in the quality of their work bolstered by a sort of management double-speak that disconnected them from the real risks they were playing with, making them foolhardy in ordering the launch”) and one can even sort of pan sideways (“why was the weather so cold that day?” or “what financial and engineering considerations made them choose this construction with these o-rings, several years ago?”). But this “middle explanation” of “it was too cold to fly that day and they flew the shuttle anyway” is a good summary that gives an entry-point to the problem, which is the real function of a root cause analysis (you want to collect information about how to fix the problem and know whether it was fixed).

The fact that this is a little bit over-sensational detracts a little from my enjoyment but I do think that the heart of the article is great and fun to think about. If folks like thinking about this stuff I really recommend Meadows's book though, systems thinking is not present in most curricula and that is a real shame.