Average Number Of Machine Failures In A Datacenter Given Just The Mean Time To Failure
Tags: math, statistics, harddisk, mttf
Published: 2019-01-20 15:18:12

---
title: Average Number Of Machine Failures In A Datacenter Given Just The Mean Time To Failure
published: true
tags: math,statistics,hard_disk,mttf
canonical_url: https://horia141.com/hard-disk-mttf.html
---

I’m currently reading [Designing Data-Intensive Applications](https://martin.kleppmann.com/2014/09/15/writing-a-book.html). Quite early in the book an interesting application of basic probability theory pops up. The author says _“Hard disks are reported as having a mean time to failure (MTTF) of about 10 to 50 years. Thus, on a storage cluster with 10,000 disks, we should expect on average one disk to die per day”_. The point they’re trying to make is that even with a component which is quite reliable, you will see a lot of issues at scale. Therefore large scale distributed systems _need_ to acknowledge this fact and be resistant to the failures. I think there’s some neat math behind the statement itself, irrespective of its sobering effect on engineers. So let’s see how we would go about deriving it.

The first thing to do is lay some groundwork for modeling the problem. Let’s say we have <script type="math/tex">N</script> computers. A computer can be in the _working_ or _failed_ states. All computers start up as working, but later can fail. Failure is temporary however, and a machine which fails will be quickly fixed. We are interested in the behavior of these computers with regards to failures across time. We consider time as flowing in discrete steps, much like it would in a [turn-based game](https://en.wikipedia.org/wiki/Turns,_rounds_and_time-keeping_systems_in_games). If a computer fails at a given time <script type="math/tex">t</script>, we can assume it will be fixed by time <script type="math/tex">t+1</script> (where it can fail again, of course).

Failure is a random phenomenon. At the individual computer level we can say that a given machine in a given day will fail with a certain probability <script type="math/tex">p</script>. The whole ensemble of computers then has a probabilistic behavior. It is more complex however, and part of that behavior well study later.

We need to make a series of simplifying assumptions. First we consider the probability of failure to be equal amongst all the computers. Second, this probability is independent of any external parameters, such as time, previous failures, or the order of the computer in the set etc. Therefore the single parameter <script type="math/tex">p</script> can describe the whole ensemble’s behaviour. Note that this is not an entirely realistic expectation. Some failures are correlated. If a whole rack in a datacenter loses power, than all of the machines in that rack will fail. The fact that the machines share some property (being in the same rack), means that their failures are correlated, or not independent. Other machines just have hidden faults which cause abnormally high failure rates after a point. If you’ve seen a failure from it, you’re likely to see another shortly. However, the model is decent enough to capture some good insights.

With this in mind, we can say that a failure for computer <script type="math/tex">i</script> at time <script type="math/tex">t</script> can be modelled as a random variable <script type="math/tex">X_{i,t}</script> with a [Bernoulli distribution](https://en.wikipedia.org/wiki/Bernoulli_distribution), with its single parameter set to <script type="math/tex">p</script>. For all <script type="math/tex">i</script> and <script type="math/tex">t</script> we get the same model, and these random variables are [independent and identically distributed](https://stats.stackexchange.com/questions/17391/what-are-i-i-d-random-variables).

At this point the question from the beginning can be stated as _“what is the expected number of failures in a certain day <script type="math/tex">t</script>, given that the probability of one machine failing is <script type="math/tex">p</script>?”_. To figure this out, we need to look at <script type="math/tex">T_{t}</script> - the number of failures in that day. <script type="math/tex">T_{t}</script> is the same for all values of <script type="math/tex">t</script>. We can write <script type="math/tex">T_{t} = X_{1,t} + X_{2,t} + \cdots + X_{N,t}</script> since each <script type="math/tex">X_{i,t}</script> produces either a <script type="math/tex">1</script> or a <script type="math/tex">0</script>, for whether the machine is in the failure or in the working state. The resulting model for <script type="math/tex">T_{t}</script> is the [Binomial distribution](https://en.wikipedia.org/wiki/Binomial_distribution) with parameters <script type="math/tex">N</script> and <script type="math/tex">p</script>. The expected or average number of machine failures on a given day is just <script type="math/tex">Np</script>.

So, for <script type="math/tex">10000</script> machines and an arbitrarily chosen low <script type="math/tex">p=0.0001</script> we’d expect to see one failure a day. We can use [Wolfram Alpha](https://www.wolframalpha.com/input/?i=binomial+distribution+n%3D10000+p%3D0.0001) to get some more insights. The graph is quite bad unfortunately, but there’s some good stuff as well. For example, the probability of at least one machine failure (which is referred to as “success” usually) is <script type="math/tex">63.21 \approx 2/3</script>. Or, one example of the numbers of failures across time is given as <script type="math/tex">1 \| 1 \| 0 \| 1 \| 0 \| 2 \| 1 \| 0 \| 1 \| 1 \| 1 \| 1 \| 0 \| 0 \| 0</script>. So on some days we even see <script type="math/tex">2</script> failures, and on others none at all.

The last step is to obtain <script type="math/tex">p</script> from the <script type="math/tex">MTTF</script>. We’ll focus on a single machine <script type="math/tex">i</script>, but look at it across time. The machine starts up in a functioning state, and given its relatively high reliability, chugs along for a number of days <script type="math/tex">F_{i}</script>. However, at <script type="math/tex">F_{i}+1</script> it stops functioning for the first time. Since this is a random process, <script type="math/tex">F_{i}</script> is itself a random variable. A good model for it is the [geometric distribution](https://en.wikipedia.org/wiki/Geometric_distribution) with its parameter set to <script type="math/tex">p</script>, which is _“the probability distribution of the number X of Bernoulli trials needed to get one success”_. The mean for this thing is <script type="math/tex">1/p</script>. But it is also our own <script type="math/tex">MTTF</script>, if we express it in days. So we can compute <script type="math/tex">p = 1 / \text{days}(MTTF)</script>.

So, for our initial <script type="math/tex">MTTF</script> of <script type="math/tex">10</script> years, we get a <script type="math/tex">p = 1 / 3650 = 0.00027</script>, which is thrice larger than our original test value of <script type="math/tex">0.0001</script>. So with our model and data we’d actually expect <script type="math/tex">2.7</script> failures a day. [Wolfram Alpha](https://www.wolframalpha.com/input/?i=geometric+distribution+p%3D0.00027) has more insights into the geometric distribution’s behaviour with these parameters. One example of the times to failures is given as <script type="math/tex">2259 \| 8792 \| 2066 \| 3770 \| 1166 \| 1179 \| 1428 \| 1720</script>. So, again, we can see there’s some variance. Some failures occur within <script type="math/tex">4</script> years, while others in <script type="math/tex">20</script>.

A special note is warranted here. If you pay attention to the sample of failure times, you’ll notice that some are quite big. Which should raise some eyebrows. Indeed, because of our earlier assumptions about independence in time, we missed out on a lot of extra complexity in the failure behaviours of these computers. Failure modelling and [reliability engineering](https://en.wikipedia.org/wiki/Reliability_engineering) is a more complex topic, and more precise results can be obtained.

Finally, let’s put things together to get a nice formula. Starting with a given <script type="math/tex">MTTF</script> in years, provided by the manufacturer, we can compute the probability of failure on a single day, under all our model’s assumptions as <script type="math/tex">p = 1/\text{days}(MTTF)</script>. From this, we can compute the average number of failures in a day as <script type="math/tex">T = Np = N/\text{days}(MTTF)</script>. Which is quite neat and hopefully easy to remember.