5 Probability Distributions Every Data Scientist Should Know
Tags: datascience, beginners, statistics, datascientist
Published: 2019-06-18 02:36:19

---
title: 5 Probability Distributions Every Data Scientist Should Know
published: true
tags: DataScience, beginners, statistics, DataScientist
canonical_url: https://www.datastuff.tech/data-science/5-probability-distributions-every-data-scientist-should-know/
cover_image: http://www.datastuff.tech/wp-content/uploads/2019/06/digital-marketing-1433427_640.jpg
---

Probability Distributions are like 3D glasses. They allow a skilled Data Scientist to recognize patterns in otherwise completely random variables.

In a way, most of the other Data Science or Machine Learning skills are based on certain assumptions about the probability distributions of your data.

This makes probability knowledge part of the basis on which you can build your toolkit as a statistician. The first steps if you are figuring out [how to become a Data Scientist](http://www.datastuff.tech/data-science/3-machine-learning-books-that-helped-me-level-up-as-a-data-scientist/).

Without further ado, let us cut to the chase.

## What are Probability Distributions?

In Probability and Statistics, a **random variable** is a thing that **takes random values** , like “the height of the next person I see” or “the amount of cook’s hairs in my next ramen bowl”.

Given a random variable _X_, we’d like to have a way of describing which values it takes. Even more than that, we’d like to characterize **how likely** it is for that variable to **take a certain value** _x_.

For instance, if _X_ is “how many cats my girlfriend has”, then there’s a non-zero chance that number could be 1. One could argue there’s a non-zero probability that value could even be 5 or 10.

However, there’s no way (and therefore no probability) that a person will have negative cats.

We therefore would like an unambiguous, mathematical way of expressing every possible value _x_ a variable _X_ can take, and how likely the event _(X= x)_ is.

In order to do this, we define a function _P_, such that _P(X = x)_ is the probability of the variable _X_ having a value of _x_.

We could also ask for P(X \< x), or P(X \> x), for intervals instead of discrete values. This will become even more relevant soon.

_P_ is the variable’s **density function** , and it characterizes that variable’s **distribution**.

Over time, scientists have come to realize many things in nature and real life tend to **behave similarly** , with variables sharing a distribution, or having the same density functions (or a similar function changing a few constants in it).

Interestingly, for _P_ to be an actual density function, some things have to apply.

- _**P(X =x)**_ **\<= 1** for any value _x_. Nothing’s more certain than certain.
- _**P(X =x)**_ **\>= 0** for any value _x_. A thing can be impossible, but not less likely than that.
- And the last one: the **sum** of _P(X=x)_ for all possible values _x_ **is 1**.

This last one means something like “the probability of X taking _any_ value in the universe, _has_ to add up to 1, since we know it will take _some_ value”.

### Discrete vs Continuous Random Variable Distributions

Lastly, random variables can be thought of as belonging to two groups: **discrete** and **continuous** random variables.

#### Discrete Random Variables

**Discrete variables** have a discrete set of possible values, each of them with a non-zero probability.

For instance, when flipping a coin, if we say

> X = “1 if the coin is heads, 0 if tails”

Then _P(X = 1) = P(X = 0) = 0.5_.

Note however, that a discrete set need not be finite.

A **geometric distribution** , is used for modelling the chance of some event with probability _p_ **happening after** _ **k** _ **retries**.

It has the following density formula.

![](https://quicklatex.com/cache3/38/ql_a1aa64858e88e4fa841ecade06d08038_l3.png)

Where _ **k** _ **can take any non-negative value with a positive probability**.

Notice how the sum of all possible values’ probabilities still **adds up to 1**.

#### Continuous Random Variables

If you said

> X = “the length in millimeters (without rounding) of a randomly plucked hair from my head”

Which possible values can _X_ take? We can all probably agree a negative value doesn’t make any sense here.

However if you said it is exactly 1 millimeter, and not 1.1853759… or something like that, I would either doubt your measuring skills, or your measuring error reporting.

A continuous random variable can take **any value** in a given (continuous) interval.

Therefore, if we assigned a **non-zero probability to all of its possible values** , their sum would **not add up to 1**.

To solve this, if _X_ is continuous, we set _P(X=x) = 0_ for all _k_, and instead assign a non-zero chance to _X_ taking a value **in a certain interval.**

To express the probability of X laying between values _a_ and _b_, we say   
_P(a \< X \< b)_.

Instead of just replacing values in a density function, to get _P(a \< X \< b)_ for _X_ a continuous variable, you’ll integrate _X_‘s density function from _a_ to _b_.

Whoah, you’ve made it through the whole theory section! Here’s your reward.

![A pug puppy.](http://www.datastuff.tech/wp-content/uploads/2019/06/pug-690566_640-e1560733151341.jpg)<figcaption>Reward puppy. Source: <a href="https://pixabay.com/photos/pug-puppy-dog-animal-cute-690566/">Pixabay</a>.</figcaption>

Now that you know what a probability distribution is, let’s learn about some of the most common ones!

## Bernoulli Probability Distribution

A Random Variable with a Bernoulli Distribution is among the simplest ones.

It represents a **binary event** : “this happened” vs “this didn’t happen”, and takes a value _p_ as its **only parameter** , which represents the **probability** that **the event will occur**.

A random variable _B_ with a Bernoulli distribution with parameter _p_ will have the following density function:

_P(B = 1) = p, P(B =0)= (1-p)_

Here _B=1_ means the event happened, and _B=0_ means it didn’t.

Notice how both probabilities add up to 1, and therefore no other value for _B_ will be possible.

## Uniform Probability Distribution

There are two kinds of uniform random variables: discrete and continuous ones.

A **discrete uniform distribution** will take a **(finite)** set of values _S_, and assign a probability of _1/n_ to each of them, where _n_ is the amount of elements in _S_.

This way, if for instance my variable _Y_ was uniform in {1,2,3}, then there’d be a 33% chance each of those values came out.

A very typical case of a discrete uniform random variable is found in **dice** , where your typical dice has the set of values {1,2,3,4,5,6}.

A **continuous uniform distribution** , instead, only takes **two values _a_ and _b_** as parameters, and assigns the same density to each value in the **interval between them**.

That means the probability of Y taking a **value in an interval** (from _c_ to _d_) is **proportional to its size** versus the size of the whole interval (_b-a_).

Therefore if _Y_ is uniformly distributed between _a_ and _b_, then

![](https://quicklatex.com/cache3/1f/ql_6428ae1ce24b50fa38db8faf6c6e211f_l3.png)

This way, if _Y_ is a uniform random variable between 1 and 2,

_P(1 \< X \< 2)=1_ and _P(1 \< X \< 1.5) = 0.5_

Python’s `random` package’s `random` method samples a uniformly distributed continuous variable between 0 and 1.

Interestingly, it can be shown that **any other distribution** can be sampled given a [uniform random values generator and some calculus](https://www.mathworks.com/help/stats/generate-random-numbers-using-the-uniform-distribution-inversion-method.html).

## Normal Probability Distribution

![](http://www.datastuff.tech/wp-content/uploads/2019/06/340px-Normal_Distribution_PDF.svg_.png)<figcaption>Normal Distributions. source: <a href="https://en.wikipedia.org/wiki/Normal_distribution">Wikipedia</a></figcaption>

**Normally distributed variables** are so commonly found in nature, they’re actually **_the norm_**. That’s actually where the name comes from.

If you round up all your workmates and measure their heights, or weigh them all and plot a histogram with the results, odds are it’s gonna approach a normal distribution.

I actually saw this effect when I showed you [Exploratory Data Analysis](http://www.datastuff.tech/data-analysis/data-analysis-pandas-seaborn-kaggle-dataset/) examples.

It can also be shown that if you **take a sample** of any random variable and **average those measures** , and repeat that process many times, that average will also have a **normal distribution**. That fact’s so important, it’s called the [fundamental theorem of statistics](https://math.tutorvista.com/statistics/fundamental-theorem-of-statistics.html).

Normally distributed variables:

- Are **symmetrical** , centered around a mean (usually called **μ** ).
- Can take **all values on the real space** , but only deviate two sigmas from the norm 5% of the time.
- Are **literally everywhere**.

Most often if you measure any empirical data and it’s symmetrical, assuming it’s normal will kinda work.

For example, rolling _K_ dice and adding up the results will distribute pretty much normally.

### Log-Normal Probability Distribution

![](http://www.datastuff.tech/wp-content/uploads/2019/06/300px-PDF-log_normal_distributions.svg_.png)<figcaption>Lognormal distribution. source: <a href="https://en.wikipedia.org/wiki/Log-normal_distribution">Wikipedia</a></figcaption>

Log-normal probability distribution is Normal Probability Distribution’s smaller, less frequently seen sister.

A variable _X_ is said to be **log-normally distributed** if the variable _Y = log(X)_ follows a normal distribution.

When plotted in a histogram, log-normal probability distributions are **asymmetrical** , and become even more so if their standard deviation is bigger.

I believe **lognormal** distributions to be worth mentioning, because **most money-based variables** behave this way.

If you look at the probability distributions of any variable that relates to money, like

- Amount sent on the latest transfer of a certain bank.
- Volume of the latest transaction in Wall Street.
- A set of companies’ quarterly earnings for a given quarter.

They will usually not have a normal probability distribution, but will behave much closer to a lognormal random variable.

_(For other Data Scientists: chime in in the comments if you can think of any other empirical lognormal variables you’ve come across in your work! Especially anything outside of finances)_.

## Exponential Probability Distribution

![](http://www.datastuff.tech/wp-content/uploads/2019/06/exp.png)<figcaption>Source: <a href="https://en.wikipedia.org/wiki/Exponential_distribution">Wikipedia</a></figcaption>

**Exponential probability distributions** appear everywhere, too.

They are heavily linked to a Probability concept called a **Poisson Process**.

Stealing straight from Wikipedia, a [Poisson Process](https://en.wikipedia.org/wiki/Poisson_point_process) is “_a process in which events occur continuously and independently at a constant average rate_“.

All that means is, if:

- You have a lot of events going.
- They happen at a certain rate (which **does not change** over time).
- Just because one happened the chances of another one happening don’t change.

Then you have a Poisson process.

Some examples could be requests coming to a server, transactions happening in a supermarket, or birds fishing in a certain lake.

Imagine a Poisson Process with a frequency rate of λ (say, events happen once every second).

Exponential random variables model the time it takes, after an event, for the next event to occur.

Interestingly, in a Poisson Process **an event can happen anywhere between 0 and infinity times** (_with decreasing probability_), in any interval of time.

This means there’s a **non-zero chance that the event won’t happen, no matter how long you wait**. It also means it could happen a lot of times in a very short interval.

In class we used to joke bus arrivals are Poisson Processes. I think the response time when you send a WhatsApp message to _some people_ also fits the criteria.

However, the λ parameter **regulates the frequency** of the events.

It will make the **expected time** it actually takes for an event to happen **center around a certain value**.

This means if we know a taxi passes our block every 15 minutes, even though theoretically we _could_ wait for it forever, it’s extremely likely we won’t wait longer than, say, 30 minutes.

### Exponential Probability Distribution: In Practice

Here’s the density function for an exponential distribution random variable:

![](https://quicklatex.com/cache3/d3/ql_1491ff4bfb47a7894aa4b4021d96f4d3_l3.png)

Suppose you have a sample from a variable and want to see if it can be modelled with an Exponential distribution Variable.

The optimum **λ parameter can be easily estimated** as the inverse of the average of your sampled values.

Exponential variables are very good for modelling any probability distributions with very infrequent, but huge (and mean-breaking) **outliers**.

This is because they can **take any non-negative value** but center in smaller ones, with decreased frequency as the value grows.

In a particularly **outlier-heavy sample** , you may want to estimate λ as the **median instead of the average** , since the median is more **robust to outliers**. Your mileage may vary on this one, so take it with a grain of salt.

## Conclusions

To sum up, as Data Scientists, I think it’s important for us to learn the basics.

Probability and Statistics may not be as flashy as [Deep Learning](http://www.datastuff.tech/machine-learning/autoencoder-deep-learning-tensorflow-eager-api-keras/) or [Unsupervised Machine Learning](http://www.datastuff.tech/machine-learning/k-means-clustering-unsupervised-learning-for-recommender-systems/), but they are the **bedrock of Data Science**. Especially Machine Learning.

Feeding a Machine Learning model with features without knowing which distribution they follow is, in my experience, a poor choice.

It’s also good to remember the **ubiquity of Exponential and Normal Probability Distributions** , and their smaller counterpart, the lognormal distribution.

Knowing their properties, uses and appearance is **game-changing when training a Machine Learning model**. It’s also generally good to keep them in mind while doing any kind of Data Analysis.

Did you find any part of this article useful? Was it all stuff you already knew? did you learn anything new? Let me know in the comments!

_Contact me on [Twitter](http://www.twitter.com/strikingloo), [Medium](http://www.medium.com/@strikingloo) of [dev.to](http://www.dev.to/strikingloo) if there’s anything you don’t think was clear enough, anything that you disagree with, or just anything that’s plain wrong. Don’t worry, I don’t bite._