Basic Mathematics for Machine Learning
Tags: deeplearning, mathematics, machinelearning, linearalgebra
Published: 2019-06-09 20:51:20

---
title: Basic Mathematics for Machine Learning
published: true
tags: deep-learning,mathematics,machine-learning,linear-algebra
canonical_url: 
---

[_EDIT :- For calculus go through my post on matrix calculus._](https://medium.com/@rohitrpatil/the-matrix-calculus-you-need-for-deep-learning-notes-from-a-paper-by-terence-parr-and-jeremy-4f4263b7bb8)

There are many reasons why mathematics is important for machine learning. Some of them are below:

1. Selecting the right algorithm which includes giving considerations to accuracy, training time, model complexity, number of parameters and number of features.
2. Choosing parameter settings and validation strategies.
3. Identifying underfitting and overfitting by understanding the Bias-Variance tradeoff.
4. Estimating the right confidence interval and uncertainty.

![](https://cdn-images-1.medium.com/max/800/1*eUYIqyg9aomn55O-Q5XRVg.png)<figcaption>What really is used!</figcaption>

### **What are the best resources for learning?**

![](https://cdn-images-1.medium.com/max/856/1*ODJPbEcabUyMrMPnCeOYKg.jpeg)

I have tried to summarized mathematics taught in above both resources. So lets begin!

### **Scalars, Vectors, Matrices and Tensors :**

- **Scalars ** : A scalar is just a single number.
- **Vectors ** : A vector is an array of numbers. The numbers are arranged in order. We can identify each individual number by its index in that ordering. _x=[x1 x2 x3 …. xn]._ We can think of vectors as identifying points in space, with each element giving the coordinate along a diﬀerent axis. Sometimes we need to index a set of elements of a vector. In this case, we deﬁne a set containing the indices and write the set as a subscript. For example, to access _x1, x3 and x6_ we deﬁne the set _S_ ={1,3,6} and write _xs._
- **Matrices ** : A matrix is a 2-D array of numbers, so each element is identiﬁed by two indices instead of just one.

![](https://cdn-images-1.medium.com/max/166/1*YIZqGzeRxmfiFW-fj8qp-Q.png)<figcaption>When we need to explicitly identify the elements of a matrix, we write them as an array enclosed in square brackets</figcaption>

The **transpose** of a matrix is the mirror image of the matrix across a diagonal line, called the main diagonal, running down and to the right, starting from its upper left corner.

![](https://cdn-images-1.medium.com/max/397/1*IfvcwFO2FLdr-xsE--BRCw.png)<figcaption>Transpose of a matrix</figcaption>

We can add matrices to each other, as long as they have the same shape, just by adding their corresponding elements: C = A + B where _Ci,j = Ai,j+ Bi,j._

We allow the addition of matrix and a vector, yielding another matrix: C=A+b, where _Ci,j = Ai,j +bj_. In other words, the vector b is added to each row of the matrix. This shorthand eliminates the need to deﬁne a matrix with b copied into each row before doing the addition. This implicit copying of b to many locations is called **_broadcasting_**.

- **Tensors :** An array of numbers arranged on a regular grid with a variable number of axes is known as a tensor.

### Multiplying Matrices and Vectors:

The matrix product of matrices A and B is a third matrix C. In order for this product to be deﬁned, A must have the same number of columns as B has rows. If A is of shape m × n and B is of shape n × p, then C is of shape m × p.

The product operation is deﬁned by

![](https://cdn-images-1.medium.com/max/195/1*UwUwRGXR4K2ulM7yLLaVfg.png)

The matrix multiplication is distributive, associative but not commutative (the condition AB =BA does not always hold), unlike scalar multiplication.

For learning more you can go through this course offered by MIT Courseware (Prof. Gilbert Strang).

[Linear Algebra](https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/)

### Probability Theory:

Probability theory is a mathematical framework for representing uncertain statements. It provides a means of quantifying uncertainty as well as axioms for deriving new uncertain statements.

Let us understand some of the terminologies used in probability theory:-

- **Random Variables ** : A random variable is a variable that can take on diﬀerent values randomly. They may be continuous or discrete. A discrete random variable is one that has a ﬁnite or countably inﬁnite number of states. A continuous random variable is associated with a real value.
- **Probability Distributions**  : A probability distribution is a description of how likely a random variable or set of random variables is to take on each of its possible states. probability distribution over discrete variables may be described using a **probability mass function (PMF)** denoted by P(_x_). When working with continuous random variables, we describe probability distributions using a **probability density function (PDF)** denoted by _p(x)_. A probability density function _p(x)_ does not give the probability of a speciﬁc state directly; instead the probability of landing inside an inﬁnitesimal region with volume δx is given by _p(x)_δx.
- **Conditional Probability**  : In many cases, we are interested in the probability of some event, given that some other event has happened. This is called a conditional probability. We denote the conditional probability that y = _y_ given x = _x_ as P(y=_y_ | x=_x_).

![](https://cdn-images-1.medium.com/max/443/1*f4LedO-X3k4cmlSeAv9_pQ.png)<figcaption>The conditional probability is only deﬁned when P(x=x) &gt;0</figcaption>

- **The Chain Rule of Conditional Probabilities :** Any joint probability distribution over many random variables may be decomposed into conditional distributions over only one variable

![](https://cdn-images-1.medium.com/max/650/1*j9BZZVw55hnb2jI0_aKM3A.png)<figcaption>Chain rule or product rule of probability</figcaption>

- **Expectation**  : The expectation, or expected value, of some function _f(x)_ with respect to a probability distribution P(_x_) is the average, or mean value, that _f_ takes on when _x_ is drawn from P.

![](https://cdn-images-1.medium.com/max/335/1*oSXW9vM5TjP5ihdhGqCfow.png)<figcaption>For discrete variables this can be computed with a summation</figcaption>

- **Variance :** The variance gives a measure of how much the values of a function of a random variable x vary as we sample diﬀerent values of _x_ from its probability distribution.

![](https://cdn-images-1.medium.com/max/418/1*0AQxWEY6sE5H09wtRiiJaw.png)

- The square root of the variance is known as the **standard deviation**.
- **Covariance**  : The covariance gives some sense of how much two values are linearly related to each other, as well as the scale of these variables:

![](https://cdn-images-1.medium.com/max/648/1*9bqZmpDeghnAE_8-Gt6fGg.png)

High absolute values of the **covariance** means that the values change very much and are both far from their respective means at the same time. If the sign of the **covariance** is positive, then both variables tend to take on relatively high values simultaneously. If the sign of the **covariance** is negative, then one variable tends to take on a relatively high value at the times that the other takes on a relatively low value and vice versa.

- **Bayes’ Rule**  : Bayes’ theorem is a formula that describes how to update the probabilities of hypotheses when given evidence. It follows simply from the axioms of conditional probability, but can be used to powerfully reason about a wide range of problems involving belief updates. We often ﬁnd ourselves in a situation where we know P(y | x) and need to know P(x | y). Fortunately, if we also know P(x), we can compute the desired quantity

![](https://cdn-images-1.medium.com/max/292/1*HBBtRBuXKvtQ6LGm7bRLMw.png)

### Common Probability Distributions:-

Some of the common probability distributions used in machine learning are as follows

- **Bernoulli Distribution**  : It is a distribution over a single binary random variable. It is controlled by a single parameter φ ∈ [0,1], which gives the probability of the random variable being equal to 1.

![](https://cdn-images-1.medium.com/max/305/1*o_yH8xdu93G68_SAbnV8PQ.png)<figcaption>Properties of Bernoulli Distribution</figcaption>

- **Multinoulli Distribution**  : The multinoulli, or categorical,distribution is a distribution over a single discrete variable with _k_ diﬀerent states, where _k_ is ﬁnite. Multinoulli distributions are often used to refer to distributions over categories of objects.
- **Gaussian Distribution**  : The most commonly used distribution over real numbers is the normal distribution, also known as the Gaussian distribution.

![](https://cdn-images-1.medium.com/max/521/1*mPhKrBsFbbH9L8kHpSNRQQ.png)

- The two parameters _µ_ ∈ R and _σ_ ∈ (0, ∞) control the normal distribution. The parameter _µ_ gives the coordinate of the central peak. This is also the **mean** of the distribution : E[x] =_µ_. The **standard deviation** of the distribution is given by _σ_, and the variance by _σ_².

![](https://cdn-images-1.medium.com/max/650/1*HRj-NCJWQyfZTGa_835ylg.png)<figcaption>Plot of the normal distribution density function</figcaption>

Khan Academy has got a very good course for statistics and probability.

[Statistics and Probability | Khan Academy](https://www.khanacademy.org/math/statistics-probability)

I will wrap here. Hope this post helps you in revising some concepts which you learned in high school. 😄 Thank You for reading!

![](https://cdn-images-1.medium.com/max/375/1*DnDyYN3E9t50nn4Iu30bZw.gif)

[You can find me on Twitter @Rohitpatil5](https://twitter.com/Rohitpatil5), or connect with me on [LinkedIn.](https://www.linkedin.com/in/rohitrpatil/)

* * *