The Matrix Calculus You Need For Deep Learning (Notes from a paper by Terence Parr and Jeremy…
Tags: calculus, mathematics, deeplearning, machinelearning
Published: 2019-06-09 20:52:26

---
title: The Matrix Calculus You Need For Deep Learning (Notes from a paper by Terence Parr and Jeremy…
published: true
tags: calculus,mathematics,deep-learning,machine-learning
canonical_url: 
---

### The Matrix Calculus You Need For Deep Learning (Notes from a paper by [Terence Parr](http://parrt.cs.usfca.edu/) and [Jeremy Howard](http://www.fast.ai/about/#jeremy))

![](https://cdn-images-1.medium.com/max/1024/1*l_ZFM3-vJMBVcZacHwCX-Q.jpeg)

#### **Table of Contents**

- [Review: Scalar derivative rules](https://medium.com/p/4f4263b7bb8#48c0)
- [Introduction to vector calculus and partial derivatives](https://medium.com/p/4f4263b7bb8#d5f1)
- [Matrix calculus](https://medium.com/p/4f4263b7bb8#3c66)
- [Generalization of the Jacobian](https://medium.com/p/4f4263b7bb8#efec)
- [Derivatives of vector element-wise binary operators](https://medium.com/p/4f4263b7bb8#59c5)
- [Vector sum reduction](https://medium.com/p/4f4263b7bb8#641f)
- [The Chain Rules](https://medium.com/p/4f4263b7bb8#ada3)
- [Resources](https://medium.com/p/4f4263b7bb8#3788)

Jeremy’s courses show how to become a world-class _deep learning practitioner_ with only a minimal level of scalar calculus, thanks to leveraging the automatic differentiation built in to modern deep learning libraries. But if you really want to really understand what’s going on under the hood of these libraries, and grok academic papers discussing the latest advances in model training techniques, you’ll need to understand certain bits of the field of matrix calculus.

### Review: Scalar derivative rules

Hopefully you remember some of these main scalar derivative rules. If your memory is a bit fuzzy on this, have a look at [Khan academy video on scalar derivative rules](https://www.khanacademy.org/math/ap-calculus-ab/ab-derivative-rules).

![](https://cdn-images-1.medium.com/max/1024/1*ZR50K2cDpl1um4S-aOeWQw.png)<figcaption>Basic rules of derivatives</figcaption>

There are other rules for trigonometry, exponential, etc., which you can find at [Khan Academy differential calculus course](https://www.khanacademy.org/math/differential-calculus).

### Introduction to vector calculus and partial derivatives

Neural network layers are not single functions of a single parameter, **_f(x)_**. So, let’s move on to functions of multiple parameters such as **_f(x,y)_**. For example, what is the derivative of xy (i.e., the multiplication of x and y)?

Well, it depends on whether we are changing _x_ or _y_. We compute derivatives with respect to one variable (parameter) at a time, giving us two different _partial derivatives_ for this two-parameter function (one for _x_ and one for _y_). Instead of using operator d/dx, the partial derivative operator is **∂/ ∂x** (a stylized _d_ and not the Greek letter δ ). So **∂(xy)/ ∂x** and **∂(xy)/ ∂y** are the partial derivatives of _xy_; often, these are just called the _partials_.

The partial derivative with respect to _x_ is just the usual scalar derivative, simply treating any other variable in the equation as a constant. Consider function f(x,y) = 3x²y. The partial derivative with respect to _x_ is written **∂(**3x²y**)/ ∂x.** There are three constants from the perspective of **∂/ ∂x** : 3, 2, and _y_. Therefore, ∂(3x²y)/ ∂x = 3y∂(x²)/ ∂x = 3y(2x) = 6xy. The partial derivative with respect to _y_ treats _x_ like a constant and we get ∂(3x²y)/ ∂y = 3x². You can learn more on [Khan Academy video on partials](https://www.khanacademy.org/math/multivariable-calculus/multivariable-derivatives/partial-derivative-and-gradient-articles/a/introduction-to-partial-derivatives).

So from above example if f(x,y) = 3x²y, then

![](https://cdn-images-1.medium.com/max/657/1*8a9yL0FQmO70v-Juo5c3Bg.png)<figcaption><em>Gradient</em> of f(x,y)</figcaption>

So the **_gradient of f(x,y) is simply a vector of its partial._**

### Matrix calculus

When we move from derivatives of one function to derivatives of many functions, we move from the world of vector calculus to matrix calculus. Let us bring one more function g(x,y) = 2x + y⁸. So gradient of g(x,y) is

![](https://cdn-images-1.medium.com/max/194/1*IUpOnL8zNi_fPcDKcFP3BQ.png)

Gradient vectors organize all of the partial derivatives for a specific scalar function. If we have two functions, we can also organize their gradients into a matrix by stacking the gradients. When we do so, we get the **_Jacobian matrix_** (or just the **_Jacobian_** ) where the gradients are rows:

![](https://cdn-images-1.medium.com/max/549/1*7xKV9D7qXX44GQQbvYlBgA.png)<figcaption><a href="https://en.wikipedia.org/wiki/Matrix_calculus#Layout_conventions">Numerator layout</a> of Jacobian</figcaption>

### Generalization of the Jacobian

To define the Jacobian matrix more generally, let’s combine multiple parameters into a single vector argument: _f_(_x,y,z_) =\> _f_( **x** ). Lowercase letters in bold font such as **x** are vectors and those in italics font like _x_ are scalars. _xi_ is the ith element of vector **x** and is in italics because a single vector element is a scalar. We also have to define an orientation for vector **x**. We’ll assume that all vectors are vertical by default of size _n_ X 1:

![](https://cdn-images-1.medium.com/max/127/1*TnlD_RDMVQhGizRLp3jd-Q.png)

With multiple scalar-valued functions, we can combine them all into a vector just like we did with the parameters. Let **y = f(x) b**e a vector of _m_ scalar-valued functions that each take a vector **x** of length n= | **x** | where | **x** | is the cardinality (count) of elements in **x**. Each _fi_ function within **f** returns a scalar just as in the previous section

![](https://cdn-images-1.medium.com/max/177/1*wyqevFkocbibjZjs1E3MOA.png)

> Generally speaking, though, the Jacobian matrix is the collection of all **_m_** X **_n_** possible partial derivatives (_m_ rows and _n_ columns), which is the stack of _m_ gradients with respect to  **x** :

![](https://cdn-images-1.medium.com/max/762/1*f75-0xoIgirN-kkL2d_vEg.png)

### Derivatives of vector element-wise binary operators

By “element-wise binary operations” we simply mean applying an operator to the first item of each vector to get the first item of the output, then to the second items of the inputs for the second item of the output, and so forth.We can generalize the element-wise binary operations with notation y= f(w) O g(x) where **_m_** = **_n_** = | **_y_** | = | **_w_** | = | **_x_** |. The O symbol represents any element-wise operator (such as +) and not the o function composition operator.

![](https://cdn-images-1.medium.com/max/1024/1*6SVHGcQcijq6aEYUoC_yZw.png)

That’s quite a furball, but fortunately the Jacobian is very often a diagonal matrix, a matrix that is zero everywhere but the diagonal.

### Vector sum reduction

Summing up the elements of a vector is an important operation in deep learning, such as the network loss function, but we can also use it as a way to simplify computing the derivative of vector dot product and other operations that reduce vectors to scalars.

Let **y=sum(f(x))** = Σ **_fi_** ( **x** ). Notice we were careful here to leave the parameter as a vector **x** because each function _fi_ could use all values in the vector, not just _xi_. The sum is over the **results** of the function and not the parameter. The gradient ( **1** X **_n_** Jacobian) of vector summation is:

![](https://cdn-images-1.medium.com/max/960/1*-MEhl1gPbjiIjXyla9C3wQ.png)

### The Chain Rules

We can’t compute partial derivatives of very complicated functions using just the basic matrix calculus rules. Part of our goal here is to clearly define and name three different chain rules and indicate in which situation they are appropriate.

The chain rule is conceptually a divide and conquer strategy (like Quicksort) that breaks complicated expressions into sub-expressions whose derivatives are easier to compute. Its power derives from the fact that we can process each simple sub-expression in isolation yet still combine the intermediate results to get the correct overall result.

The chain rule comes into play when we need the derivative of an expression composed of nested subexpressions. For example, we need the chain rule when confronted with expressions like d(sin(x²))/dx.

- [**Single-variable chain rule**](http://m.wolframalpha.com/input/?i=chain+rule) ** :-** Chain rules are typically defined in terms of nested functions, such as **_y_** = **_f_(_u_)** where **_u_= _g_(_x_) so _y_= _f_(_g_(_x_))** for single-variable chain rules.

![](https://cdn-images-1.medium.com/max/160/1*u3k7gS3vAPEXukvkWmyDzg.png)<figcaption>Formulation of the single-variable chain rule</figcaption>

To deploy the single-variable chain rule, follow these steps:

1. Introduce intermediate variables for nested sub-expressions and sub-expressions for both binary and unary operators; example, X is binary, **_sin (x)_** and other trigonometric functions are usually unary because there is a single operand. This step normalizes all equations to single operators or function applications.
2. Compute derivatives of the intermediate variables with respect to their parameters.
3. Combine all derivatives of intermediate variables by multiplying them together to get the overall result.
4. Substitute intermediate variables back in if any are referenced in the derivative equation.

- **Single-variable total-derivative chain rule :-** The total derivative assumes all variables are potentially codependent whereas the partial derivative assumes all variables but _x_ are constants.

![](https://cdn-images-1.medium.com/max/905/1*Nxfy9U1Eh7_skLznwc3U-w.png)

This chain rule that takes into consideration the total derivative degenerates to the single-variable chain rule when all intermediate variables are functions of a single variable.

> A word of caution about terminology on the web. Unfortunately, the chain rule given in this section, based upon the total derivative, is universally called “multivariable chain rule” in calculus discussions, which is highly misleading! Only the intermediate variables are multivariate functions.

- **Vector chain rule :-** Vector chain rule for vectors of functions and a single parameter mirrors the single-variable chain rule.

If **_y_= _f_(_g_(_x_))** and **x** is a vector **.** The derivative of vector **y** with respect to scalar _x_ is a vertical vector with elements computed using the single-variable total-derivative chain rule.

![](https://cdn-images-1.medium.com/max/412/1*jtPtLuhQTBVK_0Be5rIpXQ.png)

The goal is to convert the above vector of scalar operations to a vector operation. So the above RHS matrix can also be implemented as a product of vector multiplication.

![](https://cdn-images-1.medium.com/max/317/1*AIrI5dTFm_X_ybGL0UqayA.png)

That means that the Jacobian is the multiplication of two other Jacobians. To make this formula work for multiple parameters or vector **x** , we just have to change _x_ to vector **x** in the equation. The effect is that **∂g/ ∂x** and the resulting Jacobian, **∂f/ ∂x ** , are now matrices instead of vertical vectors. Our complete _vector chain rule_ is:

![](https://cdn-images-1.medium.com/max/223/1*sGJplnNYP7leOS-lNmRNFg.png)

> Please note here that matrix multiply does not commute, the order of (**∂f/ ∂x)(∂g/ ∂x) **matters.

For completeness, here are the two Jacobian components :-

![](https://cdn-images-1.medium.com/max/666/1*g5ph9kQOynRPVYfr1nYytA.png)

where **_m_** = | **_f_** |, **_n_** = | **_x_** | and **_k_** = | **_g_** |. The resulting Jacobian is **_m_** X **_n_**. (an **_m_** X **_k_** matrix multiplied by a **_k_** X **_n _** matrix).

We can simplify further because, for many applications, the Jacobians are square ( **_m_** = **_n_** ) and the off-diagonal entries are zero.

### Resources

1.The original [paper](http://parrt.cs.usfca.edu/doc/matrix-calculus/index.html#sec3).

2. There are some online tools which can differentiate a matrix for you:

- [Wolfram Alpha](http://www.wolframalpha.com/input/?i=D%5B%7Bx%5E2,+x%5E3%7D.%7B%7B1,2%7D,%7B3,4%7D%7D.%7Bx%5E2,+x%5E3%7D,+x%5D)
- [Matrix calculus Differentiator.](http://www.matrixcalculus.org/)

3. More [matrix calculus](https://atmos.washington.edu/~dennis/MatrixCalculus.pdf).

![](https://cdn-images-1.medium.com/max/201/1*xhCj5o5sYa_8VBUw_mR_Xg.jpeg)