Backprop intuition
Published: 2021-08-19 16:30:37

## Motivation

I've studied a few online courses on deep learning. Back-propagation is a fundamental part of it, but I always felt like the explanation lacked something.

My biggest pet-peeves are:
* the relation of weights and gradients is not very intuitive; it's not well-explained in general;
* all of a sudden, there is a bunch of formulas thrown at you for error propagation;
* I spent more time trying to guess the matrices shapes than understanding the whole concept.

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/thh4kju7fknw468qyhfx.png) 

I'm trying to circumvent these problems in this article. Hopefully, it's going to help you (like it helped me) to get a sense of the whole picture.

## Why does it matter?
Well, Andrej Karpathy has a whole [article](https://karpathy.medium.com/yes-you-should-understand-backprop-e2f06eab496b) on that which I strongly recommend you to read. **Spoiler**: it has a lot of side effects. Knowing how it works will certainly help you if you want to design efficient neural nets.

Feel free to watch this [class](https://www.youtube.com/watch?v=i94OvYb6noo) too. It's an invaluable piece of knowledge.

## The first step: defining the problem
Let's start simple: Suppose I create a 1-layer network with two inputs: x and y, trying to predict an outcome variable called z. For example, these variables could be: 
* x: the size of a house
* y: number of bedrooms
* z: house price predicted based on (x, y).
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5nqhzuh13qbq0akm3og7.png)

Finding a good function f(x,y) is the purpose of this optimization. To have a better prediction, I'll need to optimize this function. Based on what?

~~Lots of guessing~~ Loss functions!

I'm not going to dive into this problem of choosing a loss function because that would require another article entirely (take a look at [this](https://machinelearningmastery.com/how-to-choose-loss-functions-when-training-deep-learning-neural-networks/) one for example). I'm defining it as Mean Squared Error just for the sake of demonstration.

## Let's optimize the loss function!

Moving on, let's consider I have **N** real-life examples of housing prices to check my network's results. Here's the equation:![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vvgfkc81u9bmgbrmoee7.png)

The loss function will be:
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sa7r2k1edqjho533gbnf.png)

Remember that all xi, yi and zi (without the hat) are known values; they're training examples, and only the function is mutable; that's what we're trying to optimize.

So, let's solve this math problem. My goal is to reduce the loss function until I find its minimum value (or a satisfying local minimum).

## The gradient descent algorithm.
There is a property in calculus called the gradient. A gradient is the N-dimensional equivalent of a derivative (the slope of a function at a given point). The gradient applied to a given function F looks like this:

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/12swlmq85bhbhaj0si76.png) 

Given some conditions (for example, the function must be differentiable), you can calculate in which direction your function decreases the most in magnitude. In other words, that's the opposite direction of a function's "velocity vector" in an n-dimensional space. 

Then, you can move slightly in that direction using the following equation (gamma sufficiently small):

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dqj5z1g16t4olyvktecl.png) 

So, if you want to diminish the magnitude of said function, you can take the negative of the gradient function and take a tiny step. Thus, the norm of that function will most certainly go down, but your gradient direction will also change in the process; you'll need to re-calculate it to take another tiny step. That goes on and on until you reach a certain number of iterations. That's the gist of the gradient descent algorithm.

Okay, now back to the loss problem. I'm going to use gradient descent there to optimize the loss function. I'll specify things a bit more.

Neurons (a node in the neural network) generally consist of: 
* linear combination through **independent** weights; 
* non-linear activation function to break linearity. This activation function can be a sigmoid, tanh, ReLU, etc. 

Only the weights can change between iterations.
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t33420xuch29vlk4tpc9.png)
 
Okay, so let's keep going with our minimization problem. Our problem at hand is 2-dimensional since we have two variables: wx e wy. Let's find the derivative of L concerning wx first.

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ss2z6ppn1cte31h7ypm1.png) 
 
Similarly,
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xum9uqeu2636bzkx0qbi.png) 
 
Looking back to our gradient descent formula, we're going to update the weights as such:

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z3yazsy5gk5s1layyo29.png) 

That seemed like a lot of work for such a simple problem, right?

## Why this solution is problematic
Neural nets can and should have several neurons if you want to learn all types of complicated patterns. Imagine writing the loss function for this neural network, which is not that big, and then compute its gradients:

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lb0qswu8zltdnegu6y8d.png)
 
It's necessary to find an easier path to the solution, something that encourages a plug-and-play approach rather than solving an incredibly complicated gradient computation formula; that's where back-propagation shines.

## The chain rule
Using the gradient descent approach, we want to discover the rate of change of our loss concerning each variable (each weight) to update them and lower the overall Loss function. That means we need to find the partial derivative of the loss for each weight.

Now, suppose we have a variable x, a variable y = f(x) and another function L(x). We want to find the derivative of L with respect to x. The chain rule in calculus tells us that:

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vb1dghtjz170kt96ltpa.png) 

Calculating the loss using the weights is a nightmare unless we start creating intermediate variables along the way: then it gets much easier to calculate the gradient step-by-step (using the chain rule multiplication) until we reach the weights:

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kgr7iwi8oblcr3lw1r98.png) 
 
 
Now, this looks better. First, we start by doing the fast-forward part, computing the magnitudes of all those intermediate variables and the final output. Then, we start the back-propagation part: starting by the end, we calculate the first gradient and then start propagating it (multiplying by the subsequent ones) until we reach each weight. 

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x0fob54ae7gnjohnyqwj.png)
 

We can treat each of these neuron units as separate gates (for example, weights + ReLU is one type of gate, weights + sigmoid is another type of gate, and so on). Each one will have a fast-forward function (transform its inputs to its output) and a back-propagation function (multiplying the gradient coming from the outputs to get the gradient for each of its weights). In the case before:
 
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ihnyhx7hjz389umujsmu.png)

## Possible side-effects
* The multiplicative effect of the gradient can be problematic. It means your gradients can vanish or explode as it goes from layer to layer.
* If one of your gates gives a zero gradient at any point, it won't update its weights. Keep that in mind when initializing the weight matrices; you might be preventing a good portion of your neural network from training;

If you haven't read this [article](https://karpathy.medium.com/yes-you-should-understand-backprop-e2f06eab496b) I mentioned before, I strongly encourage you to do it. It is much more complete in this matter.

## Final considerations
Hopefully, you understand a little bit more of the math concepts behind back-propagation. If that is the case for at least one of you out there, I'll be plenty satisfied :) Feel free to comment or give me feedback so I can update this article and make it better!