Mathematical Formulae behind Optimization Algorithms for Neural Networks
Tags: python, deeplearning, beginners
Published: 2022-03-24 01:04:01

Hello, 
The following topics are covered in this blog:-

* Introduction

* Optimization Algorithms:

  **Gradient Descent (GD)**

  **Stochastic Gradient Descent (SGD)**

  **Mini-batch SGD**

  **SGD with Momentum**

  **AdaGrad**

  **AdaDelta & RMSProp**

  **Adam**

* Conclusion                   ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/spw17rq43xssner41udc.png)

_Download the whole blog from the following link:-_
https://github.com/ruthvikraja/Optimization-Algorithms.git

## _Introduction_

* Neural networks is a subset of Machine Learning in which  Neural networks adapt and learn from vast amounts of data. 

* The Neuron is the building block of a Neural network that takes some input, does some mathematical computation by multiplying the input values with their corresponding random weights and finally produces the output. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vpaq6ca9vkyzy7pryzpl.gif)

* Each node (Hidden and Output layers) in a Neural network is composed of two functions, namely linear and activation function. In the forward propagation, the Linear function is computed by summation of multiplying previously connected nodes output and corresponding weight, bias as shown in the Figure. 

* After applying the Linear function, Activation functions like Sigmoid, Relu, Leaky Relu, Parametric Relu, Swish Relu, Softplus AF’s etc are implemented based on the problem type and requirement. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5qt9gif6iv0jnepzfv3b.png)

**Role of an Optimizer**
 
* After computing the output at the output layer, the predicted value is compared with the actual value by computing Loss.

* The Loss function is used to determine the error between the actual and predicted value. The Optimization algorithm is used to determine the new weight values i.e Loss w.r.t change in weights to bring the output of the next trial closer to the actual output.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xsoffod6o8go04waom7y.gif)

## _Gradient Descent_

* The formula to compute new weights using Gradient Descent is as follows:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1ybylr5xgabx7ztci8vy.png)

* The formula to compute Loss using Gradient Descent is as follows:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cqff8zptfbausp1vkjt0.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1ics4iqduxkvq4ypiyfk.png)

## _Stochastic Gradient Descent_

* The formula to compute new weights using Stochastic Gradient Descent is   as follows:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i92jsgvzgw7ifos32fwr.png)

* The formula to compute Loss using Stochastic Gradient Descent is as follows:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/00qbuo8d8k28qtwqtdxz.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yyydh087q9pn97kalnnh.png)

## _Mini-Batch Stochastic Gradient Descent_

* The formula to compute new weights using Mini-Batch Stochastic Gradient Descent is as follows:-

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/64yixshwdn4e3jxxwb2a.png)

* The formula to compute Loss using Mini-Batch Stochastic Gradient Descent is as follows:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p381rmuf6978ozj9ooaw.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5ficq4ase8w9aiqi7w3v.png)

**Overall Comparison (GD (vs) SGD (vs) Mini-Batch SGD)**
 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jyqpo6l6c37xggkmpyt2.png)

## _Stochastic Gradient Descent with Momentum_

* The formula to compute new weights using Stochastic Gradient Descent  with Momentum is as follows:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yuvws80p8pzn1fc1tnsv.png)

* The formula to compute Loss using Stochastic Gradient Descent with Momentum is as follows:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qzgd9xbce39nm01e2tej.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ol2znfvnz2cnjypmu04b.png)

For better illustration, consider the following scenario to calculate Exponential Weighted Average:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sxs8y3gox4ctj70mqwt8.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hev601ic9vf8f3zpkhax.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gxt7katopzbhkopjzx8g.png)

* Therefore, the final updated formulae to calculate new weights & bias are as follows:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uvctcknsdtt0yhgoqkdb.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/culdfngqkzelpsw0wooi.png)

where,

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i2ih7lhz2zn12eqckkap.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x57a5pwe6fdoui69h82n.png)

## _Adaptive Gradient Descent_
    
* The formula to compute new weights using Adaptive Gradient Descent is as follows:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dedip7vzs46fp2yfexjl.png)

* The formula to compute Loss using Adaptive Gradient Descent is as follows:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c5q974br8mr1d2c0h91e.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pvk9sc4ybk3ne09xztzk.png)

where,

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qcg1ophjv76jbqbtch66.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p81mwmmibg4vqzphu9xf.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fefbw5cphx6gpyr9wfiv.png)

## _Adaptive Learning Rate Method & Root Mean Squared Propagation_

* The formula to compute new weights using AdaDelta & RMSProp is as follows:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/makpzm9i9zsb72xmj5in.png)

* The formula to compute Loss using AdaDelta & RMSProp is as follows:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xbm116a6zayv5aki4fgw.png)

where, 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n47056475lk4fmfea5t5.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4xd2zz9cum5lm0ghaevj.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vx6bcbdzkjkoy4xy8abo.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5vsafbuascvcz0vmh8ys.png)

## _Adaptive Moment Estimation_

* The formulae to compute new weights & bias using Adam are as follows:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/423q01v8y9uapfu0nhch.png)

* The formulae to compute Loss for Regression & Classification problems using Adam are as follows:- 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2v5q7g1pmzppjuizkrlk.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f7gye5x9x5brvd99bvx6.png)

where,

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r2ea5rdj7w0s57qqezl0.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rbz8zsnzc9zbv17vg82n.png)

* When utilising Exponential Weighted Averages, there is a process known as bias correction. Scientists have introduced Bias correction to get better results at the initial time stamps. Therefore, the formulae for Bias correction is as follows:-

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9nesqiksshb4tmqc1laz.png)

* The updated Weight & Bias formulae are as follows:-  

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8asi9dcl2dz0udsw8xhx.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9uo1usr9bie8iihwxfb8.png)

## _Conclusion_

* In this Presentation, different Optimization algorithms that are available in the field of Artificial Intelligence were discussed in detail to reduce the Loss function of a Neural Network.

* Overall, Adam Optimizer is comparatively better than other algorithms because it was implemented using some advanced theories.

* However, there is no guarantee that the Adam optimizer will outperform all the given datasets because it depends on several other features like the type of problem, size of the input data, number of features, etc.

* Gradient Descent and SGD algorithm work well for small datasets, Mini-batch SGD, SGD with Momentum & RMSProp can be tried on large datasets. 


## THANK YOU