Gradient Descent- Geometrical Interpretation
Tags: optimizationalgorith, optimizationtechniqu, artificialintelligen, gradientdescent
Published: 2023-03-10 05:46:34

---
title: Gradient Descent- Geometrical Interpretation
published: true
date: 2022-10-21 20:54:13 UTC
tags: optimizationalgorith,optimizationtechniqu,artificialintelligen,gradientdescent
canonical_url: 
---

One of the most critical tasks in Computer Science, Machine Learning, and computation, in general, is to optimize solutions. Not only important but it was also one of the most challenging tasks until the entry of modern computers. **Gradient Descen** t is one such computational technique that helps us to optimize problems and the good thing is that it’s trivial to understand and get the intuition of it :)

### Pre-requisites

I won't be saying that there are no prerequisites and you can learn this topic without prior knowledge this time. For you to understand the underlying mathematics of Gradient Descent you need to have a good knowledge of -

1. Differntiation and Integration
2. Maxima and Minima
3. Vector Differentiation ( Just a bit of understanding would work)
4. Your time and interest :D

### Why do we need Gradient Descent?

Let’s take the logistic loss expression with regularization for this example-

![](https://cdn-images-1.medium.com/max/1024/1*rNhk34kwnHWyzNyOVVfYTQ.jpeg)
_Logistic Loss with Regularization Expression_

From the equation itself, we can say that x\_i and y\_i are constants because these values are already present in the dataset itself. So L here is a function of W i.e the variable to find is W. So we can rewrite the equation as-

![](https://cdn-images-1.medium.com/max/1024/1*khd20MMizWqRaT4axoS06Q.jpeg)

Now to calculate the maxima or minima we need to calculate the vector differentiation of L w.r.t W (because W is a vector here). For simplicity let’s the consider

part 1 of the expression as :

![](https://cdn-images-1.medium.com/max/197/1*9Fb8ah8SW9kAFxdyCx-79g.jpeg)

and part 2 of the expression as :

![](https://cdn-images-1.medium.com/max/64/1*zuHg2DDZsPEvoe15otAwYw.jpeg)

Let’s consider the 2nd part for the moment-

if W was a scalar then-

![](https://cdn-images-1.medium.com/max/129/1*6l8tHlQJjkEXxosXX4UZ0w.jpeg)

So derivative of this will be-

![](https://cdn-images-1.medium.com/max/121/1*tZdiFhrFqrsIHPZRB7Ng3Q.jpeg)

The vector equivalent will also be the same

Now let’s consider the 1st part-

For this part, we will use the chain rule as usual and we will get the following

![](https://cdn-images-1.medium.com/max/383/1*Z9wnWKa0Mp8kRQl9A6QsIw.jpeg)

So the overall equation will be —

![](https://cdn-images-1.medium.com/max/496/1*2UzOt1piTdZEscb5woR86w.jpeg)

These problems are extremely hard to solve, so we use computational methods like Gradient Descent.

### Geometric Interpretation

The gradient Descent algorithm is iterative and this is how it works:

1. First, we take a random value of x\* where

x\*= argmin f(x) w.r.t x

If you didn’t understand, x\* by the end of the algorithm will achieve the value of argmin f(x) w.r.t x but initially, we are assigning a random value to it

2. So, x\_0= initial value of x\*

3. As more and more iterations happen we will keep on assigning newer values:

x\_1= value of x\* at iteration 1

x\_2= value of x\* at iteration 2

and so on…..

At the end of the iteration, we will have x\_k whose value will be very close to x\*

Now before showing how exactly the algorithm will work, let’s clear some basic concepts-

Let us have the following function f(x) then -

![](https://cdn-images-1.medium.com/max/1024/1*LjJIMCcCXPeyQ0YVwxNgPg.jpeg)

a) The x\* marked here is what the ideal x\* or the one at the end of the algorithm will look like approximately because we are calculating the value of x which will result in the min value of f(x).

b) In quadrant 1, all the slopes will be positive and in quadrant 2 all slopes will be negative.

c) min f(x)=- max f(x)

max f(x)=- min f(x)

### Working of the Gradient Descent Algorithm

a) Let’s pick a point x\_0 initially at random

b) Now we will find x\_1 such that x\_1 is closer to x\* and

![](https://cdn-images-1.medium.com/max/1024/1*8FWn8mmYWC6dX2E4esBmyw.jpeg)

![](https://cdn-images-1.medium.com/max/176/1*Nvi8LPAm4slnrKGQv7mVMQ.jpeg)
_Here we are calculating the derivative of f(x) at x0_

r= step-size and let r=1

Slope of f(x) at x\_0 will be positive i.e [df/dx] at x\_0 will be positive

so x\_1=x\_0–(1\*+ve) => x1 shifted towards left of x\_0 and is closer to x\*

c) Similarly we will calculate x2

![](https://cdn-images-1.medium.com/max/175/1*1ITV54bi5Ift6HFdDNxZDQ.jpeg)

Now we can generalize the whole thing as:

![](https://cdn-images-1.medium.com/max/187/1*T4V5ydBsQqvWmazQ4yQnLg.jpeg)

We will have k iterations, so we will have-

x\_0 , x\_1 ……..x\_k

![](https://cdn-images-1.medium.com/max/196/1*FUP6QSooVAnyZclOwRQXsQ.jpeg)

The termination condition of the loop will be :

if x\_{k+1} — x\_k is very small then terminate at x\*=x\_k

Intuitively what we are doing is reducing the value of the slope at each iteration.

### The problem with fixed r or step-size value

Till now we were keeping the value of step-size or r constant which is a problem and we will understand why it is so-

Let us have a function f(x)=x² and after some iterations, we reached x\_i=0.5

![](https://cdn-images-1.medium.com/max/1024/1*gAiFNr-LjPlinzvJ5sDG6Q.jpeg)

Now if we calculate x\_{i+1} we will find that it will have a value of -0.5 which is going away from x\*

Again x\_{i+2} will have a value of +0.5. Similarly x\_{i+3}=-0.5 and x\_{i+4}=+0.5 and we will keep oscillating between -0.5 and +0.5. This kind of problem arises when the r value is kept fixed. So we reduce r with each iteration and keep it dynamic.