Ordinary Least Squares in Simple Linear Regression - Unveiling the Math Behind the Line
Published: 2024-03-22 03:13:43

Today we will talk about the detailed working behind the simple linear regression. 


## Explaining Ordinary Lease Squares (OLS)

Recall that in Simple Linear Regression, our objective is to find the best-fitting line that minimizes the overall error between the data points and the line itself. The OLS method is a common approach used to achieve this. Let's break down the steps involved:

## 1. Initial State
Imagine we have a scatter plot with four data points. we will draw an initial regression line to start the things. Our goal is to minimize the **sum of the distances between these points and the line**. 
Let's take the distance from the point 1 to 4 be 
d1,d2,d3 and d4

So we need to minimize the sum of d1+d2+d3+d4

![ols-initial](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t82tj8xoqdnatdzicyd7.png)

## 2. Square of Distances

To avoid **positive and negative distances cancelling each other out**, we square the distances from each data point to the line. This ensures that all distances contribute to the error calculation, regardless of direction. 

So sum will be d1^2 + d2^2 + d3^2 + d4^2

## 3. Error Function

![squared distances](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hwg0pwlncmoell4apmg4.png)

Mathematically, the error function (E) is expressed as the sum of the squared difference
between the actual y values (yi) and the predicted values (yi_hat) estimated by the regression line: 


![Error Function](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fpvlpjcrtswpjd7uu0ts.png)

 
## 4. Dissecting the Error

The predicted value (yi_hat) for each data point can be calculated using the equation of line (y = mx + b), where m is the slope and b is the y-intercept. With this error function can be written as 


![line-equation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4lfwcs8uwnlbn5wx316r.png)

## 5. Finding the Perfect Fit

To minimize the error function and achieve the best fit, we need to find the values of m and b that make E the Smallest possible. We can achieve this by taking partial derivatives of E with respect to m and b,  and then setting those derivatives to zero. 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ncvj1n1qw58p625pgtuu.png)

Solving the above equations provides us with the formulas for the optimal slope (m) and y-intercept (b) of best-fitting line

## Building Our Own Linear Regression Model in Python: Hands-on Implementation

Now, let's implement the above theoretical understanding into practice

```
class OwnLR:
    def __init__(self):
        self.m = None
        self.b = None


    def fit(self, X_train,y_train):
        num = 0
        den = 0

        for i  in range(X_train.shape[0]):
            num = num + ((X_train[i] - X_train.mean())*(y_train[i] - y_train.mean()))
            den = den + ((X_train[i] - X_train.mean())*(X_train[i] - X_train.mean()))

        self.m = num/den
        self.b = y_train.mean()  - (self.m * X_train.mean())

    def predict(self,X_test):
        return self.m * X_test + self.b
```
 
This class defines two methods:

- fit(X_train, y_train): This method takes the training data and calculates the optimal values of m an b using OLS formulas.

- predict(X_test): This method takes a new data point as input and returns the predicted y value.


## Conclusion
Today, we have explored the Ordinary Least Squares method in Simple Linear Regression. We have learned how OLS helps us find the best-fit line by minimizing the sum of squared errors. We've also implemented our own Simple Linear Regression class in Python.

Stay tuned for new topics