Linear Regression
Tags: machinelearning, computerscience, python, datascience
Published: 2022-12-08 10:43:56

A journey of a thousand miles begins with a single step, my journey and I believe your M.L journey is going to start today and it is going to start with **linear regression**, so let's begin... :)

# Two Takeaways

1. Mathematical Explanation behind linear regression
2. Python Implementation Using Scikit-Learn

## What is Regression and why is this algorithm called Linear Regression

Regression is a statistical method that helps us find the relationship between independent and dependent variables.
To make it simple you have two columns "EXPERIENCE" and "KNOWLEDGE" and the relationship is how your knowledge increases with experience, this relationship is called regression.

####Linear Regression
The linearity assumption in linear regression means the model is linear in parameters (i.e coefficients of variables) & may or may not be linear in variables.

![Graph - Linear Regression](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uo57nzzpizp1jm1o1896.png)
(The above is a perfectly linear graph)

## Mathematical Understanding of Linear Regression

1. **Hypothesis Function**
2. **Loss and cost function**
3. **Optimisation**

### Hypothesis Function
Any machine learning algorithm is nothing but a hypothesis function, imagine this to be a function that you use to get output from a set of inputs, this hypothesis function is optimised to work for our dataset and this process of optimisation is what we call **model-training**.



![Hypothesis Function - Single Variable](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xe50dtxesiey4dfeaqo9.png)

The above equation is the hypothesis function for linear regression, and yes you would have seen this somewhere else, this equation is similar to the equation of a straight line **y=m*x + c**.
The above equation is a single variable function i.e the data for this contains just 2 columns in total, the **input** column and **output** column. 

**x** ==> **Input/Independent Variable**
**y** ==> **Output/Dependant Variable**
**teta1, teta2** ==> **Constants**

We basically try to predict **y** by giving **x** as the input and assigning optimised values for **teta1** and **teta2** (NOTE: These are also called weights and in further text teta1, teta2 would be referred to as weights).

#### Multi-Variable Hypothesis Function

![Hypothesis Function - Multi Variable](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/073mb3mmplhl6qutdcqh.png)
When we have many columns then the hypothesis function would look like this.


###Loss and Cost Functions

Loss and cost functions are not exactly the same, there is a subtle difference between them.
![Loss - Cost](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2bu4emvwing93avnrw0o.png)

When we check the deviation between the actual and predicted value of a **single data point** it is called **loss** and if I cumulatively find the deviation for the **entire dataset** it is called **cost** and the corresponding functions are called loss and cost functions.

Some common loss and const functions for linear regression are

![Loss-Cost Regression](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oefv8madsamvl7rz5ufg.png)

#### Mean Squared Error (MSE)
The cost function that we are going to see is Mean Squared Error(MSE) which could be used for optimising the hypothesis function of linear regression.


![MSE](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7vtrgvgxg13ec2xttww0.png)

Basically, the difference between the predicted value and the actual value is found, and squared, this is done for each and every data point and collated.

Now we have predicted with random weights(teta1, teta2), now a process called **training** is used to optimise the weights.

#### Training the model
Training could be defined as a process that is used to optimise the hypothesis function by optimising the weights.
This optimisation process is achieved using [gradient-descent](https://www.geeksforgeeks.org/gradient-descent-in-linear-regression/).


![Gradient descent](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ta9qfzmas01sk658o5r.png)
The above curve is called the error curve and it is the output of the mean squared error(MSE).
**Gradient descent** is the process where we update the weights of the hypothesis function either by small values or large values depending on the **learning rate**.
On reaching a particular threshold error value the gradient descent stops and we would have optimised weights to best suit our input dataset.


![Best Fit](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/azauj0f349xwyfgsnnoi.png)
The **red line** is called the **best-fit line** and this represents the final output after the linear regression model has completed training.

## Python Implementation - Linear Regression Using Scikit-Learn

Importing the required python libraries
```python
# Import the libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
```

We are going to use the [diabetes](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_diabetes.html) to understand linear regression and this dataset comes along with the [scikit-learn](pip install scikit-learn) library.

```python
# We are taking an inbuilt dataset present in sklearn 
diabetes = datasets.load_diabetes()
```
#### Dataset description

![Dataset desc](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2yr1uetsl18c9mb6hbk5.png)

Columns present in the dataset
```python
diabetes.feature_names
```

![Columns](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ex15f3c71rs34hrhu10d.png)

Storing the training data and target data separately in two different variables
```python
# Extracting the training data and target 
X = diabetes.data
Y = diabetes.target
```


```python
print(X.shape, Y.shape)
```

![Shape](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/af31kkj0cb6su6sml5xr.png)

Importing the **linear regression model** from **scikit-learn** library, the model is present as a class.
```python
from sklearn.linear_model import LinearRegression
le = LinearRegression()
```

#### Training process
```python
le.fit(train_x,train_y)
```
yes the whole training process is done in one single line of code, thanks to scikit-learn.

#### Making Predictions
```python
y_pred = le.predict(test_x)
```
the predictions on the test dataset are stored in **y_pred**.

Let's print the results by converting it into a pandas dataframe
```python
result = pd.DataFrame({'Actual': test_y, 'Model Prediction' : y_pred})
print(result.head(20))
```

![Result Df](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z64yhb2dcf1gerzmq3m1.png)

#### Visualisation
Let's take a small subset i.e 20 data points of our prediction and compare it with actual output using [matplotlib](https://matplotlib.org/) library
```python
sample_result = result.head(20)
sample_result.plot(y=["Actual", "Model Prediction"],
        kind="line", figsize=(10, 7))
```

![Output](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iicjwzlsh5qp0q7415xl.png)

#### Variance Score
```python
print('Variance score: {}'.format(le.score(test_x, test_y)))
```

![Variance Score](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/emr49ivvli3bqww28att.png)

That's all about about linear regression, thank you for your patience :))