Simple Linear Regression (basic overview)
Tags: machinelearning, linearregression, mse, gradientdescent
Published: 2023-09-03 07:41:45

This exercise shows the basics of the Linear Regression algorithm using:
- MSE and Gradient Descent
- One feature
- Code written in Python

_You can download the notebook from [https://github.com/erickbr15/ml-notebooks/blob/main/ml-linear-regression-simple-one-feature.ipynb](url)

Linear regression is a fundamental statistical technique used in data analysis and machine learning for modeling the relationship between a dependent variable (label) and one or more independent variables (features). It is a versatile tool with applications in various fields, from economics to biology and beyond.

### How Linear Regression Works:

1. Data Collection: Start by collecting a dataset with both labels and features. This process is also known as data-generating process.

2. Model Building: Choose the appropriate type of linear regression (simple or multiple) based on the problem at hand. Calculate the best-fitting line using techniques like the least squares method.

3. Model Evaluation: Assess the goodness of fit by examining metrics such as the coefficient of determination (R-squared), which tells you how well the model explains the variance in the data.

4. Prediction: Once you have a reliable model, you can use it to make predictions. For instance, you can predict house prices for new properties based on their sizes using the established linear relationship.

## 1. Data-generating process
The training and test data are generated by a probability distribution over datasets called the data-generating process. We tipically make a set of assumptions known collectively as the i.i.d. assumptions. These assumptions are that the examples in each dataset are independent from each other, and that the traning set and test set are identically distributed drawn from the same probability distribution as each other.

For this excercise we will use the library *sklearn.datasets* to generate the training and test sets.


```python
#Import necessary libraries
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
import numpy as np
```


```python
#Data-generating
X, y = make_regression(n_samples = 150, 
n_features = 1, random_state = 42, noise = 50)

print(f" X.shape = {X.shape}")
print(f" y.shape = {y.shape}")
```

     X.shape = (150, 1)
     y.shape = (150,)



```python
# EDA - Exploratory Data Analysis
plt.scatter(X, y)
plt.show()
```


![Exploratory data analysis](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hhdodtizfbu7tb3k8tyh.png)  

    


### Split dataset into training and test sets


```python
X_train, X_test, y_train, y_test = train_test_split(X, y,
 test_size = .10, random_state = 42)

print(f" X_train.shape = {X_train.shape}")
print(f" y_train.shape = {y_train.shape}")
```

     X_train.shape = (135, 1)
     y_train.shape = (135,)



```python
plt.scatter(X_train, y_train, label="Training set")
plt.scatter(X_test, y_test, label="Test set")
plt.legend()
```







![Training and test sets](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x1yo7p21jykfprp5xl8f.png)  

    


## Model building

**Linear regression** assumes a linear relationship between the label and feature variables. This means that we believe the relationship can be expressed as a straight line equation:

{% katex %}f(x) = wx + b{% endkatex %}

### Initialize a random *w* and a random *b*. This task will allow us to define a starting point to optimize the weigths and bias and find the model that better fits the training data set.


```python
m, n = X_train.shape
y_train = y_train.reshape(m, 1)
w = np.random.rand(1)
b = np.random.rand(1)

print(f"X_train shape={X_train.shape}")
print(f"y_train shape={y_train.shape}")
print(f"m={m}")
print(f"w={w}")
print(f"b={b}")
```

    X_train shape=(135, 1)
    y_train shape=(135, 1)
    m=135
    w=[0.89389793]
    b=[0.45829251]



```python
plt.scatter(X_train, y_train, label = "Training dataset")
plt.plot(X_train, 
np.dot(X_train, w) + b, color="red", label="Starting model")
plt.legend()
plt.show()
```



![Starting random model](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9j66hs1a4xpmhalmnp2x.png) 
    


Now that it is defined how the label will be predicted from the feature using the model {% katex inline %}\hat{y} = wx + b{% endkatex %} we need a definition of our performance measure. One way of measuring the performance of the model is to compute the **mean squared error** of the model on the test set. If {% katex inline %}\hat{y}{% endkatex %} gives the predictions of the model on the test set, then the mean squared error is given by

{% katex %}MSE_{test}=J(w,b)=\frac{1}{2m}\sum_{i}^{m}(f_{w,b}(x^{(i)})-y^{(i)})^{2}{% endkatex %}

This measure is called the **cost** **J(w,b)** and intuitively, one can see that this error measure decreases to 0 when {% katex inline %}\hat{y} = y{% endkatex %}. We can also see that the error increases whenever the Euclidean distance between the predictions and the targets increases.

We need to improve the weights **w** in a way that reduces {% katex inline %}MSE_{test}{% endkatex %}. One intuitive way of doing this is just to minimize the mean squared error on the training set. To minimize the cost function and find the global minimum we will use the gradient descent.

The **gradient descent** algorithm is described as *repeat until convergence:*

{% katex %}w=w-\alpha\frac{\delta J(w,b)}{\delta w}{% endkatex %}

{% katex %}b=b-\alpha\frac{\delta J(w,b)}{\delta b}{% endkatex %}

where parameters {% katex inline %}w, b{% endkatex %} are updated simultaneously and {% katex inline %}\alpha{% endkatex %} is the learning rate.

Until now we have: 

- **Linear regression model**: {% katex inline %}f_{w,b}(x) = wx + b{% endkatex %}

- **Cost function**: {% katex inline %}J(w,b) = \frac{1}{2m}\sum_{i=1}^{m}(f_{w,b}(x^{(i)})-y_{(i)})^{2}{% endkatex %}

- **Gradient descent algorithm**

{% katex %}w=w-\alpha\frac{\delta J(w,b)}{\delta w}{% endkatex %}

{% katex %}b=b-\alpha\frac{\delta J(w,b)}{\delta b}{% endkatex %}

---

Now, let's do some math to get {% katex inline%}\frac{\delta J(w,b)}{\delta w}{% endkatex %}


{% katex inline%}= \frac{\delta}{\delta w}\frac{1}{2m}\sum_{i=1}^{m}(f_{w,b}(x^{(i)})-y^{(i)})^{2}{% endkatex %}


{% katex inline%}= \frac{\delta}{\delta w}\frac{1}{2m}\sum_{i=1}^{m}(wx^{(i)}+b-y^{(i)})^{2}{% endkatex %}


{% katex inline%}= \frac{1}{2m}\sum_{i=1}^{m}(wx^{(i)}+b-y^{(i)})2x^{(i)}{% endkatex %}


{% katex inline%}= \frac{1}{m}\sum_{i=1}^{m}(f_{w,b}(x^{(i)})-y^{(i)})x^{(i)}{% endkatex %}

---

Now, to get {% katex inline%}\frac{\delta J(w,b)}{\delta b}{% endkatex %}


{% katex inline%}= \frac{\delta}{\delta b}\frac{1}{2m}\sum_{i=1}^{m}(f_{w,b}(x^{(i)})-y^{(i)})^{2}{% endkatex %}


{% katex inline%}= \frac{\delta}{\delta b}\frac{1}{2m}\sum_{i=1}^{m}(wx^{(i)}+b-y^{(i)})^{2}{% endkatex %}


{% katex inline%}= \frac{1}{2m}\sum_{i=1}^{m}(wx^{(i)}+b-y^{(i)})2{% endkatex %}


{% katex inline%}= \frac{1}{m}\sum_{i=1}^{m}(f_{w,b}(x^{(i)})-y^{(i)}){% endkatex %}

---




### Code implementation
To implement the Linear Regression using gradient descent and MSE, we will need the next functions:
- function to compute the cost
- function to compute the gradient
- function to compute the gradient descent (fit)


```python
def compute_cost(X, y, w, b, m):
    cost_sum = 0 
    for i in range(m): 
        f_wb = w * X[i] + b   
        cost = (f_wb - y[i]) ** 2  
        cost_sum += cost  
    total_cost = (1 / (2 * m)) * cost_sum
    
    return total_cost
```


```python
def compute_gradient(X, y, w, b, m):
    dj_dw = 0
    dj_db = 0
    
    for i in range(m):  
        f_wb = w * X[i] + b 
        dj_dw_i = (f_wb - y[i]) * X[i] 
        dj_db_i = f_wb - y[i] 
        dj_db += dj_db_i
        dj_dw += dj_dw_i 
    dj_dw = dj_dw / m 
    dj_db = dj_db / m 
        
    return dj_dw, dj_db
```


```python
def fit(X, y, w, b, m, learning_rate, epochs):
    W = np.array([])
    B = np.array([])
    L = np.array([])

    working_w = w
    working_b = b

    for epoch in range(epochs):
        dj_dw, dj_db = compute_gradient(X,y,working_w, working_b, m)
        
        working_w = working_w - learning_rate * dj_dw
        working_b = working_b - learning_rate * dj_db

        loss = compute_cost(X, y, working_w, working_b, m)
                        
        L = np.append(L, loss)
        W = np.append(W, working_w)
        B = np.append(B, working_b)
        
    return working_w, working_b, L, W, B
```


```python
optimized_w, optimized_b, cost_h, w_h, b_h = 
fit(X_train, y_train, w, b, m, 0.1, 100)

print(optimized_w)
print(optimized_b)
```

    [97.20911222]
    [8.58244015]


### Plotting the cost function and the progress history of the gradient descent


```python
plt.scatter(np.linspace(1, 100, 100), cost_h)
plt.title("Cost minimization over iterations")
plt.xlabel("Iterations")
plt.ylabel("J(w,b)")
plt.show()
```



![Cost minimization over iterations](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aj58hqje1fqxv8x61kww.png)
    



```python
fig, axes = plt.subplots(1, 2, figsize=(15, 5))

axes[0].plot(w_h, cost_h)
axes[0].set_title('w , J(w,b)')
axes[0].set_xlabel('w')
axes[0].set_ylabel('J(w,b)')

axes[1].plot(b_h, cost_h)
axes[1].set_title('b, J(w,b)')
axes[1].set_xlabel('b')
axes[1].set_ylabel('J(w,b)')
```





![Gradient descent tracking](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yn1bda2zyz2997e0d1y0.png)
    


### Definition of prediction function using the model


```python
def predict(X):
  return np.dot(X, 97.20911222) + 8.58244015
```

### Test the model


```python
plt.scatter(X_train, y_train, color="green", label="Train")
plt.scatter(X_test, y_test, color="red", label="Test")
plt.plot(X_test, predict(X_test), color="blue", label="Predictions")

plt.title("Train, Test and Predictions graph")
plt.xlabel("X")
plt.ylabel("Y")

plt.legend(loc='upper left')

plt.show
```



![Train, Test and Predictions graph](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6wy9zo0flyx4bvs9ydg9.png)  
    



```python
fig, axes = plt.subplots(1, 2, figsize=(15, 5))

axes[0].scatter(X, y)
axes[0].plot(X, np.dot(X, w) + b, color="red")
axes[0].set_title('Starting model (not optimized)')
axes[0].set_xlabel('X (feature)')
axes[0].set_ylabel('Y (label)')

axes[1].scatter(X,y)
axes[1].plot(X, predict(X), c = "r")
axes[1].set_title('Linear regression model (optimized)')
axes[1].set_xlabel('X (feature)')
axes[1].set_ylabel('Y (label)')

plt.show()

```



![Random staring model vs Linear regression model](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n5dotcn0g1i6obhs5ks0.png)
    


### Linear regression model ({% katex inline%}feature \to f(x)\to prediction{% endkatex %})

{% katex inline%}f(x)=wx+b{% endkatex %}

{% katex inline%}\hat{y}=wx+b{% endkatex %}

{% katex inline%}\hat{y}=97.20911222x+8.58244015{% endkatex %}

## Final comments
While linear regression is a powerful tool, it has limitations. It assumes a linear relationship, which may not hold for all problems. Additionally, it can be sensitive to outliers and may not perform well when the data has a complex, nonlinear structure.

In conclusion, linear regression is a foundational technique in data analysis and machine learning that helps us model relationships between variables and make predictions. It's a valuable tool in a data scientist's toolkit and serves as a basis for more advanced regression techniques.

Linear regression has a wide range of applications:

- Finance: Predicting stock prices based on historical data.
- Marketing: Analyzing the impact of advertising expenditure on sales.
- Medicine: Predicting patient outcomes based on clinical variables.
- Environmental Science: Studying the relationship between pollution levels and health effects.

In future posts I will come with: 
- the metrics to evaluate the model, 
- a code implementation to use optimized algorithms to operate over vectors and matrix

Thank you for reading and happy coding!


## References

[1] A. Géron, "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow," O'Reilly Media, Inc., Oct. 2022.

[2] I. Goodfellow, Y. Bengio, and A. Courville, "Deep Learning," The MIT Press.