Linear Regression for Machine Learning
Tags: technologyindustrytr
Published: 2022-12-09 12:01:57

---
title: Linear Regression for Machine Learning
published: true
date: 2022-12-08 13:45:59 UTC
tags: Technologyindustrytr
canonical_url: https://apiumhub.com/tech-blog-barcelona/linear-regression-for-machine-learning/
---

## Introduction

In this article, we will look at the Linear Regression model for [Machine Learning](https://dev.to/apium_hub/machine-learning-interview-with-gema-parreno-lead-data-scientist-at-apiumhub-1ed4), which is one of the most basic models available.

## Linear Regression

![Linear regression for machine learning](https://apiumhub.com/wp-content/uploads/2022/12/Captura-de-Pantalla-2022-12-08-a-las-13.35.47.png "Linear Regression for Machine Learning 8")

This equation shows a multi-dimension formula for linear regression, where ŷ is the predicted multidimensional value, _n_ is the number of dimensions (or commonly called features),_x_i is the _i_th feature value, and ⍬ is the _j_th model parameter or weight.

With this definition, we now must see the way to train a model following this equation. Training a model means setting its parameters so the model best fits the training set, also we must find a measure of how well the model fits the training data, for this purpose we could use the Mean Square Error.

![Equation - linear regression for machine learning](https://apiumhub.com/wp-content/uploads/2022/12/Captura-de-Pantalla-2022-12-08-a-las-13.37.34.png "Linear Regression for Machine Learning 9")

Here we are using _x_, _y_, ⍬ as vectors of size m.

Taking this into account, to train a Linear Regression model we need to find the value of ⍬ that minimizes the MSE.

## Normal Equation

There is a _closed-form_ _solution_ called Normal Equation that gives the value of ⍬ that minimizes the cost function.

![Normal equation - linear regression for machine learning](https://apiumhub.com/wp-content/uploads/2022/12/Captura-de-Pantalla-2022-12-08-a-las-13.39.17.png "Linear Regression for Machine Learning 10")

Where ⍬^is the value of ⍬ that minimizes the cost function, _y_ is the vector of target values containing _y_¹to _y_m

Let’s see an example with linear-looking data to test this equation:

![QvDOw7EZfxWkNvDZAHZJ8BDOHFg0CCnWgmqX7L3NfczhEB2hTU4YVA uz7j3fBIMDbyI68ABPsXJ8eCVRvbbiiVjwKYQ8vlAQ6nmdaGXUYxmIVwcZrLMKoKqtjYa9UDU8XVdwFJ4ElZUi2kM0umjKuQoOKgag0wj5hwYsCD Gh wNNnln07QGiVWK2TrZcjczzzRI 4](https://lh5.googleusercontent.com/QvDOw7EZfxWkNvDZAHZJ8BDOHFg0CCnWgmqX7L3NfczhEB2hTU4YVA_uz7j3fBIMDbyI68ABPsXJ8eCVRvbbiiVjwKYQ8vlAQ6nmdaGXUYxmIVwcZrLMKoKqtjYa9UDU8XVdwFJ4ElZUi2kM0umjKuQoOKgag0wj5hwYsCD-Gh-wNNnln07QGiVWK2TrZcjczzzRI-4 "Linear Regression for Machine Learning 11")

Now we can compute ⍬^using the Normal Equation:

![bdsA6eET6UR54KDpM9fwRELwP0B3a5sCnue4aPMF0kAdt5FtgtYh8hvrKPSBLvTbJjQJLp2AUu 4E8z5sKh8fc7fJMzIcNDio2a55qzoGahKZlpdiRjhtll08v2qSQcx0PdJN2RyC9FRNwl5W4O2q6iXud qmkKNMu GoKmRtm7Kt6vqJ4hlD jPkcpWRgXhehds7Vo](https://lh3.googleusercontent.com/bdsA6eET6UR54KDpM9fwRELwP0B3a5sCnue4aPMF0kAdt5FtgtYh8hvrKPSBLvTbJjQJLp2AUu-4E8z5sKh8fc7fJMzIcNDio2a55qzoGahKZlpdiRjhtll08v2qSQcx0PdJN2RyC9FRNwl5W4O2q6iXud_qmkKNMu_GoKmRtm7Kt6vqJ4hlD-jPkcpWRgXhehds7Vo "Linear Regression for Machine Learning 12")

As we can see the initial equation that we used to generate the data is:

![Equation - Liner regression](https://apiumhub.com/wp-content/uploads/2022/12/Captura-de-Pantalla-2022-12-08-a-las-14.23.01.png "Linear Regression for Machine Learning 13")

And we could have expected:

![Linear regression for machine learning](https://apiumhub.com/wp-content/uploads/2022/12/Captura-de-Pantalla-2022-12-08-a-las-13.41.46.png "Linear Regression for Machine Learning 14")

The result was close enough, nevertheless, the Gaussian noise made it impossible to recover the exact parameters of the original function. Now we can make predictions using ⍬^:

![UfOVhiHB24XI8vaeceWWQsWAYT71NvBU6SC9UIpXx65 lJfX4iqocB42K1V61RJ0fVbTrvzQM 0w0q8NVkrZmNivHNSXTIvsAgBX1E1GiSYc62FLYlaOrWInKmHXNu2UNS4FDeCPWpzeEwzqwK7ojTPDgI1dPVoMwIfegXC5ziReembbCpqO7ITFS1H YjLOOJvI7Nc](https://lh5.googleusercontent.com/UfOVhiHB24XI8vaeceWWQsWAYT71NvBU6SC9UIpXx65-lJfX4iqocB42K1V61RJ0fVbTrvzQM_0w0q8NVkrZmNivHNSXTIvsAgBX1E1GiSYc62FLYlaOrWInKmHXNu2UNS4FDeCPWpzeEwzqwK7ojTPDgI1dPVoMwIfegXC5ziReembbCpqO7ITFS1H_YjLOOJvI7Nc "Linear Regression for Machine Learning 15")

There are several ML tools to perform Linear Regression rather than compute these equations manually, for example, Scikit-Learn:

![kw g6riDkGcihJ55j0cEF2I8tmLsXgtSwhaF](https://lh5.googleusercontent.com/kw-g6riDkGcihJ55j0cEF2I8tmLsXgtSwhaF-oPEg7jcA2wMgmPdF45hMOG9tsRaj5i0FwaE8b7HKOu51clDoTY3EyjLxmgi_JFw5FlLYZijJ2x20B0AWRnLm6i12U0ovGhXbHFu6SOEAzVnwbh6HXkq5TzB1tSQFfUWMizqCxlXjrmkYCAfikqyg8gKpUEWyehBuMM "Linear Regression for Machine Learning 16")

## Standard Correlation Coefficient

The standard correlation coefficient, also called Pearson’s, gives us a measure of linear correlation between two sets of data. It is the ratio between the covariance of two variables and the product of their standard deviations:

![Standard correlation coefficient - Linear regression model for machine learning](https://apiumhub.com/wp-content/uploads/2022/12/Captura-de-Pantalla-2022-12-08-a-las-13.43.10.png "Linear Regression for Machine Learning 17")

This coefficient varies between -1 and 1. The close to 1 or -1 the more correlated the two variables (positive or negative).

We can compute this coefficient easily with Python using the pandas library. For our example, we need to transform _x_ and _y_ in one dimension variables with the concatenate function from numpy and then using the corr() function in the dataset created with pandas, we obtain the correlation matrix between these two parameters:

![i5wPOGfI2r5x354RKMHns AnpZmRl9xJ1Wpmuimzfv T4irICrYJHX1R1DmAI92PpMggduUE7sFtZy7nR51AQlmR8siWY09K7jC17PqZaFPTMm9ulhxKTpz 7L4heNYEE1 z5uhgUy1 pKlTBINirknkFeLT7KkxQO 91cfomn9hGj05Y36SzSOXhS5CQHj0V0oFCAc](https://lh5.googleusercontent.com/i5wPOGfI2r5x354RKMHns-AnpZmRl9xJ1Wpmuimzfv_T4irICrYJHX1R1DmAI92PpMggduUE7sFtZy7nR51AQlmR8siWY09K7jC17PqZaFPTMm9ulhxKTpz-7L4heNYEE1-z5uhgUy1_pKlTBINirknkFeLT7KkxQO-91cfomn9hGj05Y36SzSOXhS5CQHj0V0oFCAc "Linear Regression for Machine Learning 18")

In this case, the correlation coefficient between _x_ and _y_ is 0.855022. So we can say that they are quite correlated as expected.

## Using Linear Regression

In this section, we are going to use what we have learned previously to analyze a dataset of Boston House Prices. We can download the dataset from [Kaggle](https://www.kaggle.com/datasets/fedesoriano/the-boston-houseprice-data) (it provides us with multiple datasets for testing and learning.)

## Attribute Information

### Input features in order:

1. CRIM: per capita crime rate by town
2. ZN: proportion of residential land zoned for lots over 25,000 sq. ft.
3. INDUS: proportion of non-retail business acres per town
4. CHAS: Charles River dummy variable (1 if tract bounds river; 0 otherwise)
5. NOX: nitric oxides concentration (parts per 10 million) [parts/10M]
6. RM: average number of rooms per dwelling
7. AGE: proportion of owner-occupied units built prior to 1940
8. DIS: weighted distances to five Boston employment centers
9. RAD: index of accessibility to radial highways
10. TAX: full-value property-tax rate per $10,000 [$/10k]
11. PTRATIO: pupil-teacher ratio by town
12. B: The result of the equation B=1000(Bk – 0.63)^2 where Bk is the proportion of blacks by town
13. LSTAT: % lower status of the population
14. MEDV: Median value of owner-occupied homes in $1000’s [k$]

Our aim will be to see the correlation between prices and other variables in the dataset, using the standard correlation coefficient. We first load the dataset using pandas:

![bO9v8uGTWD1rLjzYHb fxecpeQsdBqFnfUVRmgMtU Uq ipBtBS EYrdOm3fErRVvZLuyjdSptQMaNVa7SNZ4ygdV7GJ5GU8N ZVJ0mmglV2RVKT7Kc30YrRaYEcJo8rbI sDuRGIw9Bvwdes](https://lh5.googleusercontent.com/bO9v8uGTWD1rLjzYHb_fxecpeQsdBqFnfUVRmgMtU-Uq_ipBtBS_-EYrdOm3fErRVvZLuyjdSptQMaNVa7SNZ4ygdV7GJ5GU8N-ZVJ0mmglV2RVKT7Kc30YrRaYEcJo8rbI_sDuRGIw9Bvwdes-IllgxtAVW7milEhV5i1DH5wkPldbEhFnwViH_5_adp4TcwCoTZrE "Linear Regression for Machine Learning 19")

Now we can look at how each attribute correlates with the Median value of owner-occupied homes:

![QtfqgeH3dgqNWl7oJcpJtcWN0IhF8w7 eXyOuYmaGhJ3sRu5hB fpwPOpZRCrDA99bVV4ZggY tz6w0Cscfday5ywb6Ck9C vxIT7orG7FYKPnNx77b8h7bvbLQBt5y2 37AAqiEdr LgoZl2UMVNO6UlHMPc4 DhpN4YbHNAVSy1MGqNchgWDX3SrNIP1TW3B38K60](https://lh3.googleusercontent.com/QtfqgeH3dgqNWl7oJcpJtcWN0IhF8w7-eXyOuYmaGhJ3sRu5hB_fpwPOpZRCrDA99bVV4ZggY_tz6w0Cscfday5ywb6Ck9C_vxIT7orG7FYKPnNx77b8h7bvbLQBt5y2_37AAqiEdr-LgoZl2UMVNO6UlHMPc4-DhpN4YbHNAVSy1MGqNchgWDX3SrNIP1TW3B38K60 "Linear Regression for Machine Learning 20")

The correlation coefficient varies between 1 to -1. When is close to 1, it means that there is a strong positive correlation. In this case, the price tends to go up when RM ( average number of rooms per dwelling) goes up. We can see that the correlation coefficient is 0.695 and as we observe in the chart despite some points there is a strong correlation between these two features.

## Conclusion

We have seen the basics of linear regression and its applications to machine learning. We have learned how to predict new values with linear regression models and also we have seen how to use the standard correlation coefficient matrix to obtain correlated features in datasets.