Formulas Behind Linear Regression
Tags: datascience, statistics, machinelearning
Published: 2022-07-12 16:28:00

Linear regression is an approach for predicting a quantitative response _Y_ on the basis of a single predictor variable _X_ _(or in multiple linear regression, on the basis of multiple predictors)_. It's a simple yet powerful model to estimate continuous variables. 

Here, I'll be discussing important formulas behind linear regression.
&nbsp;

## Linear model

Since linear regression is a linear model, it assumes that the dependence of _Y_ on _X<sub>1</sub>_, _X<sub>2</sub>_, …, _X<sub>p</sub>_ is also linear. Because of that, a simple linear regression can be interpreted in the form:

{% katex %} Y=\beta_0+\beta_1X+\epsilon {% endkatex %}

Where:
&nbsp;&nbsp;&nbsp;&nbsp;_β<sub>0</sub>_ = intercept term (expected value of _Y_ when _X=0_)
&nbsp;&nbsp;&nbsp;&nbsp;_β<sub>1</sub>_ = the average increase in _Y_ associated with a one-unit increase in _X_
&nbsp;&nbsp;&nbsp;&nbsp;_ϵ_ = a catch-all for what we miss with this simple model. We assume that the error term is independent of _X_

Here, _X_ is the independent variable and _Y_ is the dependent variable (also the estimated value), but what we're most interested to look at here are the coefficients _β<sub>0</sub>_ (slope) and _β<sub>1</sub>_ (intercept). 

Recall your high school geometry class where slope-intercept form was introduced. It's basically similar as to what's happening in a simple linear regression equation, where the higher the slope is, the steeper the line goes and the greater the rate of change becomes. Meanwhile, the intercept controls the location where it intersects an axis.

To build a linear model, we must find first the values of these coefficients. This is a job for the least squares approach.
&nbsp;

## Least squares and residuals

The least squares approach chooses the values of _B̂<sub>0</sub>_ and _B̂<sub>1</sub>_ _(hat symbol means estimates)_ that minimize residuals. A residual is the difference between the _i_ th observed response value and the _i_ th response value that is predicted by the linear model, or in the form:

{% katex %} e_i=y_i-\hat{y}_i {% endkatex %}

This is a single instance of a residual, and sometimes it comes up negative, so we usually get the sum of the squares of all residuals, hence the residual sum of squares (_RSS_):

{% katex %} RSS=e_1^2+e_2^2+...+e_n^2 {% endkatex %}

_<center>or</center>_

{% katex %} RSS=\textstyle\sum_{i=1}^n(y_i-\hat{y}_i)^2 {% endkatex %}

Since we're all set, we can now compute for the values of _B̂<sub>0</sub>_ and _B̂<sub>1</sub>_ using the formulas:


{% katex %} \hat{\beta_{1}}={\textstyle\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y}) \over \textstyle\sum_{i=1}^n(x_i-\bar{x})^2} {% endkatex %}

{% katex %} \hat{\beta_0}=\bar{y}-\hat{\beta_1}\bar{x} {% endkatex %}

These are derived from the formula of the regression line. Once the coefficients are calculated, we can now compute for estimates and assess the accuracy of the model.
&nbsp;

## Assessing accuracy

There are various metrics we can use to assess the accuracy of a linear model. Some of them are residual standard error (_RSE_), _R<sup>2</sup>_ statistic, and mean squared error.

The **residual standard error or _RSE_** is considered a measure of the lack of fit of the model to the data. If the predictions obtained using the model are very close to the true outcome values, then _RSE_ will be small and we can conclude that the model fits the data very well. 

_RSE_ is also an estimate of the standard deviation of _ϵ_. Roughly speaking, it is the average amount that the response will deviate from the true regression line.

It uses _RSS_ to compute its value, thus having a form of:

{% katex %} RSE=\sqrt{ {1 \over {n-2}} RSS }  {% endkatex %}


The **coefficient of determination or _R<sup>2</sup>_ statistic** is the percentage of variation in _Y_ which is explained by all the _X_ variables. In other terms, it is the square of the correlation _R_ of the data _(statistical relationship between variables)_.

_R<sup>2</sup>_ has an interpretational advantage over _RSE_ since unlike _RSE_, it always lies between 0 and 1.

The formula to find _R<sup>2</sup>_ is:

{% katex %} R^2=1-{RSS \over {\textstyle\sum_{i=1}^n}(y_i-\bar{y})^2} {% endkatex %}

If you recall in other references, the denominator is the total sum of squares (_TSS_). With that, we can rewrite the formula to:

{% katex %} R^2=1-{RSS \over TSS} {% endkatex %}

Finally, the **mean squared error or _MSE_** tells us the average squared difference between the predicted values and the actual values. The lower the MSE, the better a model fits the data.

The formula to calculate the _MSE_ is:

{% katex %} MSE={1 \over n} {\textstyle\sum_{i=1}^n}(y_i-\hat{y}_i)^2 {% endkatex %}

Upon observation, we can see that the _MSE_ can also be computed by dividing the _RSS_ to the total number of data points _n_.
&nbsp;



_Reference: 
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R (2nd ed.)._