Explanation for the notation of linear models
Tags: machinelearning, statistics, math, esl
Published: 2020-07-15 04:37:18

In the [Elements of Statistical Learning book](https://web.stanford.edu/~hastie/ElemStatLearn/), the chapter on Supervised Learning describes equation for a linear model as a vector (Eq 2.1) and matrix form (Eq 2.2). Both are equivalent.

As a vector is another word for a one dimensional array.

We can write this with parentheses going horizontal like

{% katex %}
A = (1, 5, 7, 3, 2)
{% endkatex %}

but we can also write this in a more matrix form with square brackets in a vertical form

{% katex %}
A =
\begin{bmatrix}
1 \\\
5 \\\
7 \\\
3 \\\
2
\end{bmatrix}
{% endkatex %}

Now we reach Equation 2.1 below.

{% katex %}
  \hat{Y} = \hat{\beta_0} + \sum_{j=1}^p X_j \hat{\beta_j}
{% endkatex %}

This equation is a fancy way to write out an equation you might see for linear regression like this

{% katex %}
  Y = -23 + 3x_1 + 3.6x_2
{% endkatex %}

Now going back to our Equation 2.1, what do these variables mean? Here's the equation again.

{% katex %}
\textcolor{orange}{\hat{Y}} = \hat{\beta_0} + \sum_{j=1}^p X_j \hat{\beta_j}
{% endkatex %}

First, {% katex inline %}\hat{Y}{% endkatex %} is the dependent variable value, or output vector. This will contain the values for what you want to predict. The caret symbol `^` here is called a "hat". This a hypothetical value we find from our model prediction. So we can read {% katex inline %}\hat{Y}{% endkatex %} as "Y hat".

{% katex %}
\hat{Y} = \textcolor{orange}{\hat{\beta_0}} + \sum_{j=1}^p X_j \hat{\beta_j}
{% endkatex %}

The next variable we run into is {% katex inline %}\beta_0{% endkatex %}. This is "the intercept, also known as the _bias_ in machine learning."

On an X-Y plot you have seen in grade school, this is where the line crosses the vertical y-axis. Another way to think about this intercept is the baseline value for your dependent value when your independent variable predictors are all zero.

{% katex %}
\hat{Y} = \hat{\beta_0} + \textcolor{orange}{\sum_{j=1}^p} X_j \hat{\beta_j}
{% endkatex %}

Now, let's gears to talking about some notation. There is a large "E" looking symbol with some numbers and letters, {% katex inline %}\sum_{j=1}^p{% endkatex %}. We call the "E" symbol "sigma" and is a fancy way to say "add these things up".

What things are we adding up? And how? The {% katex inline %}X_j \hat{\beta_j}{% endkatex %} is actually a representation of matrices of {% katex inline %}X_j{% endkatex %} being a row in the matrix below and {% katex inline %}\hat{\beta_j}{% endkatex %} as a single value in the column vector.

{% katex %}
X \hat{\beta} = 
\begin{pmatrix}
x_{1,1} & x_{1,2} & \cdots & x_{1,n} \\\
x_{2,1} & x_{2,2} & \cdots & x_{2,n} \\\
\vdots  & \vdots  & \ddots & \vdots  \\\
x_{m,1} & x_{m,2} & \cdots & x_{m,n} 
\end{pmatrix}
\begin{pmatrix}
\beta_{1} \\\
\beta_{2} \\\
\vdots \\\
\beta_{m} 
\end{pmatrix}
{% endkatex %}

Similarly, the {% katex inline %}\hat{\beta_0}{% endkatex %} is a column vector:

{% katex %}
\begin{pmatrix}
\beta_{1} \\\
\beta_{2} \\\
\vdots \\\
\beta_{m} 
\end{pmatrix}
{% endkatex %}

Multiplying everything together will produce a single column vector, {% katex inline %}\hat{Y}{% endkatex %}.

We all do this so that we can run equations like

{% katex %}
  Y = -23 + 3x_1 + 3.6x_2
{% endkatex %}

this over and over again across multiple sets of values, which are encoded as rows in the square matrices above {% katex inline %}X{% endkatex %}.