Supervised learning - Linear regression model
Tags: machinelearning, mathematics, linearregression, algorithms
Published: 2023-02-25 15:29:14

###Linear Regression Model
Concept map: 
Linear regression model {% katex inline %}\rightarrow{% endkatex inline %} SSE {% katex inline %}\rightarrow{% endkatex inline %} Gradient descent {% katex inline %}\rightarrow{% endkatex inline %} Training the model

**1. Linear Regression Model:**
Linear regression is a type of supervised learning algorithm used to predict a continuous target variable based on one or more input features. Other common supervised learning algorithms include logistic regression, decision trees, and neural networks. 
The goal of linear regression is to find the best linear relationship between the input features and the target variable. The model takes the form of a linear equation:
{% katex %}y = b_0 + b_1 x_1 + b_2 x_2 + ... + b_n x_n{% endkatex %}
where:
{% katex inline %}y{% endkatex inline %} is the target variable {% katex inline %}x_1, x_2, ..., x_n{% endkatex inline %} are the input features 
{% katex inline %}b_0{% endkatex inline %} is the y-intercept (also known as the bias) 
{% katex inline %}b_1, b_2, ..., b_n{% endkatex inline %} are the coefficients (also known as weights) of the input features

The objective of linear regression is to find the values of {% katex inline %}b_0, b_1, b_2, ..., b_n{% endkatex inline %} that minimize the difference between the predicted and actual values of {% katex inline %}y{% endkatex inline %}. This is typically done by minimizing the sum of squared errors (SSE) between the predicted and actual values.<br>

**2. SSE**
By definition:
{% katex %}SSE = \sum_{i=1}^n (y_i - \hat{y_i})^2{% endkatex %}
where:
{% katex inline %}y_i{% endkatex inline %} is the actual value of the target variable for the {% katex inline %}i^{th}{% endkatex inline %} data point 
{% katex inline %}\hat{y_i}{% endkatex inline %} is the predicted value of the target variable for the {% katex inline %}i^{th}{% endkatex inline %} data point
The minimization of SSE is typically achieved using gradient descent, which is an optimization algorithm that iteratively adjusts the values of {% katex inline %}b_0, b_1, b_2, ..., b_n{% endkatex inline %} to find the optimal values that minimize SSE.
<br>
**3. Gradient Descent:**
Gradient descent is an iterative optimization algorithm that is used to find the optimal values of the model parameters (weights and biases) that minimize the cost function (SSE in this case). It works by updating the parameters in the opposite direction of the gradient of the cost function with respect to the parameters.

The update rule for the weights in gradient descent is:
{% katex %}w_i = w_i - \alpha \frac{\partial Cost}{\partial w_i}{% endkatex %}
where:
{% katex inline %}w_i{% endkatex inline %} is the {% katex inline %}i^{th}{% endkatex inline %} weight 
{% katex inline %}\alpha{% endkatex inline %} is the learning rate or step size for each iteration of gradient descent 
{% katex inline %}\frac{\partial Cost}{\partial w_i}{% endkatex inline %} is the partial derivative of the cost function with respect to the {% katex inline %}i^{th}{% endkatex inline %} weight
The update rule for the bias term is similar:
{% katex %}b_0 = b_0 - \alpha \frac{\partial Cost}{\partial b_0}{% endkatex %}
where:
{% katex inline %}b_0{% endkatex inline %} is the bias term 
{% katex inline %}\frac{\partial Cost}{\partial b_0}{% endkatex inline %} is the partial derivative of the cost function with respect to the bias term

The partial derivatives of the cost function with respect to the weights and bias can be calculated using calculus.<br>

**4. Training the Model:**
To train the model, we first initialize the weights and bias to some random values. We then iteratively update the weights and bias using the gradient descent algorithm until the cost function reaches a minimum or a predefined stopping criterion is met (e.g., maximum number of iterations reached).

Once the model is trained, we can use it to make predictions on new data by plugging in the values of the input features into the linear equation and calculating the predicted value of the target variable.