Transforming Simplicity: Adapting Linear Regression to Capture Complex Non-Linear Phenomena with NumPy
Tags: machinelearning, python, numpy, learning
Published: 2024-07-12 14:59:16

In the field of Machine Learning, non-linear phenomena are ubiquitous. Often, people think that linear regression models cannot capture these complexities. However, with a few adjustments, linear regressions can be surprisingly effective in modeling non-linear relationships. In this article, we will explore how to adapt a linear regression model for non-linear phenomena using Python and NumPy.

**Linear Regression Model**
Linear regression seeks to find the best line (or hyperplane in higher dimensions) that minimizes the error between predictions and actual values. The equation of a simple linear regression model is:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tdedk0cfu9rlp7aodlou.PNG)
For multivariate models, this equation becomes:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0qnnvsecc86mpwk1lsgm.PNG)

**Transforming Variables to Capture Non-Linearity**
To adapt this model to non-linear phenomena, we can transform the input variables using non-linear functions. For example, we can include quadratic or cubic terms:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/51l3nbssumb1tf6p1xnd.PNG)
These transformations allow the linear model to learn complex non-linear relationships.

**Cost Function**
The cost function for linear regression is usually the mean squared error (MSE):
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ldq3bqjwgn4f5qfi1wj.PNG)

**Gradient Descent**
To minimize the cost function, we use the gradient descent algorithm. The parameter updates are as follows:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gf4zz0epnh9k3azqecxs.PNG)
where **𝛼** is the learning rate.

**Implementation in Python with NumPy**
Here is a simple implementation in Python:

```
import numpy as np
from sklearn.datasets import make_regression
import matplotlib.pyplot as plt
```
```
x, y = make_regression(n_samples=100, n_features=1, noise=10)
y = y + abs(y/2)
plt.scatter(x, y)
```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gt2h5puo59bgsq4ay79z.PNG)
```
X = np.hstack((x, np.ones(x.shape)))
X = np.hstack((x**2, X))
```
```
theta = np.random.randn(3, 1)
theta
```
```
def cost_function(X, y, theta):
    m = len(y)
    return 1/(2*m) * np.sum((model(X, theta) - y)**2)
```
```
def grad(X, y, theta):
    m = len(y)
    return 1/m * X.T.dot(model(X, theta) - y)
def gradient_descent(X, y, theta, learning_rate, n_iter):
    cost_history = np.zeros(n_iter)
    for i in range(0, n_iter):
        theta = theta - learning_rate * grad(X, y, theta)
        cost_history[i] = cost_function(X, y, theta)
        
    return theta, cost_history
```
```
n_iter = 1000
learning_rate = 0.01

theta_final, cost_history = gradient_descent(X, y, theta, learning_rate, n_iter)

predict = model(X, theta_final)

plt.scatter(x[:, 0], y)
plt.scatter(x[:, 0], predict, c='r')
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uwupoj1icsv3a67uagty.PNG)

By transforming the input variables, we can use linear regression models to capture non-linear relationships. This flexibility, combined with techniques such as gradient descent, allows linear models to handle much more complex phenomena than they would normally. NumPy provides a solid foundation for implementing these concepts in Python, making non-linear modeling more accessible.