Regression Modeling
Tags: machinelearning, datascience, statistics, regression
Published: 2022-04-29 11:33:22

Make sure to read this [blog](https://dev.to/daud99/5-regression-27d0) before reading this one.

## What is Regression model and which kind of problem it is used for?

A regression model predicts continuous values. For example, regression models make predictions that answer questions like the following:

- What is the value of a house in California?
- What is the probability that a user will click on this ad?

They can't be use to predict discrete values. For example:
- Is a given email message spam or not spam?
- Is this an image of a dog, a cat, or a hamster? 

Such problems are known as **Classification** problem. We will see which Machine Learning algorithms we can use for classification problem in future blogs.

> The dataset used in this blog can be found [here](https://drive.google.com/drive/folders/19d316176lgxLWDvubH7-62PszoBLBYDv?usp=sharing).

## Linear Regression Model

In Linear Regression Model, we have only one Dependent Variable (Target Variable) and one Independent Variable (Feature).

### Step-1

Loading Libraries

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn import linear_model
```

We are Scikit-Learn library to model our model. By default, this model uses the **Ordinary Least Square** method.

### Step-2

Loading Dataset.

```python
df = pd.read_csv('homeprices.csv')
```

### Step-3

We now that in order for linear regression to work. We need to know whether the data. We are fitting our model is linear or not. Let's draw a scatter plot to find it's Linear or not.

```python
plt.xlabel('Area')
plt.ylabel('Price')
plt.scatter(df['area'],df['price'],marker='+', color='red')
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/skjaaf053a7p51j2rbbq.png)

We  can see that all points plot indicate that it's almost linear data and can be fit by a stright line.

### Step-4
Modeling our Linear Regression model.

```python
linear_reg_model = linear_model.LinearRegression()
linear_reg_model.fit(df[['area']],df['price'])
``` 

You may wonder why are we using df[['area']] instead of df['area']. Thing is df['area'] returns Pandas.Series while this function takes DataFrame as an argument. So, that's why we used df[['area']] because it return the DataFrame instead of Series.

### Step-5

Testing our model on a single value

```python
linear_reg_model.predict([[3300]])
```

**Output**

> array([628715.75342466])

Our model is doing the prediction as intended.

### Step-6

Checking the regression coefficient and Y-intercept calculated.

```python
print(linear_reg_model.coef_)
print(linear_reg_model.intercept_)
```

**Output**

> [135.78767123]
> 180616.43835616432

For the equation `y=mx+b`. The **linear_reg_model.coef_** indicates the **m** and **linear_reg_model.intercept_** indicates the **b** in the equation respectively.

### Step-7

Doing prediction on testing data

```python
test_data = pd.read_csv('areas.csv')
preds = linear_reg_model.predict(test_data)
test_data['price'] = preds
plt.scatter(test_data['area'],test_data['price'],color='red')
plt.plot(test_data['area'],preds,color='green')
```

Here, **plot** methods draw a straight line while the **scatter** method draw the point on the plot. We are trying to get an idea. How well our regression lines fit the actual point of the data.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/txdlzb8b3t7yaft9khss.png) 

> Here, we don't have the actual price values. For the sake of visualization. We are treating the predicted values both as actual and predicted value. That's why we have all the point exactly on the line which is not the case otherwise.

## Multivariate Regression Model

When we have more than one independent variables. We just can't fit a Linear Model as it can't be fit by a **Straight line**. When we have more than one independent variable (features) the dimensinality of our problem increase. So, we use **plane** or **hyperplane** to fit our dataset.

### Step-1

Import libraries and Reading data.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
USAhousing = pd.read_csv('USA_Housing.csv')
```

### Step-2

We want to see whether our data is Linear. Does there exist any collineraity or noise. What it's distribution?

```python
sns.pairplot(USAhousing)
```
> Remember! It may take quite a time for larger dataset. Just be patient.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fvffw3j46251wti18uzu.png)

We can clearly see that other than Price none of the features show  collinearity. There isn't noise here as well. 

Let's now see the distribution of Price dependent variable.

```python
sns.distplot(USAhousing['Price'])
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ilgzcf2i4deojcvwmw4d.png)
 
We can clearly see that it's normally distributed just like we wanted. So, we can use Regression model here.

```python
sns.heatmap(USAhousing.corr())
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hmng1123f3tioyez9of6.png)
 
Another way of seeing the correlation between all the variable. We can see that no variable other than the target variable have any correlation with any other variable which confirms the absence of the **Collinearity**. The diagonal show the strong correlation because every variable have 100% correlation with itself.

### Step-3

Seperating Features and Target Variable.

```python
X = USAhousing[['Avg. Area Income', 'Avg. Area House Age', 'Avg. Area Number of Rooms','Avg. Area Number of Bedrooms', 'Area Population']]
y = USAhousing['Price']
```

### Step-4

Train Test split

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=101)
```

**0.4** indicates that 40% will be test data while 100-40=60% will be the Training data.

### Step-5

Creating and training the model.

```python
from sklearn.linear_model import LinearRegression
lm = LinearRegression()
lm.fit(X_train,y_train)
```

### Step-6

Model Evaluation

```python
# print the intercept
print(lm.intercept_)
coeff_df = pd.DataFrame(lm.coef_,X.columns,columns=['Coefficient'])
print(coeff_df)
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m3qtq3myzwv9v8h95jdd.png)
 
This is how we will interpret the data.

- Holding all other features fixed, a 1 unit increase in Avg. Area Income is associated with an *increase of $21.52 *.
- Holding all other features fixed, a 1 unit increase in Avg. Area House Age is associated with an *increase of $164883.28 *.
- Holding all other features fixed, a 1 unit increase in Avg. Area Number of Rooms is associated with an *increase of $122368.67 *.
- Holding all other features fixed, a 1 unit increase in Avg. Area Number of Bedrooms is associated with an *increase of $2233.80 *.
- Holding all other features fixed, a 1 unit increase in Area Population is associated with an *increase of $15.15 *.

### Step-7

Predication from our model.

```python
predictions = lm.predict(X_test)
plt.scatter(y_test,predictions)
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fd5j8h62f66kq6u8vxdf.png)

```python
sns.distplot((y_test-predictions),bins=50);
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4iqed8eim4d90ae3o147.png)

> If your residual have a NORMAL distribution it's a good indication that your choice for choosing the Regression model was correct. Otherwise, you may want to go back and see perhaps there is another model which is better for this problem.

### Step-8

Regression Evation Metrics

Here are three common evaluation metrics for regression problems:

**Mean Absolute Error** (MAE) is the mean of the absolute value of the errors:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9i8wnvb0s8nf3tdmmmix.png)

**Mean Squared Error** (MSE) is the mean of the squared errors:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e7epjhyydjyfdprw0j6f.png)
 
**Root Mean Squared Error** (RMSE) is the square root of the mean of the squared errors:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j93yszszo3mdrpvwjzda.png)
 
Comparing these metrics:

- **MAE** is the easiest to understand, because it's the average error.
- **MSE** is more popular than MAE, because MSE "punishes" larger errors, which tends to be useful in the real world. Applying square, as we know larger values have larger squares also square will make the value positive.
- **RMSE** is even more popular than MSE, because RMSE is interpretable in the "y" units.

All of these are **loss functions**, because we want to minimize them. 

```python
from sklearn import metrics
print('MAE:', metrics.mean_absolute_error(y_test, predictions))
print('MSE:', metrics.mean_squared_error(y_test, predictions))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, predictions)))
```

**Output**

> MAE: 82288.2225191
MSE: 10460958907.2
RMSE: 102278.829223

The smaller the errors (LOSS) better is the model performance.