Unraveling Multicollinearity: Causes, Implications, and Solutions
Tags: machinelearning, datascience, python, programming
Published: 2023-12-23 08:05:45

## What is multicollinearity
Multicollinearity occurs when two or more predictor variables in a regression model are highly correlated, leading to issues in estimating individual coefficients. This tangled web can introduce numerical instability, making it tricky to pinpoint the true relationship between predictors and the target variable.

But wait, how does it create an issue in estimating individual coefficients?

<img width="100%" style="width:100%" src="https://media.giphy.com/media/64afibPa7ySzhFAf00/giphy.gif">

## Why multicollinearity is bad

Suppose this is the equation for knowing the accuracy of the model

![Equation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/39l3truvg152w8zj7y6q.png)

Here `y` is the dependent variable and `x1` and `x2` are independent variables where `β1` and `β2` represent the weight of the `x1` and `x2` respectively which simply means in determining `y` how much weight does `x1` possess. So if we want to see how much `y` depends upon `x1` we can calculate `β1` by keeping other variables constant.

Now if the two variables `x1` and `x2` are correlated then we cannot isolate `x1` as it depends upon `x2` and hence we will have a problem in calculating the value of `β1`.

So in place, we need to find out how our target depends upon each feature having multicollinearity in a data is very bad.

## How to remove multicollinearity
1. We can apply feature selection. Where we identify which columns are important and drop the rest.

2. We can apply the regression technique which introduces a penalty term which is used to shrink the coefficient.

3. Calculate **VIF (Variance Inflation Factor)**. Variables with high VIF values (typically above 5 or 10) may indicate multicollinearity.

4. We can plot a scatter plot and by looking at the graph we can tell which data are correlated.

5. We can increase the size of the data. Sometimes, increasing the size of your dataset can help alleviate multicollinearity issues. More data can provide a more representative sample of the population, leading to more stable coefficient estimates.

## Code

```python
np.random.seed(42)

# These are correlated data
x1 = np.random.rand(100)
x2 = 0.8 * x1 + 0.2 * np.random.rand(100)
x3 = 0.7 * x1 + 0.3 * np.random.rand(100)
x4 = 0.6 * x1 + 0.4 * np.random.rand(100)
x5 = 0.5 * x1 + 0.5 * np.random.rand(100)

# These are uncorrelated data
x6 = np.random.rand(100)
x7 = np.random.rand(100)
x8 = np.random.rand(100)
x9 = np.random.rand(100)
x10 = np.random.rand(100)
```

Let's see if we plot the graph between two correlated data how it going to look

```python
plt.scatter(x1, x2)
```

![correlated graph](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7cly0bw91viwlllocy6n.png)

and if we plot the graph between the uncorrelated data
```python
plt.scatter(x6, x7)
```

![Uncorrelated graph](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1ifn4ucibtgfen4c24ct.png)

From the figure, we can see which columns are correlated and if we want to drop the correlated columns we can easily do that

### Now let's see what a correlation matrix is going to tell us

```python
# First we need to make our variable into a data frame
data = pd.DataFrame({
    "x1": x1,
    "x2": x2,
    "x3": x3,
    "x4": x4,
    "x5": x5,
    "x6": x6,
    "x7": x7,
    "x8": x8,
    "x9": x9,
    "x10":x10
})

matrix = data.corr()
matrix
```

![matrix](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1jmgc111xzt3p6d5wft4.png)

Here, a high value represents a very large correlation. For example, the x2 and x1 correlate 0.97 which is very high. The negative value represents the negative correlation.

We can also see this correlation using the heatmap provided by the seaborn.

```python
import seaborn as sns
plt.figure(figsize=(10, 7))
sns.heatmap(corre, annot=True, cmap='coolwarm')
plt.title("Correlation Matrix")
plt.show()
```


![Heatmap](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uc1gr2czocrsl6oop31m.png)

If the data is very large it will be very difficult to tell which columns are correlated. So let's build a function which takes threshold (amount of correlation) as a parameter and returns the column name

```python
def correlation(data, threshold):

    # so that we don't get duplicate column
    correlated_col = set()

    correlated_matrix = data.corr()
    for i in range(len(correlated_matrix.columns)):
        for j in range(i):

            # if the correlation is greater than the threshold
            if abs(correlated_matrix.iloc[i, j]) > threshold:
                related_col_name = correlated_matrix.columns[i]
                correlated_col.add(related_col_name)

    return correlated_col
```

`correlation(data, 0.9)`
Output
`{'x2', 'x3'}`
and
`correlation(data, 0.7)`
Output
`{'x2', 'x3', 'x4', 'x5'}`

Now if we want to drop the correlated column we can easily do that.

## Conclusion
By understanding and addressing multicollinearity, we can enhance the robustness and interpretability of our models, ensuring more accurate insights for data-driven decision-making.