Sampling Bias and how to fix it?
Tags: machinelearning, python, beginners, tutorial
Published: 2022-01-09 12:09:18

Pre-requisites: 

- Basic knowledge of Python 
- Working Juypter notebook for testing the code

---

## What is sampling Bias? 

To understand sampling bias let us consider an example - 

Supposedly we are a Statistician conducting a study on the effects of a new drug introduced in the market for producing more serotonin. We have been asked to identify if the new drug is effective on the male or female population more. 

We collect data on the population who have taken the drug and create more features around it where our target `(y)` is gender; 

_1 if it's a female 0 if its a male_

As you guessed it we have a **binary classification** problem at our hands.

When we load the dataset we realize - 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ye9cvsonflohksxl727u.png)

That there are 4 females and 1 male in the dataset _(Since its a hypothetical scenario and my drawing is horrible I could only make 5 stick figures)_

But imagine we get a dataset where the **ratio of the classes in target is imbalanced** - for every 4 females there is 1 male in our dataset.

> The analysis we do and the model we build would be incorrect since our **target is highly imbalanced the model gets biased towards the majority class**.

Feeling a bit stressed, eh?


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8pkce8sqj7e1prj5xcig.gif)


###To deal with imbalanced datasets we can go through 2 routes - 

1. Either we reduce our target where gender is female and make sure that we have an equal proportion of both the classes, resulting in a **case of undersampling** of the female gender.

2. Or we increase our target where gender is male and make sure that we have an equal proportion of both the classes, resulting in a **case of oversampling** of the male gender. 

There are [several techniques](https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis) to deal with oversampling such as - 

1. Random oversampling 
2. SMOTE
3. ADASYN 
4. Augmentation 

For undersampling we have - 

1. Random undersampling 
2. Cluster 
3. Tomek links 

In this article, we will learn about [imblearn](https://pypi.org/project/imbalanced-learn/) and resolve the case of sampling bias. 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m2gl6dw2ct9xamwp9dj0.gif)

> Imblearn library offers us the methods by which we can generate a data set that has an equivalent proportion of classes.

Let us build a classification model and see this working. For the  dataset and code used in the article please refer [here](https://github.com/jaindamini1111/ML-DT-Course-Files/tree/main/Article). 
_The data has been generated using Excel._


![Target distribution](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0i2o2qjwa2woxw5rdky6.jpg)

As we can see above the distribution of female vs male in our target variable is imbalanced. Let us use the sampling technique to overcome the situation.

###We will first perform oversampling. 

Top 2 rows of our data after preprocessing (steps available in the repo)
![preprocessed data](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ah0gkbrsb6har489e3jv.jpg)



```
import imblearn  #Importing all libraries. 
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from collections import Counter
from xgboost import XGBClassifier


y = data["Gender"]   #Creating target and feature set.
X = data.drop("Gender", axis=1)

#Splitting data to create train, valid & test sets.
X_train, X_test, y_train, y_test = train_test_split(X, y, shuffle = True, test_size = 0.2, random_state=42, stratify = y)
X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, shuffle = True, test_size = 0.2, random_state=42, stratify = y_train)

print(sorted(Counter(y_train).items())) #Lets see what the group count looks like in our training set prior to sampling.
```

![print result](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/isd9c4dy503qtdamterv.jpg)

As we can see before correcting the data for sampling we have 261 instances that belong to class 0 and 635 instances which belong to class 1. If we oversample 0 class we will have a balanced dataset lets do that using SMOTE.

The important thing to focus here is that 

> we fit the sampling object only on the training data and not on valid & test sets otherwise data leakage occurs.

```
ros = imblearn.over_sampling.RandomOverSampler(random_state=42)
x_ros, y_ros = ros.fit_resample(X_train, y_train) #Fitting the oversampling object to our training set.
print(sorted(Counter(y_ros).items()))
```

![Oversampling the class 0](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u1r54l5unpo2fqxmiy9o.jpg)

Now the classes are balanced. We will build an **XGBoost model** on the new data. 
```

model = XGBClassifier(objective='binary:logistic')
model.fit(x_ros, y_ros)
y_pred = model.predict(X_valid)
print(model.score(X_valid, y_valid))
```
We get a 0.59 accuracy score without any hyperparameter tuning.

###Let us perform undersampling now.

```
ros_under = imblearn.under_sampling.RandomUnderSampler(random_state=42)
x_ros_under, y_ros_under = ros_under.fit_resample(X_train, y_train) #Fitting the oversampling object to our training set.
print(sorted(Counter(y_ros_under).items()))
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6jpwxy6h0sw99wyem1sm.jpg)

```
model = XGBClassifier(objective='binary:logistic')
model.fit(x_ros_under, y_ros_under)
y_pred = model.predict(X_valid)
print(model.score(X_valid, y_valid))
```
 We get a 0.51 accuracy score on the validation dataset when we undersample the data without any hyperparameter tuning.

One reason for the accuracy to be less in undersample is that we give less data for our model to learn. Hence oversampling in our case without hyperparameter tuning wins on the validation dataset.

### Conclusion

In this article, we discussed how we can pre-process the imbalanced class data set before building predictive models. 

We first did  oversampling and then performed undersampling.You can check the documentation [here](https://imbalanced-learn.org/stable/index.html).

Before we part away most of the ensemble algorithms today offer a parameter by which we can handle imbalanced datasets. In XGBoost we have `scale_pos_weight` parameter, to handle cases but the 
room for configuration gets less using SMOTE we have the control over resolving biases.      

Edit 01: You may notice difference in accuracy score since I didn't set the `random_state` parameter while creating my classifier. 
Shoutout to Barbara Bredner for pointing this out :)