Data preprocessing
Tags: ai, machinelearning
Published: 2021-02-08 18:50:09

Anytime you build a machine learning model, you always have a data preprocessing phase to work on. You have to preprocess the data in the right manner in such a way that the machine learning model which we built can be trained in the right way on the data.

__Data set for processing__

Let us consider the data in a hospital where we have the __country, age, plateletcount, whether patient has survived or not__ as columns in the data row. This data can be used to train the system in such a way that it can predict survival rate for the following :

1. Country-wise survival rate with respect to platelets
2. Age-wise survival rate with respect to platelets
3. Country and age -wise survival rate with respect to platelets etc..

Let us consider the following dataset below :

```
Country      Age 	Platelets 	Survived
 
1.India	     23	        45000		Yes
2.India	     70	        50000		No
3.USA	     40	        70000		Yes
4.Australia  60	        100000		No	
5.USA	     60	        100000		Yes
6.India                 100000      Yes
7.India      40                     Yes
.... 
....
....
....	  
``` 

__Steps to be followed in Data Preprocessing__

1. Importing the libraries required for data preprocessing 
2. Importing the dataset required for data preprocessing
3. Take care of missing data
4. Encoding categorical data
5. Splitting the data set into the training set and the test set
6. Feature scaling

__Prerequisites for development__

Python installation and environment setup, Visual Studio Code with Jupyter Notebook extension added. 

__Step 1: Importing the libraries__

The first step is how to import the libraries and make them ready every time we build a new machine learning model. We will be importing three required libraries : 

1. __Numpy__ :
     
     Numpy will allow you to work with arrays. Most of our machine learning models will require you to work with arrays. 

2. __Matplotlib__ :

    Matplotlib will allow you to print some very nice charts based on the available dataset.

3. __Pandas__ :

    Pandas will not only allow you to import the dataset, but also create the __matrix of features__ and the __dependent variable vector__. We will see about these concepts later.

```
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
```

__Step 2: Importing the dataset__

Using the pandas library we will be reading the data from the csv file. But an additional important step is required which is __creating two new entities. :__ 

1. __Matrix of features__
2. __Dependent variable vector__

__Important principe to know :__

In any dataset with which you are going to train a machine learning model, you have the same entities which are the features and the dependent variable vector.

Feature columns are those with the help of which you are going to predict the output, the output column is the dependent variable vector. The data set itself should be prepared in such a way that the dependent variable should be in the last column. Features are also called the independent variables.

In our case, the columns __Country, age, platelets__ are the feature variables and the last column __survived__ is the dependent variable. Always remember in any machine learning model, features should be present in the beginning of the dataset and dependent variables should be in the last columns. 

```
dataset = pd.read_csv('Data.csv')
X = dataset.iloc[:, :-1].values
Y = dataset.iloc[:, -1].values
print(X)
print(Y)
```

In the above code, X will contain the features and Y will contain the dependent variables. __iLoc__ is a method used to obtain the rows and columns in a dataset by specifying the index, range etc..

__Step 3 : Taking care of missing data__

In our dataset in the last two rows, in one row we have a missing age and in the last row we have the platelets missing. Generally you shouldn't have any missing data in the dataset, as it can cause some errors while training the machine learning model. So we have to handle these missing value rows. There are several ways to handle them :

__1.Ignoring the row with missing values__ 

This method actually will work when you have a very large dataset and you have only 1% or 2% missing data, removing this data won't much affect the learning quality of your machine learning model. Sometimes when there is a large amount of missing data, we have to handle them the right way.

__2.Replace the missing value by the average of all the available values in the column__

For performing these kind of operations, we have one of the best data science libraries available called the __scikit-learn__. It is library which contains a lot of tools including 
data preprocessing tools.

```
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(missing_values=np.nan, strategy='mean')
imputer.fit(X[:,1:3])
X[:, 1:3] = imputer.transform(X[:, 1:3])
print(X)
```

The above code will replace the missing values in the age and platelets column with the mean value of all the values in their respective columns. Please note that this method can be applied only for numerical values.

__Step 4: Encoding categorical data__

As you can see one column named Country contains categories such as India, Australia and the USA. This data would cause difficulties for machine learning models to compute some correlations between the feature columns and the dependent variable (outcome).

Hence, we have to change these categories into numbers. One method is to encode USA to 0, Australia to 1 and India to 2, however if you do this, your machine learning model can understand that because USA is 0, Australia is 1 and India is 2, there is a numerical order between these three countries, and mostly it could interpret that this order matters. You need to avoid such kind of interpretation by your machine learning model. 

__One hot encoding__

This method turns the country column into three columns, in our dataset it is three columns whereas if there are more than 3 columns, there will be those many number of column generated. 

It creates __binary vectors__ for each of the countries. Very simply, for India it would be _100_, Australia will be __010__ and USA will be __001__, and hence there is not a numerical order between the countries. Please follow the below two steps for one hot encoding 

__1. Encoding the independent variable__

```
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
ct = ColumnTransformer(transformers=[('encoder', OneHotEncoder(), [0])], remainder='passthrough')
X = np.array(ct.fit_transform(X))
print(X)
```

__2. Encoding the dependent variable using Label Encoding__

```
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
Y = le.fit_transform(Y)
print(Y)
```

__Step 5: Splitting the data into Training set and Test set__

It makes two separate sets, __one training set__ where you are going to __train your machine learning model__ on existing observations, and __one test set__ with which you are going to __evaluate the performance of your model__. 


```
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size = 0.2, random_state = 1)
print(X_train)
print(X_test)
print(Y_train)
print(Y_test)
``` 

__Step 6 : Feature scaling__

It will allow you to put all your features on the same scale. 

__Why should you do this ?__

For some machine learning models, in order to avoid some features to be __dominated__ by other features in such a way that these features are not even considered by the machine learning model.

__When to apply feature scaling ?__

You need not apply feature scaling for all the machine learning models, but only for some, where feature values highly vary in range, units etc.. Feature scaling helps to weigh all the features equally in order to improve the quality of your machine learning model.

__How to apply feature scaling ?__

There are two techniques to apply feature scaling :

__1.Standardisation__ :

   This will put all the values of the feature between around -3 and +3 for example. When you apply this transformation on all the different features, all your features will take values around -3 and +3. 

__2.Normalisation__ :

   All the values of the feature will become between 0 and 1 for example. 

__Which one to opt on the above two techniques ?__

__Normalisation is recommended__ when you have a normal distribution in most of your features. __Normal distribution__ is a probability distribution which often resembles a bell shaped curve. For example: the height of kids.

__Standardisation will work well in all the cases__, mostly you will use this technique except for features having a normal distribution. This will always improve the training process. 

We will not apply feature scaling on the full dataset, instead we will apply feature scaling for the test and the training set separately. Please note that we need not apply feature scaling for the categorically encoded data, because they already are present in a certain range with less standard deviation. For example, the categorically encoded data for the country column already lies between -3 and +3. 

In our dataset, we will be applying feature scaling for the __age feature__ and the __platelets feature__. The below code will transform these two features values in between -2 and +2.
Now all the values of these two features are on the same scale and this will improve the training of most machine learning models. 

```
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train[:, 3:] = sc.fit_transform(X_train[:, 3:])
X_test[:, 3:] = sc.fit_transform(X_test[:, 3:])
print(X_train)
print(X_test)
```

Note: Please note that we will be working on huge data sets, the data set which i have prepared contains less than 10 rows of data and this is just for your understanding.