7 Essential Techniques for Data Preprocessing Using Python: A Guide for Data Scientists
Tags: datascience, python, machinelearning, productivity
Published: 2023-05-30 03:45:00

Data preprocessing is an important step in data science. Data preprocessing means cleaning, transforming and preparing raw data for analyzing. For this purpose Python programming language is used. Because python has inbuilt libraries and tools for data science and machine learning. In this blog we will see some steps in data preprocessing.

Below are some steps anyone (even beginner can understand) can use in their practice or learning.

### 1. Importing Libraries :
First step in any project is importing the necessary libraries, which will be used in entire code. Below are some common libraries.


```python
import numpy as np
import pandas as pd
import matplot.pyplot as plt
import seaborn as sns
```

### 2. Loading Data :
Next step is to load the data to process. In python pandas is a library which used for this purpose. Pandas is a very powerful library to load the data and process it. To load the data from _csv_ file there is function named `read_csv`.

```python
df = pd.read_csv("data.csv")
```

### 3. Handling Missing Values :
When we work on real world data, there might be some missing values in files. When there are missing values in data then any algorithm can't work on that data. So this step is most important in data preprocessing. So we need to handle all missing values before performing any analysis. `pandas` includes various function/methods to handle missing values. There is a function names `isnull` which provides information about missing values from data.

```python
missing_values = df.isnull()
```

For filling some value in the place of missing value `fillna` function can be used.

```python
df.fillna(0,inplace=True)
```
_here `inplace` means values will be overwritten in dataset directly_


### 4. Handling outliers :
There may be some data points in dataset which may be totally different from other data points. These datapoints are known as outliers. This process can be called as _finding the odd one out_. There is function in `seaborn` library named `Boxplot` which can be used to visualize the distribution of data and identify the outliers.

```python
sns.boxplot(x=df['column_name'])
```

`z_score` is used to identify and remove the outliers. `z_score` is the measurement of standard deviation of a data point from the mean. this function is inside the `scipy` library.

```python
from scipy import stats
df = df[(np.abs(stats.zscore(df)) < 3).all(axis=1)]
```

To more information on outliers [Click Here](https://dev.to/newbie_coder/outliers-4b10)


### 5. Encoding the Categorical variables :
Categorical variables means that data which shows categories or some memberships. Mostly this data can be in the form of string or characters. But all machine learning models works on only numerical data, so that's why we have to encode these variables in numerical format. 

There are several methods for encoding categorical variables.
we will discuss `get_dummies` method from `pandas` library.
This method creates dummy variables for categorical data. Basically it converts categorical data into 0/1 on the basis of categories from that column.

```python
df = pd.get_dummies(df,columns=['column_name'],dtype=float)
```

Demo :

![Encoding Categorical Variables](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ty0a41vpqyfv3l7v8fwq.png)

For more [Click Here](https://pandas.pydata.org/docs/reference/api/pandas.get_dummies.html)


### 6. Feature Scaling :
To work the model smoothly data should be in some range. Here comes the feature scaling. Using feature scaling data points can be brought in some scale, like 0-1. There is method named `MinMaxScaler` in `sklearn` library, which can scale the features between 0 to 1.

```python
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
df[['column_name']] = scaler.fit_transform(df[['column_name']])
```

Above code scales the data points from given column between 0 to 1.


### 7. Feature Selection :
Feature selection involves choosing the most relevant features from a dataset, aiming to enhance the accuracy and efficiency of machine learning models. `sklearn` library provides the function named `SelectKBest` which selects the top K features from the dataset using statistical tests. By opting out for most relevant features, we can optimize the performance of our models in terms of speed and accuracy.

```python
from sklearn.feature_selection import SelectKBest, chi2
x = df.drop('target_column', axis=1)
y = df['target_column']

selection = SelectKBest(chi2, k=3)
x_new = selection.fit_transform(x,y)
```


here `chi2` means chi-squared value which indicates which features is most important. Here if chi2 is higher then that feature will be selected. And k=3 means number of features to select.

For more information of chi2 - [Click Here](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html)


### Conclusion :
In conclusion, we can say that data pre-processing plays important role in data science/Machine Learning. In this post we explored some fundamental techniques for data preprocessing using python. By applying these techniques, we can clean, transform and prepare raw data for further analysis and modeling.