Pandas - Handling Missing Values - 7 Days of Pandas
Tags: beginners, python, datascience
Published: 2022-12-24 15:54:37

Welcome to the fourth article in the "7 Days of Pandas" series where we cover the `pandas` library in Python which is used for data manipulation.

In the [first article](https://dev.to/piyushraj/pandas-read-and-write-data-from-csv-files-7-days-of-pandas-1o4f) of the series, we looked at how to read and write CSV files with Pandas. 
 
In the [second article](https://dev.to/piyushraj/pandas-basic-data-manipulation-7-days-of-pandas-4c47), we looked at how to perform basic data manipulation. 

In the [third article](https://dev.to/piyushraj/pandas-basic-exploratory-data-analysis-7-days-of-pandas-3816), we looked at how to perform EDA (exploratory data analysis) with Pandas.  

In this tutorial, we will look at how to handle missing values in data.

When working on some data, it's not uncommon to find missing values in the data. Missing values can occur in data for a variety of reasons, for example, error in data capture, encoding issues, etc. It's important to deal with missing values before you proceed on further analyzing data and it's a major step in data preprocessing.

In this tutorial, we will cover the following topics:

1. Identifying missing values.
2. Handling missing values.
    * Filling missing values.
    * Removing missing values.

Before we begin, let's first import pandas and create a sample dataframe that we will be using throughout this tutorial.


```python
import pandas as pd
import numpy as np

# employee data
data = {
    "Name": ["Tim", "Shaym", "Noor", "Esha", "Sam", "James", "Lily"],
    "Age": [26, 28, 27, 32, 24, None, 33],
    "Department": ["Marketing", "Product", "Product", "HR", "Product", np.nan, "Marketing"],
    "Salary": [60000, np.nan, 82000, np.nan, 58000, 55000, 65000]
}

# create pandas dataframe
df = pd.DataFrame(data)

# display the dataframe
df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Name</th>
      <th>Age</th>
      <th>Department</th>
      <th>Salary</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Tim</td>
      <td>26.0</td>
      <td>Marketing</td>
      <td>60000.0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Shaym</td>
      <td>28.0</td>
      <td>Product</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Noor</td>
      <td>27.0</td>
      <td>Product</td>
      <td>82000.0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Esha</td>
      <td>32.0</td>
      <td>HR</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Sam</td>
      <td>24.0</td>
      <td>Product</td>
      <td>58000.0</td>
    </tr>
    <tr>
      <th>5</th>
      <td>James</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>55000.0</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Lily</td>
      <td>33.0</td>
      <td>Marketing</td>
      <td>65000.0</td>
    </tr>
  </tbody>
</table>
</div>



## Identifying missing values

To identify missing values in a pandas DataFrame, you can use the pandas `isna()` method, which returns a boolean mask indicating the presence of missing values. You can then use this mask to select the rows or columns with missing values.

For example, let's check which values in the above dataframe are missing using the `isna()` function.


```python
df.isnull()
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Name</th>
      <th>Age</th>
      <th>Department</th>
      <th>Salary</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>False</td>
      <td>False</td>
      <td>False</td>
      <td>False</td>
    </tr>
    <tr>
      <th>1</th>
      <td>False</td>
      <td>False</td>
      <td>False</td>
      <td>True</td>
    </tr>
    <tr>
      <th>2</th>
      <td>False</td>
      <td>False</td>
      <td>False</td>
      <td>False</td>
    </tr>
    <tr>
      <th>3</th>
      <td>False</td>
      <td>False</td>
      <td>False</td>
      <td>True</td>
    </tr>
    <tr>
      <th>4</th>
      <td>False</td>
      <td>False</td>
      <td>False</td>
      <td>False</td>
    </tr>
    <tr>
      <th>5</th>
      <td>False</td>
      <td>True</td>
      <td>True</td>
      <td>False</td>
    </tr>
    <tr>
      <th>6</th>
      <td>False</td>
      <td>False</td>
      <td>False</td>
      <td>False</td>
    </tr>
  </tbody>
</table>
</div>



You can see the resulting boolean mask.

To check which columns in the dataframe have missing value, apply the `any()` function on the resulting boolean dataframe with `axis=0`.


```python
df.isna().any(axis=0)
```




    Name          False
    Age            True
    Department     True
    Salary         True
    dtype: bool



We see that only the "Name" column doesn't have any missing values.

## Handling missing values

Handling missing values is an important step in the data preparation pipeline. Generally, there are two approaches to handle missing values - 
* Fill the missing value with some appropriate value (for example, a constant or mean, median, etc. for continuous variables, and mode for categorical fields).
* Remove the missing values (remove the records with missing values).

Let's now look at how to do both of them in pandas.

### Filling missing values

To fill missing values in a pandas DataFrame, you can use the pandas `fillna()` method. This method allows you to specify a value to fill the missing values with, or a method for imputing the missing values.

For example, let's see what we get if we fill the missing values with 0.


```python
df.fillna(0)
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Name</th>
      <th>Age</th>
      <th>Department</th>
      <th>Salary</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Tim</td>
      <td>26.0</td>
      <td>Marketing</td>
      <td>60000.0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Shaym</td>
      <td>28.0</td>
      <td>Product</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Noor</td>
      <td>27.0</td>
      <td>Product</td>
      <td>82000.0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Esha</td>
      <td>32.0</td>
      <td>HR</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Sam</td>
      <td>24.0</td>
      <td>Product</td>
      <td>58000.0</td>
    </tr>
    <tr>
      <th>5</th>
      <td>James</td>
      <td>0.0</td>
      <td>0</td>
      <td>55000.0</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Lily</td>
      <td>33.0</td>
      <td>Marketing</td>
      <td>65000.0</td>
    </tr>
  </tbody>
</table>
</div>



It returns a dataframe with the missing values filled with the constant value. Note that the `fillna()` function didn't modify the original dataframe in-place. It returned the resulting dataframe after filling the missing values.

You can also specify different values for different columns when filling missing values.
For exmaple, let's fill missing values in "Age" and "Salary" columns with their respective means and the missing value in the "Department" column with its mode (the most frequent value).


```python
df.fillna({'Age': df['Age'].mean(), 'Salary': df['Salary'].mean(), 'Department': df['Department'].mode()[0]})
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Name</th>
      <th>Age</th>
      <th>Department</th>
      <th>Salary</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Tim</td>
      <td>26.000000</td>
      <td>Marketing</td>
      <td>60000.0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Shaym</td>
      <td>28.000000</td>
      <td>Product</td>
      <td>64000.0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Noor</td>
      <td>27.000000</td>
      <td>Product</td>
      <td>82000.0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Esha</td>
      <td>32.000000</td>
      <td>HR</td>
      <td>64000.0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Sam</td>
      <td>24.000000</td>
      <td>Product</td>
      <td>58000.0</td>
    </tr>
    <tr>
      <th>5</th>
      <td>James</td>
      <td>28.333333</td>
      <td>Product</td>
      <td>55000.0</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Lily</td>
      <td>33.000000</td>
      <td>Marketing</td>
      <td>65000.0</td>
    </tr>
  </tbody>
</table>
</div>



### Dropping rows with missing values

Another strategy of handling missing values is to remove the rows that contain missing values. This is used when the proportion of missing values is comparitively less and we can afford to discard that data.

Use the pandas `dropna()` function to remove rows with missing values.


```python
df.dropna()
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Name</th>
      <th>Age</th>
      <th>Department</th>
      <th>Salary</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Tim</td>
      <td>26.0</td>
      <td>Marketing</td>
      <td>60000.0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Noor</td>
      <td>27.0</td>
      <td>Product</td>
      <td>82000.0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Sam</td>
      <td>24.0</td>
      <td>Product</td>
      <td>58000.0</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Lily</td>
      <td>33.0</td>
      <td>Marketing</td>
      <td>65000.0</td>
    </tr>
  </tbody>
</table>
</div>



This is how the dataframe looks after removing rows with any missing values.