Pandas - Basic Exploratory Data Analysis - 7 Days of Pandas
Tags: python, datascience, beginners
Published: 2022-12-23 07:29:36

Welcome to the third article in the "7 Days of Pandas" series where we cover the `pandas` library in Python which is used for data manipulation.

In the [first article](https://dev.to/piyushraj/pandas-read-and-write-data-from-csv-files-7-days-of-pandas-1o4f) of the series, we looked at how to read and write CSV files with Pandas. 
In the [second article](https://dev.to/piyushraj/pandas-basic-data-manipulation-7-days-of-pandas-4c47), we looked at how to perform basic data manipulation.
In this tutorial, we will look at some of the common operations that we perform on a dataframe during the exploratory data analysis (EDA phase).

Exploratory Data Analysis (EDA) helps us better understand the data at hand and can give us valuable insights. In this phase, we look at the data for insights and use descriptive statistics and visualizations to derive insights from the data.

The pandas library comes with a number of useful functions that help us explore the data. In this tutorial, we will cover the following topics:

1. Get the first and the last N rows of a dataframe.
2. Using the `info()` function.
3. Get descriptive statistics with the `describe()` function.

Before we begin, let's first import pandas and create a sample dataframe that we will be using throughout this tutorial.


```python
import pandas as pd

# employee data
data = {
    "Name": ["Tim", "Shaym", "Noor", "Esha", "Sam", "James", "Lily"],
    "Age": [26, 28, 27, 32, 24, 31, 33],
    "Department": ["Marketing", "Product", "Product", "HR", "Product", "HR", "Marketing"],
    "Salary": [60000, 70000, 82000, 55000, 58000, 55000, 65000]
}

# create pandas dataframe
df = pd.DataFrame(data)

# display the dataframe
df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Name</th>
      <th>Age</th>
      <th>Department</th>
      <th>Salary</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Tim</td>
      <td>26</td>
      <td>Marketing</td>
      <td>60000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Shaym</td>
      <td>28</td>
      <td>Product</td>
      <td>70000</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Noor</td>
      <td>27</td>
      <td>Product</td>
      <td>82000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Esha</td>
      <td>32</td>
      <td>HR</td>
      <td>55000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Sam</td>
      <td>24</td>
      <td>Product</td>
      <td>58000</td>
    </tr>
    <tr>
      <th>5</th>
      <td>James</td>
      <td>31</td>
      <td>HR</td>
      <td>55000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Lily</td>
      <td>33</td>
      <td>Marketing</td>
      <td>65000</td>
    </tr>
  </tbody>
</table>
</div>



We have a dataframe with information of some employee in an office.

## Get the first and the last N rows of a dataframe

After loading or creating a dataframe, a good first step is to look at the first few rows to see if the data is as expected or not. Or, if there are any obvious issues with the data (for example, missing fields, etc.).

You can use the pandas dataframe `head()` function to get the first n rows of the dataframe. Pass the number of rows you want from the top as an argument. By default, n is 5.


```python
# get the first five rows
df.head(5)
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Name</th>
      <th>Age</th>
      <th>Department</th>
      <th>Salary</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Tim</td>
      <td>26</td>
      <td>Marketing</td>
      <td>60000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Shaym</td>
      <td>28</td>
      <td>Product</td>
      <td>70000</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Noor</td>
      <td>27</td>
      <td>Product</td>
      <td>82000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Esha</td>
      <td>32</td>
      <td>HR</td>
      <td>55000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Sam</td>
      <td>24</td>
      <td>Product</td>
      <td>58000</td>
    </tr>
  </tbody>
</table>
</div>



You can similarly get the last n rows of the dataframe, using the pandas dataframe `tail()` function. Pass the number of rows you want from the bottom as an argument. By default, n is 5.


```python
# get the last five rows
df.tail(5)
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Name</th>
      <th>Age</th>
      <th>Department</th>
      <th>Salary</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2</th>
      <td>Noor</td>
      <td>27</td>
      <td>Product</td>
      <td>82000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Esha</td>
      <td>32</td>
      <td>HR</td>
      <td>55000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Sam</td>
      <td>24</td>
      <td>Product</td>
      <td>58000</td>
    </tr>
    <tr>
      <th>5</th>
      <td>James</td>
      <td>31</td>
      <td>HR</td>
      <td>55000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Lily</td>
      <td>33</td>
      <td>Marketing</td>
      <td>65000</td>
    </tr>
  </tbody>
</table>
</div>



## Use the `info()` function

You can use the [pandas dataframe `info()` function](https://datascienceparichay.com/article/pandas-get-dataframe-summary-with-info/) to get a concise summary of the dataframe. It gives information such as the column dtypes, count of non-null values in each column, the memory usage of the dataframe, etc.


```python
# summary of the dataframe
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 7 entries, 0 to 6
    Data columns (total 4 columns):
     #   Column      Non-Null Count  Dtype 
    ---  ------      --------------  ----- 
     0   Name        7 non-null      object
     1   Age         7 non-null      int64 
     2   Department  7 non-null      object
     3   Salary      7 non-null      int64 
    dtypes: int64(2), object(2)
    memory usage: 352.0+ bytes


## Get descriptive statistics with the `describe()` function

The [pandas dataframe `describe()` function](https://datascienceparichay.com/article/pandas-dataframe-describe-function/) returns some descriptive statistics for a dataframe. For example, for numerical columns, it returns the count, mean, standard deviation, min, max, percentile values, etc.


```python
# get dataframe's descriptive statistics
df.describe()
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Age</th>
      <th>Salary</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>count</th>
      <td>7.000000</td>
      <td>7.000000</td>
    </tr>
    <tr>
      <th>mean</th>
      <td>28.714286</td>
      <td>63571.428571</td>
    </tr>
    <tr>
      <th>std</th>
      <td>3.352327</td>
      <td>9778.499252</td>
    </tr>
    <tr>
      <th>min</th>
      <td>24.000000</td>
      <td>55000.000000</td>
    </tr>
    <tr>
      <th>25%</th>
      <td>26.500000</td>
      <td>56500.000000</td>
    </tr>
    <tr>
      <th>50%</th>
      <td>28.000000</td>
      <td>60000.000000</td>
    </tr>
    <tr>
      <th>75%</th>
      <td>31.500000</td>
      <td>67500.000000</td>
    </tr>
    <tr>
      <th>max</th>
      <td>33.000000</td>
      <td>82000.000000</td>
    </tr>
  </tbody>
</table>
</div>



Note that the pandas dataframe `describe()` function, by default includes only the numeric columns when generating the dataframe’s description.

You can, however, specify other columns types (or all the columns) to include the statistics for using the `include` parameter. 


```python
# get descriptive statistics for object type the columns
df.describe(include='object')
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Name</th>
      <th>Department</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>count</th>
      <td>7</td>
      <td>7</td>
    </tr>
    <tr>
      <th>unique</th>
      <td>7</td>
      <td>3</td>
    </tr>
    <tr>
      <th>top</th>
      <td>Tim</td>
      <td>Product</td>
    </tr>
    <tr>
      <th>freq</th>
      <td>1</td>
      <td>3</td>
    </tr>
  </tbody>
</table>
</div>



For `object` type columns, we get the information about the count, number of unique values, top (the most frequent value), and freq (the count of the most frequent value in the column).

These descriptive statistics give us valuable insights into the distribution of the data in different columns.