Pandas - Data Manipulation and Analysis
Tags: python, machinelearning, datascience, programming
Published: 2022-10-28 17:50:38

### Introduction

Python's Pandas package is used to manipulate data collections. It offers tools for data exploration, cleaning, analysis, and manipulation. Wes McKinney came up with the name "Pandas" in 2008, and it refers to both "Panel Data" and "Python Data Analysis.”

### How to Install Pandas

To install Pandas, first, make sure that Python and Pip is already installed in your system. If they are installed, then you can install Pandas just by running this command on the command line.

```python
>> pip install pandas
```

You can use other Python distributions like Spyder or Anaconda where pandas are already installed.

### Write your first code using Pandas


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a05qm01hvlv8gcv2vc0n.png)

To use Pandas in our code, first, we have to import it. Here we are importing pandas as `pd`. Here, `pd` is an alias for `panda`.  

### Series

A series in pandas is like a column in a table that can hold data like a one-dimensional array of any type. To create a Pandas series, we can write this code:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b7ijjqxibafcmanis4qk.png)

Here we have used a method called `Series()` to convert a list into a series. After that, we can print the series using `print()` function. If we observe the output, we will see that there are 5 rows and 2 columns. In the end, it’s showing the data type of the elements in the list that we have converted into a series. The first column is the column of labels. If nothing else is specified, the values are labeled with their index number. The first value has index 0, second value has index 1 etc. This label can be used to access a specified value.


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3opoahky46hplsl72h4a.png)

Now, its clear that we can use use the labels as an index and access a specific value using it. The fun thing is, we can name our own labels using the `index` argument.


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vd88lyadgv1qk925tkpm.png)

After creating our labels, we can access an item by referring to the label like we access a dictionary value using the key:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t4q8bteiqiveq24sy97c.png)

**Key/Value Objects as Series**

You can also use a key-value object, like a dictionary, when creating a series.


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b2u9fevwy36g42sd7x7g.png)

Use the index option to specify only the words you wish to be included in the Series, leaving out the rest of the words in the dictionary.


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ymv1ifcz1ks4fsm56ldp.png)

Now let's see what happens when we add two series.


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ojdwmh6aas3i8cgte3l4.png)

It is creating a new series by adding the same labeled data together and converting them into floats. If the same index is missing, then the output will be `NaN`.

### **DataFrames**

A Pandas DataFrame is a 2-dimensional data structure, like a 2-dimensional array, or a table with rows and columns. To understand it better, first, let’s compare it with the pandas series.


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ttuaizsff0lz6agf7n4k.png)

The first visible difference here is that a series has a specific data type, but a data frame doesn’t.

**Dictionary and Data Frame**


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8m24f4c1bnbiv9g7v68z.png)

Here we can see that the keys of the dictionary are used as the titles of the columns, and the lists are the data of the columns. Now, what is happening here is that we are passing a list and it is creating a data frame. It also defines its index and column names on its own. If we want to use our own name as an index or column name, we have to pass it as an argument through the `DataFrame()` method.

**Generate Data Frame**


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h2uaoki1a78tiflpk8r0.png)

In the example above, we are passing a 2D array, index, and column name. After that, the `DataFrame()` method automatically allocates the index and column name.

**Selection and Indexing**

Firstly, DataFrame columns are just Series:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bwcm9mfpvdtfjwzq3w85.png)

Now, let’s learn the various methods to grab data from a DataFrame


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2i4fx5dmxhyltthr4ibh.png)

We can grab any column by using the column name. To access multiple columns, we have to pass a list of column names like that:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8m0zyo5lm1m4ahjanrak.png)

We can also use the SQL Syntax but it is not recommended as it creates confusions:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ujapc09j7yo9v0aqqp42.png)

**Creating a new column:**

We can add a new column to our DataFrame like below. We can assign either Array or Series to define the new column. In the example below, we are adding two columns, which are basically two series, and assigning a new column named `‘new’`:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gh3548go6do9z0x57od5.png)

**Removing Columns**

We have to use`.drop()` method to drop a column or row. Here `axis=0` means row and `axis=1` means column. We have to mention it if we want to remove the column. The default value is `0`. 


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xy2tqdize8r1kfyx97fo.png)

But there is a problem. If we try to print `df` again, we will see that column `new` is still there.


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x4zokc2rb0i4ngyuvbsz.png)

But why? Because to remove the column completely from the DataFrame, we have to use the `inplace` attribute. The Python developers kept that feature so that we do not have to face unwanted data loss.


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rhzvaz0cg5b717no2dfe.png)

Congratulations! The `new` column is completely removed from the DataFrame. We can also drop a row by changing the `axis` from `1` to `0`:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2wfkanck0ch4wjq893tw.png)

**Locate Rows**

We already know that a dataframe is like a table with rows and columns. If we want to display or store a specific row, we have to use the `loc` method:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x7dobaz0fo8arq49kpt0.png)

Alternatively, you can choose based on position rather than the label using `iloc`:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sz8ksxbz8evt66gxykd3.png)

We can also select a subset of rows and columns by the following method:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/96nqgmk9l8u406ygpc7v.png)

**Conditional Selection**

An important feature of pandas is conditional selection using bracket notation, very similar to Numpy:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5nagpprluav2rlm2o5uc.png)

Here we can see a truth table based on the conditions we have given. When a cell satisfies our conditions, it's giving `True`. Otherwise, it's giving `false`. Now, if we want to print the value instead of true or false, we can write our code like this:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r39iu3y6uapdg6hpk9h1.png)

Now, it is printing only the values that satisfy our conditions.

What if our condition is only based on a specific column and we want to print the values according to the data of that column? In that case, we have to mention that specific column using this notation:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zdpdxulqt2jxfimd4nf3.png)

For printing a specific column of our new filtered data frame, we just have to mention this like that:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3loy02y5b7gr9pvnsb78.png)

For two conditions, you can use `|` and `&` with parenthesis:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cb0f37uqf4wpqvkygkw2.png)

**More Index Details**
Let's discuss some more features of indexing, including resetting the index or setting it to something else. We'll also talk about index hierarchy!

By using the `.split()` function, we can create a new column. It will create a list, and then we will create a new column using that:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nhuucz3ysxpxqx3yomom.png)

We can also set an existing column as an index by using the `set_index()` method and passing the name of the column through it:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hpt9an2wdhlkcu4aql5i.png)

Again, we have to use the `inplace` argument if we want to make the change permanent.

**Missing Data**

Let's show a few convenient methods to deal with missing data in pandas:

First, let’s create a new data frame using a dictionary:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5fai02mno4zj6m9ko2yz.png)

Here what we are doing is, we are intentionally putting some `nan` data to our data frame and assuming that those are missing data. 

If we want to remove those missing values from our data frame, there is a method called `.dropna()`. It will remove all the rows that contain at least one missing data point. We can use the `asix` argument if we check the missing data column-wise.


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mcs743zfrebwjamgbi4k.png)

We can use the `thresh` attribute if we need to keep some missing values by mentioning how many missing values we will consider:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lqrlrt434ha9daxuwjt5.png)

Here, `thresh=2` means - keep only the rows with at least 2 non-NA values.

If we want to fill our missing values with something else, we have to write our code like that by using the `.fillna()` method:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lvdcbvfgwy95hkzfj7mv.png)

Sometimes we have to fill in our missing data using a mean value. For that we can code like that:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kzcqohjuzri4bgl5tq2b.png)

**Groupby**

Groupby allows you to group together rows based on a column and perform an aggregate function on them. Here aggregate function means a function that takes some data and returns may be the sum of those data or the mean value of those data.

Firstly, let us create a data frame using a dictionary:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/795j904np0tqngx3mghm.png)

Now you can use the `.groupby()` method to group rows together based off of a column name. For instance, let's group based on `company`. This will create a `DataFrameGroupBy` object:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0sq1pwm4nzxcilq6vokm.png)

You can save this object as a new variable:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lcwtorzme33gz5yyt0ru.png)

And then call aggregate methods on the object:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gry583mor2diry8l9odt.png)

Here, we are using the variable to call aggregate methods on the object. We can also do this directly like that:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vo6s7lw09i93aqik2i49.png)

**More examples of aggregate methods:**


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/61j6ykebiezhgxifmqsh.png)

You can also do things such as max and min.


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fgydnvwrbrfw80gy016x.png)


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1b1hogi1udpz7r2of6i7.png)

Some other useful aggregate functions that you may find yourself doing are things such as count which just counts the number of instances or column. In this case it was able to return the person column because it's able to count how many instances of a person occur in that column or company. So we have two people and they have two sales each and that's makes sense.


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w9tl0t745y2scucua769.png)

One last useful thing I want to show you with `groupby` is the `describe()` method and that gives you a bunch of useful information all at once.


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wyxydhal2j198hng2ngr.png)

And if you don't like this format, you can actually transpose this. So, you can say something like:


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mieh4zxravyey1qmq3qz.png)

So, whatever format you like better you can describe to that and then you can actually just call column names of this if you're just interested in a single column.


![Python Pandas](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8tpchskm104tew4g6i3w.png)