Pandas-Data Manipulation and Analysis
Published: 2023-01-21 18:51:40


** Introduction to Pandas**

Pandas is basically a Python library that is used for analyzing, cleaning, exploring, and manipulating data. Pandas allow us to analyze big data and make conclusions based on statistical theories. Pandas can clean messy data sets, and make them readable and relevant. Relevant data is very important in data science.


**Installation and importing Pandas.**

First of all, open your jupyter notebook, then go to your command panel and write “pip install Pandas”. Here we go, you are done with the installation of the Pandas library. Now, to use the Pandas library you need to import Pandas in your notebook, to do so write “import Pandas as PD”. You are ready to use Pandas Library.
Finally, check if the installation was successful. 

import pandas as pd
print(pd.__version__)

**Getting Started with Pandas**

## Series

A series can actually hold pretty much almost all types of data
objects with python as its data point. Series also can take lots of parameters (data, index, dtype, name, copy, fastpath). To create a pandas series we just need a list initially it will generate  random index we can also manipulate index by sending a list of index with the index parameter.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z3ug2huxue9ignsfnsnj.png)
Just look at the picture above. we have used an integer list and series automatically generated index but in the 2nd and 3rd series we have customized the index as we wanted that's the beauty of Pandas.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eikvusia5xmd31w0icki.png)
We can also use a dictionary and Pandas series will use its key as index .


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mkoyya4x73i8wf6yve3f.png)
Use the index option to specify only the words you wish to be included in the Series, leaving out the rest of the words in the dictionary. Two series can added.

**Data Frames**
Data Frame is known as the main tool of Pandas. It takes multiple parameters like data, column, rows, dtype, copy. To start with Data Frames we have some necessary setting in our Jypyter. As we don't have any specific data so, we will use random data to learn Data Frames.

```
import numpy as np
import pandas as pd
from numpy.random import randn
np.random.seed(101)
df=pd.DataFrame(randn(5,4),["A","B","C","D","E"],["X","Y","Z","P"])
df
```
output

```

       X	       Y	       Z	      P
A	2.706850	0.628133	0.907969	0.503826
B	0.651118	-0.319318	-0.848077	0.605965
C	-2.018168	0.740122	0.528813	-0.589001
D	0.188695	-0.758872	-0.933237	0.955057
E	0.190794	1.978757	2.605967	0.683509
```
Data Frames is just bunches of series. we can access those series by following command.

```
 **df["X"]
output
	X	       Y
A	2.706850	0.628133
B	0.651118	-0.319318
C	-2.018168	0.740122
D	0.188695	-0.758872
E	0.190794	1.978757
```
more than one series is a data frames

```
**df[["X","Y"]]
        X	      Y
A	2.706850	0.628133
B	0.651118	-0.319318
C	-2.018168	0.740122
D	0.188695	-0.758872
E	0.190794	1.978757
```
we can also add or drop any column.
To add any column.

```
df["updated"]=df["X"]+df["Y"]
df["updated"]

A    3.334983
B    0.331800
C   -1.278046
D   -0.570177
E    2.169552
```
it will add the number column x and y and create a column which will be added as updated.
To drop any column.

```
df.drop("updated",axis=1,inplace=True)
df
         X	        Y	       Z		p
A	2.706850	0.628133	0.907969	0.503826	
B	0.651118	-0.319318	-0.848077	0.605965	
C	-2.018168	0.740122	0.528813        -1.278046
D	0.188695	-0.758872	-0.933237	0.955057	
E	0.190794	1.978757	2.605967	0.683509	
```
as we have set inplace as true so the updated column will drop permanently.
we can also add or drop any rows or columns by using this command.
`df.drop("D")`
we just have to put the row or column name and it will be droped automatically.
we can check dimension of a data frames by
`df.shape`
 

## Conditional selection in Data Frames

```
df>0
output
      X	    Y	    Z	     P	    updated
A	True	True	True	True	True
B	True	False	False	True	True
C	False	True	True	False	False
D	True	False	False	True	False
E	True	True	True	True	True
```
we can also specify only false value

```
booldf=df>0
df[booldf]
output
       X	        Y	        Z	         P	
A	2.706850	0.628133	0.907969	0.503826	
B	0.651118	NaN	         NaN	        0.605965
C	NaN	        0.740122	0.528813	NaN
D	0.188695	NaN	         NaN	       0.955057	
E	0.190794	1.978757	2.60596 	2.169552
```
we can also use specific rows or column or both

```
df["X"]>0
```
starting  with multiple conditions

```
df[(df["X"]>0) | (df["Y"]>1)]
```
** skipping index reset it's bit harder to explain index reset in that blog.

## Pipe Index Oparetor
  index level

```
a="G1 G1 G1 G2 G2 G2".split()
outside=a
inside=[1,2,3,1,2,3]
higher_index=list(zip(outside,inside))
higher_index=pd.MultiIndex.from_tuples(higher_index)
```
now we can use that pipe index to built a multi level index

```
df=pd.DataFrame(randn(6,2),higher_index,["A","B"])

```
to get sub data frames from a multilevel index

```
df.loc["G1"]
df.loc["G1"].loc[1]
```

## Missing Data

```
d={"A":[1,2,np.nan],"B":[1,np.nan,np.nan],"C":[1,2,3]}
df=pd.DataFrame(d)
output
       A	B	C
0	1.0	1.0	1
1	2.0	NaN	2
2	NaN	NaN	3
```
to drop any row that hold missing data

```
df.dropna()
it will drop that rows that has one or more NaN value 
df.dropna(axis=1)
it will drop that coloum that has one or more NaN value  
```
to fill any missing data

```
df.fillna(value="FIll  VALUE")
```
to chnage specific value

```
df["A"].fillna(value=3)
```
mean function

```
df["A"].fillna(value=df["A"].mean())
```

## GROUP BY
all functions of group by


```
data={"Company":["GOOG","GOOG","ghgh","fdff"],"Persoon":["Mridul","Mohosin","Sabid","Raihan"],"Sales":[10,12,34,45]}
df=pd.DataFrame(data)
df
bycomp=df.groupby("Company")
to find minimun valuee
bycomp.mean()
```

total sell of a company

```
bycomp.sum()
```
standered division

```
bycomp.std()
```
to see sells of a specific company

```
bycomp.sum().loc["GOOG"]
```

```
df.groupby("Company").sum().loc["GOOG"]
df.groupby("Company").count()
df.groupby("Company").max()
df.groupby("Company").min()
df.groupby("Company").describe()
df.groupby("Company").describe().transpose()
df.groupby("Company").describe().transpose()["GOOG"]
```

## merging,joining,concatenating


```
dic={
    'col1':[1,2,3,4,5],
    "col2":[45,5,77,88,5],
    "col3":["abc","def","ghi","jkl","mno"],
}
df=pd.DataFrame(dic)
df.head()
```
#to find unique values in a specific coloum

```
df["col2"].unique()
```
#it will return a arrays with unique values
#we can aslo check the lenght of that array by len function

```
len(df["col2"].unique())

```
# nunique is an alternatives of len function 

```
df["col2"].nunique()

```

unique value count method

```
df["col2"].value_counts()
```
filtering data using conditional statement

```
df[df["col1"]>2]
```
pandas can use  customizer function

### Thank you