Pandas Basics
Tags: pandas, beginners, python
Published: 2022-02-08 17:49:38

Pandas are used hand in hand with NumPy in data science. In this article, we will explore pandas and form a solid understanding of the library.

#Introduction
Pandas is a an open-source Python library used to read, write, manipulate and analyze data.
Pandas integrates well with other data visualization libraries.
To use pandas, you have to import it using:

`import pandas as pd`
pd is the conventional alias for pandas.

Pandas has two main data structures; Data frames and Series.

Series is a one-dimensional array that can contain elements of different data types. A series is similar to a list in python but is displayed as a column in a table.

Dataframes are two dimensional table structured with labeled axes. It is a collection of series.

##Creating a Series 
You can create a series using pd.Series() and passing a list or dictionary as an argument.

1.Using a list
```python
import pandas as pd
my_list = [1,4,5,8,9,3]
series_A = pd.Series(my_list)
print(series_A)
```
Output:

```
0    1
1    4
2    5
3    8
4    9
5    3
dtype: int64
```
2.Using a dictionary
The key, value pair in the dictionary become the index and value in the series.
```python
my_dict ={'one':'Jane','two':'Tom','three':'Kamau'} 
series_dict = pd.Series(my_dict)
print(series_dict)
```
Output:
```
one       Jane
two        Tom
three    Kamau
dtype: object
```
##Creating DataFrames

1.Use a list or NumPy array. You can specify column and row indexes.

```python
patient_details = [[101,'Julia','Johns'],
[102,'Jessica','Watkins'],
[103,'Amanda','Elis']]
patient_dataframe = pd.DataFrame(patient_details,columns=['ID','FirstName','LastName'],index=[1,2,3])
print(patient_dataframe)
```
Output:

```
  ID FirstName LastName
1  101     Julia    Johns
2  102   Jessica  Watkins
3  103    Amanda     Elis
```
2.Using a dictionary

```python
data = {'Name':['Kris', 'Kate', 'Gao', 'Anita'],
        'Age':[27, 24, 22, 32],
        'Major':['Statistics', 'Accounting', 'Economics', 'Telecoms']}
df = pd.DataFrame(data)
print(df)
```
Output:

```
Name  Age       Major
0   Kris   27  Statistics
1   Kate   24  Accounting
2    Gao   22   Economics
3  Anita   32    Telecoms
```
The dataframe consists of columns with different datatypes
`python df.dtypes
`
Output:

```
Name     object
Age       int64
Major    object
dtype: object
```
#Reading and writing CSV files
We often work with already existing data which can be used to create data frames. Most data exist as CSV files, therefore it is important to understand how to read and write CSV files.

The `read_csv()` function in pandas is used for reading CSV files stored locally or from a URL.

```python
df2 = pd.read_csv("https://raw.githubusercontent.com/dphi-official/Datasets/master/Standard_Metropolitan_Areas_Data-data.csv")
df2
```
Output:

```
land_area	percent_city	percent_senior	physicians	hospital_beds	graduates	work_force	income	region	crime_rate
0	1384	78.1	12.3	25627	69678	50.1	4083.9	72100	1	75.55
1	3719	43.9	9.4	13326	43292	53.9	3305.9	54542	2	56.03
2	3553	37.4	10.7	9724	33731	50.6	2066.3	33216	1	41.32
3	3916	29.9	8.8	6402	24167	52.2	1966.7	32906	2	67.38
4	2480	31.5	10.5	8502	16751	66.1	1514.5	26573	4	80.19
...	...	...	...	...	...	...	...	...	...	...
94	1511	38.7	10.7	348	1093	50.4	127.2	1452	4	70.66
95	1543	39.6	8.1	159	481	30.3	80.6	769	3	36.36
96	1011	37.8	10.5	264	964	70.7	93.2	1337	3	60.16
97	813	13.4	10.9	371	4355	58.0	97.0	1589	1	36.33
```

The `to_csv()` function converts a data frame into a CSV file.

syntax: `df.to_csv('filename.csv')`

You can also specify index=False to import csv file without the index.

`df.to_csv('filename.csv', index=False)`

#Attributes and Methods
1.The **_shape attribute_** shows the shape of the dataframe; the number of rows and columns.

Syntax : `df.shape`

2._**dtype attribute:**_ shows the data types in the columns.

Syntax : `df.dtypes`
3._**axes:**_ Returns a list with the row axis labels and column axis labels.

Syntax : `df.axes`

4._**empty:**_ Returns True if dataFrame is entirely empty.

Syntax : `df.empty`

5._**ndim :**_ Returns the number of dimensions of the underlying data, by definition 1.
Syntax : `df.ndim`

6._**size:**_ Returns the number of elements in the dataFrame. Product of the rows and columns.
Syntax : `df.size`


To display the first five observations from the dataframe use the _**head()**_ method.

`df2.head()`

Output:
```
	land_area	percent_city	percent_senior	physicians	hospital_beds	graduates	work_force	income	region	crime_rate
0	1384	78.1	12.3	25627	69678	50.1	4083.9	72100	1	75.55
1	3719	43.9	9.4	13326	43292	53.9	3305.9	54542	2	56.03
2	3553	37.4	10.7	9724	33731	50.6	2066.3	33216	1	41.32
3	3916	29.9	8.8	6402	24167	52.2	1966.7	32906	2	67.38
4	2480	31.5	10.5	8502	16751	66.1	1514.5	26573	4	80.19
```

To display the last five observations from the dataframe use the **tail()** method. 

You can also specify the number of rows to be displayed by passing an argument to the tail() and head() methods.

`df2.tail()`
Output:

```
	land_area	percent_city	percent_senior	physicians	hospital_beds	graduates	work_force	income	region	crime_rate
94	1511	38.7	10.7	348	1093	50.4	127.2	1452	4	70.66
95	1543	39.6	8.1	159	481	30.3	80.6	769	3	36.36
96	1011	37.8	10.5	264	964	70.7	93.2	1337	3	60.16
97	813	13.4	10.9	371	4355	58.0	97.0	1589	1	36.33
98	654	28.8	3.9	140	1296	55.1	66.9	1148	3	68.76
```

The info() method returns a summary of the dataframe.

`df2.info()
`
Output:

```
#   Column          Non-Null Count  Dtype  
---  ------          --------------  -----  
 0   land_area       99 non-null     int64  
 1   percent_city    99 non-null     float64
 2   percent_senior  99 non-null     float64
 3   physicians      99 non-null     int64  
 4   hospital_beds   99 non-null     int64  
 5   graduates       99 non-null     float64
 6   work_force      99 non-null     float64
 7   income          99 non-null     int64  
 8   region          99 non-null     int64  
 9   crime_rate      99 non-null     float64
dtypes: float64(5), int64(5)
```
The describe() method returns a summary of the numerical columns only. 

To get a summary of the categorical columns, we use the include parameter.
Syntax : `df.describe(include='object')`

You can also use include='all' to get a summary of all the columns.
Syntax : `df.describe(include='all')`

The mean() function returns the mean of each numerical column.
`df2.mean()`
Output:

```
land_area         2615.727273
percent_city        42.518182
percent_senior       9.781818
physicians        1828.333333
hospital_beds     6345.868687
graduates           54.463636
work_force         449.366667
income            6762.505051
region               2.494949
crime_rate          55.643030
dtype: float64
```
The median() function returns the median of each numerical column.
`df2.median()`

Output:

```
land_area         1951.00
percent_city        39.50
percent_senior       9.70
physicians         774.00
hospital_beds     3472.00
graduates           54.00
work_force         257.20
income            3510.00
region               3.00
crime_rate          56.06
dtype: float64
```

The value_counts() function returns the number of unique entries in the data.
`df2.value_counts()
`Output:

```
land_area  percent_city  percent_senior  physicians  hospital_beds  graduates  work_force  income  region  crime_rate
47         41.9          11.9            745         3352           36.3       258.9       3915    1       51.70         1
2966       26.9          10.3            2053        6604           56.3       450.4       6966    1       56.55         1
2766       67.9          7.7             679         3873           56.3       224.0       2598    3       63.22         1
2737       45.0          10.5            602         1462           71.3       131.4       1980    4       63.44         1
2710       63.7          6.2             357         1277           72.8       110.9       1639    4       63.10         1
                                                                                                                        ..
1490       33.1          11.9            827         3818           47.4       300.2       4144    1       30.59         1
1489       58.8          9.5             911         5720           56.5       175.1       2264    3       70.55         1
1465       30.3          6.8             598         6456           50.6       164.7       2201    3       70.66         1
1456       46.7          10.4            2484        8555           56.8       710.4       10104   2       44.64         1
27293      25.3          12.3            2018        6323           57.4       510.6       7399    4       76.03         1
Length: 99, dtype: int64
```
The unique() function returns all unique categories in a column.
`df.Age.unique()
`Output
`array([27, 24, 22, 32])`

To drop columns or rows use df.drop().
Drop values from rows (axis=0)
Drop values from columns(axis=1)

#Indexing

The loc() and iloc() methods are used for indexing. 

loc() is label based selection and iloc() is index based selection.

`df.iloc[1]
`
Output:

```
Name           Kate
Age              24
Major    Accounting
Name: 1, dtype: object
```
You can specify the row and column to access specific elements. 

You can also use negative indexing.
`df.iloc[1,2]
`
Output:
`'Accounting'
`
You can also use slicing to access a range of items.
`df.iloc[:3,1]
`
Output:

```
0    27
1    24
2    22
Name: Age, dtype: int64
```
Using loc() you can specify the columns 
`df.loc[0:2,['Name','Age']]
`
Output:

```
	Name	Age
0	Kris	27
1	Kate	24
2	Gao	22
```
#Selecting and Assigning data

##Attribute based selection
You can use dot selection to select a column using the following syntax: `df.columnName` 
`df.Name`
Output:

```
0     Kris
1     Kate
2      Gao
3    Anita
Name: Name, dtype: object
```
You can also use the bracket based selection to select a column or multiple columns. 
When selecting multiple columns we use double square brackets.
df[['Age','Major']]

Output:

```
	Age	Major
0	27	Statistics
1	24	Accounting
2	22	Economics
3	32	Telecoms
```

##Conditional Selection.
To select a row or rows that satisfy a certain condition, we can use conditional operators such as ==.
`df.Age == 27
`
Output:

```
0     True
1    False
2    False
3    False
Name: Age, dtype: bool
```
Assigning Data
To assign data to a given column, or row select and provide the new data to be replaced.

```python
df.Name= 'Kamau'
df
```
Output:

```
	Name	Age	Major
0	Kamau	27	Statistics
1	Kamau	24	Accounting
2	Kamau	22	Economics
3	Kamau	32	Telecoms
```