Reading and Manipulating Your Dataset With Pandas (2)
Tags: python, datascience, analytics, beginners
Published: 2020-12-19 03:28:25

# Manipulation

Let's say you need to see only one column of your dataframe. To see the 'fixed acidity' column of our dataset, you need to write:

```
df['fixed acidity']
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/h72efcbi0ovivmd9w1pb.png)

If you add a condition to this column, for example, if you want to see the rows that has a fixed acidity higher than 9:

```
df[df['fixed acidity']>9]
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/mln4d8tu70y8obn77ja9.png)

Sometimes you might need rows with multiple conditions added to columns:
```
df[(df['fixed acidity']>9) & (df['citric acid']>0.5)]
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/mzhxr6ghuey36snfgexl.png)

If you need to find specific columns:
```
df.loc[:,['volatile acidity', 'chlorides']]
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/0g408hsf38b6ppfohdbt.png)

You may want to add conditions with them too, for example, you may want to see the 'volatile acidity' and 'chlorides' content of those rows that have a 'fixed acidity' of 9.2:

```
df.loc[df['fixed acidity'] == 9.2, ['fixed acidity','volatile acidity', 'chlorides']]
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/we2hmdmwzigx8n6ql9kt.png)

You can view the rows for specific indices (as discussed in the previous chapter) too, like this:

```
df.loc[0:3, ['volatile acidity', 'chlorides']]
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/1ah7gxs2gm4g748vc8ld.png)

Now, if you want to locate a specific value, for example, the alchohol content of the wine of 0th row:

```
df['alcohol'].loc[0]
```
and you will get a value of 9.4

You can find locate a row using its index too:
```
df.iloc[100]
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/m1pu3dx11d7bsua26vdf.png)

Now if you want to pinpoint a value within this, for example, the 1st attribute (volatile acidity in this case) of the 100th row, try:

```
df.iloc[100][1]
```
and you will get 0.61 as expected.

You can locate specific consecutive rows and columns using this iloc command, for example, first three columns of 3rd to 7th row:

```
df.iloc[3:8, 0:3]
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/8zyq9tazgavq2qs1jike.png)

and non consecutive rows and columns too:
```
df.iloc[[71, 122, 400], [0, 2]]
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/y25ihctt5qvbdn7z0mqw.png)

What if you want to add a new column to your dataframe? Let's add a 'new column' containing the word 'hi' for all rows:

```
df['new column'] = 'hi'
df.head()
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/7otyirlsqo39ho3fw07e.png)

Let's try changing the value of 'new column' of 0th index of the dataframe using iloc from 'hi' to 'bye':

```
df.iloc[0, df.columns.get_loc('new column')]= 'bye'
df.head()
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/e2109eya7v7z403wyv9i.png)

Now let's try to find the word starts with 'by' (that we just have added) and replace it with 'hello':

```
df['new column'].loc[df['new column'].str.startswith('by')] = 'hello'
df.head()
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/oymfdq8nfkquf5r4ocyj.png)

You can also replace null values of your data using pandas. We do not have any null values here, so let's introduce a null value first. Let's replace the string 'hello' with null. To do so, we would need the numpy library.

```
import numpy as np
df['new column'].loc[df['new column'].str.startswith('hel')] = np.nan
df.head()
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ja4y8gzk9235033fnh0v.png)

To check the number of null values, you can use the isna() method like this:

```
df.isna().sum()
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/2p0jbhii1ldzc49gz5av.png)

This isna() method can also be used to locate the null value like this:

```
pd.isna(df.head())
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/2n5w7vuutgq1t88ptam0.png)

Let's replace the null value with 'hey'.

```
df.fillna(value='hey', inplace=True)
df.head()
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/pvby0arnc5di35z2e8tq.png)

If you want to drop null values, use the dropna() method.

Now we will try to create a new dataframe using a loop, where one column of the new dataframe would look the same as the 'new column' of our dataframe df.

```
rows = []
for i in range(df.shape[0]):
     rows.append(['hi', 'bye'])
df_new = pd.DataFrame(rows, columns=["new column 2", "new column 3"])
df_new.iloc[0, df_new.columns.get_loc('new column 2')]= 'hey'
df_new.head()
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/7e66sspkpf59nk03q5qs.png)

You can merge these two dataframes using their common attributes:

```
df_merged = df.merge(df_new, left_on='new column', right_on='new column 2')
df_merged.head()
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/2r0p5rkk9rs355fwpcz0.png)

You can make necessary variations in your merging operations by dropping mismatched attributes, or by using a column with common name and so on.

You can also group your dataframes:

```
df.groupby(['volatile acidity', 'chlorides']).count().head()
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/3i3j3nlxyq2is6vrtxcn.png)

You can also group the dataframes using other attributes like sum. 

When you are done with manipulation of your dataframes, you are ready to visualize your data.