Exploratory Data Analysis: Part C
Tags: machinelearning, datascience, python, beginners
Published: 2020-07-09 13:16:38

In this article, we will delve into various aspects of plotting and analyzing numerical and categorical variables in bivariate and a multivariate manner.

### Correlation

Exploratory data analysis in many modeling projects (whether in data science or in research) involves examining correlation among predictors and between predictors and a target variable.

*Variables X and Y (each with measured data) are said to be positively correlated if high values of X go with high values of Y, and low values of X go with low values of Y.* 

We use *Pearson's Correlation Coefficient* as a de-facto method for computing correlation among numerical variables.

Following is the mathematical formula of the same:

![Correlation](https://dev-to-uploads.s3.amazonaws.com/i/m4wuiox5krlgzg1gfx0c.png)

The correlation coefficient always lies between +1 (perfect positive correlation) and –1 (perfect negative correlation); 0 indicates no correlation.

```python
#Reading the data
data = pd.read_csv('winequality-red.csv', sep = ';')
# pandas dataframe has .corr() method to compute a correlation table
data.iloc[:, :-1].corr()
```
The output of the following code is a square matrix with each numerical variable's correlation computed against every other in the data.

![CorrTable](https://dev-to-uploads.s3.amazonaws.com/i/hhlevlf1ohj0ebdvd919.png)

For visualization purposes, we use seaborn's *heatmap* for better inferences and data storytelling.

```python
plt.figure(figsize= (10,7))
sns.heatmap(data.iloc[:, :-1].corr(), vmin= -1, vmax= 1, cmap= sns.diverging_palette(20, 220, as_cmap=True))
```


![heatmap](https://dev-to-uploads.s3.amazonaws.com/i/lmmro9723p1zlqasvdyu.png)

Like the mean and standard deviation, the correlation coefficient is sensitive to outliers in the data.

Note: There are various other correlation coefficients devised by statisticians: *Spearman’s rho* or *Kendall’s tau*. 
These are correlation coefficients based on the rank of the data. Since they work with ranks rather than values, these estimates are robust to outliers and can handle certain types of nonlinearities.

#### Scatterplots

The standard way to visualize the relationship between two measured data variables is with a scatterplot. The x-axis represents one variable and the y-axis another, and each point on the graph is a record.

```python
ax = data.plot.scatter(x = 'citric acid', y = 'fixed acidity', figsize = (10,6))
ax.axhline(0, color='grey', lw=1)
ax.axvline(0, color='grey', lw=1)
ax.grid()
```

![scatterplot](https://dev-to-uploads.s3.amazonaws.com/i/vzvo5vs3yhmxa9g5b2gm.png)

The plot shows a fairly positive relation between *Citric Acid* and *Fixed Acidity*, where we can conclude safely that increase of citric acid results in a corresponding increase of acidity levels. 

### Exploring Two or More Variables

Familiar estimators like mean and variance look at variables one at a time (univariate analysis).
In this section, we look at additional estimates and plots, and at more than two variables (multivariate analysis).

#### Hexagonal Binning and Contours

Scatterplots are fine when there is a relatively small number of data values.
For data sets with hundreds of thousands or millions of records, a scatterplot will be too dense, so we need a different way to visualize the relationship.
Rather than plotting points, which would appear as a monolithic dark cloud, we grouped the records into hexagonal bins and plotted the hexagons with a color indicating the number of records in that bin. 

In Python, hexagonal binning plots are readily available using the pandas data frame method *hexbin*

```python
ax = data.plot.hexbin(x = 'citric acid', y = 'fixed acidity', gridsize= 40, figsize = (10,6))
ax.set_xlabel('Citric Acid')
ax.set_ylabel('Fixed Acidity')
```

![Hexplot](https://dev-to-uploads.s3.amazonaws.com/i/f70t0acy6ezq0foux5xh.png)

Another method to analyze dense data is to plot the density contours. In Python, *seaborn* has the method *kdeplot* 

```python
plt.figure(figsize=(10,10))
sns.kdeplot(data['citric acid'], data['fixed acidity'])
```

![contourplot](https://dev-to-uploads.s3.amazonaws.com/i/cwv5k0vot11n4zwex5jq.png)

#### Two Categorical Variables

A useful way to summarize two categorical variables is a contingency table - *a table of counts by category.*
Contingency tables can look only at counts, or they can also include column and total percentages. 

The pivot_table method creates the pivot table in Python. The aggfunc argument allows us to get the counts.

```python
crosstab = adult_data.pivot_table(index = 'education', columns= 'sex', aggfunc= lambda x: len(x) , margins= True)

df = crosstab.copy()

df.loc[:, 'workclass']
```

![ContingencyTable](https://dev-to-uploads.s3.amazonaws.com/i/cu6wwtbka6ih3z9wybrf.png)

As we can observe that, for two categorical variables : *education* and *sex*, we have computed a contingency table. 
From the table, we have used 'workclass' column for the output. 

#### Categorical and Numerical Variables

*Boxplots* are a simple way to visually compare the distributions of a numeric variable grouped according to a categorical variable.

The *pandas* boxplot method takes the by an argument that splits the data set into groups and creates the individual boxplots.

```python
ax = adult_data.boxplot(by = 'race', column = 'hours-per-week', figsize=(10,10))

```

![boxpair](https://dev-to-uploads.s3.amazonaws.com/i/bxzmwqcafjhb08sbr500.png)

From the above visualization, we can observe that, we have grouped the data by the categorical variable *race* and have plotted against *hours-per-work*

A **violin plot** is an enhancement to the boxplot and plots the density estimate with the density on the y-axis. 
The density is mirrored and flipped over, and the resulting shape is filled in, creating an image resembling a violin. 
*The advantage of a violin plot is that it can show nuances in the distribution that aren’t perceptible in a boxplot*

```python
plt.figure(figsize=(10,5))
sns.violinplot(x= adult_data['race'], y= adult_data['hours-per-week'], inner = 'quartile')
```

![violenplot](https://dev-to-uploads.s3.amazonaws.com/i/x3gceax0ms847dr80grj.png)

We created a violin plot with similar features as the aforementioned boxplot.


#### Closing Remarks

*Exploratory data analysis (EDA)* set a foundation for the field of data science. The key idea of EDA is that the first and most important step in any project based on data is to look at the data. By summarizing and visualizing the data, you can gain valuable intuition and understanding of the project.

Fin.