Exploratory Data Analysis – A Key Step in Machine Learning
Published: 2020-05-20 12:36:06

*This blog is part of MSP Developer Stories initiative by Microsoft Students Partner (India) Program - https://studentpartners.microsoft.com/ which is aimed for student communities to Learn, Lead and Empower.*

The goal of this post is to emphasize the role of **Exploratory Data Analysis** while solving business problems with Machine Learning and Artificial Intelligence with a detailed case study walkthrough.

**A 360° data mindset** In this information-driven age, a 360° view has to be taken for the extraordinary volume of data that is being available – historic, current and predictive – so that right data has to be extracted to make better business decisions.

**Exploratory Data Analysis (EDA)** is an observational approach to understand the characteristics of the data. EDA is essential for a well-defined and structured data science project and it should be performed before any machine learning modelling phase. This helps in Identifying patterns and develop hypotheses.

**Case Study** : A medium size bikes & cycling accessories manufacturing consultancy is keen on growing the business. We’ll help them analyze their customer and transaction data to optimize marketing strategy.

**Preliminary Data Exploration – Identify ways to improve the quality of data**

_**Environment and Code Readiness**_
* Create a Jupyter Notebook hosted on Azure
* Import pandas package to read and write excel data
* Import matplotlib & seaborn for data visualization
* Upload the Customer data into the Azure Notebook path

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ackfm0mj4ul46ifqsx0x.jpg)

_**Let’s put the below analysis into various data quality dimensions in a table**_

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/9jmvytolhh6mysv99bp3.jpg)

_**Identify Missing Values**_

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/76eelziivzn9krpsqtkl.jpg)

_**Column can be dropped if no relevance**_

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/8qcp6b3cw5rcvs8xpblb.jpg)

_**Gender data to be consistent, should be either Male or Female**_

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/fkchubhr2yw14immn63y.jpg)

_**Check for validity of Transactions data : product first sold date data type float to be converted into date time format**_

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/kjdaossz5jhy6xtcmg62.jpg)

Follow the above code and output for other data sets

Here is the Data **Quality Analysis Summary**

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/id5cxspnuzdgwhmi36xu.jpg)

**Data Exploration, Model Development and Interpretation** :  Understanding the data distributions, feature engineering, data transformations, modelling, results interpretation and reporting.

**Customer Age & Gender Distribution** : Female category is more than Male; New customers are recommended between 30 to 60 years old

_Calculate the age of the customers from date of birth for plotting the graph_

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/up10twhgv1o2aojw4tnb.jpg)
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/fuq2zv1ngam4g3ktmksz.jpg)

**Number of Mass Customers under the Wealth Segment are the highest**

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/zvgxq8bepra3s1pkmbon.jpg)

**New customers are from Manufacturing & Finance industry**

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ss70jbuo0n92782dm7zg.jpg)

**Customer cars owned data**

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/tj4nyd8g0xsz4oa7ahem.jpg)

**Visualizations & Interactive Dashboard** : Help us highlight key findings and convey the ideas in a more succinct manner. Below dashboards have been built in Power BI desktop. _Walkthrough of the building of dashboards in Power BI is out of scope for this blog._

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/he01k9lj0e4mlgma1iba.jpg)
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/g94g89fguiyj7039eqzf.jpg)

Conclusion, **Exploratory Data Analysis** is a key process in Machine Learning / Data Science projects. The main pillars of EDA are data cleaning, data preparation, data exploration, and data visualization.