Simplify Data Prep with AWS Glue DataBrew
Tags: aws, machinelearning, datascience
Published: 2021-02-01 02:59:17

Business users always format, transform and summarize data to derive insights. Even with data being ETL to Data Warehousing systems; users had to apply additional transformations in BI tools. 

As modern enterprises are increasingly adopting Lake House architectures, there is an even greater need to simplify this process for data prep on large data sets that can scale.

##### AWS Glue DataBrew
> AWS Glue DataBrew is a new visual data preparation tool that makes it easy for data analysts and data scientists to clean and normalize data to prepare it for analytics and machine learning.

It allows you to:
* Profile 
* Clean and Transform
* Track Lineage
* Orchestrate and Automate

In the remaining sections, I will demonstrate these capabilities using a simple customer segmentation dataset - [Link] (https://www.kaggle.com/vjchoudhary7/customer-segmentation-tutorial-in-python)

##### Data Profiling
First thing we do on a dataset is to profile - either with code or here with a few clicks in Data Brew. 

To begin, select the dataset and "*Run data profile*" button.  
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/fgh5eiceqza4ftnnqhgf.JPG)

This would take us to "*Create job*" page where we create a "*Profile Job*" and this will generate summary statistics on our data.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/wk596s0ts2yip0c01h1m.JPG)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/qjr5r8y9v94pzue61nj5.JPG)

Once Profile job is complete, it provides a profile overview, detailed column statistics and lineage.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/48sdclnnh7ab36c6k8hz.JPG)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/7wo5puckxrts9phk3nj6.JPG)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/qao65qg9b1wu3xs8ydhw.JPG)

##### Projects
Next, we will transform the data by creating a project on the dataset. We can create Project from the Datasets view by specifying Project Name and Recipe details. For recipe we can either create a new one or use from existing list.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/zh1j0tf5r7hu7rttp5nb.jpg)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/8h6akfu1arrx9zlo1ua0.JPG)

While creating recipes, transforms are done with sample data. For this Project, we will select 5000 random rows. 

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/aumlrw1regkfyucb3fn7.JPG)

##### Project Landing Page
Default view of this page shows us sample rows, distribution and recipe steps.It also shows the available list of transformations that can be done on the dataset.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/yc7juu5x4adbwswnvafz.JPG)

Schema View-
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/pmj8mw1v26vmmttbflnt.JPG)

Transformations steps can be added and together makes a recipe.
> DataBrew provides from over 250 built-in transformations to visualize, clean, and normalize your data with an interactive, point-and-click visual interface.

For this dataset, we will create a recipe with two transformations:
1. Filter Customers with age greater than equal to 30
2. Encode Gender to 'M' or 'F'

##### 1. Filter Customers with age greater than 30

From Filter category > By Condition > Greater than or equal to:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/sy4t9na19gt3kgele0kb.JPG)

Select the source column and the filter value
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/d0yzxyiag7jl9f2h98wh.JPG)

Preview shows the result of the step and helps verify the outcome before applying it
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/lqhjwwgwqdpw4yuz18yd.JPG)

##### 2. Encode Gender to 'M' or 'F'
From the clean category > select Replace value or Pattern

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ehrr362n394qjc3vg3s9.JPG)

Select source column and the value to be replaced
**Note** - We need two steps here, one for each value to be replaced.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ocj2xnl7picvybdf6bpk.JPG)

Preview changes and apply
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/q1n4kawbrkfwi5gilgdk.JPG)

Then, transform the next value
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/mcurevr2ds282vsmyira.JPG)

As we are done with all the steps, we are good to publish our recipe:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/all88tr2smaw4e7pfm8f.JPG)

##### Jobs
We had earlier created a Profile Job. We will now create a recipe job that will transform the entire dataset.

We will specify the Dataset, Recipe and Output details.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/n6m7qvffekbt71ygpumk.JPG)

Once job is created, we get the transformed dataset, job run and history details.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/0a5q0o73tvxgy7sn9asl.JPG)

##### Data Lineage
Lineage shows the various components and their relationships to the output.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/9cateshcne5kq6iliag7.JPG)

##### Summary
Data Brew provides a powerful abstraction for data preparation. As explained, it fastens and simplifies the Data Prep process allowing Users to focus on the insights and decisions that drives their business.
 .