How to check for quality? Evaluate data with AWS Glue Data Quality
Tags: aws, etl, analytics, tutorial
Published: 2023-07-07 09:47:56

## Data is the new oil

The Women in Data Science (WiDS) Conference 2017 [trailer](https://www.youtube.com/watch?v=fU8qgpFOCaI) from Stanford University aimed to inspire the audience in the field of data science.

In the trailer, the panelist mentioned the power of data.

> Who gets to control data, controls the society that's the reality. -- Bell Wei (Carolyn Guilding Chair in Engineering and Innovative Learning, San Jose State University)

Having access to the right data and understanding your data will empower you to make data-driven decisions for your organization. **Data understanding** is Step 2 of the Data Analytics and Data Science workflow called Cross-industry standard process for data mining (i.e. CRISP-DM). 

You may read more about CRISP-DM [here](https://www.datascience-pm.com/crisp-dm-2/). The second step:


> Data understanding – What data do we need or is available? Is it clean? 

![data understandinf](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rn9ywbowuno6bqxsew10.png)

This is the data science workflow using CRISP-DM explaining all the stages and the image is from [Wikipedia](https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining):

![cr dm](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/82t7rnrxoc47m1s1eopj.png)

### Learning Objectives

In this lesson you will learn the following:

* How to create rules with DQDL
* What is AWS Glue?
* What is AWS Glue Data Quality?
* Solution Architecture
* How to check for data quality results?

### Who owns data quality?

Is data quality the responsibility of the data engineer building your ETL data pipelines? Is it the responsibility of the data analyst building the data modelling? Or the data scientist building hypotheses and machine learning models responsible for data quality?

Take a look at this Twitter post on data quality [here](https://twitter.com/BM_DataDowntime/status/1666141241434841088).

In fact we can all play part to ensure that with good quality data, we can improve our data analysis and machine learning model performance to make business decisions.

### What is AWS Glue?

AWS Glue is an AWS service that allows data engineers, business analysts, data analysts, developers and data scientists to integrate data from multiple sources and also perform ETL.

It is serverless [data integration service](https://aws.amazon.com/glue/) to allow you to easily scale your workloads in preparing data and moving transformed data into a target location. 

You do not need to provision any servers or clusters before using AWS Glue with a managed serverless experience.

You may bring in your own code or notebook to create an ETL job on demand, batch or streaming. You may create an AWS Glue data catalog to make data available for others to use.

You may also use AWS Glue Studio a visual editor to create your ETL pipeline.

### What is AWS Glue Data Quality?

AWS Glue Data Quality is a new product feature of AWS Glue that is now [generally available](https://aws.amazon.com/about-aws/whats-new/2023/06/aws-glue-data-quality-generally-available/) since 6 June 2023.

* AWS Glue Data Quality allows you to measure and monitor 'good or bad data' in your ETL pipelines before it enters your data lake or data warehouse to ensure high quality data is available to make data-driven decisions.

### What are the benefits?

In the [AWS Glue Developer Guide](https://docs.aws.amazon.com/glue/latest/dg/glue-data-quality.html) for AWS Glue Data Quality, the benefits include the following:

* Serverless – there is no installation, patching or maintenance.

* Get started quickly – AWS Glue Data Quality quickly analyzes your data and creates data quality rules for you.

* Improvise your rules to check the integrity and accuracy of the data

* Evaluate data quality and make confident business decisions 

* Zero in on bad data with errors

* Pay as you go - There are no fixed costs and you pay for your usage when you use AWS Glue Data Quality

* Data quality checks - you can implement checks in the AWS Glue data catalogue

* You may create rules to check the profile of your dataset.

* You may get started quickly

### Solution Architecture

Below is my own solution overview of the new product feature of AWS Glue Data Quality.



![overview](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4r4de7m6oq80dsa3x6mk.png)

### Dataset

Let's examine the Amazon Data Science Books Dataset from [Kaggle.com](https://www.kaggle.com/datasets/die9origephit/amazon-data-science-books)


![kaggle](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xym2nkb35y00nzs6p4qe.png)

### Pre-requisite

* You may read this [blog](https://aws.amazon.com/blogs/big-data/getting-started-with-aws-glue-data-quality-for-etl-pipelines/) to learn how to get started with AWS Glue Data Quality.

* You may read this blog [How to ETL with AWS Glue - Part 1](https://dev.to/aws-builders/how-to-etl-with-aws-glue-and-amazon-glue-studio-to-transform-your-data-328b).

* You may dive deeper with a practical example in [Part 2](https://dev.to/aws-builders/how-to-etl-with-aws-glue-and-amazon-glue-studio-to-transform-your-data-part-2-3ng5).

* You have an AWS Account, if you do not have an account you may learn how to create one [here](https://dev.to/aws-builders/getting-started-with-aws-a-sweet-journey-5cjj).

### Tutorial 1: Add the Evaluate Data Quality transform to the visual job in AWS Glue Data Studio

In this [tutorial](https://docs.aws.amazon.com/glue/latest/ug/tutorial-data-quality.html) you may refer to the instructions from the AWS Glue User Guide.

Step 1: Log into your AWS account as an [IAM Admin User](https://dev.to/aws-builders/creating-your-first-iam-admin-user-and-user-group-in-your-aws-account-machine-learning-part-1-3cne).

Step 2: Navigate to Amazon S3 and [create your bucket](https://dev.to/aws-builders/creating-your-first-amazon-s3-bucket-and-new-feature-release-2c03) in an AWS region of your preference and click **Create bucket**.

![bucket](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3ahkyxhc2b6rq1z9agv9.png)

Step 3: Upload the *Amazon data science books dataset* into your Amazon S3 bucket and click **Upload**.


![upload](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2tzss9rttnyatd5gh9w5.png)

![up data](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qkis39or0i9ju1c8h6g9.png)

Step 4: Navigate to AWS Glue dashboard.

![Glue](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bdc7sr5ay0it7no1ypfb.png)

Click **Create a job to evaluate data quality**

![Icreate](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dle58efzifdigmqcpde9.png)

Step 5: Click on **Visual** and select **Evaluate Data Quality**.

Step 6: Add the data quality node.

On the AWS Glue Studio console, choose **Visual with a source and target** from the Create job section. **Choose Create**.

![create](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oaok2anrraaqx81y7e6i.png)

Step 7: Choose a node on which to apply the data quality transformation. Currently there is no node selected.

![ no node](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ifiev12uhz5zh38578bg.png)

You may select the transform node (i.e. Transform - ApplyMapping).

![ApplyMapping](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tazyotnyju45nfmvochi.png)

Step 8: On the left-hand side click the blue plus sign and select from the drop-down menu **Evaluate Data Quality** towards the bottom section. You may name this job and be sure to save it e.g. GlueDataQuality_Tutorial. 

![Evaluate data quality](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qer4jowzqhf67ygz4es6.png)

Step 9: The selected *Evaluate Data Quality transform* node will be displayed in the visual editor.

On the right-hand side you may inspect if you would like to retain the current parent node or change it from the drop-down menu. (Note: The parent node is connected to the Evaluate Data Quality node).

![new node](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/smd9efb2ohy2plp49s1s.png)

Step 10: You may validate data quality rules across multiple datasets. The rules that support multiple datasets [include](https://docs.aws.amazon.com/glue/latest/ug/tutorial-data-quality.html):

* Referential Integrity
* DatasetMatch
* SchemaMatch
* RowCountMatch
* AggregateMatch

When you add multiple inputs to the Evaluate Data Quality Transform, you need to select your *primary* input.

Select your primary input which is the *Amazon data science books dataset* to validate data quality for. 

All other nodes or inputs are considered as references.

Use *Evaluate Data Quality transform* to identify specific records that failed data quality checks. 

(Note: New columns flagged as bad records are added to the primary dataset).

![transform right](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/73s5a5qv2q1u1mkflnwt.png)

Step 11:  In the *Data Source - S3 bucket* select the S3 bucket where the dataset or primary input is saved.


![s3 data source](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5tnc6an0r9velr2nbasp.png)


Step 12: Click on *Output schema tab* to modify any data types e.g. change price from string to integer.


![output schema](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l4jbop1z33utgy9o2h3m.png)

Click **Apply** to change the data types.

![changed data types](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l31y3mjhsomglulmbdwd.png)


Step 13: Select the *Data target - S3 bucket* and save the transformed dataset in this location.


![target](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qo2y52tbtlp3z3snsqay.png)

### Tutorial 2: Create rules using DQDL builder

Step 1: Preview the *Amazon data science books dataset* and let's create a rule using the DQDL rule builder to check for the completeness of data.

You may browse the available data quality rules from the *Rules type* tab which include the following:

* ColumnCount
* ColumnLength
* ColumnExists
* ColumnDataType
* ColumnValues
* ColumnNameMatchesPattern
* Completeness
* CustomSql
* DataFreshness
* DatasetMatch
* DistinctValuesCount
* Entrophy
* IsComplete
* IsPrimaryKey
* Sum
* Uniqueness
* ReferentialIntegrity
* Mean
* RowCount
* RowCountMatch
* StandardDeviation
* UniqueValueRatio
* SchemaMatch

I selected the data quality rule for **Completeness**, because I would like to check the percentage of missing data greater than 80% for the following variables:

* Price
* Price (that includes used books)
* Number of book reviews
* Average rating

Firstly preview the [primary data source](https://www.kaggle.com/datasets/die9origephit/amazon-data-science-books) to understand the data.

![preview data](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7l82llkwejkcxj9tkdmx.png)

The following rules were created in the DQDL Builder in the * Schema* tab.

```
Rules= [
    Completeness "avg_reviews"> 0.8, Completeness "n_reviews"> 0.8, Completeness "price" > 0.8, Completeness "price (including used books)" > 0.8
]    
```

![schema rule](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rs756ymzbjz9wfiehqes.png)

The Completeness rule will check for the specified columns if there is greater than 80% of non-null values present in the primary data source.

### Tutorial 3: Configure Data Quality Outputs

Step 1: After the data quality rules are created, you can select additional options to be included in the data quality results output.

I have selected two additional options:

* Action: publish actions to Cloudwatch
* Data Quality Results: to flag fail or pass results


![output options](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lxlwyrdz5goivxgythch.png)

Below is an image of the rule outcomes node.


![rule outcomes](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yumgk4xkv5lwhcg6mv3d.png)

Step 2: Under *Data Quality Transformation Output*, I also checked the box for *Original data* as this will also append additional columns to the primary dataset to indicate bad errors.

![original data](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7t2g5aafnpg3rri8da5h.png)

### Tutorial 4: Configure Data Quality actions

After a data quality rule is created, you may select actions for CloudWatch to publish metrics or stop jobs based on a [criteria](https://docs.aws.amazon.com/glue/latest/ug/tutorial-data-quality.html).

Actions in CloudWatch are also published to Amazon Eventbridge and can be used to create alert notifications.

* On ruleset failure

* Fail job after loading data to target 

* Fail job without loading to target data 

### Tutorial 5: View data quality results

Click **Save** and initiate the AWS Glue job by selecting **Run**.


![start job](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kp1h71xyet6mhx5gxkdk.png)

You may click **Run details** to inspect the progress of the Glue job.

![run detials](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mjrl8gtid3wuv27hmajv.png)

After the job has completed, select the **Data Quality New** tab to inspect the results.

You will be able to see that the data quality rules have passed successfully and you may click **Download results** as a csv file

![rules passed](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mjyz1j8mjzck6zxh06m4.png)


```
[
	{
		"ResultId": "dqresult-xxxxxxxxxxxxxxxxxxxxxxxxxx",
		"Score": 1,
		"RulesetName": "EvaluateDataQuality_nodexxxxxxxxxx",
		"EvaluationContext": "EvaluateDataQuality_nodexxxxxxxxx",
		"StartedOn": "2023-07-07T08:57:48.117Z",
		"CompletedOn": "2023-07-07T08:58:08.203Z",
		"JobName": "GlueDataQuality_tutorial",
		"JobRunId": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
		"RuleResults": [
			{
				"Name": "Rule_1",
				"Description": "Completeness \"avg_reviews\" > 0.8",
				"EvaluatedMetrics": {
					"Column.avg_reviews.Completeness": 1
				},
				"Result": "PASS"
			},
			{
				"Name": "Rule_2",
				"Description": "Completeness \"n_reviews\" > 0.8",
				"EvaluatedMetrics": {
					"Column.n_reviews.Completeness": 1
				},
				"Result": "PASS"
			},
			{
				"Name": "Rule_3",
				"Description": "Completeness \"price\" > 0.8",
				"EvaluatedMetrics": {
					"Column.price.Completeness": 1
				},
				"Result": "PASS"
			},
			{
				"Name": "Rule_4",
				"Description": "Completeness \"price (including used books)\" > 0.8",
				"EvaluatedMetrics": {
					"Column.price (including used books).Completeness": 1
				},
				"Result": "PASS"
			}
		]
	}
]
```
If you are running multiple data quality jobs, you may filter the data quality results by date and time.

If you navigate to the tab **Script** you will be able to see that AWS Glue Studio automatically created Python code for the  transformation steps that you could easily download for reusability.


![python](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3598x3zb7q318dz8marw.png)

### Conclusion

You have learnt how to set up data quality rules in AWS Glue Data Quality using the visual editor of AWS Glue Studio. You have explored how to create an ETL job and examine the data quality results for a pass or fail from rules that were created.

Until the next lesson happy learning! 😀

### AWS Glue Data Quality Quick Start videos on Youtube

If you would like to learn more you may watch the following videos

* [AWS Glue Data Quality Overview](https://www.youtube.com/watch?v=pgue7jZrVNU)

![quality](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wll0ggprz1wg97c0g2i2.png)

* [Measure and Monitor Data Quality of your Datasets in AWS Glue Data Catalog](https://www.youtube.com/watch?v=rK1NGQu4Sgg)

![Youtube](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oy9ff5z9hwdpxjbdk498.png)

* [Introducing AWS Glue Data Quality for ETL Pipelines](https://www.youtube.com/watch?v=m4OKjfgsZ00)

![glue data](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/meg1dck9h5vvkrafujhd.png)

### Reference

* [AWS Glue Data Quality](https://aws.amazon.com/about-aws/whats-new/2023/06/aws-glue-data-quality-generally-available/)

* [Getting started with AWS Glue Data Quality for ETL pipelines](https://aws.amazon.com/blogs/big-data/getting-started-with-aws-glue-data-quality-for-etl-pipelines/)

* [Data Quality Definition Language reference](https://docs.aws.amazon.com/glue/latest/dg/dqdl.html#dqdl-syntax)

* [AWS Glue User Guide](https://docs.aws.amazon.com/glue/latest/ug/gs-data-quality-chapter.html)

* [Top Announcements of AWS re:invent 2022](https://aws.amazon.com/blogs/aws/top-announcements-of-aws-reinvent-2022/)

* [Join the Preview AWS Glue Data Quality](https://aws.amazon.com/blogs/aws/join-the-preview-aws-glue-data-quality/)

### New announcements from AWS Glue

* [AWS Glue for Ray](https://aws.amazon.com/about-aws/whats-new/2023/06/aws-glue-ray-generally-available/)

* [AWS Glue Studio](https://aws.amazon.com/about-aws/whats-new/2023/06/aws-glue-studio-data-previews-glue-streaming-jobs/)

### Next Week: AWS Builders Online Series - 13 July 2023

You may [register](https://aws.amazon.com/events/builders-online-series/) to join AWS Builders Online Series and learn from AWS experts on architectural best practices.

![builders online](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ntsoeq2k4l8opkvhs1l2.png)