Explore Customized Glue Parameters! Database, Table and Partition through CloudFormation
Tags: cloudformation, s3, glue, aws
Published: 2022-11-22 16:48:45

“ I have checked the documents of AWS to create a glue database, table and partition parameters easily in a multi account scenario without any serverless function . So I checked for various solutions and found out that it can be possible using cloudformation stack. Pricing of S3 bucket as per storage size and  glue costs cheaper.”

AWS Glue is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development. AWS Glue provides all of the capabilities needed for data integration so that you can start analyzing your data and putting it to use in minutes instead of months. AWS Glue provides both visual and code-based interfaces to make data integration easier. Users can easily find and access data using the AWS Glue Data Catalog.

In this post, you will get to know how to explore customized glue parameters! Database, table and partition through cloudformation. Here I have created a s3 bucket, glue through a cloudformation stack with customized parameters.

#Prerequisites
You’ll need an Amazon Simple Storage Service for this post. [Getting started with Amazon Simple Storage Service](https://aws.amazon.com/s3/getting-started/) provides instructions on how to create a bucket in simple storage service.

#Architecture Overview
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fk1opwtolpwcvzc2akb7.png)
The architecture diagram shows the overall deployment architecture with data flow, amazon s3, aws glue and cloudformation.

#Solution overview
The blog post consists of the following phases:
1. Create of Glue Database, Table and Partition through Cloudformation 
2. Output of Stack “AWS Glue”

I have a s3 bucket as below →
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aar85kyp3660m24ap672.png)

##Phase 1: Create of Glue Database, Table and Partition through Cloudformation
1. Open the cloudformation console and create a stack for the glue database and table. In the template file, various parameters are defined such as: database parameters, table(schema) parameters and partition parameters.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0up01rli83jl6znemk8f.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6zaoe73s85vk0u2v2m7y.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9p9qvpgg40p1o9hkg1pr.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c7ex1fgkd1mtpp609pig.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pbpfk4yg7710e6vfqsmo.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0cdkhg6utf1km4nhau8o.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5qph1z2lzl7g68fgiopn.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/16mpa4luf9tqeatzlarp.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3sobzn857yr02unvbuea.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rlqpwiq1lonr7wryn3lb.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/93y6bq6z6eh74oh5j5fw.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/okw107jqvs17e1ahmqog.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iuuwxu5llv4eyxhwly26.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q4ecs0upli6eoxo5d4xi.png)

##Phase 2: Output of Stack “AWS Glue” 
1. Open the glue console and verify the database, table and partition created.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1vao49uka4ts695l5ijo.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hez8lo0yaqr4jg8hciun.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2sixjs6l7g9zo9xbo12b.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/32gswcyybj3a8iszciq9.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x8gjo86husgbdc6m4pwc.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ybieiuhm3o69bos7wlt1.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b8on9xm8yb8gcs8izym6.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f2lutgj7b1z0qaqbwnik.png)

#Clean-up
Delete S3 Bucket and Cloudformation Stack.

#Pricing
I review the pricing and estimated cost of this example. 

Cost of Data Transfer = $0.0

Cost of Glue = $0.0

Cost of S3 = $0.1

Total Cost = $0.1

#Summary
In this post, I showed “how to explore customized glue parameters! Database, table and partition through cloudformation”.

For more details on AWS Glue, Checkout Get started AWS Glue, open the [AWS Glue console](https://ap-south-1.console.aws.amazon.com/glue/home?region=ap-south-1#). To learn more, read the [AWS Glue documentation](https://docs.aws.amazon.com/glue/?id=docs_gateway).

Thanks for reading!

Connect with me: [Linkedin](https://www.linkedin.com/in/gargee-bhatnagar-6b7223114)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/38sdi4nsp89fnui8h7dk.png)