Build a graph database with Amazon Neptune Part 1a - AWS CloudFormation
Tags: beginners, tutorial, database, aws
Published: 2022-08-14 08:08:00

## How are we all connected?

You may have heard of the term *6 degrees of separation*, perhaps you are a 'friend of a friend' on Facebook, you catch a plane from Sydney to San Francisco every month, you purchase your favourite coffee from Starbucks every morning or you purchase a machine learning textbook from Amazon.com and your preferences are similar to other people in your demographic profile. 

This is where graph databases may be helpful because relationships cannot be stored in a traditional relational database or queried using structured query language (SQL).

### Learning Outcomes

In the AWS Management console, you will learn how to:

* Create an AWS account
* Attach IAM policies to IAM Admin user for CloudFormation and EC2
* Create a graph database using Amazon CloudFormation

### What is Amazon Neptune?

Amazon Neptune is a fully managed graph database. It can be very useful when you need to describe and display **relationships** within your [connected](https://aws.amazon.com/neptune/?c=db&sec=srv) datasets.

### What is a graph?

A graph consists of **nodes** (or vertices) which are like tables for SQL, they are connected by **edges** which represent relationships.

![flight](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qiuowre8o5a09lduaekh.jpg)

For example, in the case of flight paths, *nodes* are represented by planes and *edges* are arrival and departures cities such as London.

In another example, in the case of social networking, *nodes* are represented by people and *edges* determine if a friendship exists.

### Key Benefits:

* Infrastructure is fully managed by AWS

* You only pay for the queries that you use

* You do not need to provision hardware

* Can store billions of relationships

* Designed with greater than 99.99 percent availability

* Amazon Neptune supports open source query language such as  openCypher, Apache TinkerPop Gremlin and the World Wide Web Consortium’s (W3C) SPARQL 1.1


### What is Amazon Neptune used for ?

Amazon Neptune can be used to describe relationships such as:

* Social networking

* Map customer journeys for customer experience

* Fraud detection 

* Knowledge graphs used to fight global terrorism

* Recommendation engines

* Identity graphs to build a *Single View of Customer*

### Architecture

The diagram below is provided by Amazon Web Services outlining the solution overview for building a graph database using Amazon Neptune:


![Iarchitect](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e9q7oc4n76uiizin4eb8.jpg)

The workflow describes:

Step 1: Bulk loading of a csv file into Amazon Neptune

Step 2: Amazon Neptune stores the data which contains relationships

Step 3: Query the data 

### Solution overview

The image below is my high level architecture of the solution:

![IMy solution overview](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a8rfm4pj0ps6wxchkgf7.jpg)

### How do you get started?

If you are a new user, you have instant access to **AWS Free Tier** for **30 days** with 750 hours of  t3.medium or t4g.medium instance usage. 


### Prerequisites 

* Add permissions to an IAM Admin user to work with [Amazon Neptune DB cluster](https://docs.aws.amazon.com/neptune/latest/userguide/get-started-create-cluster.html).

* Create an Amazon EC2 Key Pair to use for launching a [Neptune cluster using AWS CloudFormation](https://docs.aws.amazon.com/neptune/latest/userguide/get-started-prereqs.html).

* Add IAM policies to grant [permissions](https://docs.aws.amazon.com/neptune/latest/userguide/get-started-prereqs.html) needed to use the AWS CloudFormation template.

* Use AWS CloudFormation stack to create an [Amazon Neptune DB cluster](https://docs.aws.amazon.com/neptune/latest/userguide/get-started-cfn-create.html).

### Tutorial 1: Create a service-linked role policy for Amazon Neptune

Create an IAM user with permissions to access Amazon Neptune by following intructions [here](https://docs.aws.amazon.com/neptune/latest/userguide/manage-console-iam-user.html).

Step 1: Create an AWS account and sign in with IAM Admin user. You may follow instructions [here](https://dev.to/aws-builders/getting-started-with-aws-a-sweet-journey-5cjj) to create an account if you do not have one.

Step 2: Login with your IAM Admin user credentials.


![Iam](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/enwyobji370lcj3juh37.jpg)

Step 3: Download and save a copy of the [Amazon Neptune User Guide](https://docs.aws.amazon.com/neptune/index.html) on your computer or follow [here](https://docs.aws.amazon.com/neptune/latest/userguide/intro.html).


![user guide](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/11ipsag0zpaudzsfcv17.jpg)

Step 4: Sign to the IAM console, type in the navigation bar 'IAM' or this resource might be saved in your bookmarks.


![IAM](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rgg8psyl8ay8q7gycz5f.jpg)

Step 5: Click **Policies** on the left-hand side menu.


![polkicies](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dx3p8utcpn41zx71t5bp.jpg)

Step 6: Click the blue button **Create policy**.


![create policy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dxzhc5tv2453epxw1h40.jpg)

Step 7: Copy this code and paste it into the **json** tab


![Icode](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gfzgp6wfnl71ogqjrf9w.jpg)


![json](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ydcrgcuzhobxlhflzuye.jpg)

Step 8: Under **Tags** retain the default settings and click **Next review**.


![next reiew](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o95r1uteeobefm7f1v9o.jpg)

Step 9: Under **Review policy**, provide a name for the role *NeptuneServiceLinked* and click **Create policy**.


![service linked name](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/feqga2qjj2bghm9oj559.jpg)

Step 10: The service linked role was successfully created.


![role](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cx28uv8n6w3375zy3v1w.jpg)

### Tutorial 2: Create a new IAM user with all necessary permissions for access to Amazon Neptune

Create an IAM user with permissions to access Amazon Neptune by following intructions [here](https://docs.aws.amazon.com/neptune/latest/userguide/manage-console-iam-user.html).

Step 1: Repeat Steps 1-4 from Tutorial 1 and click **Users** on the left-hand menu from the IAM console.


![usres](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/artrxxt1zpowert05egi.jpg)

Step 2: Click **Add users**.


![Add users](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/004and7p4s6o5fxytv1l.jpg)


Step 3: Provide a name for the user, click the check box for 'Access Key: Programmatic access' and finally click **Next:Permissions**.


![neptune user](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/txmuhhvsifxe2328ehci.jpg)

Step 4: Under **Set permissions**,  click on the box **Attach policies directly** and copy and under the search bar type the word 'Neptune' and check the boxes to provide policy access to:

* NeptuneConsoleFullAccess
* NeptuneFullAccess
* NeptuneReadOnlyAccess


![filter policies](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y9zs7x60u8g0va9mb5k8.jpg)

Step 5: On the same page type 'VPC' in the search bar and check the box to provide access to:

*AmazonVPCFullAccess

![VPC](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uxuxmmxn6o4kfugbium4.jpg)

Click **Next:tags**.

Step 6: You do not need to include any tags. Click **Next:review**


![tag](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dxt89q4qwuqaoaktld98.jpg)

Step 7: Review the access details. Click **Create user**


![create user](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wttsh6zi7lqce0354n2t.jpg)

Step 8:  Click **Download as a csv** file to save the new user's access key ID and secret access key for programmatic access to interact with AWS using the CLI.


![download as csv](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lbm9krhg684g64zkvly1.jpg)

### Tutorial 3: Create an Amazon EC2 Key Pair for Windows instance to launch a Neptune DB cluster using AWS CloudFormation template

You may follow the instructions [here](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/create-key-pairs.html)to create An Amazon EC2 Key Pair name for Windows instances with pem. 


Step 1: Type 'EC2' into the search bar to navigate to the EC2 console.


![ec2 console](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u62ypxmqb7scwgyklabr.jpg)
 
Step 2: On the left-hand menu **Network & Security** click **Key Pairs**


![Ikeypairs](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1ce96eyvphupw8zhmg6n.jpg)


Step 3: Click **create key pair**


![create kp](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pgckb9c3am2ku6v9swqh.jpg)


Step 4: Provide a descriptive name under **Name**. For Windows, select 'Key Pair Type' **RSA** and thirdly select 'Private key file format' as **.pem** for use with openSSH. Click **Create key pair**.



![enter kp](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h3s4mouhldybu651yc0e.jpg)

Step 5: Key pair is successfully created. Download and save the private key file details.



![success](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rigjmkrxmb80f7wxhvk9.jpg)

### Tutorial 4: Add IAM policies to grant permissions to use the AWS CloudFormation template

Follow the instructions from [AWS documentation](https://docs.aws.amazon.com/neptune/latest/userguide/get-started-prereqs.html) to grant permissions to use AWS CloudFormation.

Step 1: With the newly created **user** from **Tutorial 2**, attach a managed policy 'AWSCloudFormationReadOnlyAccess' to provide access to AWS CloudFormation.

Click on **Users** on the left-hand menu and select **Add permissions**.


![add permissions](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/spd1wh990u6npndnvhwv.jpg)

Step 2: Under **Grant Permissions**, type *AWSCloudFormationReadOnlyAccess* to filter the policy, check the box and click **Next:review**


![cloud formationn](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qyy4uqjkyztdobv6uxga.jpg)

Step 3: Click **Add permissions**

![Add permission](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yaj8p26svelyvex937s2.jpg)

Step 4: You may view the 5 policies applied to the User.


![Ipolicies](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a6hf4jcei5on5dzwewa1.jpg)

Step 5: Navigate to **Policies** on the left-hand menu and click the button **Create policy**.


![policies](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jj4qlbbp0gmj0mskxzzi.jpg)

Step 6: Under *Create policy*, click on the **json** tab and copy the code from [here](https://docs.aws.amazon.com/neptune/latest/userguide/get-started-prereqs.html) and insert into the json. Click **Next:tags**.

![copy code](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lt6anoweac92lfpr81yl.jpg)


![json tab code](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ldyv886l8fxfmgpgoxge.jpg)

Step 7: Retain the default settings for tags and click **Next:review**.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/255jqxdj366bymj2yh57.jpg)

Step 8: Under *Review policy*, provide a name for the role and click **Create Policy**.


![create policy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ijfubnsgtvpc1ydrh8e8.jpg)

Step 9: Click *Users* on the left-hand menu and click **Add permissions**. Click **Attach policies directly** and **Next:review**.


![neptune rolen](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sx3e4pojqnwqg6bzsbtr.jpg)

Step 10: Click **Add permissions**


![add p](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gic9tq1cs24ywg0xxinr.jpg)

Step 11: You can view the policies attached to the IAM Admin user that provides access to AWS CloudFormation to access Amazon Neptune.


![7 policies](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r7kvpibmdemzalu3htpq.jpg)

### Tutorial 5: Creating an Amazon Neptune DB cluster using AWS CloudFormation template

AWS CloudFormation allows you to manage a [collection](https://aws.amazon.com/cloudformation/features/?pg=ln&sec=hs) of AWS services with a template to assist in provisioning and managing services with infrastructure as code.

An AWS CloudFormation stack will be used to create an Amazon Neptune DB cluster to describe AWS resources and other dependencies to launch and configure a [stack](https://aws.amazon.com/cloudformation/features/?pg=ln&sec=hs).


Step 1: From the Amazon Neptune User Guide, use Amazon CloudFormation template to create an Amazon Neptune DB Cluster by clicking **Launch Stack** for your AWS region from the link [here](https://docs.aws.amazon.com/neptune/latest/userguide/get-started-cfn-create.html)


![launch stack](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g2bff28udn6ya2gr2136.jpg)

For example, I selected the launch stack for *Asia-Pacific(Sydney)*.


![Isydney](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5dbgby37ywr8wchpu72k.jpg)

Step 2: From the AWS CloudFormation console, under *Template is ready* click **Next**.


![next](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jy2lqn4myplni38vbjbs.jpg)

Step 3: On the *Specify Stack details* page, enter the key pair name in the box **EC2SSHKeyPairName**. Click **Next**.


![enter EC2 kp](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7y1gy0wc4arcdm2a18jp.jpg)


Step 4: On this page of **Options**,click **Next**.


![optionsn](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hh0bqgay255v6scnlxdd.jpg)

Step 5: On the page **Review Neptune QuickStart**,  check the two boxes shown in the image to acknowledge that IAM resources will be created by AWS CloudFormation. Click **Create Stack**.

 

![create stack](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9uo8axre4mrl74wq7mpj.jpg)

Step 6: AWS CloudFormation is initiating the stack.


![creating](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0u41tjjpyyw04z1u84z8.jpg)

The stack creation process was approximately 25 minutes to create an Amazon Neptune DB cluster.

![stack progress](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3ydtelezpgenx4mkca82.jpg)

### Final Thoughts

AWS CloudFormation is particularly useful providing:

* Template snippets
* Template by AWS region
* Specific to an AWS Service e.g. Amazon Redshift
* Sample solutions e.g. integration with Microsoft Sharepoint and Active Directory.


![region](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/isrewo0p2o8hx8hazt17.jpg)

###  AWS re:Invent 2020: Deep dive on Amazon Neptune with Brad Bebee, GM of Amazon Neptune

{% embed https://www.youtube.com/watch?v=_j419Lo7Q7k %}

### AWS re:Invent 2020: New capabilities to build graph apps quickly with Amazon Neptune

{% embed https://www.youtube.com/watch?v=v0NAbeHj63c&t=401s %}

### References

* [Simplify Your Infrastructure Management Using AWS CloudFormation](Simplify Your Infrastructure Management Using AWS CloudFormation)

* [Amazon Neptune Developer Resources](https://aws.amazon.com/neptune/developer-resources/)

* [ETL Process for Transforming and Loading data into Amazon Neptune](https://github.com/aws-samples/amazon-neptune-samples/blob/master/gremlin/etl-from-relational-model/README.md)

* [Query results in Amazon Neptune workbench](https://aws.amazon.com/blogs/database/visualize-query-results-using-the-amazon-neptune-workbench/)

* [Amazon Neptune User Guide](https://docs.aws.amazon.com/neptune/latest/userguide/graph-get-started.html)

* [Getting started with Amazon Neptune](https://aws.amazon.com/neptune/?c=db&sec=srv)

* [Introducing Amazon Neptune Global Database](https://aws.amazon.com/blogs/database/introducing-amazon-neptune-global-database/)


### Additional Resources

* [Launch an EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/get-set-up-for-amazon-ec2.html)

* [Build a friend recommendation engine](https://aws.amazon.com/getting-started/hands-on/recommendation-engine-for-games-amazon-neptune/?trk=gs_card)

* [Amazon Neptune features](https://docs.aws.amazon.com/neptune/latest/userguide/feature-overview.html)

* [AWS Graph Notebooks](https://github.com/aws/graph-notebook)

* [About Amazon Neptune](https://aws.amazon.com/neptune/)

* [Learning Path for Amazon Neptune](https://pages.awscloud.com/AWS-Learning-Path-Getting-Started-with-Amazon-Neptune_2020_LP_0009-DAT.html)

* [Keep up to date withe the latest news from AWS Database Blogs for Amazon Neptune](https://aws.amazon.com/blogs/database/category/database/amazon-neptune/)

* [Amazon Neptune simplifies graph analytics and machine learning workflows with Python integration](https://aws.amazon.com/about-aws/whats-new/2022/06/amazon-neptune-simplifies-graph-analytics-machine-learning-workflows-python-integration/)

* [Neptune tools](https://github.com/wendy-wong/amazon-neptune-tools)

* [AWS Graph notebooks](https://github.com/aws/graph-notebook/tree/main/src/graph_notebook/notebooks)

###  AWS Innovate Online Conference APJ - 23 & 25 August 2022


![AWS Innovate](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/st6r8cnixap7o4log7yw.jpg)

Register at this [link](https://aws.amazon.com/events/aws-innovate/apj/data/)


### 🚀 Hot off the press: Introducing Amazon Neptune Global Database 

From 27 July 2022, AWS Neptune Global Database can build graph relationships in multiple regions including US East (N. Virginia), US East (Ohio), US West (N. California), US West (Oregon), Europe (Ireland), Europe (London), and Asia Pacific (Tokyo) Regions.

You can read about this announcement from the AWS Database Blog written by author Navtanay Sinha [here](https://aws.amazon.com/blogs/database/introducing-amazon-neptune-global-database/).

### Register for Australia's biggest data engineering conference DataEngBytes:

* Melbourne: 27 September 2022
* Sydney: 29 September 2022

Register here: https://dataengconf.com.au/

![Data Eng Bytes](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qzzqs2s8tte2hzqrf6n8.jpg)

Until the next lesson, happy learning ! 😁

### Next Lesson

The Part 2, the next instalment will explore data transformation and loading data into Amazon Neptune.