Quickly Installing And Running Neo4j Using Ansible On AWS Cloud
Tags: ansible, devops, aws, database
Published: 2019-10-28 10:45:13

![](https://i2.wp.com/blog.innomizetech.com/wp-content/uploads/2019/10/neo4j-aws-ansible.png?resize=800%2C300&ssl=1)

<!-- wp:paragraph -->
<p>Recently, we have worked on a project that uses Neo4j to store and process large graph data for our client. Our client has been asked for a solution to launch, install and configure a <strong>Neo4j single node</strong> (for the development environment and <strong>High Availability</strong> <strong>Neo4j cluster</strong> (for production environment).

<!-- wp:more -->
<!--more-->
<!-- /wp:more -->

<!-- wp:paragraph -->
<p>Before getting started, I just wanted to give you a note that there are a couple of options to deploy <strong>Neo4j</strong> on <strong>AWS</strong>, so you might take a look before looking for details or select the best option that works for you:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li><a rel="noreferrer noopener" href="https://aws.amazon.com/marketplace/pp/B071P26C9D?qid=1530283441386&amp;sr=0-1&amp;ref_=srh_res_product_title" target="_blank">Neo4j Community AWS Marketplace</a>, easy to launch instances and configure networking such as VPCs, storage.</li><li><a rel="noreferrer noopener" href="https://aws.amazon.com/marketplace/pp/B07D441G55?qid=1530283441386&amp;sr=0-2&amp;ref_=srh_res_product_title" target="_blank">Neo4j Enterprise Causal Cluster</a>, which deploys multi-node causal clusters.</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>This article provides a step by step guild on how to launch, install and configure high availability Neo4j cluster (aka HA cluster) using Ansible on AWS. We use <strong><a href="https://aws.amazon.com/">AWS</a></strong> for demonstration, but you are able to custom the playbooks and configurations for other cloud vendors such as <strong>Google Cloud</strong>, <strong>Azure</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Pre-requisites</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>In order to use this Ansible playbook on AWS, the following is needed:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>An <strong><a href="https://aws.amazon.com/">AWS</a></strong> account with a user's access key and secret key.</li><li>An IAM policy attached to the above user that allows launching new EC2 instances, authorize ports in security groups.</li><li>An EC2 Key-Pair to allow SSH to EC2 instances.</li><li><a href="https://git-scm.com/"></a><a href="https://git-scm.com/"></a>git installed on your machine</li></ul>
<!-- /wp:list -->

<!-- wp:heading {"level":3} -->
<h3>The steps</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>To deploy Neo4j, what are we going to build is the following deployment flow:</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>Setup security groups and authorize ports communication</li><li>Launch EC2 instance(s) - optional</li><li>Update OS</li><li>Install Neo4j Enterprise on EC2 instance(s)</li><li>Install <a href="http://www.haproxy.org/">HAProxy</a>  and configure HA cluster on EC2 instance(s) - only required for HA cluster</li></ol>
<!-- /wp:list -->

<!-- wp:heading {"level":3} -->
<h3>Getting started</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Before deploying, a security group needs to be created that the Neo4j cluster/instance will use. In fact, you can create multiple security groups for different purposes such as allow SSH to instance, allow Neo4j communication between each other. But to simplify the process, we will use one security groups that allow the following ports:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>22 (SSH)</li><li>Neo4j Ports listed on this <a href="https://neo4j.com/docs/operations-manual/current/configuration/ports/">page</a></li><li>8000 - HA admin port for HA cluster deployment</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Login to <a href="http://console.aws.amazon.com/">AWS Console Management</a> portal and create a security group and open the above ports like below screenshot:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":217,"align":"center"} -->
<div class="wp-block-image"><figure class="aligncenter"><img src="https://blog.innomizetech.com/wp-content/uploads/2019/10/Screen-Shot-2019-10-28-at-4.54.21-PM-1024x257.png" alt="Security group inbound ports" class="wp-image-217"/></figure></div>
<!-- /wp:image -->

<!-- wp:heading {"level":4} -->
<h4>Project structure</h4>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>A well-defined <em>project structure</em> will help us easy to understand each part of the solution, allow reuse, and customizable. If you have experience in working with Ansible, you should know how to organize the Ansible project. I followed the alternative approach mentioned in&nbsp;<a rel="noreferrer noopener" href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html#directory-layout" target="_blank">this</a>&nbsp;article, feel free to select your own approach.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":219,"align":"center"} -->
<div class="wp-block-image"><figure class="aligncenter"><img src="https://blog.innomizetech.com/wp-content/uploads/2019/10/Screen-Shot-2019-10-28-at-4.54.54-PM.png" alt="Ansible project structure" class="wp-image-219"/><figcaption>Ansible Project Structure</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>The above project structure contains the following:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li><em>extension/setup</em>: contain scripts to install <strong>Ansible</strong> and required python packages</li><li><em>inventories/[env]</em>: define all variables for playbooks that allow us custom for each environment</li><li><em>roles:</em> predefined and reusable roles for our playbooks. In this solution, we use the following roles:<ul><li><em>common</em>: the common role to install common package or update the latest OS version.</li><li><em>haproxy</em>: the role to install and configure HAProxy.</li><li><em>launch-ec2</em>: the role to launch EC2 instances in multiple AZ.</li><li><em>neo4j</em>: install and configure Neo4j on a single instance.</li></ul></li><li><em>templates: </em>template files for configuring Neo4j instances as well as HAProxy config files</li><li>There are two main playbooks:<ul><li><em>neo4j.single.yml:</em> the playbook to launch and install a single Neo4j node.</li><li>neo4j.cluster.yml: the playbook to launch and install an HA Neo4j cluster.</li></ul></li></ul>
<!-- /wp:list -->

<!-- wp:heading {"level":4} -->
<h4>Preparation</h4>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><strong>Step 1 - Clone/download source code from Github using GIT</strong></p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>clone https://github.com/innomizetech/neo4j-ansible.git</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>change directory into the newly created directory</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>cd neo4j-ansible</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p><strong>Step 2 - Install Ansible and required python packages</strong></p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>chmod +x extension/setup/setup.sh
./extension/setup/setup.sh</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p><strong>Step 3 - Decrypt <a href="https://docs.ansible.com/ansible/latest/user_guide/vault.html">Ansible Vault</a> file</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>A&nbsp;vault&nbsp;file&nbsp;contains&nbsp;sensitive&nbsp;information&nbsp;so&nbsp;that&nbsp;we&nbsp;shouldn't&nbsp;commit&nbsp;to&nbsp;source&nbsp;control&nbsp;in&nbsp;plaintext. So we need to encrypt it before committing to source control. Using&nbsp;<a href="https://docs.ansible.com/ansible/latest/user_guide/vault.html#decrypting-encrypted-files">Ansible&nbsp;vault</a>&nbsp;so&nbsp;this&nbsp;problem. In this repo, we committed the password file for demo purpose, <strong>please note that you should not commit the password file into Source control.</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Run below command to decrypt the<em> vault.yml</em> file in the inventory directory:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>ansible-vault decrypt inventories/dev/group_vars/vault.yml --vault-password-file ansible-vault.pass</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p><strong>Step 4 - Update vault.yml file</strong></p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>---
# Sensitive variables here are applicable to deploy application

aws_access_key: &lt;&lt;your access key>>
aws_secret_key: &lt;&lt;your secret access key>>

# The security group id to be attached to new instance
security_group: &lt;&lt;your security group id>>
# An Amazone Linux image
image: &lt;&lt;AMI id i.e. ami-048a01c78f7bae4aa>>
# The first subnet to launch instance, it should be public subnet if you allowed public access
vpc_az1_subnet_id: &lt;&lt;your subnet 1 id>>
# The second subnet to launch instance, it should be public subnet if you allowed public access
vpc_az2_subnet_id: &lt;&lt;your subnet 2 id>>

# Set initial password for Neo instances
initial_password: &lt;&lt;your password>>

# HAProxy configuration (requires for cluster mode with HAProxy)
stats_user: &lt;&lt;your HAProxy username>>
stats_pass: &lt;&lt;your HAProxy user password>></code></pre>
<!-- /wp:code -->

<!-- wp:heading {"level":4} -->
<h4>Deploy a single Neo4j Node</h4>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><strong>Step 1 - Update groups variables</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Review and update variables in the <em>inventory\dev\group_vars\neo4j-single.yml</em> file, below are some important variables:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li><em>region</em>: an AWS region to launch and deploy Neo4j</li><li><em>keypair</em>:  an existing key-pair on the above region</li><li><em>other variables</em>: feel free to update according to your requirement</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p><strong>Step 2- Run Ansible playbook</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Run below command to deploy a single Neo4j instance for <code>dev</code> environment. Replace <code>dev</code> to any existing inventory in the <code>inventories</code> directory (i.e. staging, prod)</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>ansible-playbook neo4j.single.yml -e env=dev --vault-password-file ansible-vault.pass

#  or we can use -b -K to enter SUDO password (sudo su)
ansible-playbook neo4j.single.yml -e env=dev --vault-password-file ansible-vault.pass -b -K</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Wait until the command finished and access to Neo4j browser at <a href="http://public-ip:7474"></a><a href="http://public-ip:7474">http://public-ip:7474</a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Deploy an HA Neo4j cluster </strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Execute the same steps above with</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li><em>neo4j-cluster.yml</em> group variables file</li><li><em>neo4j.single.yml</em> the Ansible playbook</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Check out the result of each case by watching those videos on our Youtube channel:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li><a href="https://youtu.be/wZccrA7ReU0">Deploy single Neo4j on AWS using Ansible</a></li><li><a href="https://youtu.be/MIZ6knCkGq8">Deploy HA Neo4j cluster on AWS Using Ansible</a></li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
If you have any issues when practicing this instruction, feel free to let us know by giving us our comments.</p>
<!-- /wp:paragraph -->

Visit our [blog](https://innomizetech.com/blog) for more interesting articles. If you have any questions or need help you can contact me via [Twitter](https://twitter.com/hoangleitvn).