How To Automate Your Deployments To AWS EC2 Using CircleCI And Ansible
Tags: cicd, devops, ansible, cloud
Published: 2023-10-21 21:41:13

### Introduction
In today's world of fast paced releases and distributed teams working across flexible hours, the importance of automated deployments cannot be overstated. In this tutorial I will guide you through the steps to configure  CircleCi and Ansible for automated deployments to AWS EC2.

### Prerequisite
To get the most out of this tutorial and follow along it will be nice to have:

* Basic programming experience and be comfortable with the command line.
* A Github account and access to command line with git installed.
* An AWS account with EC2 server set up.
* A CircleCI account.
* A code editor.

This tutorial will focus on configuration and deployment only, we won't be setting up new servers. I will be using a backend server I wrote a while ago. You can find the starter code [here](https://github.com/AbuchiKings/aws_ec2_auto_deploy/tree/starter). Fork and clone the repository. Switch to the `starter` branch to follow along.

### CircleCI Set-up
Head over to [circleci.com](https://circleci.com) and login. Ensure you are within your personal organization. On the "Projects" page search for `aws_ec2_auto_deploy` repo and just click "Set Up Project". Select "Use the `. circleci/config.` yml in my repo" option and set up project. Set up your project environment variables by copying the default values in `.env.example` file over to `Project Settings > Environment Variables` on CircleCI project dashboard. Now, let's get down to brass tacks.

#### Writing `config.yml`
On your editor, open the `config.yml` file inside `.circleci` folder.

CircleCI uses YAML files to configure pipelines which is usually made up of the following blocks of YAML codes:

| YAML Block | Description |
| ---------- | ----------- |
| `version`    | This specifies the version of CircleCI API you wish to use |
| `commands`   | Contains list of reusable commands |
| `orbs`      | Makes it possible to use certain prewritten functionalities in a job |
| `jobs`      | List of jobs |
| `workflow`      | Defines the order to run jobs |

Before deploying your code to the servers, it is good practice to always ensure that all tests are passing. Update your `config.yml` file with the code below.

```
version: 2.1

jobs:
  test:
    docker:
      - image: cimg/node:16.17.0
      - image: redis
        name: redis
      - image: mysql:8.0
        environment:
          MYSQL_ROOT_PASSWORD: 12345
          MYSQL_DATABASE: lend
          MYSQL_USER: abuchikings
          MYSQL_PASSWORD: 12345
    steps:
      - checkout
      - restore_cache:
          keys: [packages]
      - run:
          name: Test MYSQL connection
          command: |
            for i in `seq 1 30`;
            do
              nc -z 127.0.0.1 3306 && echo Success && exit 0
              echo -n .
              sleep 1
            done
            echo Failed waiting for MySQL && exit 1

      - run:
          name: Run tests
          command: |

            npm install
            npm run migrate:latest
            npm run test

      - save_cache:
          paths:
            - "./node_modules"
          key: packages

workflows:
  default:
    jobs:
      - test
```

Here is a break down of what is happening in the code above:

- The `jobs` keyword indicates that the code block is going to contain jobs.
- The very first job is named `test` and the `docker` keyword tells CircleCI that this job is going to be executed within a docker environment.
- `cimg/node` is CircleCI's official nodejs image on docker hub. Node version 16.17.0 is used for this project. MySQL 8.0 database and redis are required to test the application.
- `steps`: As its name suggests defines the steps required to run a specific job. 
- `checkout`: This checks out the source code from your version control repository.
- `restore_cache`: Restores cached dependencies. Caching is a mechanism that allows you to save and reuse dependencies (e.g., npm packages) to speed up the build process. In this case, it restores cached npm packages.
- `run`: This step runs a series of commands. It's responsible for running the actual CI tasks.
- `name`: This provides a descriptive name for this step, indicating that the following commands are related to running tests. 
- `command`: Specifies the commands to be executed. The `|` symbol is used to define a multi-line command block, allowing the execution of multiple commands within this step. The first command utilizes `nc` (netcat) utility to check if a MySQL database server running on the localhost (127.0.0.1) and listening on port 3306 is available. The second command runs migration and automated tests on the code. 
- `save_cache`: This is a built-in CircleCI command used to save cached dependencies or artefacts. It ensures that the specified files or directories are cached for future builds.
- `workflows`: This section defines one or more workflows. A workflow is a series of jobs that need to be executed in a particular order or under specific conditions. 

Now you can commit your changes and push them to remote repository. Head over to CircleCI dashboard to see your pipeline run. 

Next update your `config.yml` with the `setup_ansible` job below. Remember to to also update your workflow as shown in the code snippet below.

```
  setup_ansible:
    docker:
      - image: amazon/aws-cli
    steps:
      - checkout
      - run:
          name: Install dependencies
          command: |
            yum install -y tar gzip
      - run:
          name: Add Backend IP To Ansible
          command: |
            backend_ip=$(aws ec2 describe-instances \
            --query "Reservations[*].Instances[*].PublicIpAddress" \
            --filters "Name=tag:Name,Values=ec2_auto_deploy_test" \
            --output text)
            echo "$backend_ip" >> ~/project/.circleci/ansible/inventory.txt
            cat ~/project/.circleci/ansible/inventory.txt
      - persist_to_workspace:
          root: ~/
          paths:
            - project/.circleci/ansible/inventory.txt

workflows:
  default:
    jobs:
      - test
      - setup_ansible:
          requires: [test]

```
**Remember to change `ec2_auto_deploy_test` to the name of your EC2 server on AWS.**

The `setup_ansible` job uses aws-cli to get the IP address of an EC2 server with name tag `ec2_auto_deploy_test`. The IP address is saved to the `inventory.txt` file within the Ansible folder and persisted to the workspace for use by the next job. Utilities `tar` and `gzip` are required to persist file to workspace. 

Before pushing your changes to github, head over to the project page on CircleCI => Project Settings => Environment Variables and add the following environment variables:

- `AWS_SECRET_ACCESS_KEY`
- `AWS_DEFAULT_REGION`
- `AWS_ACCESS_KEY_ID`

which can be obtained from your aws account.

The final job configuration on the `config.yml` file will be responsible for executing an Ansible playbook that uses ssh to access the server, pulls code from github and builds an updated docker image. See code below. 

```
  deploy_server:
    docker:
      - image: python:3.11.6-alpine3.18
    steps:
      - checkout
      - add_ssh_keys:
          fingerprints: ["9f:66:26:48:fd:251:62:5d:73:c4:d5:90:2b"]
      - attach_workspace:
          at: ~/
      - run:
          name: Install dependencies
          command: |
            apk add --update ansible openssh-client
            eval $(ssh-agent -s)
      - run:
          name: Deploy Server
          command: |
            cd ~/project/.circleci/ansible
            echo "Contents  of the inventory.txt -------"
            cat inventory.txt
            ansible-playbook -i inventory.txt deploy-server.yml
```
On your CircleCI project settings page, go to `SSH Keys`, add a new ssh key by copying and pasting the ssh key for your EC2 server. This will generate a key fingerprint. Copy the SSH key fingerprint and replace `fingerprints` on the `deploy_server` job.

Also update your workflow with the following code:

```
workflows:
  default:
    jobs:
      - test
      - setup_ansible:
          requires: [test]
      - deploy_server:
          requires: [setup_ansible]

```


### Ansible Set-up
Ansible is a powerful open-source automation tool and configuration management system that streamlines and simplifies the automation of routine IT tasks and the management of complex systems. It provides a framework for IT professionals to define and execute automation tasks through human-readable playbooks. 

With Ansible, you can automate the provisioning, configuration, deployment, and maintenance of servers, applications, and various infrastructure components. You should check out [Ansible docs](https://docs.ansible.com/) for more on this interesting automation tool.


#### Writing A Playbook
To automate Ansible, we use simple YAML files called playbooks to define automation tasks. The following shows the tree structure for the `Ansible` folder which contains the playbook to be executed. 

```
├── ansible
│   ├── ansible.cfg
│   ├── deploy-server.yml
│   ├── inventory.txt
│   └── roles
│       └── deploy
│           └── tasks
│               └── main.yml
└── config.yml
```

Here is a breakdown of each file and its function:
- `ansible.cfg`: A configuration file used to customize the behaviour of Ansible. This allows you to override built-in configuration settings like SSH connection options, default inventory file, define remote user privileges and so on.
- `deploy-server.yml`: This is an Ansible playbook. It defines a set of tasks or a task to be executed on remote hosts. In this project, I used roles to define playbook tasks.
- `inventory.txt`: This serves as a critical component in Ansible for defining the hosts and groups of hosts that Ansible will manage. It is a configuration file that specifies the remote servers or systems where Ansible tasks and playbooks will be executed. In this file, IP addresses of the servers or machines to be managed by Ansible are listed.
- `roles`: Roles are a way to organize and structure tasks, handlers, variables, and other content for better code reuse and maintainability. This contains the tasks to be executed in playbooks. These tasks are defined within the `main.yml` file.

Update `deploy-server.yml` with the following code:

```
---
- name: "Deploy Play"
  hosts: web
  user: ubuntu
  gather_facts: false
  vars:
    - ansible_python_interpreter: /usr/bin/python3
    - ansible_host_key_checking: false
    - ansible_stdout_callback: yaml
  roles:
    - deploy
```

- `name`: This is a user-defined name for this playbook that makes it easy for us to understand its function.
- `hosts`: This specifies the target hosts or hosts group on which the tasks in this playbook will be executed. In this case it is set to web. When you open the `inventory.txt` file, it contains a single line, `[web]`. This defines a host group called `web` and all IP Addresses listed under `[web]` are the hosts or remote machines we wish to manage with Ansible. To add an IP address to the inventory we are using `aws-cli` in the `setup_ansible` job to pull the IP Address of our EC2 server and dynamically update the inventory file. 
- `user`: This specifies the remote user that Ansible should use when connecting to the target hosts. In this case it is `ubuntu`. You may need to change that if your ssh user is different.
`gather_facts`: This tells Ansible whether to collect facts about the target hosts, such as hardware details, network configuration, and more. Here it is set to false to improve playbook execution speed.
- `vars`: This defines variables to be used in the playbook. `ansible_python_interpreter` specifies the python intepreter Ansible should use when running jobs on the host. `ansible_host_key_checking` is set to false to disable host key checking. Host key checking is a prompt that comes up on your command line when you are executing ssh login to a server with a key for the first time. This is a security measure to ensure the remote host identity.
- `roles`: This specifies which roles should be included in this job. In this case, it's a role named `deploy`. When the playbook is executed, Ansible will look for a directory named "deploy" within the `roles` path and execute the tasks and configurations defined in that role. Let's write the tasks for the deploy role.

Copy the code below to `main.yml` within the `deploy/tasks` role folder.

```
---
- name: Perform Git Pull
  become: false
  git:
    repo: git@github.com:AbuchiKings/aws_ec2_auto_deploy.git # Replace with github repo you prefer
    dest: ~/apps/aws_ec2_auto_deploy
    accept_hostkey: yes
    clone: false
    version: main # Replace with the desired branch or tag

- name: Run Docker build
  become: true
  shell: 
    chdir: apps/aws_ec2_auto_deploy
    cmd: make "{{ item }}"
  loop: 
      - down
      - up-d
```

The first task uses Ansible's `git` module to perform a git pull from the project's git repository. 
- `become` is a boolean field that tells Ansible to perform a task as a root user.
- `dest` is the destination or directory where the fetched items should be added. 
- To prevent a fresh cloning of the repo, `clone` is set to false. 
- `version` defines the branch to pull from.

In the second task, the `make` command is used on two different arguments defined inside a `Makefile` in the project root folder:
- `down` which maps to `docker-compose -f docker-compose.yml -f docker-compose.prod.yml down` takes down the current running container.
- `up-d` which maps to `docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build` starts up the container with a new build in a detached mode. 

Finally, commit your changes and push to remote repository.
 
### Conclusion
This is by no means means an exhaustive tutorial on CircleCi and Ansible. There are a wide range of automation tasks you could perform with Ansible. Nonetheless, I hope you find this helpful. Thank you for your time.