CircleCI + AWS How to create CI/CD pipeline from scratch Part 1 - Build and push images
Tags: devops, aws
Published: 2020-07-25 09:35:34

## Prepare environment for containers

### Create an IAM user

Right after the creation of an AWS account, you are logging in under the root user. You have full access to every service and the billing management console. To secure interaction with AWS it is a good practice to create a new user inside the group that has only required permissions. 
    
A few words about managing permissions. There are two main ways to add them to users through [groups](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_groups.html) and [roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html). The main difference is in that groups are a collection of users with the same policies. Roles, in turn, can be used to delegate access not only to users but also to other services and applications, we will use both. Let's create them.

<figure class="image">
  <img src="https://dev-to-uploads.s3.amazonaws.com/i/u820ir2b0h6d17w0nt3h.png">
  <figcaption>Create group window</figcaption>
</figure>

On the second step select the next policies

* AmazonEC2ContainerRegistryFullAccess 
* AWSCodeDeployRoleForECS 
* AmazonEC2ContainerServiceFullAccess 
* AmazonECSTaskExecutionRolePolicy 

<figure class="image">
  <img src="https://dev-to-uploads.s3.amazonaws.com/i/pnsb3cbigw9y6aph2bzd.png">
  <figcaption>Group after creation</figcaption>
</figure>

Then create the role that we will give to ECS to deploy our containers to EC2 instances.

<figure class="image">
  <img src="https://dev-to-uploads.s3.amazonaws.com/i/xnhhkaoebhl469ay2rl3.png">
  <figcaption>Create role</figcaption>
</figure>

and on the second step select in policies `AmazonEC2ContainerServiceforEC2Role`

<figure class="image">
  <img src="https://dev-to-uploads.s3.amazonaws.com/i/bbh6q9gjcmjwnuyxcyhs.png">
  <figcaption>Result</figcaption>
</figure>

More about it is showed [there](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html)

And finally, let's add a new user and add to the previously created group:

<figure class="image">
  <img src="https://dev-to-uploads.s3.amazonaws.com/i/ayi69xqennnmfk0kw6a7.png">
  <figcaption>New user</figcaption>
</figure>

This user should have only programmatic access because you will use it only from the circleci. After it, generate access keys and save them, they will need you later.

### Create ECR

[ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html) a place where you will store containers from they will be deployed. Just go to the ECR and click on the "Create repository" button. You will see the window where you should select the name for the repository. Other 
settings use by default

<figure class="image">
  <img src="https://dev-to-uploads.s3.amazonaws.com/i/6kx7227c33esz22b3w28.png">
  <figcaption>ECR Create</figcaption>
</figure>

Great! You have a repository and all required credentials to build and push images. Time to automatize it.

<figure class="image">
  <img src="https://dev-to-uploads.s3.amazonaws.com/i/yvbzmmqqgfcw4ixfycs0.png">
  <figcaption>ECR After creation</figcaption>
</figure>

## Configuring Circle CI

The main idea is to run tests after each commit for all branches and deploy after changes in the development and master.

Before you start to configure the pipeline, you will need to prepare the application
following this fantastic [getting started](https://circleci.com/docs/2.0/getting-started/#section=getting-started) page.

### Tests

The most popular use case of the Circle CI that I've seen is running tests (not all developers trust to external
services to deploy applications). To run them you should define [a job](https://circleci.com/docs/2.0/jobs-steps/#jobs-overview)
and add it as a step to [a workflow](https://circleci.com/docs/2.0/workflows/). There is an example of `test` workflow
for the `simple_plug_server` application:

```yaml
version: 2.1
jobs:
  test:
    docker:
      - image: elixir:1.10
        environment:
          MIX_ENV: test
    working_directory: ~/repo
    steps:
      - checkout
      - run: mix local.hex --force
      - run: mix local.rebar --force
      - run: mix deps.get
      - run: mix deps.compile
      - run: mix test
      - store_test_results:
          path: _build/test/lib/simple_plug_server
workflows:
  version: 2
  test:
    jobs:
      - test
```

It has only the one workflow `test` with the one job `test`. This job has three parts: 
* docker - where is defined as a container inside which you will deploy test environment and run tests
* working_directory - the name of the folder where everything is happening
* steps - the set of commands where you download code, setup dependencies and finally run tests. 
You can also [cache dependencies](https://circleci.com/docs/2.0/caching/) on this step.

We can also improve the representing of the failed tests, for it you should add a JUnit formatter for test results
(for elixir it is the hex package [JUnitFormatter](https://github.com/victorolinasc/junit-formatter)) and specify
the directory containing subdirectories of JUnit XML or Cucumber JSON test metadata files. 
More information about it and how to add support for other languages and test frameworks look [here](https://circleci.com/docs/2.0/collect-test-data/).


### Build and push containers

On the previous steps we created the ECR repository and the user that can push images, time to setup CircleCI config.

For work with images we will use the official orb for ECR [circleci/aws-ecr@6.9.1](https://circleci.com/orbs/registry/orb/circleci/aws-ecr)
It significantly simplifies building and pushing images, let's add the new step to our config file:

```yaml
version: 2.1
orbs:
  aws-ecr: circleci/aws-ecr@6.9.1
  aws-ecs: circleci/aws-ecs@1.2.0
jobs:
  test:
    docker:
      - image: elixir:1.10
        environment:
          MIX_ENV: test
    working_directory: ~/repo
    steps:
      - checkout
      - run: mix local.hex --force
      - run: mix local.rebar --force
      - run: mix deps.get
      - run: mix deps.compile
      - run: mix test
      - store_test_results:
          path: _build/test/lib/simple_plug_server
workflows:
  version: 2
  test-and-build:
    jobs:
      - test
      - aws-ecr/build-and-push-image:
          repo: "simple_plug_server"
          tag: "${CIRCLE_BRANCH}_${CIRCLE_SHA1},${CIRCLE_BRANCH}_latest"
          requires:
            - test
          filters:
            branches:
              only:
                - master
                - development
```

Briefly about the steps of this job:
* repo - the name of the repository (last part of the `815991645042.dkr.ecr.us-west-2.amazonaws.com/simple_plug_server`)
* tag - tags that we apply to the built container, for the master branch it will add two tags: 
master_02dacfb07f7c09107e2d8da9955461f025f7f443 and master_latest
* requires - there you should describe the previous necessary steps, in this example we build an image only 
if all tests pass
* filters - describe for which branches this job should execute. There are a lot of other [filters](https://circleci.com/docs/2.0/configuration-reference/#filters-1)
that you can use to customize a workflow

But before you start to run this workflow you should add the next environment variables:
* AWS_ACCESS_KEY_ID - access key for `circleci` that you obtained on [this step]()
* AWS_SECRET_ACCESS_KEY - secret key for `circleci` that you obtained on [this step]()
* AWS_REGION - region where placed your ECR instance
* AWS_ECR_ACCOUNT_URL - url of the ECR(looks like 815991645042.dkr.ecr.us-west-2.amazonaws.com)

<figure class="image">
  <img src="https://dev-to-uploads.s3.amazonaws.com/i/cyb0myv33xnqv93a2rqv.png">
  <figcaption>CircleCI ENV Settings example</figcaption>
</figure>

After successful build of the development and master branches you will see something like there: 

<figure class="image">
  <img src="https://dev-to-uploads.s3.amazonaws.com/i/frjverkv167wd4qcquu2.png">
  <figcaption>ECR after the successful push</figcaption>
</figure>

Great! You automatized process of running tests and building images in the next chapter you will see how to
setup servers on the AWS infrastructure and redeploy them after successfully passed tests.