Tag-based monorepo docker image build on AWS
Tags: aws, monorepo, devops, awsthai
Published: 2021-09-07 09:09:22

![Monorepo to AWS ECR](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6dxe1y23s46sc9y4ysqq.png)
My team adopted [monorepo](https://www.atlassian.com/git/tutorials/monorepos) for our backend microservices and we need a simple solution to build docker images so we can use it for test or deploy anywhere. Because we use [Amazon Elastic Container Repository (ECR)](https://aws.amazon.com/ecr/) to store docker images, I decided to use git tag to trigger docker image build on [AWS CodeBuild](https://aws.amazon.com/codebuild/) and push it to ECR.

The condition to trigger build is adding tag `service-name/version-number` and push it to GitHub or Bitbucket. AWS CodeBuild will receive webhook and start building new docker image using `version-number` provided in tag and push it to ECR. And lastly.... store build artifact in S3.

### The REPO
Firstly, we need a monorepo on GitHub or Bitbucket, I prepared a sample repo with 2 projects at https://github.com/teer823/monorepo-aws-docker.

> Each projects is just a simple express service with Dockerfile ready to build docker image and include buildspec.yml for AWS CodeBuild to use, however, I also added 2 buildspec file inside buildspec folder to demonstrate that it's possible to separate buildspec.yml file per project as needed.

The buildspec.yml file is an instruction for AWS CodeBuild and separated to phases. Here's what I do at each phase.
* **pre_build** extract TAG_VERSION_NUMBER from tag and create version.txt file to store as artifact then login to ECR
* **build** as the name say... build and tag docker image. Please note that I also add `latest` tag here as well, doing so prevent me from using ECR's tag immutability feature.
* **post_build** push docker image to ECR

```
version: 0.2

phases:
  pre_build:
    commands:
      - echo Pre Build Phase...
      - export TAG_VERSION_NUMBER=$(echo $CODEBUILD_WEBHOOK_TRIGGER | sed 's/.*\///')
      - echo $TAG_VERSION_NUMBER
      - >
        PACKAGE_VERSION=$(cat $REPO_FOLDER/package.json | grep version | head -1 | awk -F: '{print $2}' | sed 's/[",[:space:]]//g')
      - echo $PACKAGE_VERSION > version.txt
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
  build:
    commands:
      - echo Build Phase...
      - cd $REPO_FOLDER
      - docker build -t $IMAGE_REPO_NAME:$TAG_VERSION_NUMBER -t $IMAGE_REPO_NAME:latest .
      - docker tag $IMAGE_REPO_NAME:$TAG_VERSION_NUMBER $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$TAG_VERSION_NUMBER
      - docker tag $IMAGE_REPO_NAME:latest $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest 
  post_build:
    commands:
      - echo Post Build Phase...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$TAG_VERSION_NUMBER
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest
artifacts:
  files:
    - '$REPO_FOLDER/**/*'
    - 'version.txt'
  name: '$IMAGE_REPO_NAME-$TAG_VERSION_NUMBER'
  discard-paths: no
```

For more info about `buildspec.yml` please refer to [HERE](https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html)

### AWS ECR
As I want to push docker image to ECR, I need to pre-create container repositories for each service beforehand. Just go and create a private repository and please don't enable **Tag immutability** or build will failed as `latest` tag cannot be overwritten. (Or just change the build spec and remove `latest` tag)

![AWS ECR Create Repo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lvhlsxf3bnrnpksphkfa.png)

### AWS S3

We want CodeBuild to store build artifact on AWS S3, so we need to prepare a bucket in advance. Let's create a bucket! (Leave the rest of configuration default or up to you.., i recommend turn on server-side encryption for *BEST PRACTICE* reason)

![S3 create bucket](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tknd6gyz7byfhhjmuoov.png)

### AWS CODEBUILD
Now we have Code Repo , Container Repo, Artifact bucket.. let's create AWS CodeBuild project.
* **Project configuration** your choices..
    ![Config Project](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/psqynyfbr11c92ie1ql1.png)
* **Source** connect to your GitHub or Bitbucket and select *repository in my GitHub account* and select our monorepo repository. Please note that we need to rely on webhook so public repo won't do.
    ![Config Sourdce](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pakgclklg40w9uziw7kn.png)
* **Primary source webhook events** Here's what important. Enable webhook by checking the box. And config build condition
    * Build type - Single build
    * Event type - PUSH and PULL_REQUEST_MERGED
    * Condition - Start build when receiving tag, in HEAD_REF add a regular expression `^refs/tags/service-name/([\w\.-]+)$`, don't forget to replace `service-name` accordingly.
    ![Config Webhook](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q6ohu3r6mlywhswi79bs.png)

> If you want to use other method to trigger build, here's the place to try and modify build condition according to your usecases.

* **Environment** I personally prefer Ubuntu, so it's up to you how you want to config your build machine, however what's important for me here are the following
    * *Privileged* Enable the flag because we want to build docker images
    ![Priviledge](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uxrrwiq4aorxaomtkrwr.png)
    * *Service role* Select your preferred name, I personally like to reuse the role in same project so I name it with project instead of service and next time I can just select it from existing service role. However, please remember the Role name because we will need it later.
    ![Service Role](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/owyh4il0gat2d78a0b18.png)
    * *Environment Variable* If you noticed in `buildspec.yml` earlier, I used some environment variable. you will need to set it here
        * AWS_DEFAULT_REGION - your ECR region
        * AWS_ACCOUNT_ID - your ECR account Id
        * IMAGE_REPO_NAME - name of ECR remove created above
        * REPO_FOLDER - folder name for service we're building

        ![Environment Variable](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/13yqokgf1efra9oh4c6b.png)
* **Buildspec** when using root's `buildspec.yml` just select *Use a buildspec file* will do. 
![Default buildspec](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e78cdxo7gxz3d8x6vyfg.png)
>In case you want to try using individual buildspec file, add path to project-buildspec file in Buildspec name field
![Custom buildspec file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9xlydegsrx2zdhzilvln.png)

* **Artifacts** Select Amazon S3 and bucket created above, use Zip packaging and use `service-name` as path.
![Build Artifact](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wjmtyvm00ljqeeua8eem.png)

Hit *Create build project* and we're done...then add another one for 2nd service.

### PERMISSION
One last step before start tagging and building. our service role don't have permission to push docker image to ECR yet. so we need to add it in IAM. Go to your Role created above and Attach Managed Policy name `AmazonEC2ContainerRegistryPowerUser` to our role and Done!
![IAM Role Allow ECR](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/adj73us7a8114avqkn9a.png)

### TAGGING TIME!
Now it's time to add tag to monorepo and push it to origin
```
❯ git tag ingest-service/0.1.0
❯ git push origin ingest-service/0.1.0
```
Go back to CodeBuild project and see it work!
![Build in progress](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n7vs2c1wtmu7sop4xbcj.png)

When building complete, find your new docker image in ECR!
![New Docker Image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7nt1362uv86p0rd3iv3t.png)

### Conclusion
That's all about it. I hope this is useful for anyone looking to make a simple build pipeline like my team :)