Setup CICD pipeline for Terraform using GitHub Actions
Tags: githubactions, aws, terraform
Published: 2023-08-28 01:39:05

In the tutorial, I will walk you through how to setup a CICD pipeline using GitHub Actions to deploy AWS resources using Terraform.
The tutorial only focuses on the GitHub Actions part. I assume that you already have some experience on Terraform, GitHub Actions and want to know how to deploy Terraform projects to AWS using Github Actions. I will use a simple terraform project that only deploy a SNS topic to AWS for demo purpose. However, there is a batch of AWS resources and modules defined in a real terraform project with more complex terraform configuration. To deploy a larger terraform project, more complex GitHub Actions workflows are necessary. There will be another tutorial to introduce how to organize and deploy a larger Terraform project automatically using GitHub Actions workflows.
You can find the demo source code from [Source Code](https://github.com/camillehe1992/terraform-cicd-pipeline-on-aws)

## Preprequisties
1. VSCode or other IDE as you prefer
2. AWS account

## Code Structure
Here is the main code structure which includes GitHub Actions workflows, Terraform configuration infrastructure (S3 Bucket and DynamoDB table), Terraform definitions and settings.

```bash
├── .github
│   └── workflows            # several workflows for different purposes
│       ├── tf-deploy-to-dev.yml
│       ├── tf-deploy-to-prod.yml
│       ├── tf-destroy.yml
│       └── tf-unit-tests.yml
├── cloudformation
│   └── infrastructure.yaml # the cloudformation for terrform state bucket and lock table
├── main.tf                 # Terraform configuration and resources 
├── outputs.tf
├── variables.tf
├── settings                # Multiple environments settings
│   ├── dev
│   │   ├── backend.conf
│   │   └── variables.tfvars
│   └── prod
│       ├── backend.conf
│       └── variables.tfvars
```

## GitOps for Multiple Environments
From the code structure above, I enabled multiple environment deployment, dev for development environment, and prod for production environment. Meanwhile, I created the specific Github Actions deployment workflow for each environment, and two branches for the corresponding environment. I'm going to implement the GitOps scenario as below.

![process-workflow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4i6gtybxxbdeceiq5o75.png)

## GitHub Repo Branches
There are two branches in the GitHub repository. The **main** branch is the default and release branch, which targets the production environment. The **dev** branch is the development branch, which targets the development environment. When you start to implement a new feature, you should create a feature branch from the **main** branch and name it as, for example, **feature/add-a-new-page-on-ui**. Then, you code and test on the local environment. You create a PR to merge code change from **feature/add-a-new-page-on-ui** to **dev** branch. After merging, deploy code on dev branch to development environment and validate the change. If everything works as expected, you can merge the change from **dev** to **main** branch, and wait for the release to production. This is the common lifecycle for the software development. 
One of the biggest benefits is any change has to be validated in the development environment before moving to the production environment. Besides, you get a dedicated source code branch for each environment. The **dev** branch can be changed frequently, and release to production environment may happen weekly or bi-weekly.

![multiple-branches](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jxjz71ecqiw78pdmnsct.png)

## Terraform Settings
There are environment specific Terraform backend configurations and variables under settings directory. In the GitHub Actions workflow, I use `-backend-config` to specify backend configuration.

```bash
terraform init -reconfigure -backend-config=settings/dev/backend.conf
```
For `plan` command, use `-var-file` to specify Terraform variables. 

```bash
terraform plan -var-file=settings/dev/variables.tfvars
```

## GitHub Environment
GitHub has an  environment concept for deployment. Environments are used to describe a general deployment target like production, staging, or development. You can configure environments with protection rules and secrets. When a workflow job references an environment, the job won't start until all of the environment's protection rules pass. A job also can not access secrets that are defined in an environment until all the deployment protection rules pass. 

![setup-github-envs](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5n2g2id6dvdrxanzv9ty.png)

In the demo. I created two environments, one named **development**, another named **production** via GitHub console as above. In each environment, there are two environmental secrets and one environmental variable, which are injected into workflow as below in env block. Then these environmental variables can be accessible within the entire workflows. You are also allowed to define env at job level as needed for security.

```yaml
env:
  AWS_ACCESS_KEY_ID: "${{ secrets.AWS_ACCESS_KEY_ID }}"
  AWS_SECRET_ACCESS_KEY: "${{ secrets.AWS_SECRET_ACCESS_KEY }}"
  AWS_REGION: "${{ vars.AWS_REGION }}"
```

## GitHub Actions Workflows
There are four workflows under .github/workflows directory. You can get the purpose from the names. I created specific workflow for deployment to the target environment. One for development, another for production. A destroy workflow to remove Terraform resources for the given environment. A unit test workflow that will be triggered automatically on each push action.

![workflows](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sjlghm1fy93zfexa4a44.png)

### Workflow: tf-deploy-to-dev and tf-deploy-to-prod
Let's go through the deployment workflow from top to bottom.
#### 1. name
The name of the workflow, which will be displayed on the list of All workflows under Actions tab.

![workflow-name](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sxmjf8mt4arrxzeg9jtn.png)

#### 2. on
A workflow run is triggered for any workflows that have on: values that match the triggering event. Some events also require the workflow file to be present on the default branch of the repository in order to run. The script below means the workflow run is triggered when:
1. There is a push event on the dev branch, or
2. There is a pull_request event on the dev branch, or
3. There is a manual trigger event (workflow_dispatch) on the dev branch

```yaml
on:
  push:
    branches:
      - dev
  pull_request:
    branches:
      - dev
  workflow_dispatch:
    branches:
      - dev
```
#### 4. permissions
The permissions you need to grant to the workflow so that it can take specific action. For example, if you want to write output to your PR, you have to grant **write** access to **pull-requests** actions.
Find the details from [Permissions](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#permissions) and [Define Access](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#defining-access-for-the-github_token-scopes).
```yaml
permissions:
  contents: read
  pull-requests: write
```

#### 5. env
Define AWS authentication related environment variables here. Two are secret , one is variable.  These environment variables are saved in [Github](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#env).

#### 6. jobs
A workflow run is made up of one or more [jobs](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobs). I created two jobs with multiple steps in each. **terraform-plan** is used to create a plan, save it to artifact. **terraform-apply** is used to download the plan from artifact, and apply the plan to environment. 

![deploy-workflow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4fnfe5fnauhkir6qnl82.png)
In **terraform-plan** job, there are multiple steps. Each step fulfills a particular task. For example, **Checkout** step pulls source code into job runner workspace.

![terraform-plan](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8s8vodf38lmlr8z0g4no.png)
For prod deployment workflow, the only difference is the workflow can only be triggered manually from GitHub console.

### Workflow: tf-destroy
I created a dedicated workflow for destroying Terraform resources. This is optional, but should be useful when there is something wrong with Terraform resources, and you have to destroy resources from the environment to redeploy/recreate the infrastructure manually via workflow.

### Workflow: tf-unit-test
Optional, but it should be better to keep your code high quality and system availability. 

## Summary
In this tutorial, I walked you through how to create a multiple environments Terraform project using GitHub Actions workflow. You can take it as an example, and build your project with more complex and advanced features. If you are newbie to GitHub Actions (well I'm a newbie as well), the [official documents](docs.github.com) are the best reference to learn GitHub Actions.

That's all. 

I'm always looking forward to any comments and suggestions. Thank you. Happy learning!