CI/CD Pipeline for Terraform Workflow Using Amazon CodeCatalyst
Tags: aws, cicd, codecatalyst, terraform
Published: 2023-11-22 04:25:20

[Terraform workflow](https://developer.hashicorp.com/terraform/intro/core-workflow) can be ran using several methods. One of them is running Terraform workflow inside CI/CD pipeline. Running Terraform workflow inside CI/CD pipeline can have several benefits, such as: automate creation or provision resources, simplify collaboration between engineers/developers, etc.

## Table of Contents

1. [Scenario](#scenario)
2. [Prerequisites](#prerequisites)
3. [Configuration Steps](#configuration-steps)
    - [Directory Structure](#directory-structure)
    - [Terraform Backend Configurations](#terraform-backend-configurations)
    - [CodeCatalyst Workflow File Configurations](#codecatalyst-workflow-file-configurations)
    - [Example of Terraform Workflow Process in CodeCatalyst CI/CD](#example-of-terraform-workflow-process-in-codecatalyst-cicd)
4. [Summary](#summary)

## Scenario

In this blog, I will explain how to run Terraform workflow inside CI/CD pipeline which in this blog is Amazon CodeCatalyst. GitHub is used as source code repository and has been connected with CodeCatalyst. Next in CodeCatalyst pipeline/workflow, We will provision resources via Terraform. In this blog, Terraform is using S3 backend. Here is the topology used in this blog:

![topology](https://ludesdeveloper.files.wordpress.com/2023/10/topology.png)

This blog is using reference from tutorial [“Bootstrapping your Terraform automation with Amazon CodeCatalyst”](https://community.aws/tutorials/bootstrapping-terraform-automation-amazon-codecatalyst) written by [@Cobus Bernard](https://github.com/cobusbernard) with some adjustment in configuration.

If You want to know more details about CodeCatalyst, please check [CodeCatalyst documentation](https://docs.aws.amazon.com/codecatalyst/latest/userguide/welcome.html).

## Prerequisites

- Before starting configuration, if you have not got [AWS Builder ID](https://docs.aws.amazon.com/signin/latest/userguide/sign-in-aws_builder_id.html) account, please sign up first
- After sign up, go to [Amazon CodeCatalyst](https://codecatalyst.aws/explore) web console to start project and configuration. Apply initial configuration that needed by CodeCatalyst such as:
    - Creating space
    - Connect CodeCatalyst space with AWS account
    - Create and connect IAM role that will be used for running CodeCatalyst workflow/pipeline
    - Connect GitHub source code repository
    - Define environment that will be used by workflow/pipeline
    - Please check my blog [“Build and Release Container Image to Amazon Elastic Container Registry (ECR) via Amazon CodeCatalyst”](https://dev.to/aws-builders/build-and-release-container-image-to-amazon-elastic-container-registry-ecr-via-amazon-codecatalyst-3a2p) to find reference for configuration above. Please check section `Amazon CodeCatalyst Pipeline/Workflow Configuration`
- Prepare S3 bucket and DynamoDB lock table (optional) that will be used by Terraform backend

## Configuration Steps

### Directory Structure

Below is directory structure used in this blog:

```
.
├── .codecatalyst/
│   └── workflows/
│       └── tf-sbx2-vpc-apse1.yml
└── sandbox2/
    ├── vpc/
    │   └── ap-southeast-1/
    │       ├── main.tf
    │       ├── resources.tf
    │       └── variables.tf
    └── tf-backend/
        ├── main.tf
        ├── resources.tf
        └── variables.tf
```

### Terraform Backend Configurations

Configuration start with provisioning resources that will be needed by Terraform backend such as S3 bucket and DynamoDB table. Beside that, We also need to prepare IAM role that will be needed to provision resources from CodeCatalyst workflow/pipeline.

1. At first, We haven’t had any resource for storing Terraform state in S3, so that we need to run and store terraform state on our local, for example: laptop. Later, We will move that state from local to S3. We start by defining configuration in **sandbox2/tf-backend/main.tf**

    ```go
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 5.11.0"
        }
      }
     
      required_version = ">= 1.3.0"
    }
     
    provider "aws" {
      region = var.aws_region
    }
    ```
2. Define variable region in file **sandbox2/tf-backend/variables.tf**

    ```go
    variable "aws_region" {
      type        = string
      description = "AWS Region"
      default     = "ap-southeast-1"
    }
    ```

3. Define resources S3, DynamoDB and IAM that needed by CodeCatalyst in file **sandbox2/tf-backend/resources.tf**. In this blog, IAM role CodeCatalyst has administrator access just for demo purposes. Please bear in mind to always use least privilege method on your production to prevent any security breaches

    ```go
    ######
    # S3 #
    ######
     
    resource "aws_s3_bucket" "my_sandbox2_tfbucket" {
      bucket = "my-sandbox2-tfbucket"
     
      lifecycle {
        prevent_destroy = true
      }
    }
     
    resource "aws_s3_bucket_versioning" "my_sandbox2_tfbucket" {
      bucket = aws_s3_bucket.my_sandbox2_tfbucket.id
     
      versioning_configuration {
        status = "Enabled"
      }
    }
     
    resource "aws_s3_bucket_server_side_encryption_configuration" "my_sandbox2_tfbucket" {
      bucket = aws_s3_bucket.my_sandbox2_tfbucket.id
     
      rule {
        apply_server_side_encryption_by_default {
          sse_algorithm = "AES256"
        }
      }
    }
     
    resource "aws_s3_bucket_public_access_block" "my_sandbox2_tfbucket" {
      bucket                  = aws_s3_bucket.my_sandbox2_tfbucket.id
      block_public_acls       = true
      block_public_policy     = true
      ignore_public_acls      = true
      restrict_public_buckets = true
    }
     
    ############
    # DynamoDB #
    ############
     
    resource "aws_dynamodb_table" "my_sandbox2_tflocks" {
      name         = "my-sandbox2-tflocks"
      billing_mode = "PAY_PER_REQUEST"
      hash_key     = "LockID"
     
      attribute {
        name = "LockID"
        type = "S"
      }
    }
 
    #######
    # IAM #
    #######
     
    data "aws_iam_policy_document" "codecatalyst_assume_role_policy" {
      statement {
        actions = ["sts:AssumeRole"]
        principals {
          type = "Service"
          identifiers = [
            "codecatalyst.amazonaws.com",
            "codecatalyst-runner.amazonaws.com"
          ]
        }
      }
    }
     
    resource "aws_iam_role" "codecatalyst_admin" {
      name               = "codecatalyst-admin"
      assume_role_policy = data.aws_iam_policy_document.codecatalyst_assume_role_policy.json
    }
     
    resource "aws_iam_role_policy_attachment" "codecatalyst_admin" {
      role       = aws_iam_role.codecatalyst_admin.name
      policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
    }
    ```

4. Refer AWS keys that needed for running terraform workflow commands, for example by exporting environment variables as explained in this [document](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#environment-variables)

5. Run terraform cli commands that needed for provisioning resources

    ```
    terraform init
    terraform plan
    terraform apply
    ```

6. Verify terraform apply is successful and resources successfully provisioned

7. Back to file **sandbox2/tf-backend/main.tf**, add additional configurations related to S3 backend

    ```go
    ###########################
    # Terraform Configuration #
    ###########################
     
    terraform {
      backend "s3" {
        bucket         = "my-sandbox2-tfbucket"
        key            = "path/to/terraform.tfstate"
        region         = "ap-southeast-1"
        dynamodb_table = "my-sandbox2-tflocks"
        encrypt        = true
      }
    }
    ```

8. Run `terraform init` command again. Terraform will detect changes related to S3 backend and will move terraform state to S3 bucket that has been defined. Choose `yes` if asked for input

    ```
    terraform init
    
    Initializing the backend...
    Do you want to copy existing state to the new backend?
      Pre-existing state was found while migrating the previous "local" backend to the
      newly configured "s3" backend. No existing state was found in the newly
      configured "s3" backend. Do you want to copy this state to the new "s3"
      backend? Enter "yes" to copy and "no" to start with an empty state.
    
      Enter a value: yes
    
    Releasing state lock. This may take a few moments...
    
    Successfully configured the backend "s3"! Terraform will automatically
    use this backend unless the backend configuration changes.
    ```

9. Go to Amazon Web Services (AWS) web console, go to S3 service menu. Verify `terraform.tfstate` file has been copied to defined S3 bucket/path

### CodeCatalyst Workflow File Configurations

After configurations for terraform backend has been finished as described above, next we continue configure CodeCatalyst workflow/pipeline.

Before starting workflow configuration, please make sure to configure all configurations that needed by CodeCatalyst as mentioned in [Prerequisites](#prerequisites).

As mentioned in document [Build, test, and deploy with workflows in CodeCatalyst](https://docs.aws.amazon.com/codecatalyst/latest/userguide/flows.html), CodeCatalyst use workflow definition file as reference workflow/pipeline configuration and this file is saved in directory `~/.codecatalyst/workflows/`.

In this blog, I will create basic resources that needed by VPC on AWS. As mentioned on [Directory Structure](#directory-structure) above, pipeline will be ran when there is changes inside directory `sandbox2/vpc/ap-southeast-1`.

Below is full content of workflow file `codecatalyst/workflows/tf-sbx2-vpc-apse1.yml` that used as example in this blog.

```yaml
Name: tf-sbx2-vpc-apse1
SchemaVersion: "1.0"
 
Triggers:
  - Type: PUSH
    Branches:
      - master
    FilesChanged:
      - sandbox2\/vpc\/ap-southeast-1\/.*
 
Actions:
  terraform-plan:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform fmt -check -no-color
      - Run: terraform init -no-color
      - Run: terraform validate -no-color
      - Run: terraform plan -no-color -input=false
    Compute:
      Type: EC2
  wait-period:
    DependsOn:
      - terraform-plan
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Steps:
      - Run: echo "Please wait for a while before terraform apply"
      - Run: echo "If you wish to cancel terraform apply, please stop this run"
      - Run: sleep 60
    Compute:
      Type: EC2
  terraform-apply:
    DependsOn:
      - wait-period
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform init -no-color
      - Run: terraform apply -auto-approve -no-color -input=false
    Compute:
      Type: EC2
```

Next, I will explain more detail each sections that defined on workflow file above.

1. Define workflow name. Then define how workflow will be ran. In this blog, workflow will be ran when there is a push to master branch but only ran when changes is under directory `sandbox2/vpc/ap-southeast-1/` (regular expression or regex is used to detect changes)

    ```yaml
    Triggers:
      - Type: PUSH
        Branches:
          - master
        FilesChanged:
          - sandbox2\/vpc\/ap-southeast-1\/.*
    ```

2. Define action. Action will be divided into 3 sections. The first one is used to run workflow `terraform plan`. In this section, action uses identifier `aws/build@v1`. Then action will be ran on sandbox environment that has been connected with AWS account and also has IAM role associated with that account. In `Configuration` section, has been defined that action will be ran on top of container using specific version of terraform public image that pulled from DockerHub. Also has defined working directory and several terraform commands which include `terraform plan`. Lastly, define compute type which is `EC2`

    ```yaml
    Actions:
      terraform-plan:
        Identifier: aws/build@v1
        Inputs:
          Sources:
            - WorkflowSource
        Environment:
          Name: sandbox
          Connections:
            - Name: lanandra-sandbox
              Role: codecatalyst-admin
        Configuration:
          Container:
            Registry: DockerHub
            Image: hashicorp/terraform:1.5.7
          Steps:
          - Run: cd sandbox2/vpc/ap-southeast-1
          - Run: terraform fmt -check -no-color
          - Run: terraform init -no-color
          - Run: terraform validate -no-color
          - Run: terraform plan -no-color -input=false
        Compute:
          Type: EC2
    ```

3. Next action that defined is called `wait-period`. This action is defined so that there is an interlude between `terraform plan` and `terraform apply`. In case there is something that unwanted in terraform plan, so that we have a time to not continue to next action or process which is terraform apply. This action is workaround because by the time this blog is released, there is no out of the box solution provided by CodeCatalyst, for example manual approval. So in case there is something unwanted, we can abort the workflow. In this action, I created a logic to pause the workflow for 60 seconds and this action is depends on previous action which is `terraform plan`

    ```yaml
    wait-period:
        DependsOn:
          - terraform-plan
        Identifier: aws/build@v1
        Inputs:
          Sources:
            - WorkflowSource
        Environment:
          Name: sandbox
          Connections:
            - Name: lanandra-sandbox
              Role: codecatalyst-admin
        Configuration:
          Steps:
          - Run: echo "Please wait for a while before terraform apply"
          - Run: echo "If you wish to cancel terraform apply, please stop this run"
          - Run: sleep 60
        Compute:
          Type: EC2
    ```

4. Next action that defined is action to run `terraform apply`. This action more or less similar with action `terraform-plan`, the difference is just terraform command that declared on configuration. In this section, action will run commands that needed for `terraform apply`. This action is depends on previous action which is `wait-period`

    ```yaml
    terraform-apply:
        DependsOn:
          - wait-period
        Identifier: aws/build@v1
        Inputs:
          Sources:
            - WorkflowSource
        Environment:
          Name: sandbox
          Connections:
            - Name: lanandra-sandbox
              Role: codecatalyst-admin
        Configuration:
          Container:
            Registry: DockerHub
            Image: hashicorp/terraform:1.5.7
          Steps:
          - Run: cd sandbox2/vpc/ap-southeast-1
          - Run: terraform init -no-color
          - Run: terraform apply -auto-approve -no-color -input=false
        Compute:
          Type: EC2
    ```

### Example of Terraform Workflow Process in CodeCatalyst CI/CD

After configure CodeCatalyst workflow file, next I will create example of Terraform codes. In this blog, I will create VPC resources using public module [terraform-aws-modules/vpc/aws](https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest) as reference. Here is the initial configuration/codes that defined:

**sandbox2/vpc/ap-souhteast-1/main.tf**

```go
###########################
# Terraform Configuration #
###########################
 
terraform {
  backend "s3" {
    bucket         = "my-sandbox-tfbucket"
    key            = "path/to/terraform.tfstate"
    region         = "ap-southeast-1"
    dynamodb_table = "my-sandbox-tflocks"
    encrypt        = true
  }
}
 
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.11.0"
    }
  }
 
  required_version = ">= 1.3.0"
}
 
provider "aws" {
  region = var.aws_region
}
```

**sandbox2/vpc/ap-souhteast-1/variables.tf**

```go
variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "ap-southeast-1"
}
 
variable "env_name" {
  type    = string
  default = "sbx2"
}
```

**sandbox2/vpc/ap-souhteast-1/resources.tf**

```go
#######
# VPC #
#######
 
 
data "aws_availability_zones" "available" {}
 
locals {
  azs = slice(data.aws_availability_zones.available.names, 0, 3)
 
  tags = {
    "myTag:environment" = "sbx2"
    "myTag:managedBy"   = "terraform"
  }
}
 
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.1.2"
 
  name = var.env_name
  cidr = "10.255.8.0/21"
 
  azs             = local.azs
  private_subnets = ["10.255.8.0/24", "10.255.9.0/24", "10.255.10.0/24"]
  public_subnets  = ["10.255.15.0/24", "10.255.14.0/24", "10.255.13.0/24"]
 
  enable_nat_gateway = false
  single_nat_gateway = true
 
  tags = local.tags
}
```

1. To conduct testing of Terraform workflow in CI/CD, I will change 1 line of code inside file **sandbox2/vpc/ap-souhteast-1/resources.tf**. I will activate NAT gateway with changing this line of code that current value is `false`

    ```go
    enable_nat_gateway = true
    ```

2. As defined in workflow file, all changes inside directory **sandbox/vpc/ap-southeast-1** will trigger workflow run. To verify, go to CodeCatalyst web console, go to space and project where codes resided. Then go to CI/CD menu, choose Workflows. Go to tab Runs, verify there is new run that currently running.

    ![new-run](https://ludesdeveloper.files.wordpress.com/2023/10/1.-new-run.png)

3. Go to run details by clicking name/id of run. Then We will be redirected to overview page of run. Click action name `terraform-plan`, on the right will be showed tray box that display details configuration/steps of action. For example We want to see details of `terraform plan`, go to tab `log`, then click step `terraform plan -no-color -input-false`. Verify action succeeded

    ![tf-plan-overview](https://ludesdeveloper.files.wordpress.com/2023/10/2.-tf-plan-overview.png)

4. Verify terraform plan show expected result. Terraform will create resources releated to NAT gateway as mentioned in point number 1

    ![tf-plan-details](https://ludesdeveloper.files.wordpress.com/2023/10/3.-tf-plan-details.png)

5. As defined in workflow file, after `terraform-plan` action ran successfully, next workflow will proceed next action which is `wait-period` to give some interlude whether terraform plan will be executed to next action which is `terraform-apply` or apply will be aborted. If plan has met expectation and We want to proceed to `terraform apply`, no action needed, just wait 60 seconds and then verify action succeeded

    ![wait-period-successful](https://ludesdeveloper.files.wordpress.com/2023/10/4a.-wait-period-successful.png)

6. But, if want to abort workflow and don’t want continue to `terraform apply`, click stop button when action still in progress

    ![stop-wait-period](https://ludesdeveloper.files.wordpress.com/2023/10/4b.-stop-wait-period.png)

7. Let’s say We want to continue to `terraform apply` process, then `terraform-apply` action will be ran. We can see the details of the action inside tray box similar with previous actions. To verify resources has been provisioned via `terraform apply`, click step `terraform apply -auto-approve -no-color -input=false`

    ![tf-apply-overview](https://ludesdeveloper.files.wordpress.com/2023/10/5.-tf-apply-overview.png)

8. Verify terraform apply has been expected and action is succeeded

    ![tf-apply-details](https://ludesdeveloper.files.wordpress.com/2023/10/6.-tf-apply-details.png)

9. Go to VPC service page in AWS web console to verify NAT gateway has been provisioned successfully

    ![verify-nat-gw](https://ludesdeveloper.files.wordpress.com/2023/10/7.-verify-nat-gw.png)

## Summary

We have reached the last section of this blog. Here are some key takeaways that can be summarized:

- Amazon CodeCatalyst can act as alternative for CI/CD engine/tools that can be used to run Terraform workflow
- Amazon CodeCatalyst use IAM role to interact with AWS services. By using this method, Terraform doesn’t need to inject static credentials such as AWS Access Key and AWS Secret Key into the pipeline. This can’t help prevent security breaches
- Amazon CodeCatalyst can give seamless experience if We want to deploy application to AWS environments
- But some workflow or logic haven’t been provided by CodeCatalyst by the time this blog is released. Maybe there will be some improvement in the future

Please check out my GitHub repository to see source code example that used in this blog:

- Terraform backend configuration: [tf backend](https://github.com/lanandra/mock-aws-infrastructure/tree/master/sandbox2/tf-backend)
- Amazon CodeCatalyst workflow file: [codecatalyst workflow](https://github.com/lanandra/mock-aws-infrastructure/tree/master/.codecatalyst/workflows)
- Example of Terraform AWS VPC configuration: [tf aws vpc](https://github.com/lanandra/mock-aws-infrastructure/tree/master/sandbox2/vpc/ap-southeast-1)

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!