Leveraging CodePipeline to deploy Terraform
Tags: aws, codepipeline, terraform, cloud
Published: 2023-01-22 18:25:49


I've been on the [CloudFormation side](https://www.chrisfarris.com/terraform-vs-cloudformation/) of the IaC Wars since 2014, when I started working in AWS. I've dabbled in terraform but never made it my _primary_ IaC choice. For the Fooli Meme Factory, I needed to mess things up and then quickly revert the state to what it was at deployment time. So for SECCDC 2023, I ported Meme Factory almost entirely over to [Terraform](https://www.terraform.io/).

This led me to two problems. The first was the perennial issue I've had with Terraform from day one: *"How do I manage state?"*. The second issue was how do I leverage some form of CI/CD tooling to allow me to leverage one of Terraform's biggest strengths - the `terraform plan` capability. Since Fooli is an AWS product, I figured that I should be able to use AWS native tools for this. I've used CodePipeline in the past to preview change-sets with [aws-org-formation](https://github.com/org-formation/org-formation-cli), so I thought it would be easy to find a well-worn pattern from AWS on doing it.

**Apparently, there is no canonical way to use Terraform in CodeBuild, with CodePipeline as the method to review plans before applying them!!!** Seriously, WTF?

![Loki Saying Very Sad, Anyway](https://www.chrisfarris.com/post/tf-codepipeline/Loki-VerySadAnyway.JPG#floatright)

This made me sad. So I decided to do it my damn self. And now I'm documenting it here for everyone else.

This solution will provide the following:
1. [CloudFormation](https://aws.amazon.com/cloudformation/) Template (CFT) to deploy a CodePipeline, CodeBuild Projects, and an S3 Bucket for state and artifact handling.
2. [Buildspec files](https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html) to perform the `terraform plan` and `terraform apply` steps.
3. [Makefiles](https://opensource.com/article/18/8/what-how-makefile) because I'm old school like that.

Why a CloudFormation template for step 1? To get around the chicken-and-egg problem with state. The CFT will deploy and do the needful to get the account setup for the terraform pipeline without needing a terraform pipeline already in place

## How it works.

When a push is made to a monitored GitHub repo, the CodePipeline will trigger. AWS's [CodeStar Connections](https://aws.amazon.com/blogs/devops/using-aws-codepipeline-and-aws-codestar-connections-to-deploy-from-bitbucket/) are used to manage the integration between GitHub and CodePipeline. (As an aside: CodeStar connections are so under-the-radar I can't even find a product page to link to. Just some API docs and the above blog post.) Anyway - CodeStar is a much better solution than previous methods that required overly-privileged GitHub _Personal_ Access Tokens to be uploaded into _shared_ AWS Accounts.

So CodeStar Connections will monitor GitHub and fire the pipeline on a push to the specified branch. After that, the pipeline will execute the following stages:
1. **Source Stage** - where it downloads the code package from GitHub and stores it in the S3 Bucket.
2. **Terraform Plan Stage** - where CodeBuild will execute the `terraform plan` and copy the `tfplan` into S3
3. **Review Stage** - sends a message to SNS, which (if configured) will email someone to review the output of the Terraform plan in CodeBuild.
4. **Apply Stage** - If approved, this stage will fire up CodeBuild to do the `terraform apply` on the preexisting `tfplan` file.

![CodePipeline](https://www.chrisfarris.com/post/tf-codepipeline/Pipeline.png "CodePipeline as seen in the console")

## CodePipeline & CodeBuild

The CodePipeline is defined entirely in the [CloudFormation Template](https://gist.github.com/jchrisfarris/acc0cd200f2fe50ea56874b2f9ab93b7). The CodeBuild projects are created by the CloudFormation Template, but the commands to be executed are defined in the [build spec files](https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html).

### Terraform Plan

The definition of the Plan stage in CodePipeline is:

```yaml
- Name: terraform-plan
  Actions:
    - Name: terraform_plan
      RunOrder: 1
      Namespace: TfPlan
      InputArtifacts:
        - Name: GitHubCode
      OutputArtifacts:
        - Name: TerraformPlan
      ActionTypeId:
        Category: Build
        Provider: CodeBuild
        Owner: AWS
        Version: '1'
      Configuration:
        ProjectName: !Ref TerraformPlanProject
        EnvironmentVariables: !Sub |
          [
            {"name": "EXECUTION_ID",    "value": "#{codepipeline.PipelineExecutionId}"},
            {"name": "BRANCH",          "value": "#{GitHubSource.BranchName}"},
            {"name": "REPO",            "value": "#{GitHubSource.FullRepositoryName}"},
            {"name": "COMMIT_ID",       "value": "#{GitHubSource.CommitId}"},
            {"name": "env",             "value": "${pEnvironment}"}
          ]
```

And the Project is:

```yaml
TerraformPlanProject:
  Type: AWS::CodeBuild::Project
  Properties:
    Name: !Sub ${AWS::StackName}-tf-plan
    Artifacts:
      Type: CODEPIPELINE
    Source:
      Type: CODEPIPELINE
      BuildSpec: buildspec-tf-plan.yaml
    Environment:
      ComputeType: BUILD_GENERAL1_SMALL
      Type: LINUX_CONTAINER
      Image: !Ref BuildImageName
    ServiceRole: !GetAtt ProjectServiceRole.Arn
```

The `EnvironmentVariables` in the pipeline are passed into the CodeBuild Project. CodePipeline substitutes environment variables that begin with a `#` at execution, and the ones beginning with `$` are substituted by CloudFormation at deployment. The BuildSpec exports some environment variables, and they're stored in the pipeline under the `TfPlan` Namespace (Line 5). The `BuildSpec` part of the CodeBuild Project (Line 9) defines the commands that CodeBuild will execute. [That file](buildspec-tf-plan.yaml) looks like:

```yaml
version: 0.2

env:
  exported-variables:
    - BuildID
    - BuildTag

phases:
  install:
    commands:
      - "curl -s https://releases.hashicorp.com/terraform/1.3.6/terraform_1.3.6_linux_amd64.zip -o terraform.zip"
      - "unzip terraform.zip -d /usr/local/bin"
      - "chmod 755 /usr/local/bin/terraform"
  pre_build:
    commands:
      - "make tf-init"
  build:
    commands:
      - "make tf-plan"
      - "export BuildID=`echo $CODEBUILD_BUILD_ID | cut -d: -f1`"
      - "export BuildTag=`echo $CODEBUILD_BUILD_ID | cut -d: -f2`"

artifacts:
  name: TerraformPlan
  files:
    - terraform/$env-terraform.tfplan
```

Lines 4-6 indicate that we will export two environment variables we want to pass back to CodePipeline, `BuildID` and `BuildTag`. These are needed to build the URL for reviewing the plan. The `artifacts` section on line 23 defines the files created that CodeBuild/CodePipeline should store in S3 and pass between the pipeline stages.

### Review Stage
The review stage consists of a manual step and looks like this:
```yaml
- Name: Review-Plan
  Actions:
    - Name: review-plan
      RunOrder: 1
      ActionTypeId:
        Category: Approval
        Provider: Manual
        Owner: AWS
        Version: '1'
      Configuration:
        NotificationArn: !Ref PipelineNotificationsTopic
        CustomData: "Review the Terraform Plan"
        ExternalEntityLink: !Sub "https://${AWS::Region}.console.aws.amazon.com/codesuite/codebuild/${AWS::AccountId}/projects/#{TfPlan.BuildID}/build/#{TfPlan.BuildID}%3A#{TfPlan.BuildTag}/?region=${AWS::Region}"
```

Here we construct the `ExternalEntityLink` from the `BuildID` and `BuildTag` from the plan stage. Again variables that begin with a `#` are substituted by CodePipeline at execution and the ones beginning with `$` are substituted by CloudFormation at deployment. We send a message to the `PipelineNotificationsTopic` which triggers an email to the user:
![Email from CodePipeline](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ay8113ge815w1knd64uw.png "The email from CodePipeline telling me I have changes to review")

### Apply Stage.

The apply stage is similar to the plan. In CodePipeline it looks like this:
```yaml 
- Name: ExecuteTerraform
  Actions:
    - Name: terraform-apply
      RunOrder: 1
      InputArtifacts:
        - Name: GitHubCode
        - Name: TerraformPlan
      ActionTypeId:
        Category: Build
        Provider: CodeBuild
        Owner: AWS
        Version: '1'
      Configuration:
        ProjectName: !Ref ExecuteTerraformProject
        PrimarySource: GitHubCode
        EnvironmentVariables: !Sub |
          [
            {"name": "EXECUTION_ID",    "value": "#{codepipeline.PipelineExecutionId}"},
            {"name": "BRANCH",          "value": "#{GitHubSource.BranchName}"},
            {"name": "REPO",            "value": "#{GitHubSource.FullRepositoryName}"},
            {"name": "COMMIT_ID",       "value": "#{GitHubSource.CommitId}"},
            {"name": "env",             "value": "${pEnvironment}"}
          ]
```

In this case, we're inputting the input artifacts from both GitHub and the plan (lines 5-7). We set the GitHubCode as the `PrimarySource` on line 15, and that becomes the working directory. The other files are written to a different directory, and we have to move them in the BuildSpec file (line 9 below).

The BuildSpec for the apply looks like this:

```yaml
version: 0.2

phases:
  install:
    commands:
      - "curl -s https://releases.hashicorp.com/terraform/1.3.6/terraform_1.3.6_linux_amd64.zip -o terraform.zip"
      - "unzip terraform.zip -d /usr/local/bin"
      - "chmod 755 /usr/local/bin/terraform"
      - "mv $CODEBUILD_SRC_DIR_TerraformPlan/terraform/$env-terraform.tfplan terraform"
  pre_build:
    commands:
      - "make tf-init"
  build:
    commands:
      - "make tf-apply"
```

The buildspec file installs terraform, moves the tfplan file back to where it's expected, runs `make tf-init` (because this is a new container), and then `terraform apply`


## Makefiles

I use Makefiles to simplify the process of deploying both in CodeBuild and when deploying the terraform locally.

The root Makefile for the repo looks like this:
```bash
# Copyright 2022 - Chris Farris (chrisf@primeharbor.com) - All Rights Reserved
#
ifndef env
$(error env is not set)
endif

include config.$(env)
export

#
# Terraform
#
tf-init:
	cd terraform && $(MAKE) tf-init

tf-plan:
	cd terraform && $(MAKE) tf-plan

tf-apply:
	cd terraform && $(MAKE) tf-apply

```

And the Makefile in the `terraform` directory is:

```bash
# Copyright 2022 - Chris Farris (chrisf@primeharbor.com) - All Rights Reserved
#
tf-init:
	terraform init -backend-config=../$(env).tfbackend -reconfigure

tf-plan:
	terraform plan -out=$(env)-terraform.tfplan -no-color

tf-apply:
	terraform apply $(env)-terraform.tfplan
```

The `config.env` file contains all the TF_VAR exports to feed variables to terraform similar to:
```bash
export TF_VAR_mail_relay_ami=ami-0e03dcd66f...
```

The `$(env).tfbackend` contains the line to define the bucket:
```
bucket="fooli-tf-state-test"
```

## Done!

There you have it, a complete solution to deploy Terraform in CodePipeline with CodeBuild and a manual review of the changes to be made. You can tweak the makefiles and buildspec files as you see fit. Here is the entire [CloudFormation Template](https://gist.github.com/jchrisfarris/acc0cd200f2fe50ea56874b2f9ab93b7).

I'm surprised that there isn't a CodeBuild container from Hashicorp or AWS with Terraform pre-installed. The effort of making one would be less than the expense of everyone curling the terraform binary and providers twice on every build.