Boost your terraform automation 
Tags: terraform, cloud, aws, devops
Published: 2020-12-21 00:39:31

---
title: Boost your terraform automation 
published: true
description: Terraform tools to automate cost management, tests, security, and more, right on your CI/CD pipeline.
tags: terraform, cloud, AWS, devops
---

After some time working with [CloudFormation](https://aws.amazon.com/cloudformation/) to manage infrastructure on AWS, a few months ago I switched jobs and started to work with [terraform](https://www.terraform.io/). I feel that the learning curve was really smooth, despite [the differences between these tools](https://acloudguru.com/blog/engineering/cloudformation-terraform-or-cdk-guide-to-iac-on-aws). 
After using the two, I feel that [HCL](https://www.terraform.io/docs/configuration/syntax.html) is great to avoid all the mess one can do with yaml and if you are using multiple cloud services, for example [Fastly](https://www.fastly.com/) or [Cloudflare](https://www.cloudflare.com/) for [CDN](https://en.wikipedia.org/wiki/Content_delivery_network) instead of [Cloudfront](https://aws.amazon.com/cloudfront/), or running a couple of workloads on different cloud providers such as [GCP](https://console.cloud.google.com/?hl=pt&pli=1) or [Azure](https://azure.microsoft.com/pt-pt/), terraform is a great tool to help you manage everything with a unified tool and language. 
I was also surprised with the amount of useful tools and resources available to work with terraform (check the [Awesome Terraform repository](https://github.com/shuaibiyy/awesome-terraform)!).

# Terraform workflow

The typical terraform workflow might look similar to this:
![terraform workflow](https://dev-to-uploads.s3.amazonaws.com/i/loz4oxcq1x67kol6peiw.jpg)

You start by opening a pull request to a repository that contains the infrastructure code and the first step is to check if the files are well formatted by running `terraform fmt --check`. After that, you run `terraform init` to initialize the working directory by [downloading the different terraform modules](https://www.terraform.io/docs/commands/init.html#child-module-installation) you might be using and [initialize the backend](https://www.terraform.io/docs/commands/init.html#backend-initialization), where the status is stored. 
The third step is to run `terraform plan` that outputs the changes that will be done on the infrastructure based on the modifications you did on the code. You can then analyze this output and see what will be modified, created and/or deleted. If you and your peers are ok with the output, you get your pull request approved, merge it to the main branch and go for the `terraform apply` that will apply the changes. Otherwise, if you are not ok with the output of the plan, you do another commit to fix what is wrong and the workflow starts from the beginning. Pretty straightforward, right?

## Apply and... Fail!

![terraform apply fail](https://i.imgur.com/yZweNwH.gif)

However, even if the plan executes successfully and the output seems fine to you and your peers, it is possible that the `terraform apply` command fails and, since it is not transactional, your infrastructure will be in an inconsistent state until someone applies a patch to fix the problem. The apply command can fail for multiple reasons such as typo on the instance type or missing permissions on an iam role that makes it impossible to do a specific operation. 

### So, what can we do to reduce the likelihood of failing at the `apply` step? 

In this blog post I will do a brief introduction to some of the tools you can add to your terraform workflow to help you to avoid this problem and, on top of that, add automated checks on security and infrastructure costs.

# [TFLint](https://github.com/terraform-linters/tflint)

{% github terraform-linters/tflint no-readme %}

Let's start with [TFLint](https://github.com/terraform-linters/tflint), a static code analysis tool focused on catching possible errors and enforcing best practices. Let's check the following example:

![Tflint example](https://i.imgur.com/4CRR7wj.png)

In this pull request, just by simply adding the [reviewdog github action for TFLint](https://github.com/reviewdog/action-tflint), I was able to get an automated review reporting that the instance type, the ebs volume type and also the AMI id were invalid. These kind of errors are easy to catch, but also easy to miss in the review process and would fail in the `apply`, as previously described.

TFLint has more than [700+ rules](https://github.com/terraform-linters/tflint/tree/master/docs/rules) available and also supports custom rules, for example to enforce patterns in S3 bucket names. It supports AWS, Azure and GCP. 

# [Terratest](https://github.com/gruntwork-io/terratest)

{% github gruntwork-io/terratest no-readme %}

And what if we could actually test the `apply` step to check if it will run successfully? :thinking: By running the `apply`step on a different environment and making automated assertions on the side effects of it, we can also detect problems before running it in the production environment. For this we have [Terratest](https://github.com/gruntwork-io/terratest), a [Go](https://golang.org/) library to write automated tests for infrastructure code.

{% github localstack/localstack no-readme %}

The first time I tried [Terratest](https://github.com/gruntwork-io/terratest), I used it with [Localstack](https://github.com/localstack/localstack) that is an AWS "emulator" you can run locally on your machine or anywhere else. The main reasons for this experiment were to avoid having a different AWS account to run the tests and to be able to run it locally and discard everything easily just by destroying localstack's docker container. 

In order to make terraform execute against your localstack's container, you have to add some settings on your provider block, as you can see in the following example. The most important part is the `endpoints` block that should contain the url for the different AWS services you are interacting with on your terraform code. 

```hcl
provider "aws" {
  region                      = "us-east-1"
  access_key                  = "mock_access_key"
  secret_key                  = "mock_secret_key"
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  s3_force_path_style         = true
  endpoints {
    ec2 = "http://localhost:4566"
    iam = "http://localhost:4566"
    s3  = "http://localhost:4566"
  }
}
```

And here's an example of a simple test that checks if the created bucket with the name "awesome-bucket" has versioning enabled:

```go
terraformOptions := &terraform.Options{
    TerraformDir: "../../local",
    EnvVars: map[string]string{
        "AWS_REGION":         awsRegion,
    },    
}
    
defer terraform.Destroy(t, terraformOptions)

terraform.InitAndApply(t, terraformOptions)

actualStatus := aws.GetS3BucketVersioning(t, 
    awsRegion, "awesome-bucket")
expectedStatus := "Enabled"
assert.Equal(t, expectedStatus, actualStatus)
```

Unfortunately, this test does not run successfully with localstack (yet!). The assertion to check if the bucket has versioning enabled ignores the endpoint set on terraform and calls the real aws endpoint, therefore failing the assertion. This issue is actually open on the terratest repository as you can see below:

{% github https://github.com/gruntwork-io/terratest/issues/494 %}

There's already an ongoing fix for this problem, which I tried following the instructions described in detail [here](https://jq1.io/posts/go_mod_terratest_localstack/), which will basically require you to set again the endepoints on the terratest code like this:

```go
var LocalEndpoints = map[string]string{
    "iam":            "http://localhost:4566",
    "s3":             "http://localhost:4566",
    "ec2":             "http://localhost:4566",
}
aws.SetAwsEndpointsOverrides(LocalEndpoints)
```

On top of this "bump on the road", I also had other issues such as some flakiness on the tests because of the container getting stuck during resource creation and, in the end, I felt that I was spending more time dealing with localstack than writing new tests and taking value out of it. I hope that in the future it will be more reliable but for now, if you wish to use terratest to test your infrastructure, I recommend you to run it against a sandbox AWS account where you can run [aws-nuke](https://github.com/rebuy-de/aws-nuke) or [cloud-nuke](https://github.com/gruntwork-io/cloud-nuke) once in a while.

# [Checkov](https://github.com/bridgecrewio/checkov)

{% github bridgecrewio/checkov no-readme %}

Just like [TFLint](https://github.com/terraform-linters/tflint), [Checkov](https://github.com/bridgecrewio/checkov) is also a static analysis tool that has more than [400 rules](https://github.com/bridgecrewio/checkov/blob/master/docs/3.Scans/resource-scans.md) focused on security and compliance best practices for AWS, Azure and Google Cloud.

It can report situations such as the following, where I created an ebs volume without having encryption enabled:
![EBS encrypted check](https://i.imgur.com/RsFN7sj.png)

Or this one, where I created an S3 bucket without access logging, that could be handy for further analysis of suspicious assesses to the bucket:
![S3 Access logging](https://i.imgur.com/og6ygJO.png)

What I enjoy most in this tool is that every issue reported has a [url](https://docs.bridgecrew.io/docs/s3_13-enable-logging) with the rationale, so you can learn why it is important to fix and how to do it. Kudos to [BridgeCrew](https://bridgecrew.io/) for the great work!

# [Infracost](https://github.com/infracost/infracost)

{% github infracost/infracost no-readme %}

This tool really got me "wow, this is cool" 😲. Basically, [Infracost](https://github.com/infracost/infracost) estimates the costs automatically in every pull request, based on the changes you did on your terraform code. How many times you didn't realize that the changes you were doing were going to cost you 2x more in the end of the month? Or you suspected that it would be expensive but you had to jump to [AWS Pricing calculator](https://calculator.aws/#/) and do the math? 
In the following example, I just increased the instance type from t3.micro to t3.xlarge and automatically (powered by this [infracost github action](https://github.com/marketplace/actions/run-infracost)), I got a comment on my pull request telling me that the monthly bill would increase from ~114$ to ~239$:

![Infracost example](https://i.imgur.com/NrluN7k.png)

# That's it! :wave:
If you're interested and want to quickly try these tools, I set up a Github Actions workflow with all of them in this [repository](https://github.com/bmbferreira/awesome-terraform-pipeline), feel free to use it!

{% github bmbferreira/awesome-terraform-pipeline no-readme %} 

I hope that you find at least some of these tools useful and please tell me on the comments what other tools are you using in your day-to-day to improve your work with terraform! :smile: