Infrastructure as Code with terraform for multiple environments
Published: 2020-12-10 09:35:32

Working with Infrastructure as Code (IaC) is incredible. Figuring out the correct repository structure, including a CI/CD pipeline, is not so simple.

Here are my two cents on the subject.

## Motivation

1. Using git to audit changes and revert to the previous state
1. Promoting environments with git's pull-requests (dev-->stg-->prd)
1. Comfortable maintenance of variables per environment

![infrastructure-as-code-with-terraform-for-multiple-environments-meme](https://s3-eu-west-1.amazonaws.com/meirg.co.il/infrastructure-as-code-with-terraform-for-multiple-environments-meme.png)

## Offered Structure

To examine a live example visit [unfor19/terraform-multienv](https://github.com/unfor19/terraform-multienv). This repository can also be used [as a template](https://github.com/unfor19/terraform-multienv/generate).

```bash
.
├── .github
│   └── workflows
│       ├── terraform-apply.yml
│       └── terraform-plan.yml
├── cloudformation
│   └── cfn-tfbackend.yml
├── live
│   ├── backend.tf.tpl
│   ├── main.tf
│   ├── providers.tf
│   └── variables.tf
└── scripts
    ├── prepare-backend.sh
    ├── prepare-files-folders.sh
    └── terraform.sh
```

## Assumptions

- Branches names are aligned with environments names, for example `dev`, `stg` and `prd`
- The CI/CD tool supports the variable `${BRANCH_NAME}`, for example `${DRONE_BRANCH}`
- The directory `./live` contains infrastructure-as-code files - `*.tf`, `*.tpl`, `*.json`

- Multiple Environments
  - All environments are maintained in the same git repository
  - Hosting environments in different AWS account is supported (and recommended)

- Variables
  - ${app_name} = `your-app-name`
  - ${environment} = `dev` or `stg` or `prd`

## Remote Backend

#### Creating The Resources

This can be done manually in the AWS Console or by using a terraform module, such as [cloudposse/terraform-aws-tfstate-backend](https://github.com/cloudposse/terraform-aws-tfstate-backend).

Initially, I used the terraform module, but then I found out I'm stuck in the "chicken and the egg" situation. I'm using terraform to create resources that will save the state of those resources; what?

I found it easier to do this task with CloudFormation, which makes it totally a separate process, and it is done once per environment.

I've created the script [scripts/prepare-backend.sh](https://github.com/unfor19/terraform-multienv/blob/dev/scripts/prepare-backend.sh), which creates an S3 bucket for the state file and DynamoDB table for [state lock](https://www.terraform.io/docs/backends/types/s3.html). The script uses aws-cli and deploys the [cloudformation/cfn-tfbackend.yml](https://github.com/unfor19/terraform-multienv/blob/dev/cloudformation/cfn-tfbackend.yml) template.

*NOTE*: Remote Backend resources are created per environment (branch).

#### Configuration Per Environment

One of the major pain points of this configuration is setting the [Terraform Remote Backend](https://www.terraform.io/docs/backends/types/s3.html) per environment. Each environment must have its own "backend settings", and the values must be hard-coded since you can't use variables in the `terraform backend` code block.

> Only one backend may be specified, and **the configuration may not contain interpolations.** Terraform will validate this. [Source](https://www.terraform.io/docs/backends/config.html)

This part was solved by creating a template, which gets modified before executing `terraform apply`.

[live/backend.tf.tpl](https://github.com/unfor19/terraform-multienv/blob/dev/live/backend.tf.tpl)
```ruby
terraform {
  backend "s3" {
    region         = "AWS_REGION"
    bucket         = "APP_NAME-state-ENVIRONMENT"
    key            = "terraform.tfstate"
    dynamodb_table = "APP_NAME-state-lock-ENVIRONMENT"
    encrypt        = false
  }
}
```

The script that applies the modification is [scripts/prepare-files-folders.sh](https://github.com/unfor19/terraform-multienv/blob/dev/scripts/prepare-files-folders.sh). It's using a simple `sed` command to find and replace `AWS_REGION`, `APP_NAME` and `ENVIRONMENT`. After applying these changes, the file `backend.tf.tpl` is renamed to `backend.tf`, making it available to the terraform CLI due to its `tf` extension.

This script `prepare-files-folders.sh` has a bigger purpose, and we'll dive into it in a few moments.

## Environment Per Branch

#### One Dir To Rule Them All

The goal is to have a single directory that contains all of the IaC files (tf, JSON, and so on) and then execute `terraform apply` from this directory. And again, [scripts/prepare-files-folders.sh](https://github.com/unfor19/terraform-multienv/blob/dev/scripts/prepare-files-folders.sh) comes to the rescue.

```bash
meirgabay@~/terraform-multienv (dev)$ ./scripts/prepare-files-folders.sh 
[LOG] Prepared files and folders for the environment - dev
total 48
drwxr-xr-x   8 meirgabay  staff   256B Dec  9 23:06 .
drwxr-xr-x  19 meirgabay  staff   608B Dec  9 23:06 ..
-rw-r--r--   1 meirgabay  staff   229B Dec  9 23:06 backend.tf
-rw-r--r--   1 meirgabay  staff   775B Dec  9 23:06 main.tf
-rw-r--r--   1 meirgabay  staff    72B Dec  9 23:06 outputs.tf
-rw-r--r--   1 meirgabay  staff    41B Dec  9 23:06 providers.tf
-rw-r--r--   1 meirgabay  staff   1.2K Dec  9 23:06 variables.tf
terraform {
  backend "s3" {
    region         = "eu-west-1"
    bucket         = "tfmultienv-state-dev"
    key            = "terraform.tfstate"
    dynamodb_table = "tfmultienv-state-lock-dev"
    encrypt        = false
  }
}
```

A new directory `dev` was created in the project's root directory, and it contains the files that are needed to deploy the infrastructure. The example above is for `dev`, but the result would be the same for `stg` and `prd`; the only thing that really changes is the Remote Backend configuration.

#### Variables Per Environment

This is where [variables](https://www.terraform.io/docs/configuration/variables.html) and [local values](https://www.terraform.io/docs/configuration/locals.html) come into play.

The variable `BRANCH_NAME` is assigned to `environment`, according to the branch that will be deployed with `terraform apply`. The CI/CD process sets the value of `BRANCH_NAME` (we'll get to that).

```bash
BRANCH_NAME=dev
$ (dev) terraform --apply -var environment="$BRANCH_NAME"
```

[live/variables.tf](https://github.com/unfor19/terraform-multienv/blob/dev/live/variables.tf)

```ruby
variable "environment" {
  type        = string
  description = "dev, stg, prd"
}

variable "cidr_ab" {
  type = map
  default = {
    dev = "10.1"
    stg = "10.2"
    prd = "10.3"
  }
}

locals {
  vpc_cidr = "${lookup(var.cidr_ab, var.environment)}.0.0/16"
}
```

[live/main.tf]()
```ruby
module "vpc" {
  source             = "terraform-aws-modules/vpc/aws"
  cidr               = local.vpc_cidr
  # omitted arguments for brevity
}
```

*NOTE*: I prefer using local values in my resources and modules instead of variables. I even wrote a [blog post](https://dev.to/prodopsio/mvc-design-pattern-in-terraform-g99) about it. It's not mandatory, but I consider it as best practice.

## CI/CD

This part is fully covered in the README of [terraform-multienv](https://github.com/unfor19/terraform-multienv#getting-started). This is an abstract of the whole process.

1. Deploying the infrastructure - Commit and push changes to your repository

   ```
   git checkout dev
   git add .
   git commit -m "deploy dev"
   git push --set-upstream origin dev
   ```

2. CI/CD service (GitHub Actions in my case) is triggered
   - On pull-request to `stg` or `prd`, trigger [terraform-plan.yml](https://github.com/unfor19/terraform-multienv/blob/dev/.github/workflows/terraform-plan.yml)
   - On push to `dev`, `stg`, `prd`, trigger [terraform-apply.yml](https://github.com/unfor19/terraform-multienv/blob/dev/.github/workflows/terraform-apply.yml)

3. Promote `dev` environment to `stg`
   - Create a PR from `dev` to `stg`
   - The plan to `stg` is added as a comment by the [terraform-plan](https://github.com/unfor19/terraform-multienv/blob/dev/.github/workflows/terraform-plan.yml) pipeline, here's [an example](https://github.com/unfor19/terraform-multienv/pull/10#issuecomment-691517521)
   - Merge the changes to `stg`, and check the [terraform-apply](https://github.com/unfor19/terraform-multienv/blob/dev/.github/workflows/terraform-apply.yml) pipeline in the Actions tab

## Alternatives

- [Terraform Cloud](https://www.terraform.io/docs/cloud/index.html) - Integration to git repository, the "environment" is referred as [Workspace](https://www.terraform.io/docs/cloud/workspaces/index.html)
- [terragrunt](https://terragrunt.gruntwork.io/) - Provides extra tools for keeping your configurations DRY, working with multiple Terraform modules, and managing remote state

## References

- [Getting started with Terraform in AWS
  ](https://www.youtube.com/watch?v=cBDmoC7QonA)
- [MVC design pattern in Terraform](https://dev.to/prodopsio/mvc-design-pattern-in-terraform-g99)
- [Terraform Dynamic Subnets](https://dev.to/prodopsio/terraform-aws-dynamic-subnets-2cgo)

## Final Words

You might have noticed I didn't mention anything about [scripts/terraform.sh](https://github.com/unfor19/terraform-multienv/blob/dev/scripts/terraform.sh). Since you've got this far, you deserve to know what it does
1. `terraform init`
1. `terraform plan` - save the plan to a markdown file `plan.md`
1. `terraform apply` - run only if the plan contains changes

I hope you find this useful, and if you do, don't forget to clap/heart and share it with your friends and colleagues.
Got any questions or doubts? Let's start a discussion! Feel free to comment below.

---
Originally published at [meirg.co.il](https://www.meirg.co.il/2020/12/10/infrastructure-as-code-with-terraform-for-multiple-environments/) on December 10, 2020