Mastering AWS deployments with Terragrunt
Tags: iac, terragrunt, terraform, aws
Published: 2022-09-16 07:22:36

_Terraform offers a robust, declarative way to describe your cloud infrastructure as code. And unlike some other IaC tools, Terraform also does a decent job comparing differences between your current version of IaC code, the last deployment stored in Terraform state, and the current state of the deployed cloud resources. But when it comes to managing and deploying multiple copies of the same infrastructure, an additional tool is needed. And that tool has the name Terragrunt._

In this article, I assume that you understand the basics of Terraform. If you’d like to check the Terraform basics first, this article is for you (article in Finnish): https://www.nordhero.com/posts/google-cloud-terraformilla/.

Alright. So first, let’s get two acronyms right before moving forward.

**IaC** means Infrastructure as Code. The idea is that you don’t have to manually log in to the AWS Console and set up the infrastructure by selecting services in the console and clicking the systems up. The manual approach could be acceptable if you only had one environment for testing purposes. But if you would need to set it up again, or if you had more than one environment that should have the same resources and configurations (e.g., development, testing, and production environments), you shouldn’t try to manage those manually. Instead, you probably would like to use an IaC tool like Terraform to set up the infrastructure configurations as code to be deployed quickly and repeatably.

**DRY** means Don’t Repeat Yourself. The downside of a declarative language like Terraform is that it’s not that easy to manage variations of the same code if you need the deploy the same infrastructure with a few different flavors depending on the use case. You quickly end up making multiple copies of the infrastructure code to manage various similar kinds of deployments. And that’s what Terragrunt is here to solve.

## How to keep Terraform code DRY?

The key idea of Terragrunt is to write needed infrastructure code only once utilizing Terraform (TF files) and to separate environment-specific values as variables to be defined in Terragrunt configuration files (HCL files). I prefer a folder structure where I have the Terraform infrastructure code in the project’s infrastructure folder and Terragrunt configurations in the deployments folder.

A typical Terragrunt configuration setup is to save environment-specific configuration files in a three-level folder structure that describes the AWS Accounts, AWS Regions under the accounts, and deployable environments under the regions. Each folder level can contain Terragrunt configuration files for account, region, or environment-specific variables.

The following figure illustrates an imaginary deployments folder setup where one production account has production environments both in the us-east-1 (N. Virginia) and eu-central-1 (Frankfurt) regions and an additional demo environment in the eu-central-1 region. There’s also a staging environment on its own account utilizing the eu-central-1 region and a dev account with a similar setup. As I’m working with both feature development and performance testing, I have two environments deployed on my sandbox account. The sandbox-pekka-perftest environment has production-kind infrastructure resources configured on it — for example, having larger Fargate clusters or heavier EC2 instances with provisioned IOPS SSD volumes. As the performance testing environment has more resources, it also generates more costs. Therefore it is easy for me to tear down the stack when the test session ends and re-deploy it again when needed without affecting the standard sandbox environment residing on the same account and region.


![An example deployment folder structure for Terragrunt](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7b6eypdt23ekffbiop1m.jpg)
_An example deployment folder structure for Terragrunt_

## Install Terraform and Terragrunt

First, install Terraform on your desktop. I have macOS, and I like to use the Homebrew package manager (https://brew.sh/) for the job, so the commands below use **brew**. There are multiple ways to install the Terraform on different operating systems, and you can find the right ingredients from https://www.terraform.io/.

Here are the magic commands for macOS/Homebrew installation of Terraform:

```
brew tap hashicorp/tap
brew install hashicorp/tap/terraform
terraform -version
```

If you get a Terraform version info, you have succeeded in the installation.

And next one goes for Terragrunt (more installation options at https://terragrunt.gruntwork.io/):

```
brew install terragrunt
terragrunt -version
```

If you get the Terragrunt version number, you are good to go to the next phase.

## The infrastructure with Terraform IaC

Let’s first create our infrastructure code. In our example I have a small stack including only one S3 Bucket with data encryption utilizing a customer-managed KMS Key. I have included only a few lines of Terraform code here to point out how Terraform works with Terragrunt. If you wish to try out with my example code, you can find the whole Terraform code example in the [GitHub repository](https://github.com/pmalmirae/terragrunt-demo).

First, create a new folder on your computer for our project (you can name it as you wish) and create a folder **infrastructure** in the project root. Under the newly created folder, add all needed Terraform files. For example here is the code for the S3 bucket configurations (in file **infrastructure/s3.tf**):

```
/******************************
S3 bucket with encryption and public access block configurations 
******************************/
# The S3 bucket
resource "aws_s3_bucket" "demo_bucket" {
  bucket_prefix = "${var.name}-${var.environment}-demo-bucket"
}
# Let's make the bucket private
resource "aws_s3_bucket_acl" "demo_bucket_acl" {
  bucket = aws_s3_bucket.demo_bucket.id
  acl    = "private"
}
/******************************
More configurations in the actual file, please check the Github repo.
******************************/
```

As you see, I have used Terraform variables **name** and **environment** in the bucket’s prefix. That will let us automatically change the bucket prefix per environment. And as you might already guess, those variables are managed with Terragrunt.

In addition to utilizing variables in Terraform code, we need to define the variables from Terraform perspective. Create a file named **infrastructure/vars.tf**:

```
/******************************
Variables to be used with the infrastructure code
******************************/
variable "name" {
  type        = string
  description = "Name of the company or the platform to build, etc." 
}
variable "environment" {
  type        = string
  description = "Name of the environment/stack"
}
/******************************
More variables in the actual file, please check the Github repo
******************************/
```

Next to the main course!

## Set up Terragrunt configurations

Let’s start the Terragrunt part by creating a folder named deployments in the project root. As you remember from the DRY chapter, a typical Terragrunt configuration set has a three-level folder structure: _account/region/environment_. So, create the folder structure under the **deployments** folder for your first environment. In my example, I have created a folder structure **deployments/sandbox-pekka/eu-central-1/sandbox-pekka**.

The **deployments/terragrunt.hcl** file is a key configuration file for Terragrunt. Please go ahead and create the file with the following contents:

```
/******************************
TERRAGRUNT CONFIGURATION
******************************/
locals {
  # Load account, region and environment variables 
  account_vars      = read_terragrunt_config(find_in_parent_folders("account.hcl"))
  region_vars       = read_terragrunt_config(find_in_parent_folders("region.hcl"))
  environment_vars  = read_terragrunt_config(find_in_parent_folders("env.hcl"))
  # Extract the variables we need with the backend configuration
  aws_region      = local.region_vars.locals.aws_region
  environment     = local.environment_vars.locals.environment
  state_bucket    = local.environment_vars.locals.state_bucket
  dynamodb_table  = local.environment_vars.locals.dynamodb_table
}
/******************************
Configure the Terragrunt remote state to utilize a S3 bucket and state lock information in a DynamoDB table. 
And encrypt the state data.
******************************/
remote_state {
  backend   = "s3"
  generate  = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config    = {
    bucket         = "${local.state_bucket}"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "${local.aws_region}"
    encrypt        = true
    dynamodb_table = "${local.dynamodb_table}"
  }
}
/******************************
Combine all account, region and environment variables as Terragrunt input parameters.
The input parameters can be used in Terraform configurations as Terraform variables.  
******************************/
inputs = merge(
  local.account_vars.locals,
  local.region_vars.locals,
  local.environment_vars.locals,
)
```

The **deployments/terragrunt.hcl**

- loads account, region, and environment level variables (we will create those files next)
- extracts the variables that are needed in Terragrunt backend configurations
- configures the Terragrunt backend to utilize the state bucket and state lock table
- merges all variables as input parameters to be fed to Terraform.

Next, let’s create the needed files for Terragrunt variables. Please create the following .hcl files and replace the folder names and variable values with your own _account/region/environment_ information:

**deployments/sandbox-pekka/account.hcl**:

```
# Set AWS Account -wide variables
locals {
  account_name   = "sandbox-pekka"
  aws_account_id = "REPLACE_WITH_YOUR_ACCOUNT_ID"
}
```

**deployments/sandbox-pekka/eu-central-1/region.hcl**:

```
# Set common variables for the AWS Region
locals {
  aws_region = "eu-central-1"
}
```

**deployments/sandbox-pekka/eu-central-1/sandbox-pekka/env.hcl**:

```
# Set common variables for the environment
locals {
  name           = "nordhero"
  environment    = "sandbox-pekka"
  state_bucket   = "nordhero-terragrunt-demo-state-sandbox-pekka" # Replace with your preferred unique S3 bucket name 
  dynamodb_table = "nordhero-terragrunt-demo-locks-sandbox-pekka" # Replace with your preferred dynamodb table name
}
```

The **name** and **environment** variables will be utilized in the Terraform code created in the chapter _The infrastructure with Terraform IaC_. And the **state_bucket** and **dynamodb_table** values will be used to create the Terragrunt state bucket and state lock table for the environment.

And one last thing to the Terragrunt configurations. Create one more folder and file: **deployments/sandbox-pekka/eu-central-1/sandbox-pekka/infra/terragrunt.hcl**:

```
/******************************
TERRAGRUNT CONFIGURATIONS
******************************/
/******************************
Include the root terragrunt.hcl configurations gathering together
the needed variables and backend configurations
******************************/
include "root" {
  path = find_in_parent_folders()
}
locals {
  # Expose the base source path
  base_source = "${dirname(find_in_parent_folders())}/..//infrastructure"
}
# Set the location of Terraform configurations
terraform {
  source = local.base_source
}
```

This is the Terragrunt configuration file that we will execute later on. What it does is that it first takes in the common **terragrunt.hcl** configurations we created in the **deployments** folder and then configures the Terraform infrastructure folder to use with the deployment. As you see, it is possible to configure different versions of Terraform code, having different **base_source** to use with different environments (not that DRY approach, thou). Or, more importantly, you could split your Terraform code into multiple modules and select which modules to deploy to this particular environment. For example, if your full stack includes the OpenSearch service (formerly ElasticSearch), but you don’t need OpenSearch in your development sandbox, you could choose to deploy all other modules but not to deploy the module that contains OpenSearch configurations.

Now we are ready. You should now have the following kind of folder structure and following files in your **deployments** folder:

- deployments/terragrunt.hcl
- deployments/sandbox-pekka/account.hcl
- deployments/sandbox-pekka/eu-central-1/region.hcl
- deployments/sandbox-pekka/eu-central-1/sandbox-pekka/env.hcl
- deployments/sandbox-pekka/eu-central-1/sandbox-pekka/infra/terragrunt.hcl

## Deploying with Terragrunt

Deploying the infrastructure with Terragrunt is very similar to deploying with Terraform. You have the same commands in use. The main difference is that you need to run the Terragrunt commands from the respective **deployments/your_account/your_region/your_env/infra** folder you wish to deploy. So please cd to your infra folder:

```
cd deployments/sandbox-pekka/eu-central-1/sandbox-pekka/infra
```

Before running any Terragrunt commands, we must ensure we have successfully connected to the right AWS Account with AWS CLI. If you don’t already have the AWS CLI configured, please follow the instructions: http://docs.aws.amazon.com/cli/latest/userguide/. After configuring AWS CLI, test the connection by running the following command:

```
aws sts get-caller-identity
```

You should get a response containing your UserId, AWS Account Id, and an IAM Role ARN if successfully connected. Check once more that the Account is the one you desire to deploy infrastructure to and that you have the same information in your **deployments/your_account/account.hcl** file.

Next, we are ready to rock ‘n roll. Let’s first initialize the Terragrunt:

```
terragrunt init
```

When running the init for the first time, Terragrunt recognizes that the S3 state bucket does not yet exist and asks whether Terragrunt should create the bucket for you. Please allow Terragrunt to create the state bucket. Terragrunt will automatically create also the state lock DynamoDB table at the same time. When everything is ready, you should receive a green response stating that the backend has been successfully configured to use S3 and that Terraform has been successfully initialized to use the hashicorp/aws provider plugin.

Next, let’s plan our deployment and save the plan in the **tfplan** file:

```
terragrunt plan -out tfplan
```

Terragrunt will now describe the resources it plans to create. Please check that there are six resources to be added and that no errors or warnings have been raised. If everything looks ok, we can next deploy the infrastructure plan saved in the **tfplan** file:

```
terragrunt apply "tfplan"
```

You should now get a green message stating “Apply complete!” with the number of resources created, changed and destroyed.

Congratulations! You have now successfully set up Terragrunt and deployed your first infrastructure stack.

## Deploying the infrastructure to another region

Now to the desserts. The benefits of Terragrunt start accumulating when setting up the next copy of the infrastructure. Let’s assume we would like to set up the same infrastructure for the same account but in a different region. The greatness is that we don’t need to touch the **infrastructure/*.tf** files at all.

What we need to do is:

1. Copy the current **deployments/your_account/your_region** folder and rename the copied folder with the new region name, e.g. **eu-north-1**
2. Edit the **deployments/your_account/new_region/region.hcl** file and replace **aws_region** value with the new region name
3. Edit **deployments/your_account/new_region/your_environment/env.hcl** file and replace the **state_bucket**, and **dynamodb_table** values with new bucket and table names to store the state of the new environment
4. Cd to the **deployments/your_account/new_region/your_environment/infra** folder and repeat the **terragrunt init/plan/apply** commands

Now try it yourself!

## Last lines

That was a nice ride! We needed to create a bunch of configuration files. Still, in a typical development project where you have multiple sandbox environments and a deployment pipeline with development, staging, production, and demo environments, it starts to pay off quickly. And in real life, you would probably like to automate the deployment pipeline from your source code repository so that, depending on the repository branch, you would automatically select the deployment folder to use with Terragrunt deployment.

NordHero is there to help you set up the infrastructure, manage the multi-environment platforms and automate the deployments with your selected GitOPS platform. Give us a call/email/LinkedIn message if you would like to hear more!

P.S. You can download the whole demo project in the [Github repository](https://github.com/pmalmirae/terragrunt-demo).