Exploring GCP With Terraform: Adding Terragrunt
Tags: terraform, terragrunt, tutorial, devops
Published: 2023-09-20 08:44:36

Finally, this is the last step in making the Terraform project manageable. I'll add [Terragrunt](https://terragrunt.gruntwork.io/) to the project. Terragrunt is a thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules. It's an excellent tool for making your Terraform code DRY and reusable.

I'm starting with the same project as in [the previous post](https://rnemet.dev/posts/gcp/gcp_tf_refactor/). The current state of [code is available here](https://github.com/robert-nemet/pcne/tree/add_modules) in the branch `add_modules`.

## Prerequisites

Well, you need to install Terragrunt. You can find installation instructions [here](https://terragrunt.gruntwork.io/docs/getting-started/install/). That is all.

## Adding Terragrunt to the back-office network and vms modules

I'm starting with the `back-office` module. What I do in this module will be easily copied to other modules.

Notice that common code in `back-office/vms` and `back-office/network` is related to the provider and state file configurations. This code is common for all modules. The only thing that differs is the `backend.prefix`. Its value differs from module to module.

So, the first thing is to make a global configuration on the `dev` level. To do that, I'm creating a `terragrunt.hcl` file in the `dev` folder:

```hcl
generate "provider" {
    path = "provider.tf"
    if_exists = "overwrite_terragrunt"
    contents = <<EOF
provider "google" {
  project = var.project_id
  region  = var.region
  zone    = var.zone
}

terraform {
  required_version = ">=1.5.7"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.77.0"
    }
  }
}
EOF
}

remote_state {
    backend = "gcs"
    generate = {
        path = "backend.tf"
        if_exists = "overwrite_terragrunt"
    }
    config = {
        bucket = "terraform-states-network-playground-382512"
        prefix = "terraform/state/dev/${path_relative_to_include()}"
    }
}

terraform {
    extra_arguments "common" {
        commands = get_terraform_commands_that_need_vars()

        arguments = [
            "-var-file=${path_relative_from_include()}/common.tfvars",
        ]
    }
}
```

This file has three sections. The first section is `generate "provider"`. This section will generate a `provider.tf` file in each module. Suppose it already exists and is managed by Terragrunt. In that case, it will be re-generated if there are any changes to the configuration. This file will contain provider configuration.

The second section is `remote_state`. This section will generate a `backend.tf` file in each module. This file will contain state file configuration. Notice that `prefix` is set to `terraform/state/dev/${path_relative_to_include()}`. This setting will store the state file in the `terraform/state/dev` folder, and each module will have its subfolder. That is important because each module will have its state file.

The third section is `terraform`. This section will add extra arguments to Terraform commands. In this case, it will add `-var-file=${path_relative_from_include()}/common.tfvars`. So, when I run any Terraform command requiring variables, Terragrunt will add this argument. That allows me to have one `common.tfvars` file in the `dev` folder, and all modules will use it.

Now, I need to add the `common.tfvars` file to the `dev` folder:

```hcl
project_id = "network-playground-382512"
region = "europe-west1"
zone = "europe-west1-b"
```

I need to add a `terragrunt.hcl` file to each module to use this. I'm adding it to the `back-office` module:

```hcl
include {
    path = find_in_parent_folders()
}
```

Now, I can remove the `provider.tf` and `terraform.tfvars` files from the `back-office` module. This time, I'll be running `terragrunt plain` instead of `terraform plan` in the `back-office` module:

```bash
terragrunt plan
Acquiring state lock. This may take a few moments...
google_service_account.back_office_fw_sa: Refreshing state... [id=projects/network-playground-382512/serviceAccounts/back-office@network-playground-382512.iam.gserviceaccount.com]
module.back-office.module.vpc.google_compute_network.network: Refreshing state... [id=projects/network-playground-382512/global/networks/back-office]
module.back-office.module.subnets.google_compute_subnetwork.subnetwork["us-central1/back-office-private"]: Refreshing state... [id=projects/network-playground-382512/regions/us-central1/subnetworks/back-office-private]
module.back-office.module.subnets.google_compute_subnetwork.subnetwork["us-central1/back-office"]: Refreshing state... [id=projects/network-playground-382512/regions/us-central1/subnetworks/back-office]
module.back-office.module.firewall_rules.google_compute_firewall.rules_ingress_egress["back-office-icmp"]: Refreshing state... [id=projects/network-playground-382512/global/firewalls/back-office-icmp]
module.back-office.module.firewall_rules.google_compute_firewall.rules_ingress_egress["back-office-iap"]: Refreshing state... [id=projects/network-playground-382512/global/firewalls/back-office-iap]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
```

Running the `terragrunt plan` in the `back-office/network` module will generate two files: `provider.tf` and `backend.tf`, according to the `terragrunt.hcl` file in the `dev` folder.

For the `back-office/vms` module, I need to add the `terragrunt.hcl` file to the `back-office/vms` folder:

```hcl
include "root" {
  path = find_in_parent_folders()
}

dependencies {
  paths = ["../network"]
}

dependency "network" {
  config_path = "../network"
}

inputs = {
  vpc_back_office_id = dependency.network.outputs.vpc_back_office_id
  vpc_back_office_subnetwork = dependency.network.outputs.vpc_back_office_subnetwork
  back_office_fw_sa = dependency.network.outputs.back_office_fw_sa
}
```

Now, this `terragrunt.hcl` not only includes files from parent folders but also defines dependency on the `../network` module. That means that the `back-office/vms` module depends on the `back-office/network` module. A network needs to be created before virtual machines. That is important because the `back-office/vms` module needs to know some outputs from the `back-office/network` module. The section `inputs` defines inputs for the `back-office/vms` module. These inputs are outputs from the `back-office/network` module. The last thing allows me to remove in addition to `provider.tf` and `terraform.tfvars`, and `inputs.tf` files from the `back office/vms` module.

I need to add additional variables to the `variables.tf` file in the `back-office/vms`:

```hcl
variable "back_office_fw_sa" {
  description = "back office firewall service account"
  type        = string
}

variable "vpc_back_office_id" {
  description = "back office vpc id"
  type        = string
}

variable "vpc_back_office_subnetwork" {
  description = "back office vpc subnetwork"
  type        = any
}
```

Running `terragrunt plan` should give me no changes needed. Also, I can now run `terragrunt apply` instead of `terraform apply` in `back-office/vms` module.

What I did in the `back-office` module, I need to do in the `service` and `strorage` modules. Simple. Remember to properly define dependencies and inputs in other `vms` modules.

When it comes to the `peering` module, I need to add the `terragrunt.hcl` file to the `peering` folder:

```hcl
include "root" {
  path = find_in_parent_folders()
}

dependencies {
  paths = ["../back-office/network", "../services/network", "../storage/network"]
}

dependency "back-office" {
  config_path = "../back-office/network"
}

dependency "services" {
  config_path = "../services/network"
}

dependency "storage" {
  config_path = "../storage/network"
}

inputs = {
  vpc_back_office = dependency.back-office.outputs.vpc_back_office
  vpc_services = dependency.services.outputs.vpc_services
  vpc_storage = dependency.storage.outputs.vpc_storage
}
```

That will ensure that the `peering` module will be created after the `back-office`, `services`, and `storage` modules. Also, it will provide inputs for the `peering` module. These inputs are outputs from `back-office`, `services`, and `storage` modules.

Interesting would be generating a graphical representation of the dependencies:

```bash
terragrunt graph-dependencies | dot -Tpng > dependencies.png
```

![dependencies](/images/dependencies.png)

The `terragrunt graph-dependencies` command will generate a graph of the dependencies between modules. The `dot -Tpng > dependencies.png` command will convert the graph to a PNG file, that is [graphviz](https://graphviz.org/). That is a great way to visualize dependencies between modules.

Try from `dev` folder `terragrunt run-all plan`. The Terragrunt will split the modules into groups. In each group, it will run `plan`, but first, it will run it in `network` modules because they are dependencies for other modules. Then, it will run `plan` in parallel for other modules in the other group.

## Conclusion

Adding Terragrunt to the project is relatively easy. It requires some changes to the project structure and the code. But it is worth it. It makes the project more manageable: code is DRY, and common code is generated in one place. I can run Terraform commands from the root folder and do not need to visit each module, etc...

## References

- [Terragrunt](https://terragrunt.gruntwork.io/)