CI/CD Workflow for AWS ECS via Terragrunt and GitHub Actions
Tags: terraform, terragrunt, github, aws
Published: 2020-12-29 13:28:42

This project leverages [Terragrunt](https://github.com/gruntwork-io/terragrunt), [Terraform](https://www.terraform.io/), and [GitHub Actions](https://github.com/features/actions) to deploy a basic web app (dockerized JS frontend and dockerized Python API) to [AWS ECS](https://aws.amazon.com/ecs/).

Check out the full source code on GitHub: https://github.com/visini/terragrunt-github-actions-aws-ecs

## Initial Setup

[Terragrunt](https://github.com/gruntwork-io/terragrunt) is a thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules, remote state, and locking. It also provides a powerful and flexible way to hierarchically provide configuration to Terraform, without duplicating code across environments, AWS regions, and AWS accounts – [keeping your Terraform config DRY](https://blog.gruntwork.io/terragrunt-how-to-keep-your-terraform-code-dry-and-maintainable-f61ae06959d8?gi=703957a5f669).

![Managing Terraform config across accounts, regions, and environments with Terragrunt](https://dev-to-uploads.s3.amazonaws.com/i/ihwlfpf9l0d6zpvbub3c.jpg)
 
The following hierarchy is proposed (aligned with directory structure):

- `terragrunt.hcl` with configuration for remote_state and AWS provider
- `common.terragrunt.hcl` defining common, project-specific variables
  - `account.terragrunt.hcl` for each account
    - `region.terragrunt.hcl` for each region within an account
      - `environment.terragrunt.hcl` for each environment within a region

This allows flexible configuration, just add additional folders and adjust the configuration files, for instance configuring...

- Accounts `main` and `secondary`
- Regions `eu-west-1` and `us-east-1` in `main` vs. `us-east-1` in `secondary`
- Environments `prod` in `main` regions vs. `stage` and `dev` in `secondary` regions

## Workflow via GitHub Flow

This project leverages [GitHub Flow](https://guides.github.com/introduction/flow/) for gradually merging changes existing on experimental branches and deployed to experimental environments, towards more mature branches and environments.

![Workflow and Deployment – GitHub Flow](https://dev-to-uploads.s3.amazonaws.com/i/3577466bemvsmljym1ks.jpg)
 
The [companion repository](https://github.com/visini/terragrunt-github-actions-aws-ecs) contains functionality to deploy code to AWS ECS simply by adopting GitHub Flow principles. All integration and deployment steps are managed by GitHub Actions workflows, including: Unit testing, building and pushing Docker images, and releasing new images to the correct ECS cluster via Terraform and Terragrunt. Create a branch, push, create a pull-request, and, after verifying checks, merge all changes - these are the only steps needed to deploy new features by adopting this approach.

Assuming a running staging and production environment, here's how to deploy changes made for a recent feature "foo" to staging and production environments:

**Step 1** - Deployment to staging environment `stage` via branch `dev`

- Create a new branch `feature/foo` and check it out
  - `git checkout -b feature/foo`
- Push to remote and set up to track remote branch `feature/foo` from `origin`
  - `git push --set-upstream origin feature/foo`
- Open pull request from branch `feature/foo` to branch `dev` to plan deployment to `stage` environment
- Wait for checks to complete: Workflow `terragrunt` will post terraform `plan` as a comment to pull request
- Additional checks may include unit tests: See `pytest-api.yml`
- After verifying terraform `plan`, merge pull request into branch `dev`
- Workflow `terragrunt` will run again and `apply` deployment for `stage` environment
- Code (and infrastructure) changes to branch `feature/foo` are now released to `stage` environment

**Step 2** - Deployment to production environment `prod` via branch `main`

- After verifying deployment to `stage` environment (e.g., e2e-testing), open pull request from branch `dev` to branch `main` to plan deployment to `prod` environment
- Wait for checks to complete: Workflow `terragrunt` will post terraform `plan` as a comment to pull request
- After verifying terraform `plan`, merge pull request into branch `main`
- Workflow `terragrunt` will run again and `apply` deployment for `prod` environment
- Code (and infrastructure) changes to branch `dev` originating from branch `feature/foo` are now released to `prod` environment

## Configure Infrastructure and Deployment Targets

The hierarchical configuration via Terragrunt is enabled by a main configuration file in which all other more granular configuration files are imported. In `terragrunt.hcl`, both remote state and AWS provider are defined according to values in more specific configuration files.

Both remote state and provider are dynamically defined for each deployment target (e.g., `prod` vs. `stage` environment) and AWS account ID (e.g., `main` vs. `secondary` account). This means, `prod` and `stage` environments (which may even be residing on two separate AWS accounts) adopt separate remote state backend configurations, depending on which environment subfolder `terragrunt` commands are executed from. Review this file and the nested Terragrunt configuration files in the [companion repository](https://github.com/visini/terragrunt-github-actions-aws-ecs) for the detailed implementation.

```hcl
locals {
    common = read_terragrunt_config(find_in_parent_folders("common.terragrunt.hcl"))
    account = read_terragrunt_config(find_in_parent_folders("account.terragrunt.hcl"))
    region = read_terragrunt_config(find_in_parent_folders("region.terragrunt.hcl"))
    environment = read_terragrunt_config(find_in_parent_folders("environment.terragrunt.hcl"))
}

remote_state {
    backend = "s3"
    # remote_state dynamically configured based on:
    # local.region.locals.aws_region
    # local.account.locals.aws_account_id
    # local.common.locals.app_name

    # ...
}

generate "provider" {
    # AWS provider dynamically configured based on:
    # local.region.locals.aws_region
    # local.account.locals.aws_account_id

    # ...
}

# The following variables apply to all configurations in this subfolder
# and are automatically merged into the child `terragrunt.hcl` config
# via the include block.

inputs = merge(
    local.common.locals,
    local.account.locals,
    local.region.locals,
    local.environment.locals
)
```

Common variables, such as the app name, base domain name and hosted zone name, which apply to the project are configured in a separate file:

```hcl
# The following define common variables for the project.
# They are automatically pulled in in the root terragrunt.hcl
# configuration to feed forward to the child modules.

locals {
    app_name = "example-app"
    app_domain_name = "app.example.com"
    route53_hosted_zone_name = "example.com"
    use_existing_route53_hosted_zone = true
    github_sha = "will_be_automatically_set_by_github_actions_or_manual_script"
}
```

One or any number of AWS accounts may be configured, each related to any number of regions and environments to be deployed via this AWS account:

```hcl
# Set account-specific variables. They are automatically
# pulled in to configure the remote state bucket in the root
# terragrunt.hcl configuration.

locals {
  account_name   = "main"
  aws_account_id = "123456789000"
  aws_profile    = "default"
}
```

Similarly, the region configuration is provided in the nested level:

```hcl
# Set region-specific variables. They are automatically
# pulled in to the root terragrunt.hcl configuration to
# feed forward to the child modules.

locals {
    aws_region = "eu-west-1"
}
```

Environment configuration regarding both infrastructure and containers are provided in the nested level. To illustrate a common use case: The `stage` environment may override variables previously defined in the parent-hierarchy, in `tf/common.terragrunt.hcl`, and for instance add a prefix `stage.*` to `app_domain_name`.

```hcl
# Set environment-specific variables. They are automatically
# pulled in to the root terragrunt.hcl configuration to
# feed forward to the child modules.

locals {
    common = read_terragrunt_config(find_in_parent_folders("common.terragrunt.hcl"))
    account = read_terragrunt_config(find_in_parent_folders("account.terragrunt.hcl"))
    region = read_terragrunt_config(find_in_parent_folders("region.terragrunt.hcl"))

    # Configure environment
    environment = "stage"
    app_domain_name = "stage.${local.common.locals.app_domain_name}"

    app_name = local.common.locals.app_name
    aws_account_id = local.account.locals.aws_account_id
    aws_region = local.region.locals.aws_region

    parameter_group = "${local.app_name}/${local.environment}"

    service_configuration = {
        # ...
    }
}
```

## Configure Container Environment and Secrets

Environment variables for the respective deployment target (e.g., for `stage` environment) are provided alongside terragrunt configuration in JSON files, following the naming `.service.environment.json`, and by specifying both keys and values. These files are committed to source control, since they do not contain any sensitive data. Adding a description key-value pair will inform development and ensure consistency of variable assignment.

```json
{
  "DEBUG": {
    "value": "true",
    "description": "API debug environment"
  }
  // ...
}
```

Similarly, JSON files following the naming `.service.secrets.json` provide _keys_ (not values) for all container secrets, which are injected by AWS Systems Manager Parameter Store into the service's ECS tasks (containers) as _environment variables_. Adding a description key-value pair will inform development and ensure consistency of variable assignment.

No secrets are present in code or source control — secrets such as database passwords or secret keys are generated as terraform resources, and stored within Systems Manager Parameter Store. Following the [design of terraform state](https://www.terraform.io/docs/state/sensitive-data.html), they are additionally stored within remote state backend. While S3 backend supports encryption at rest, remote state is to be considered sensitive data.

```json
{
  "SECRET_KEY": {
    "description": "Secret key required for API JWT authentication"
  }
  // ...
}
```

## Integration via GitHub Actions – Pytest

Easily add continuous integration workflows for unit, integration, and e2e tests by adding a GitHub Action workflow. The [companion repository](https://github.com/visini/terragrunt-github-actions-aws-ecs) includes an example for testing the API service via Pytest.

```yaml
name: Pytest API

on:
  pull_request:

jobs:
  # ----------------------------------------------------------------
  # Checkout code, install dependencies, run pytest for API
  # ----------------------------------------------------------------
  pytest: #...
```

## Deployment via GitHub Actions – Terragrunt

To support branch-aware continuous deployment of code to the respective environment, a GitHub Actions workflow is provided.

The workflow is based on [Terraform's guide](https://learn.hashicorp.com/tutorials/terraform/github-actions) for automating Terraform with GitHub Actions, and adds support for nested configuration via Terragrunt. With this workflow, pull-requests will trigger `terraform plan`, and merging these pull requests will trigger `terraform apply` to deploy to the correct environment: Branch `dev` will deploy to environment `stage` and branch `main` will deploy to environment `prod` – all following the specified hierarchical configuration defined in `*.terragrunt.hcl` files. Review this file in the [companion repository](https://github.com/visini/terragrunt-github-actions-aws-ecs) for the detailed implementation.

```yaml
name: Terragrunt

on:
  push:
    branches:
      - main
      - dev
  pull_request:

jobs:
  # ----------------------------------------------------------------
  # Checkout code, setup target_push and tf_env
  # Configure AWS credentials via secrets in repo settings
  # ----------------------------------------------------------------
  setup: #...
  # ----------------------------------------------------------------
  # Push to ECR if target_push
  # Tag image with github.sha of commit
  # ----------------------------------------------------------------
  ecr: #...
  # ----------------------------------------------------------------
  # Terraform for deployment targets
  # Based on tf_env (pull request -> plan; push to branch -> apply)
  # ----------------------------------------------------------------
  terraform: #...
```

## Conclusion

This article describes a flexible CI/CD workflow for AWS ECS based projects. Using GitHub Flow, changes to the infrastructure and/or codebase are deployed to the intended deployment targets.

Thanks for reading. I'm curious to hear your thoughts on this topic – don't hesitate to reach out to me on Twitter or start a discussion in the [companion repository](https://github.com/visini/terragrunt-github-actions-aws-ecs) on GitHub!