How to use Terraform in AWS multi-account via AzureDevOps pipelines
Tags: devops, terraform, aws, azuredevops
Published: 2022-09-16 05:48:18

In this session, we will see how to provision infrastructure using **Terraform in AWS cloud multi-accounts, cross-account IAM roles, and AzureDevOps pipelines** to automate the deployment.

Let's break down the topics below

1. Architecture diagram
2. AWS multi-account creation setup
3. Terraform infrastructure code
4. AzureDevOps pipelines as code to automate deployment

## Architecture diagram

![AWS, Terraform, Azure DevOps](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z0tlt1efloj3sqix2jja.png)

From the above architecture diagram, let's list down the workflow. 

**Workflow 1**

We need to create AWS multi accounts, basically 3 accounts based on this example. The below steps need to be completed manually by creating via AWS console or AWS CLI commands.

**- Management/DevOps account**

1. In this account we will manage the IAM user(automation user for ex: **vscode**) and we will generate the secret key and access id for this user which will be used by Terraform

2. Assign an Administrator role or least privilege role by attaching only STS policy, so that the user 'vscode' can assume a role in trusting account like Development or Production account

**- Development Account**

1. Create a role for ex: ManageInDevAsAdmin with Administrator permission or least privilege that is required to manage this account and mention trusted account ARN number of Management/DevOps account

**- Production Account**

1. Create a role for ex: ManageInProdAsAdmin with Administrator permission or least privilege that is required to manage this account and mention trusted account ARN number of Management/DevOps account

Once these above steps are completed, Our terraform automation able to use those credentials for **vscode** and assume the IAM role to provision the resources in those different AWS accounts. 

**Workflow 2 and 3**
Azure DevOps tool has a complete solution to manage source code repository, pipelines to automation deployment, and much more. In this use case, we use both repository and pipelines. 

Azure DevOps Repository to keep the source code for Terraform infrastructure provisioning and Azure pipelines to execute the terraform commands.

**Pre-requisites:**
1. We need to upload a secure file in the Azure DevOps portal which will be used in pipelines for the Terraform to use the secret key and access id in the profile section. 

Let's use azure-pipelines.yml pipelines as a code to automate the deployment using Terraform. The code has stages for Dev and Prod and it contains terraform commands.

The credentials file looks like below.

![Credential file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d6lvi5likuhd8s5vexk9.png)

Secure file upload in AzureDevOps portal

![Secure file upload](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/au4qlo13alemroavj0vy.png)

azure-pipelines.yml file content. 

```javascript
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- main

pool:
  vmImage: ubuntu-latest

stages:    
  - stage: Dev
    displayName: Dev
    jobs:
      - job: plan
        pool:
          vmImage: ubuntu-latest
        displayName:  Terraform Dev Init and Apply
        steps:
        - task: DownloadSecureFile@1
          name: aws_credentials
          displayName: 'Download AWS Credentials'
          inputs:
            secureFile: 'credentials'
        - task: PowerShell@2
          displayName: 'New directory to keep the aws credentials'
          inputs:
            targetType: 'inline'
            script: |
              New-item -Path "~/" -Name ".aws" -ItemType "directory"
        - task: CopyFiles@2
          displayName: 'Copy aws credentials to home directory '
          inputs:
            SourceFolder: "$(Agent.TempDirectory)"
            contents: 'credentials'
            targetFolder: '~/.aws/'
        - task: PowerShell@2
          inputs:
            targetType: 'inline'
            script: 'terraform init -reconfigure -backend-config="dev-backend.conf"'
            workingDirectory: '$(Build.Repository.LocalPath)/saas_app'
          displayName: Terraform init for plan
        - task: PowerShell@2
          inputs:
            targetType: 'inline'
            script: 'terraform plan --var-file="dev-app.tfvars" -var createdBy="dev-pipeline" -var tags="asd" -out main.tfplan'
            workingDirectory: '$(Build.Repository.LocalPath)/saas_app'
          displayName: "Terraform Plan"
      - job: approve
        dependsOn: plan
        condition: and (succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
        pool: server
        displayName: Wait for approval
        steps:
        - task: ManualValidation@0
          timeoutInMinutes: 15 # task times out in 15 mins
          inputs:
            notifyUsers: saasproject86@outlook.com
            instructions: 'Please validate the build configuration and resume'
            onTimeout: 'reject'
      - job: apply
        dependsOn: approve
        pool:
          vmImage: ubuntu-latest
        displayName:  Terraform Dev Init and Apply
        steps:
        - task: DownloadSecureFile@1
          name: aws_credentials
          displayName: 'Download AWS Credentials'
          inputs:
            secureFile: 'credentials'
        - task: PowerShell@2
          displayName: 'New directory to keep the aws credentials'
          inputs:
            targetType: 'inline'
            script: |
              New-item -Path "~/" -Name ".aws" -ItemType "directory"
        - task: CopyFiles@2
          displayName: 'Copy aws credentials to home directory '
          inputs:
            SourceFolder: "$(Agent.TempDirectory)"
            contents: 'credentials'
            targetFolder: '~/.aws/'
        - task: PowerShell@2
          inputs:
            targetType: 'inline'
            script: 'terraform init -reconfigure -backend-config="dev-backend.conf"'
            workingDirectory: '$(Build.Repository.LocalPath)/saas_app'
          displayName: Terraform init for apply
        - task: PowerShell@2
          inputs:
            targetType: 'inline'
            script: 'terraform apply --var-file="dev-app.tfvars" -var createdBy="dev-pipeline" -var tags="asd" --auto-approve'
            workingDirectory: '$(Build.Repository.LocalPath)/saas_app'
          displayName: "Terraform Apply"
  - stage: Prod
    displayName: Prod
    jobs:
      - job: plan
        pool:
          vmImage: ubuntu-latest
        displayName: Terraform Prod Init and Apply
        steps:
        - task: DownloadSecureFile@1
          name: aws_credentials
          displayName: 'Download AWS Credentials'
          inputs:
            secureFile: 'credentials'
        - task: PowerShell@2
          displayName: 'New directory to keep the aws credentials'
          inputs:
            targetType: 'inline'
            script: |
              New-item -Path "~/" -Name ".aws" -ItemType "directory"
        - task: CopyFiles@2
          displayName: 'Copy aws credentials to home directory '
          inputs:
            SourceFolder: "$(Agent.TempDirectory)"
            contents: 'credentials'
            targetFolder: '~/.aws/'
        - task: PowerShell@2
          inputs:
            targetType: 'inline'
            script: 'terraform init -reconfigure -backend-config="prod-backend.conf" -no-color'
            workingDirectory: '$(Build.Repository.LocalPath)/saas_app'
          displayName: Terraform init for plan
        - task: PowerShell@2
          inputs:
            targetType: 'inline'
            script: 'terraform plan --var-file="prod-app.tfvars" -var createdBy="prod-pipeline" -var tags="asd"  -out main_prod.tfplan -no-color '
            workingDirectory: '$(Build.Repository.LocalPath)/saas_app'
          displayName: "Terraform Plan"
      - job: approve
        dependsOn: plan
        condition: and (succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
        pool: server
        displayName: Wait for approval
        steps:
        - task: ManualValidation@0
          timeoutInMinutes: 15 # task times out in 15 mins
          inputs:
            notifyUsers: saasproject86@outlook.com
            instructions: 'Please validate the build configuration and resume'
            onTimeout: 'reject'
      - job: apply
        dependsOn: approve
        pool:
          vmImage: ubuntu-latest
        displayName: Terraform Prod Apply
        steps:
        - task: DownloadSecureFile@1
          name: aws_credentials
          displayName: 'Download AWS Credentials'
          inputs:
            secureFile: 'credentials'
        - task: PowerShell@2
          displayName: 'New directory to keep the aws credentials'
          inputs:
            targetType: 'inline'
            script: |
              New-item -Path "~/" -Name ".aws" -ItemType "directory"
        - task: CopyFiles@2
          displayName: 'Copy aws credentials to home directory '
          inputs:
            SourceFolder: "$(Agent.TempDirectory)"
            contents: 'credentials'
            targetFolder: '~/.aws/'
        - task: PowerShell@2
          inputs:
            targetType: 'inline'
            script: 'terraform init -reconfigure -backend-config="prod-backend.conf" -no-color'
            workingDirectory: '$(Build.Repository.LocalPath)/saas_app'
          displayName: Terraform init for apply
        - task: PowerShell@2
          inputs:
            targetType: 'inline'
            script: 'terraform apply --var-file="prod-app.tfvars" -var createdBy="prod-pipeline" -var tags="asd" --auto-approve -no-color'
            workingDirectory: '$(Build.Repository.LocalPath)/saas_app'
          displayName: "Terraform Apply"
```

**Workflow 4 and 5**
Let's walk through the stages in the pipeline. 

**Dev stage job tasks**

1. Download a secure file

This task will download the secure file into the agent's temporary path

2. Copy secure file

This task will copy the file secure file from the temporary path to the agent home directory because we referred to this path in terraform provider section.

terraform `providers.tf` file content

```javascript
terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
  backend "s3" {}
}

provider "aws" {
  region                   = var.aws_region
  shared_credentials_files = ["~/.aws/credentials"]
  profile                  = "vscode"
  assume_role {
    role_arn = var.aws_assume_role_arn 
  }
}
``` 
**3. Terraform init**

For terraform initialization, we will use the S3 bucket as the backend state.

**Pre-requisites:**

- Create an S3 bucket for Development and Production AWS account to maintain the terraform state for those accounts

- Create dynamodb for Development and Production AWS account to maintain the terraform lock for those accounts

- Role name should be present in backend configuration

`dev-backend.conf` file content

```javascript

bucket         = "example-tf-dev-states"
key            = "example-dev-state/terraform.tfstate"
region         = "ap-southeast-2"
dynamodb_table = "example-tf-dev-locks"
encrypt        = true
profile        = "vscode"
role_arn = "arn:aws:iam::<AWS Development env account number >:role/ManagementInDEVAdminRole"
```

`prod-backend.conf` file content

```javascript

bucket         = "example-tf-prod-states"
key            = "example-prod-state/terraform.tfstate"
region         = "ap-southeast-2"
dynamodb_table = "example-tf-prod-locks"
encrypt        = true
profile        = "vscode"
role_arn = "arn:aws:iam::<AWS Development env account number >:role/ManagementInProdAdminRole"
```

**Terraform init command:**
`terraform init -reconfigure -backend-config="dev-backend.conf"`

Here we are using `-reconfigure` with the backend config, this will correctly set the terraform state 
s3 bucket for the corresponding AWS account based on the environment configuration file.

**4. Terraform plan**
Terraform plan is used to compare the state of the infrastructure and the current desired state from the code changes, It presents the changes that are going to happen in our infrastructure. 

**Terraform plan command:**
`terraform plan --var-file="dev-app.tfvars" -var createdBy="dev-pipeline" -var tags="asd" -out main.tfplan`

Here we need to pass Development environment-specific values 

`dev-app.tfvars` file content

```yaml
# Common
project = "example-project"

# General 
aws_region  = "ap-southeast-2"
profile     = "vscode"
environment = "dev"
aws_assume_role_arn = "arn:aws:iam::<Development AWS account number>:role/ManagementInDEVAdminRole"

# Terraform Backend State
backend_s3_bucket      = "example-tf-dev-states"
backend_dynamodb_table = "example-tf-dev-locks"
backend_s3_key = "example-dev-state/terraform.tfstate"

# VPC
create_vpc           = true
cidr                 = "10.123.0.0/16"
instance_tenancy     = "default"
enable_dns_hostnames = true
enable_dns_support   = true
create_igw           = true

# Subnet
create_subnet         = true
subnet_cidr_block = "10.123.1.0/24"
subnet_map_public_ip_on_launch = true
subnet_availability_zone = "ap-southeast-2a"
```
Once the plan is completed, we can see the result.


![Terraform plan](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/crd3nljt0s7ut4uf6nct.png)

**4. Azure DevOps Terraform plan review and approve or reject**
This approval gate helps us to review the terraform plan and understand the changes we are going to make in our infrastructure.


![Approval](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/30a43yplc8vw13hfsgpr.png)

**5. Terraform apply**
In Azure DevOps pipeline, we need to define the download, copy a secure file, and terraform init task because we are using a new job definition. 

**Terraform apply command**
`terraform apply --var-file="dev-app.tfvars" -var createdBy="dev-pipeline" -var tags="asd" --auto-approve`

Here we need to pass `auto-approve` because we already reviewed the terraform plan. 

Once the terraform apply is completed, we can see the AzureDevOps pipeline status.

![Terraform apply](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zfatencj26b0h4547vqf.png)


![Result](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5m1iqe3pcrqtf05kchto.png)

**NOTE:** For the Prod stage as well, we will be using the same concept and we need to pass different environment variable files.