Speeding Up Your Terraform Projects with S3-backend Remote States
Tags: terraform, aws, bash, productivity
Published: 2021-08-05 17:09:23

**Update 8/6/2021**: Abstracted out the `Resource` property in the `s3_policy.json` file, since the script below handles replacing that. 🙂

Ever wonder if there's a quicker way to initialize your infrastructure with Terraform on a remote state? My go-to is the [S3 backend](https://www.terraform.io/docs/language/settings/backends/s3.html), but manually setting that up can take more time versus making a script to do the work for us.

Assuming you already have your AWS credentials setup already, we need to create two files. The first file is our S3 policy to let Terraform view the items, and to get and update our Terraform state.

Our S3 policy, `s3_policy.json`:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "RESOURCE",
      "Principal": {
        "AWS": "ARN"
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "RESOURCE/KEY",
      "Principal": {
        "AWS": "ARN"
      }
    }
  ]
}
```

So, if you're wondering about the `ARN`, `RESOURCE`, and `KEY`, that's okay. Our script will handle those by replacing those via `sed` magic, so when you change the bucket name in the bash script, you don't need to update the policy file. 😀 

The second file is our actual bash script, let's say `tf-setup.sh` (remember to `chmod +x` this file so you can execute it!):

```bash
#!/bin/bash

# Create S3 Bucket
MY_ARN=$(aws iam get-user --query User.Arn --output text 2>/dev/null)
aws s3 mb "s3://tf-remote-bucket" --region "us-east-1"
sed -e "s/RESOURCE/arn:aws:s3:::tf-remote-bucket/g" -e "s/KEY/terraform.tfstate/g" -e "s|ARN|${MY_ARN}|g" "$(dirname "$0")/s3_policy.json" > new-policy.json
aws s3api put-bucket-policy --bucket "tf-remote-bucket" --policy file://new-policy.json
aws s3api put-bucket-versioning --bucket "tf-remote-bucket" --versioning-configuration Status=Enabled
rm new-policy.json

# Create DynamoDB Table
aws dynamodb create-table \
  --table-name "tf-lock-table" \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1 \
  --region "us-east-1"
```

Now, let's configure our `main.tf`:

```hcl
terraform {
  required_version = ">= 1.0.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.29"
    }
  }

  backend "s3" {
    encrypt        = true
    bucket         = "tf-remote-bucket"
    dynamodb_table = "tf-lock-table"
    region         = "us-east-1"
    key            = "terraform.tfstate"
  }
}

provider "aws" {
  region = "us-east-1"
}
```

A `terraform init` outputs:

```
Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
```

You should be all set to run `terraform plan` and `terraform apply`! I hope this comes as a useful tool in your toolbox and/or pipeline. If you have any questions or run into any problems, please let me know in the comments!