Speed up and save cost for Terraform Deployments with AWS CodeBuild's Lambda Compute
Tags: terraform, aws, codebuild, finops
Published: 2024-03-21 04:04:37

> _This article is all about how you can make your **terraform deployments faster** and **reduce your AWS billing** using AWS **CodeBuild's Lambda Compute**._
>
> _This article will help those who are using Terraform for their IaC on AWS and are aiming to optimise the AWS infrastructure provisioning process._ 
>
> _I wanted to go ahead for my readers so I have shared a strategy using the concepts of this blog which will help them to make their CI/CD pipelines better._
 

## Motivation

- AWS has recently released **two new updates** related to CodeBuild and these updates were the reason for coming with this blog.

- **[ Nov 6, 2023 ]** [AWS CodeBuild now supports AWS Lambda compute](https://aws.amazon.com/about-aws/whats-new/2023/11/aws-codebuild-lambda-compute/)

  - **What does that mean**: **_AWS CodeBuild users now have the option to employ AWS Lambda for building and testing their software packages._** 
  - This new compute mode choice offers **quicker builds thanks to Lambda's almost instantaneous start-up times**. Additionally, customers can benefit from **cost savings since Lambda compute is billed per second of usage**. 

> Until now, The only compute option was to use Ec2 as compute but with this update we can use LAMBDA as compute with CodeBuild.

- **[ Mar 19, 2024 ]** [AWS CodeBuild now supports custom images for AWS Lambda compute](https://aws.amazon.com/about-aws/whats-new/2024/03/aws-codebuild-custom-images-lambda-compute/)

  - **What does that mean**: **AWS CodeBuild has enhanced its capabilities by enabling the utilization of container images stored in Amazon ECR repositories for projects configured to operate on Lambda compute**. 
  - Previously, users were restricted to leveraging managed container images supplied by AWS CodeBuild. These AWS-managed container images come equipped with support for tools like AWS CLI, AWS SAM CLI, and a range of programming language runtimes.

> Until now, With LAMBDA as compute there was only the option to use AWS-managed images but now we can use custom images for LAMBDA as compute with CodeBuild.


## Prerequisites

- Knowledge of Terraform
- Basic working of CodeBuild
- Understanding of core AWS and its operations

## Let's get started.

### Create ECR repository

- As mentioned in the update we need to create an ECR repository to host custom images. I will name it as `terraform`.

![Terraform ecr repository](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yidml8kgo80237rz21fl.png)

### Pull the Terraform docker image 

- There is an [official terraform docker image](https://hub.docker.com/r/hashicorp/terraform/tags) which I use as a custom image and push to ecr repository.

```sh

docker pull hashicorp/terraform  

```

### Push image to ECR repository

- Retrieve an authentication token and authenticate your Docker client to your registry. _(change account id as per your aws account id)_

```sh
aws ecr get-login-password --region eu-west-1 --profile cicd | docker login --username AWS --password-stdin 123456789.dkr.ecr.eu-west-1.amazonaws.com
```

- Tag your image so you can push the image to this repository:

```sh
docker tag hashicorp/terraform 123456789.dkr.ecr.eu-west-1.amazonaws.com/terraform:latest
```
- Push terraform(custom) image 

```sh
docker push 123456789.dkr.ecr.eu-west-1.amazonaws.com/terraform:latest
```

![ECR with terraform image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/18qm81o3mr59o713xbeu.png)

## Create s3 buckets for terraform code and backend

- In this step, will create two **versioned s3 buckets**; one for storing terraform code and another for terraform backend for storing state files with all the default settings **except versioning**. I will enable versioning for both buckets.

- I will call them `terraform-deployment-test-lambda-compute` and `terraform-deployment-test-lambda-compute-backend`

![s3 bucket for tf](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k86hvjeqodx12opicpcf.png)

![s3 bucket for tf backend](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/awu23ruzhacui6zysv16.png)


## Create a CodePipeline

- To automate and share the strategy I will create a small pipeline which will be crucial for automation.


![codepipeline create](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xwdrnq857yataktpvfof.png)

- Add source stage as s3.
- Select `terraform-deployment-test-lambda-compute`.
- Give an arbitrary name for the file name. Since I will be uploading a zip file, I will use `tf.zip`


![codepipeline source stage](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1qmrgqvnmhqf6fhwjkxv.png)


## Create CodeBuild Project with Lambda as Compute

- After creating the source stage for the pipeline, you will see the next screen to add the build stage. Select `CodeBuild` and select `create new project` or choose existing if you already have CodeBuild project.

![Pipeline screen](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y1o57vtiww5ovnbg44vi.png)

- While creating the CodeBuild Project, use the Project name as `terraform-deployment-test-lambda-compute`.
- Codepipeline's source stage will provide the input artifact which is `tf.zip` for CodeBuild.

![codebuild creation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v0mppa68dxy67znfwgtl.png)

- For Environment, Environment Image as `Custom`. Compute as `Lambda`, Environment type as `Linux Lambda`.

![codebuild Environment](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7vz94krcv4xbsg4fodfp.png)

- Choose Image registry as `ECR`. Select `terraform` as ecr repository created in first step.

- Choose ECR Image as `latest`.

- Create Service role, choose default setting as `create new service role`.

![codebuild environment ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kan5wqrrh2rq7tu78v4r.png)

- For buidspec, choose `insert build commands` and add the following yaml. _All it does is initialise the terraform and apply the terraform configuration._

```yaml
version: 0.2

phases:
  build:
    commands:
      - terraform init
      - terraform apply -auto-approve -no-color -input=false

```
![buildspec](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f4pah6v3u6ua3hjggjl0.png)


![lambda codebuild project config](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cx8dd4f3gz7q03x9hfjz.png)

- Rest choose everything as default and `select create build project` at the bottom. After creation, you will return to the CodePipline screen with CodeBuild project name auto-filled. Select Next

![codebuild project created screen](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7i3oibd8p4f5bu3ktuhs.png)


- As next step Skip Deploy Stage

![skip deploy stage](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8htcs0uf1pf77j93852t.png)

- After Reviewing, select `Create` to create the Pipeline.

## Modify service role for Codebuild project

- Go to the Codebuild project, find the `terraform-deployment-test-lambda-compute` and click on the service role link created automatically by CodeBuild.

![codebuild service link role policies](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vvgvkre0lcn3v7cs6q5k.png)

- Since CodeBuild will deploy terraform resources across the account in this case we need to give **`Administrator access policy` so that it can deploy any kind of resources.**

- **It will make more sense to use this method for cross- account deployment as in that case you can restrict permissions by permitting only to assume to role of the account terraform wants to deploy resources.**


## Terraform code

```terraform

terraform {
  backend "s3" {
    bucket = "terraform-deployment-test-lambda-compute-backend"
    region = "eu-west-1"
    key    = "terraform-deployment.tfstate"
  }
}

resource "aws_s3_bucket" "deploy-bucket" {
  bucket = "test-lambda-compute-backend-test-bucket"

  tags = {
    Project     = "For CI CD deploy test"
    Environment = "Dev"
  }
}

resource "aws_s3_bucket" "deploy-bucket-two" {
  bucket = "test-lambda-compute-backend-test-bucket-two"

  tags = {
    Project     = "For CI CD deploy test"
    Environment = "Dev"
  }
}

```

- Here I am using backend bucket for storing terraform state.

- In this terraform code, I am creating two s3 buckets.

- You can of course divide your code into modules just like any other regular terraform project.

```sh
zip -r tf.zip ./     
```

- **_Then I am converting this terraform file to a zip file called tf.zip and upload to the s3 bucket. This upload will trigger CodePipline and run `terraform apply` inside Codebuild using Lambda to create two s3 buckets as per configuration._**

![upload tf.zip](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c7lwe1cbu7b0xvvnn9p7.png)


## Pipeline in Action


- After uploading **tf.zip**, the pipeline gets automatically triggered and runs.

![pipeline state](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uci0s8zwpa07lb91yx3o.png)

- We can also see the action details of codebuild where it initialises s3 backend for the terraform state and creates two s3 buckets.

![terraform init and backend](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pt0kicdwt5h5s3fnfqj4.png)

![terraform apply](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4tmrdypk64wwhu7uo2na.png)

- We can confirm the created buckets in the console too.

![s3 buckets](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iawlzm3j0fbzhi7bq7ff.png)


## EC2 compute v/s Lambda compute

- Documentation says _lambda compute promotes faster builds and saves cost so I wanted to test this same operation with the Ec2 Compute CodeBuild project_.


- Buildspec commands for `CodeBuild project with Compute as EC2` will be as follows. It's not the same because with LAMBDA our docker image is from Terraform which already has Terraform command but in the EC2 compute case we need to install Terraform command.

```terraform
version: 0.2
env:
  variables:
    TERRAFORM_VERSION: 1.5.4

phases:
  install:
    commands:
      - echo "======== install terraform ========"
      - wget --quiet https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip
      - unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip
      - mv terraform /usr/local/bin/

  build:
    commands:
      - terraform init -no-color -input=false
      - terraform apply -auto-approve -no-color -input=false
 
```


![Ec2 project](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vui9j7womgfjfkldqvmk.png)

- Edit the Ec2 codebuild project service role and add Administrator policy as it will be used to create resources.

- All the other steps remain the same(except buildspec file), just create another CodeBuild project with Compute as EC2 and replace the build action in the pipeline with Lambda compute codeBuild Project.


![edit the pipeline](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ebghg4j7yfgzri7umhl.png)

![add ec2 codebuild project](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oi60uuo4dhfzr28g23ev.png)

- Delete the buckets by commenting out in the terraform code, regenerate `tf.zip` and upload to s3. 
- Once buckets are deleted, uncomment the terraform regenerate `tf.zip` and upload to s3. Pipeline will Run

![ec2 state](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nuecsmzp80rkdhh0kskl.png)

- we can see that even though Lambda's default memory configured is 2GB whereas EC2 compute is configured with a default memory of 3GB still** LAMBDA-based deployment(below image) is faster in every sense**.


![lambbda deployment state](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kf07q0hp33tyj4cjllv0.png)


## Limitations to LAMBDA as compute

- [According to Docs](https://docs.aws.amazon.com/en_us/codebuild/latest/userguide/lambda.html#lambda.limitations), there are some use cases where EC2 as compute should be used.


![limitations](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4p3ufps6aj314cr70z55.png)



**AWS Lambda lacks support for tools needing root permissions,  Docker, writing outside /tmp, lambda timeout error after 15 minutes, LINUX_GPU_CONTAINER environment, and features like caching, VPC connectivity, EFS, or SSH access.**


## Bonus: Strategy


- As you have seen, **If your infrastructure deployment workload is something less than 15 minutes codeBuild's LAMBDA as compute with CodePipeline can be used to create a completely secure serverless CD pipeline**.

- A pipeline which can be used to deploy infrastructure cross-account. You can go crazier by adding alerts on failure and success, adding manual approval actions and handling approvals with lambda functions, and integrating with Slack using CodePipeline features.

- If your workload increases by more than 15 minutes then you can strategise to split codeBuild steps into multiple stages(subsections) within CodePipeline.


## From DevOps Perspective

- With this feature, the possibility of doing operations for [code-based events](https://docs.aws.amazon.com/en_us/codebuild/latest/userguide/lambda-samples.html) becomes endless (of course considering the limitations of LAMBDA).

- You can not only build the application coder artifact but also deploy infrastructure cost-effectively and more quickly thanks to AWS LAMBDA.


**_I would like to hear how other members are planning to use this amazing update and in what way. Please try this out, feel free to post your feedback or any questions you didn't understand._**