Build Scheduled AWS Batch Job Infrastructure using Terraform
Tags: aws, batch, terraform
Published: 2023-09-10 06:22:56

In this post, I will walk you through how to build a scheduled AWS Batch job infrastructure using Terraform. It focuses on the Terraform infrastructure and modules definition, and how they work together to build the entire workflow. It doesn't cover the advanced features that AWS Batch provided, and only use [Terraform aws official provider](https://registry.terraform.io/providers/hashicorp/aws/5.0.0) 
and basic Terraform features. In short, this post helps you setup a basic AWS Batch job infrastructure quickly, and you can add more features and enhance the structure as needed.

## Prerequisite
1. AWS CLI (V2) is installed on the local machine.
2. AWS credentials setup. We use the credentials to deploy Terraform resources to the target AWS Account. In this demo, I use the same AWS profile for S3 remote backend configuration and Terraform apply.
3. Terraform CLI (1.3.4) is installed in the local machine. You can loosen the Terraform version restriction in `versions.tf` to use other close versions, however I'm not sure if the code works as expected without verification.
4. I'm working on Mac (MacOS Monterey) with Apple M2 Chip. So the `aws provider` installed is `.terraform/providers/registry.terraform.io/hashicorp/aws/5.0.0/darwin_arm64/terraform-provider-aws_v5.0.0_x5`. If you are in other operating system, you should remove `.terraform.lock.hcl` file from source code, and let Terraform CLI install the well-matched `aws provider` version according to your OS.
5. In the demo source code, I use default VPC, subnets and security group for EC2 instances. You can use customized network resources, however you should take care of the network availability if your Batch needs internet access, for example, the docker image that is used is hosted in public docker registry. 

You can find the demo source code from https://github.com/camillehe1992/on-demand-job-in-aws-batch

## Terraform Structure
The Terraform structure contains several components shows as below.

```bash
.
├── config.tfbackend  # Remote backend config file
├── data.tf           # File for Terraform data source 
├── main.tf           # The reference of modules
├── outputs.tf        # The outputs of Terraform resources
├── terraform         
│   ├── dev           # Environment specified variables
│   └── modules       # Terraform modules
├── variables.tf      # Terraform input variables that should be passed to the arch module
└── versions.tf       # Defines the versions of Terraform and providers

```
### Remote Backend Configuration (S3)
For best practice, I save the Terraform state files in the remote location, which is S3 bucket here. The configuration below specifies the S3 bucket name, region, and profile used to access S3 bucket. Another parameter is `key`, which we 
```
# config.tfbackend
region         = "cn-north-1"
bucket         = "tf-state-756143471679-cn-north-1"
profile        = "service.app-deployment-dev-ci-bot"
```

Another parameter is `key` that is provided via `-backend-config` command in `terraform init` as there is a `ENVIRONMENT` specific path and we have to inject it dynamically instead of hard-coding in `config.tfbackend` file. You can use other configurations settings as needed.  

```
terraform init -reconfigure \
    -backend-config=$BACKEND_CONFIG \
    -backend-config="key=$NICKNAME/$ENVIRONMENT/$AWS_REGION/terraform.tfstate"
```
### Terraform Main File
The `main.tf` in the root directory is the core component and entrance in Terraform structure. I define all resources here, including IAM roles, Batch, CloudWatch Events (EventBridge), and SNS Topic. You can separate them into specific files, for example, `iam.tf`, `batch.tf`, `events.tf`, `sns.tf`, etc. I keep them in one file as the project is not that complex, and it's easy to understand the relationships between resources/modules. You should make a better decision according to your project structure.

In the `main.tf` file, I define 8 resources, 3 IAM roles, 1 secrets group, 1 batch, 2 CloudWatch events, 1 SNS topic. All of them compose the AWS architecture as the diagram shows

![batch-arch](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rhojifkiwg5gpzii9355.png)

Workflow steps:

1. User creates a docker image, uploads the image to the Amazon ECR or another container registry (for example, DockerHub), and creates a job definition, compute environment and job queue in AWS Batch. In this repo, we use an AWS official image `public.ecr.aws/amazonlinux/amazonlinux:latest` for demo purpose.
2. Batch job is submitted using job definition into the job queue in AWS Batch by CloudWatch Event regularly as scheduled.
3. AWS Batch launches an EC2 instance in computing environment, pulls the image from the image registry and create an container.
4. The container should implement some tasks on your behave. An email notification will be triggered if the job is failed.
5. After done, the container will be stopped and removed. EC2 instance is shutdown automatically by AWS Batch. 

### Terraform Modules
For well-architecting and organizing Terraform structure, I define modules for each resource group. `Modules` is a key feature in Terraform which helps users manage their own resources efficiently. Let's dive into the details of each module in this section.

**Batch Module**: Defines resources in Batch service, including computing environment, job queue, job definition.
**EventBridge Module**: Defines resources in EventBridge, including event rule, rule target. One named `submit_batch_job_event` is used to submit Batch job as scheduled, another named `capture_failed_batch_event` is used to send out an alert email if a Batch job is failed.
**IAM Module**: Defines IAM resources, including roles, policies, instance profile. These roles are used by AWS Batch resources and EventBridge rules.
**SecretManager Module**: Defines secret token that may be used in Job container. It's not required in the demo project, but for your reference.
**SNS Module**: Defines resource in SNS, including topic and subscription.

## Apply/Destroy Terraform Resources
I create a `Makefile` and shell script to simplify the apply/destroy process in one command. You can find the shell script from `/scripts/apply.sh` in demo code.

```bash
# apply Terraform infrastructure
make apply

# destroy Terraform infrastructure
make destroy
```

## Submit Batch Job Manually
After applying Terraform resources successfully using `make apply`, you can submit a Batch job for testing the entire workflow. The Batch job is submitted/triggered by CloudWatch Event (EventBridge) per day regularly as scheduled. However, you are allowed to submit a job manually via [AWS CLI](https://docs.aws.amazon.com/cli/latest/reference/batch/submit-job.html) as below. Or from AWS Console directly.
> Don't forget to update job definition revision in `--job-definition` if you have a new revision created. Only the latest revision is ACTIVE.

```bash
# Setup AWS_PROFILE with permission to submit batch job
export AWS_PROFILE=service.app-deployment-dev-ci-bot

# Submit a job using CLI
aws batch submit-job \
  --job-name triggered-via-cli \
  --job-definition arn:aws-cn:batch:cn-north-1:756143471679:job-definition/dev-helloworld-jd:1 \
  --job-queue arn:aws-cn:batch:cn-north-1:756143471679:job-queue/dev-helloworld-jq
```

After submitted successfully, go to `AWS Console -> Batch -> Jobs`. Select the target job queue from the dropdown list, then your new submitted job will be listed on the top. It will spend a few minutes for a job to complete, according to the job processing time, and whether you allocate an EC2 instance resource in advance by giving variable `desired_vcpus` a number greater than 0 or not. If the job failed, an email notification will be sent out to the Topic subscribers you provided in variable `notification_email_addresses`.

For cost saving, I set the `desired_vcpus` as `0` as default, which means a new EC2 instance will be launched when a new job is submitted and shut down immediately after completed. The screenshot below shows the latest job that submitted by CloudWatch Event (EventBridge) at 04:00 AM (UTC).

## Logging
The logging data is saved to CloudWatch Logs automatically. You can find the logs on the bottom of the job details (some delay to sync logs from CloudWatch Logs). In the job details view page, it also provides a link to the log stream of current job. AWS creates a CloudWatch Logs group named `/aws/batch/job` automatically when you submit a Batch job at the first time in the same region.

## Notes:
1. In the `compute_resources` block of resource `aws_batch_compute_environment`, the `instance_role` argument is the ARN of the IAM role profile, not the IAM role.
2. You may met a ClientException that failed to delete Computing Environment as it has a relationship with Job Queue. This is a well-known issue in `Terraform aws provider` and lots of people raised on the Internet. A few workarounds came up with, and you can find one that fits your needs if you google it. 
3. Don't forget to accept the email subscription request when you first deploy the SNS subscription, then your subscribed email address is able to receive failed job alert. 
4. For secret token, NEVER EVER checkin the secret token in your source code. For demo purpose, I export the token in `Makefile` using `export TF_VAR_my_secret=replace_me`, but you should save the secret tokens in a better place, for example GitHub Secrets, and inject them as environment variables in runtime. 

## Reference
- [What is Batch](https://docs.aws.amazon.com/batch/latest/userguide/what-is-batch.html)
- [Terraform Batch Compute Environment Resources](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/batch_compute_environment#compute_resources)
- [Terraform SNS Topic](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sns_topic)
- [Batch CloudWatch Events](https://docs.aws.amazon.com/batch/latest/userguide/batch_cwe_events.html)
- [Batch CloudWatch Events Target](https://docs.aws.amazon.com/batch/latest/userguide/batch-cwe-target.html)
- [Batch with SNS](https://docs.aws.amazon.com/batch/latest/userguide/batch_sns_tutorial.html)


Done. I'm always appreciating your commands and ideas. Happy learning!