Mastering ECS Task Scheduling: Effective Strategies to Reduce Costs
Tags: devops, terraform, webdev, aws
Published: 2024-06-01 08:33:03

## Introduction
As we moved our service deployment from Lambda to ECS on Fargate, we aimed to minimize costs outside of the production environment. We implemented several cost-saving measures:
- Configuring vCPU and memory to the minimum viable settings (0.25 vCPU, 0.5 GB)
- Utilizing Fargate Spot for cost efficiency

These settings were the extent of what we could optimize on the ECS side. However, considering that costs accrue per hour based on vCPU and memory usage, we developed a system using Lambda and EventBridge to stop ECS tasks outside of business hours.

I have detailed the Lambda function code and EventBridge Terraform configuration below, so you can easily replicate and utilize them to reduce costs.

[AWS Fargate Pricing](https://aws.amazon.com/fargate/pricing/?nc1=h_ls)

## Lambda Function
The integration with EventBridge allows for scheduled execution, making Lambda an ideal environment for stopping and starting ECS tasks.

Considering the Single Responsibility Principle, I initially thought about separating the stop and start processes into different functions. However, since the code does not change frequently and to keep management simple, I decided to consolidate them into a single function.

In some cases, tasks need to be stopped or started manually rather than on a schedule, so I included the `action` parameter to select between `scheduled` and `manual` modes.

Below is the complete code, which you can copy and paste by simply changing the ECS cluster and service names specified in `clusters_services_scheduled` and `clusters_services_manual`.

```python
import boto3
from botocore.exceptions import ClientError

def update_service(cluster_name, service_name, desired_count):
    client = boto3.client('ecs')
    application_autoscaling_client = boto3.client('application-autoscaling')

    # Control the minimum number of tasks in the AutoScaling policy
    scalable_targets = application_autoscaling_client.describe_scalable_targets(
        ServiceNamespace='ecs',
        ResourceIds=[f'service/{cluster_name}/{service_name}'],
        ScalableDimension='ecs:service:DesiredCount'
    )['ScalableTargets']

    for scalable_target in scalable_targets:
        application_autoscaling_client.register_scalable_target(
            ServiceNamespace='ecs',
            ResourceId=f'service/{cluster_name}/{service_name}',
            ScalableDimension='ecs:service:DesiredCount',
            MinCapacity=0 if desired_count == 0 else 1,
            MaxCapacity=scalable_target['MaxCapacity']
        )

    # Update the service
    service_update_result = client.update_service(
        cluster=cluster_name,
        service=service_name,
        desiredCount=desired_count
    )
    print(service_update_result)

def lambda_handler(event, context):
    try:
        client = boto3.client('ecs')
        elbv2_client = boto3.client('elbv2')

        action = event.get('action')  # 'stop' or 'start'
        environment = event.get('environment')  # 'scheduled' or 'manual'

        clusters_services_scheduled = [
            ('example-cluster', 'example-service-stg')
        ]

        clusters_services_manual = [
            ('example-cluster', 'example-service-dev')
        ]

        if environment == 'scheduled':
            clusters_services = clusters_services_scheduled
        elif environment == 'manual':
            clusters_services = clusters_services_manual
        else:
            raise ValueError("Invalid environment specified")

        desired_count = 0 if action == 'stop' else 1

        for cluster_name, service_name in clusters_services:
            update_service(cluster_name, service_name, desired_count)

        if action == 'start':
            for cluster_name, service_name in clusters_services:
                # Retrieve new task IDs and register them with the target group
                tasks = client.list_tasks(
                    cluster=cluster_name,
                    serviceName=service_name
                )['taskArns']

                task_descriptions = client.describe_tasks(
                    cluster=cluster_name,
                    tasks=tasks
                )['tasks']

                # Retrieve target group information associated with the service
                load_balancers = client.describe_services(
                    cluster=cluster_name,
                    services=[service_name]
                )['services'][0]['loadBalancers']

                for load_balancer in load_balancers:
                    target_group_arn = load_balancer['targetGroupArn']

                    for task in task_descriptions:
                        task_id = task['taskArn'].split('/')[-1]
                        elbv2_client.register_targets(
                            TargetGroupArn=target_group_arn,
                            Targets=[{'Id': task_id}]
                        )

    except ClientError as e:
        print(f"Exception: {e}")
    except ValueError as e:
        print(f"Exception: {e}")
```

## Code Explanation

### Task Count Control in Services
The task count of ECS services is updated within the `update_service` function:

```python
    service_update_result = client.update_service(
        cluster=cluster_name,
        service_name=service_name,
        desiredCount=desired_count
    )
```

This section uses the `update_service` method to dynamically update the `desiredCount` for the specified cluster and service. The `desiredCount` determines the number of tasks aimed for the service, facilitating the stop or start operations.

### Scaling Settings Control
The `update_service` function adjusts the number of instances in an ECS service and sets the `MinCapacity` in the Auto Scaling policy to either 0 or 1:

```python
def update_service(cluster_name, service_name, desired_count):
    client = boto3.client('ecs')
    application_autoscaling_client = boto3.client('application-autoscaling')
    ...
```

Even if the service's task count is set to 0, if `MinCapacity` remains at 1, the system will attempt to launch tasks repeatedly. Therefore, this setting is crucial if Auto Scaling is configured.

### Task Launch and Target Group Registration
If you're integrating ELB (ALB) with ECS, when tasks are launched, you need to retrieve new task IDs and register them with the ALB's target group:

```python
if action == 'start':
    for cluster_name, service_name in clusters_services:
        # Retrieve new task IDs and register them with the target group
        tasks = client.list_tasks(cluster=cluster_name, serviceName=service_name)['taskArns']
        ...
```
Even during task shutdown, the ECS service remains linked to the target group, allowing you to retrieve the target group in the following section:

```python
load_balancers = client.describe_services(
    cluster=cluster_name,
    services=[service_name]
)['services'][0]['loadBalancers']

for load_balancer in load_balancers:
    target_group_arn = load_balancer['targetGroupArn']
```

## Terraform Code Example
The configuration for deploying the Lambda function via a ZIP file is as follows:

```terraform
resource "aws_lambda_function" "ecs_task_scheduler" {
  function_name    = "ecs-task-scheduler"
  s3_bucket        = var.s3_bucket_lambda_functions_storage_bucket
  s3_key           = "ecs-task-scheduler.zip"
  handler          = "app.lambda_handler"
  runtime          = "python3.12"
  role             = var.iam_role_ecs_task_scheduler_lambda_exec_role_arn
  timeout          = 300 # 5 minutes
}
```

Place the `Dockerfile` and `build.sh` in the same directory as `app.py`. Run `./build.sh` to create the `ecs-task-scheduler.zip`. Upload this ZIP file to the designated S3 bucket and execute `terraform apply` to complete the deployment.

```Dockerfile
FROM public.ecr.aws/lambda/python:3.12

# Install Python dependencies
COPY requirements.txt /var/task/
RUN pip install -r /var/task/requirements.txt --target /var/task

# Copy the Lambda function code
COPY app.py /var/task/

# Set the working directory
WORKDIR /var/task

# Set the CMD to your handler
CMD ["app.lambda_handler"]
```

```bash
#!/bin/bash

# Build the Docker image
docker build -t ecs-task-scheduler-build.

# Create a container from the image
container_id=$(docker create ecs-task-scheduler-build)

# Copy the contents of the container to a local directory
docker cp $container_id:/var/task ./package

# Clean up
docker rm $container_id

# Zip the contents of the local directory
cd package
zip -r ../ecs-task-scheduler.zip .
cd ..

# Clean up
rm -rf package
```

## Manual Execution Method
To manually execute the Lambda function, use the testing tab in the console. Paste the JSON formatted request into the test event body and press the test button to execute the function.

```json
{
  "action": "start",
  "environment": "manual"
}
```

![Screenshot 2024-06-01 17.10.46.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/569054/6450ace1-2758-a249-9bb0-91f5db76d221.png)

## EventBridge
Use the Cron expression in EventBridge to schedule the automatic execution of the Lambda function on specific days and times. This feature allows you to build a schedule to stop ECS tasks during weekday nights and all weekend.

**Weekday Schedule (22:00 - 5:00 Stop)**
- **Stop**: Weekdays at 22:00 (UTC 13:00)
- **Start**: Weekdays at 5:00 (UTC 20:00)

**Weekend Schedule (All Day Stop)**
- **Stop**: Saturday at 00:00 (UTC 15:00 previous day)
- **Start**: Monday at 5:00 (UTC 20:00)

Below is an example of Terraform code to set these schedules. Define the EventBridge rules and set the appropriate Lambda function as the target. Note that time settings are in UTC, so adjust according to your region.

### Weekday Schedule

```hcl
# Weekday stop schedule (every day at 22:00 JST)
resource "aws_cloudwatch_event_rule" "ecs_weekday_stop_tasks_schedule" {
  name                = "ECSWeekdayStopTasksSchedule"
  description         = "Schedule to stop ECS tasks on weekdays at 22:00 JST"
  schedule_expression = "cron(0 13 ? * MON-FRI *)"  # Weekdays at 22:00 JST
}

resource "aws_cloudwatch_event_target" "ecs_weekday_stop_tasks_target" {
  rule      = aws_cloudwatch_event_rule.ecs_weekday_stop_tasks_schedule.name
  target_id = "ecsTaskSchedulerWeekdayStop"
  arn       = var.lambda_function_ecs_task_scheduler_arn

  input = jsonencode({
    action      = "stop"
    environment = "scheduled"
  })
}

resource "aws_lambda_permission" "ecs_weekday_stop_tasks_allow_eventbridge" {
  statement_id  = "AllowEventBridgeInvokeLambdaWeekdayStop"
  action        = "lambda:InvokeFunction"
  function_name = var.lambda_function_ecs_task_scheduler_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.ecs_weekday_stop_tasks_schedule.arn
}

# Weekday start schedule (every day at 5:00 JST)
resource "aws_cloudwatch_event_rule" "ecs_weekday_start_tasks_schedule" {
  name                = "ECSWeekdayStartTasksSchedule"
  description         = "Schedule to start ECS tasks on weekdays at 5:00 JST"
  schedule_expression = "cron(0 20 ? * MON-FRI *)"  # Weekdays at 5:00 JST
}

resource "aws_cloudwatch_event_target" "ecs_weekday_start_tasks_target" {
  rule      = aws_cloudwatch_event_rule.ecs_weekday_start_tasks_schedule.name
  target_id = "ecsTaskSchedulerWeekdayStart"
  arn       = var.lambda_function_ecs_task_scheduler_arn

  input = jsonencode({
    action      = "start"
    environment = "scheduled"
  })
}

resource "aws_lambda_permission" "ecs_weekday_start_tasks_allow_eventbridge" {
  statement_id  = "AllowEventBridgeInvokeLambdaWeekdayStart"
  action        = "lambda:InvokeFunction"
  function_name = var.lambda_function_ecs_task_scheduler_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.ecs_weekday_start_tasks_schedule.arn
}
```

### Weekend Schedule

```hcl
# Saturday stop schedule (at 0:00 JST)
resource "aws_cloudwatch_event_rule" "ecs_weekend_stop_tasks_schedule" {
  name                = "ECSWeekendStopTasksSchedule"
  description         = "Schedule to stop ECS tasks on Saturday at 00:00 JST"
  schedule_expression = "cron(0 15 ? * SAT *)"  # Saturday at 0:00 JST
}

resource "aws_cloudwatch_event_target" "ecs_weekend_stop_tasks_target" {
  rule      = aws_cloudwatch_event_rule.ecs_weekend_stop_tasks_schedule.name
  target_id = "ecsTaskSchedulerWeekendStop"
  arn       = var.lambda_function_ecs_task_scheduler_arn

  input = jsonencode({
    action      = "stop"
    environment = "scheduled"
  })
}

resource "aws_lambda_permission" "ecs_weekend_stop_tasks_allow_eventbridge" {
  statement_id  = "AllowEventBridgeInvokeLambdaWeekendStop"
  action        = "lambda:InvokeFunction"
  function_name = var.lambda_function_ecs_task_scheduler_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.ecs_weekend_stop_tasks_schedule.arn
}

# Monday start schedule (at 5:00 JST)
resource "aws_cloudwatch_event_rule" "ecs_weekend_start_tasks_schedule" {
  name                = "ECSWeekendStartTasksSchedule"
  description         = "Schedule to start ECS tasks on Monday at 05:00 JST"
  schedule_expression = "cron(0 20 ? * MON *)"  # Monday at 5:00 JST
}

resource "aws_cloudwatch_event_target" "ecs_weekend_start_tasks_target" {
  rule      = aws_cloudwatch_event_rule.ecs_weekend_start_tasks_schedule.name
  target_id = "ecsTaskSchedulerWeekendStart"
  ar

n       = var.lambda_function_ecs_task_scheduler_arn

  input = jsonencode({
    action      = "start"
    environment = "scheduled"
  })
}

resource "aws_lambda_permission" "ecs_weekend_start_tasks_allow_eventbridge" {
  statement_id  = "AllowEventBridgeInvokeLambdaWeekendStart"
  action        = "lambda:InvokeFunction"
  function_name = var.lambda_function_ecs_task_scheduler_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.ecs_weekend_start_tasks_schedule.arn
}
```

## IAM Role
Here's an example of Terraform code to define the execution role for the Lambda function scheduling ECS tasks. This role enables the Lambda function to call APIs from ECS and related AWS services.

- **ECS Service Management**: Update services, list tasks, and fetch detailed information about tasks and services.
- **Auto Scaling Management**: Register and deregister scalable targets, retrieve information about scalable targets.
- **Elastic Load Balancing (ELB) Management**: Register and deregister targets, fetch detailed information about target groups and listeners.
- **Log Management**: Create log groups and streams, post log events.

```terraform
resource "aws_iam_policy" "ecs_task_scheduler_policy" {
  name        = "ecs-task-scheduler-policy"
  description = "Policy for ECS task scheduler Lambda function"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = [
          "ecs:UpdateService",
          "ecs:ListTasks",
          "ecs:DescribeTasks",
          "ecs:DescribeServices"
        ],
        Resource = "*"
      },
      {
        Effect   = "Allow",
        Action   = [
          "application-autoscaling:RegisterScalableTarget",
          "application-autoscaling:DeregisterScalableTarget",
          "application-autoscaling:DescribeScalableTargets"
        ],
        Resource = "*"
      },
      {
        Effect   = "Allow",
        Action   = [
          "elasticloadbalancing:RegisterTargets",
          "elasticloadbalancing:DeregisterTargets",
          "elasticloadbalancing:DescribeTargetGroups",
          "elasticloadbalancing:DescribeListeners",
          "elasticloadbalancing:DescribeRules"
        ],
        Resource = "*"
      },
      {
        Effect   = "Allow",
        Action   = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ],
        Resource = "arn:aws:logs:*:*:*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "ecs_task_scheduler_policy_attach" {
  role       = aws_iam_role.ecs_task_scheduler_lambda_exec_role.name
  policy_arn = aws_iam_policy.ecs_task_scheduler_policy.arn
}
```

## Conclusion
As we continue to aim for further cost reductions, the next step will be to introduce Compute Savings Plans.

[Compute Savings Plans Pricing](https://aws.amazon.com/savingsplans/compute-pricing/?nc1=h_ls)