Deploying Django Application on AWS with Terraform. ECS Autoscaling
Tags: aws, terraform, autoscaling, ecs
Published: 2022-08-23 13:18:22

--- series: Django + AWS ---

This is the 7th part of the "Deploying Django Application on AWS with Terraform" guide. You can check out the previous steps here:

- [Part 1: Minimal Working Setup](https://dev.to/daiquiri_team/deploying-django-application-on-aws-with-terraform-minimal-working-setup-587g)
- [Part 2: Connecting PostgreSQL RDS](https://dev.to/daiquiri_team/deploying-django-application-on-aws-with-terraform-connecting-postgresql-rds-2j0i)
- [Part 3: GitLab CI/CD](https://dev.to/daiquiri_team/deploying-django-application-on-aws-with-terraform-gitlab-cicd-13am)
- [Part 4: Namecheap Domain + SSL](https://dev.to/daiquiri_team/deploying-django-application-on-aws-with-terraform-namecheap-domain-ssl-19a5)
- [Part 5: Celery and SQS](https://dev.to/eugen1j/deploying-django-application-on-aws-with-terraform-setting-up-celery-and-sqs-38ik)
- [Part 6: Connecting to Amazon S3](https://dev.to/daiquiri_team/deploying-django-application-on-aws-with-terraform-connecting-to-amazon-s3-2a23)

In this part, we'll make our Django web application scalable using [ECS Autoscaling](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html). 

Autoscaling is the ability to increase or decrease the number of running instances. It allows you to handle traffic spikes and save money for low intensive periods of time. 

When you enable autoscaling for ECS service, AWS creates [Cloudwatch alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html) to determine whether we need to add a new instance or remove a redundant one.

Let's see how it works in practice.

## ECS Autoscaling configuration

First, create a new `autoscale.tf` with the following content:

```terraform
resource "aws_appautoscaling_target" "prod_backend_web" {
  max_capacity       = 5
  min_capacity       = 1
  resource_id        = "service/${aws_ecs_cluster.prod.name}/${aws_ecs_service.prod_backend_web.name}"
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace  = "ecs"
}

resource "aws_appautoscaling_policy" "prod_backend_web_cpu" {
  name               = "prod-backend-web-cpu"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_appautoscaling_target.prod_backend_web.resource_id
  scalable_dimension = aws_appautoscaling_target.prod_backend_web.scalable_dimension
  service_namespace  = aws_appautoscaling_target.prod_backend_web.service_namespace

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }
    target_value = 80
  }

  depends_on = [aws_appautoscaling_target.prod_backend_web]
}
```

Here we defined:

- [AWS AppAutoscaling Target](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/appautoscaling_target). We want to scale the `prod_backend_web` service by instance count dimension from 1 to 5.
- [AWS AppAutoscaling Policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/appautoscaling_policy). We want to scale `prod_backend_web` up when the `ECSServiceAverageCPUUtilization` metric exceeds 80%.

We are ready to apply changes, but first, let's think about load balancer health checks.

## Load Balancer Health Checks

Now we have a quire aggressive health checks. If the container fails to respond twice with a timeout of 2 seconds, Load Balancer considers this container unhealthy and removes them.

It could be an okay solution for the little amount of traffic. But if many requests reach the container and CPU usage goes up to 100%, the container will fail to respond to Health Checks. So, Load Balancer kills them, and we will face an even worse situation: there will be no containers to handle the traffic at all

The possible solution is to increase health checks `timeout` and `unhealthy_threshold`. Thus, there will be more possibility for overloaded containers to survive.

I think it's not a perfect solution, but it will work for this test. If you know a more elegant way to keep overloaded containers running, feel free to leave a comment.

Go to the `load_balancer.tf` and increase `unhealthy_threshold`, `timeout`, and `interval` parameters.

```terraform
# Target group for backend web application
resource "aws_lb_target_group" "prod_backend" {
  ...

  health_check {
    ...
    unhealthy_threshold = 5
    timeout             = 29
    interval            = 30
    ...
  }
}
```

Let's apply our changes and check them at the AWS console.

## CloudWatch Alarms

First, go to the [ECS console](https://console.aws.amazon.com/ecs/home) and check the autoscaling policy for the `prod_backend_web` ECS Service. Select `prod` ECS cluster, select `prod-backend-web` service and click "Update". Pass to the step "Set Auto Scaling" and click on the `prod-backend-web-cpu` autoscaling policy.

![Autoscaling Policy](https://s3.eu-central-1.wasabisys.com/daiquiri-external-content/blog/eugen1j/django-aws/img/part7/autoscaling_policy.png)


Here we see that autoscaling becomes effective when average CPU utilization reaches 80%. But what is the condition for scaling down? Let's check CloudWatch alarms associated with this autoscaling policy.

Go to the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/home) and look at the alarms. 

![Cloudwatch Alarms](https://s3.eu-central-1.wasabisys.com/daiquiri-external-content/blog/eugen1j/django-aws/img/part7/cloudwatch_alarms.png)

Here we see that we scale up when the average CPU load exceeds 80% during 3+ minutes. And, we scale down when the average CPU load goes less than 72% for 15 minutes.

Such specific numbers, but how can we adjust them to our case? For this, you need to create and use custom metrics for alarms with [customized_metric_specification](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/appautoscaling_policy#customized_metric_specification) param in `aws_appautoscaling_policy`. 

Also, you can change `AlarmHigh` and `AlarmLow` metrics manually in the console. It's not a preferable way to create a repeatable setup, but it's okay for our test. So, I'll change the `AlarmLow` metric to 50% and 10 minutes.

![alarm low updating](https://s3.eu-central-1.wasabisys.com/daiquiri-external-content/blog/eugen1j/django-aws/img/part7/alarm_low_updating.png)
 

## Stress Testing

Let's move to the tests. I'll use [ApacheBenchmark](https://httpd.apache.org/docs/2.4/programs/ab.html) for stress testing. This tool can send a lot of requests to our service, so the CPU load goes up.

First, ensure that now the web service has only one container running.

![only one web container](https://s3.eu-central-1.wasabisys.com/daiquiri-external-content/blog/eugen1j/django-aws/img/part7/ecs_web_pre_test.png)

Also, you need to increase the limit of open files with `ulimit -n 10000`.


Now we are ready to run the benchmark. We'll use the health-check URL for this test:

```shell
$ ab -n 100000 -c 1000 https://api.example53.xyz/health/
```

Where `-c 1000` concurrent number of requests, `-n 100000` is the total number of requests.

Check the CloudWatch metrics and ECS Service for the next 10-15 minutes.  

![CPU Burst](https://s3.eu-central-1.wasabisys.com/daiquiri-external-content/blog/eugen1j/django-aws/img/part7/cpu_burst.png)

First, you should see the CPU spike in the charts. After 3 minutes, ECS autoscale starts to spawn new instances.

Then, the average CPU drops below 80%. There were 3 ECS tasks at this moment of time.

After some time, CPU load exceeds 80% again, and ECS autoscale creates the 4th instance. You can see them on the ECS console.

![ecs web scale up](https://s3.eu-central-1.wasabisys.com/daiquiri-external-content/blog/eugen1j/django-aws/img/part7/ecs_web_scale_up.png)

So, scale up works; let's check scale down. Stop ApacheBenchmark and wait for 10-15 minutes to wait for scale down.

You'll see how CPU load drops to zero and ECS scales down the web service to 1 instance.

![cloudwatch cpu goes down](https://s3.eu-central-1.wasabisys.com/daiquiri-external-content/blog/eugen1j/django-aws/img/part7/cloudwatch_cpu_goes_down.png)

Recheck the ECS console to ensure that we have only one web task running:

![ecs web post test](https://s3.eu-central-1.wasabisys.com/daiquiri-external-content/blog/eugen1j/django-aws/img/part7/ecs_web_post_test.png)

So, scale down works too. Let's commit and push our changes to the `infrastructure` repository. 

## The end

Congratulations! In this part, we added ECS autoscaling for the web service. We increase Health Check timeout and period to prevent killing overloaded containers. Then, we run a stress test and verify that number of instances increases when CPU load goes up and decreases when CPU load goes down.

You can find the source code of backend and infrastructure projects [here](https://gitlab.com/django-aws/django-aws-backend) and [here](https://gitlab.com/django-aws/django-aws-infrastructure).

If you need technical consulting on your project, check out our [website](https://daiquiri.team/services/technical-consulting?utm_medium=referral&utm_source=dev_to&utm_campaign=django_aws_7) or connect with me directly on [LinkedIn](https://www.linkedin.com/in/yevhen-bondar/).