Practical ECS scaling: vertically scaling an application with a memory leak
Tags: aws, containers, cdk, scaling
Published: 2023-12-20 18:59:37

The [previous article](https://dev.to/aws-builders/practical-ecs-scaling-vertically-scaling-a-cpu-heavy-application-105c) looked at how changing the performance envelope for a CPU-heavy application affected its performance. This article shows whether vertically scaling an application with a memory leak is effective.

- [The endpoint under test](#the-endpoint-under-test)
- [Running tests](#running-tests)
- [Results](#results)
   * [Container 1 (1GB of memory)](#container-1-1gb-of-memory)
   * [Container 2 (2GB of memory)](#container-2-2gb-of-memory)
- [How effective is it to vertically scale an application that has a memory leak?](#how-effective-is-it-to-vertically-scale-an-application-that-has-a-memory-leak)

***

## The endpoint under test

Our [mock application](https://dev.to/ivicak/practical-ecs-scaling-2b05-temp-slug-3558989?preview=609977daf9ecaef438eaca84344f22a40049ed64d0c4f120e2451035ac73faf886c11952d19be6aacd95fc4ed114f985bc762efbd07d1406ead73497#meet-the-app) comes with this REST API endpoint:

* `/memory_leak`, simulating a [memory leak](https://en.wikipedia.org/wiki/Memory_leak).

When this endpoint is invoked, the application calculates the square root of `64 * 64 * 64 * 64 * 64 * 64 ** 64` and returns the result. Due to a bad code merge, 1MB of memory is added to a Python list on each request.

```python
leaked_memory = []

@app.route("/memory_leak")
def memory_leaky_task():
    global leaked_memory

    # appends approx. 1MB of data to a list on each
    # request, creating a memory leak
    leaked_memory.append(bytearray(1024 * 1024 * 1))

    return _build_response(do_sqrt())
```

## Running tests

> It is better to be roughly right than precisely wrong.
  — Alan Greenspan

To load-test this application, I used [hey](https://github.com/rakyll/hey) to invoke the endpoint with 5 requests per second for 30 minutes using `hey -z 30m -q 1 -c 5 $URL/memory_leak`

To be able to compare results, I ran the same application in two containers with different hardware configurations:

|             	| CPUs 	| Memory (GB) 	|
|-------------	|------	|-------------	|
| Container 1 	|  0.5 	|     1.0     	|
| Container 2 	|  1.0 	|     2.0     	|

## Results

### Container 1 (1GB of memory)

Looking at the summary of `hey`, we notice that not all requests were successful:

```text
Summary:
  Total:        1800.0255 secs
  Slowest:      10.1465 secs
  Fastest:      0.0088 secs
  Average:      0.1860 secs
  Requests/sec: 4.3499

Status code distribution:
  [200] 6148 responses
  [502] 341 responses
  [503] 1211 responses
  [504] 130 responses
```

Roughly 21% of all requests had a non-200 status code :disappointed: This is not a great user experience.

Looking at ECS task details we notice that there are 7 tasks in total, only one of which is running at the moment - other 6 are stopped.

![Several stopped tasks](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d4nwwt1bv12d88xhemhr.png)

To get more details, we can describe one of the stopped tasks:

```
aws ecs describe-tasks \
  --cluster ecs-scaling-cluster \
  --tasks 7f0872485e6e421e8f83a062a3704303 |
  jq -r '.tasks[0].containers[0].reason'

OutOfMemoryError: Container killed due to memory usage
```

**OutOfMemoryError: Container killed due to memory usage** becomes obvious when looking at the memory utilization metric:

![Memory utilization](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y09c6wo0pkrkn9265m2g.png)

The sawtooth pattern reveals the problem: our application is exceeding the performance envelope for the "Memory" dimension! Each request leaks approx. 1MB of memory, and because the container is given 1GB of memory, serving roughly a 1000 requests leads to the container running out of memory. At that point the ECS service forcefully stops the container that is out of memory and starts a fresh one.

### Container 2 (2GB of memory)

Running another container with double the memory (1GB → 2GB) and load testing it in the same way we get very similar results, with approx. 7% of all requests having a `5xx` status code:

```text
Summary:
  Total:        1800.0240 secs
  Slowest:      10.0976 secs
  Fastest:      0.0119 secs
  Average:      0.0850 secs
  Requests/sec: 4.7249
  
Status code distribution:
  [200] 7868 responses
  [502] 177 responses
  [503] 405 responses
  [504] 55 responses
```

In this instance only 4 tasks were started, 3 of which were forcefully stopped:

![Several stopped tasks](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sn05ickg2ux71563zr6a.png)

```
aws ecs describe-tasks \
  --cluster ecs-scaling-cluster \
  --tasks c35b7029c38c4383b26e768aec3c77f2 |
  jq -r '.tasks[0].containers[0].reason'

OutOfMemoryError: Container killed due to memory usage
```

And again, the sawtooth memory utilization reveals that we have a memory leak:

![Sawtooth memory utilization](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q396e5gr08051h79qh4r.png)

## How effective is it to vertically scale an application that has a memory leak?

Not at all.

> A memory leak can not be fixed by scaling. You can’t vertically or horizontally scale yourself out of a memory leak. The only way to fix this is to fix the application code. You cannot have scalability with a memory leak. [Source](https://containersonaws.com/presentations/amazon-ecs-scaling-best-practices/)

Regardless of its memory configuration, a container with an application that has a memory leak will sooner or later run out of memory.

AWS re.Post has an article on [troubleshooting OutOfMemory errors](https://repost.aws/knowledge-center/ecs-resolve-outofmemory-errors). [This blog post](https://aws.amazon.com/blogs/containers/how-amazon-ecs-manages-cpu-and-memory-resources/) explains how containers and how containers (in general, and those running on ECS) consume CPU and memory.

***

Next up: Should you horizontally scale your application based on response times?

You also might want to check how [changing the performance envelope for a CPU-heavy application affects its performance](https://dev.to/aws-builders/practical-ecs-scaling-vertically-scaling-a-cpu-heavy-application-105c).