How to deploy django app to ECS Fargate Part2
Tags: aws, docker, tutorial, webdev
Published: 2020-07-03 12:00:25

> This post was originally published on notion. Click [here](https://www.notion.so/kokospapa/How-to-deploy-django-app-to-ECS-Fargate-part2-93a372ebd8b941cea99d03de8ca98863) if you prefer to read in notion page which has better readability.

# Introduction

This blog post illustrate development cycle using django app container. I assume readers are already somewhat familar with docker and docker-compose. Although I used django for app development, it is language-agnostic since the post is about containerized application deployment.

Walkthough is devided into three parts consisting three different environemt respectively. First part, describes the architecture of the app(api and async worker) and how they are deployed on `local` enviroment. Second part is how to deploy the docker containers on cloud using single ec2 instance with on `staging` environment. Third part, illustrate how to convert traditional ec2 deployment into ECS using fargate with github actions on `prod` environment.

`local` - run docker containers on desktop/laptop with sqlite and redis server using docker-compose

`stating` - run docker containers on single ec2 instance with mysql RDS and ElasticCache

`prod` - convert stagning setup to ECS Fargate

For part1, click here.
{% link https://dev.to/kokospapa8/how-to-deploy-django-app-to-ecs-fargate-part1-2ga %}

You can skip to part3 if you are already familiar with AWS architectures.
{% link https://dev.to/kokospapa8/how-to-deploy-django-app-to-ecs-fargate-part3-3i7p %}


# Staging Infra Setup

Before going straight to ECS deployment, we will setup up a `staging` env to test the application using ec2 compute instance and other AWS cloud services.  If you are familar with AWS infra and confident with ecs setup, you can skip this part. However I will be using same vpc, redis and mysql for `production` env so it would be worth while to take a look at the setup. (you should use different vpc, redis and mysql for your actual `production` deployment)

The `staging` cloud architecture will consist following AWS services.

- VPC with public and private subnet
- EC2 instance (in public subnet)
- ALB in front of ec2 instance
- RDS mysql (in private subnet)
- ElasticCache - Redis (private subnet)

![staging infra](https://dev-to-uploads.s3.amazonaws.com/i/v942mc1r16rjzd6kpbp8.png)

Refer to the cloudformation for detailed configurations.


[kokospapa8/ecs-fargate-sample-app](https://github.com/kokospapa8/ecs-fargate-sample-app/blob/master/config/cloudformation/cf-ecs-sample.json)

- Preview
    - Network - VPC
        ```json
        "VpcEcsSample": {
              "Type": "AWS::EC2::VPC",
              "Properties": {
                "CidrBlock": "172.10.0.0/16",
                "InstanceTenancy": "default",
                "EnableDnsSupport": "true",
                "EnableDnsHostnames": "true",
                "Tags": [
                  {
                    "Key": "Name",
                    "Value": "ecs-sample"
                  }
                ]
              }
            },
            "EcsSamplePrivate1": {
              "Type": "AWS::EC2::Subnet",
              "Properties": {
                "CidrBlock": "172.10.11.0/24",
                "AvailabilityZone": "us-west-2a",
                "VpcId": {
                  "Ref": "VpcEcsSample"
                },
                "Tags": [
                  {
                    "Key": "Name",
                    "Value": "ecs-sample-private 1"
                  }
                ]
              },
              "DependsOn": "VpcEcsSample"

            },
        "EcsSamplePublic1": {
              "Type": "AWS::EC2::Subnet",
              "Properties": {
                "CidrBlock": "172.10.1.0/24",
                "AvailabilityZone": "us-west-2a",
                "VpcId": {
                  "Ref": "VpcEcsSample"
                },
                "Tags": [
                  {
                    "Key": "Name",
                    "Value": "ecs-sample-public 1"
                  }
                ]
              },
              "DependsOn": "VpcEcsSample"
            },
            "IgwEcsSample": {
              "Type": "AWS::EC2::InternetGateway",
              "Properties": {
                "Tags": [
                  {
                    "Key": "Name",
                    "Value": "ecs-sample IGW"
                  }
                ]
              },
              "DependsOn": "VpcEcsSample"
            },
        "NATGateway": {
                "Type": "AWS::EC2::NatGateway",
                "Properties": {
                    "AllocationId": {
                        "Fn::GetAtt": [
                            "EipNat",
                            "AllocationId"
                        ]
                    },
                    "SubnetId": {
                        "Ref": "EcsSamplePublic1"
                    }
                },
                "DependsOn": [
                    "EcsSamplePublic1",
                    "EipNat",
                    "IgwEcsSample",
                    "GatewayAttachment"
                ]
            },
        ```

    - RDS

        ```json
        "RDSCluster" : {
              "Type": "AWS::RDS::DBCluster",
              "Properties" : {
                  "MasterUsername" : {
                      "Ref": "DBUsername"
                  },
                  "MasterUserPassword" : {
                      "Ref": "DBPassword"
                  },
                  "DBClusterIdentifier" : "ecs-sample",
                  "Engine" : "aurora",
                  "EngineVersion" : "5.6.10a",
                  "EngineMode" : "serverless",
                  "ScalingConfiguration" : {
                      "AutoPause" : true,
                      "MinCapacity" : 4,
                      "MaxCapacity" : 8,
                      "SecondsUntilAutoPause" : 1000
                  }
              }
            },
        ```

    - Redis

        ```json
        cacheecssample001": {
              "Type": "AWS::ElastiCache::CacheCluster",
              "Properties": {
                "AutoMinorVersionUpgrade": "true",
                "AZMode": "single-az",
                "CacheNodeType": "cache.t2.micro",
                "Engine": "redis",
                "EngineVersion": "5.0.6",
                "NumCacheNodes": "1",
                "PreferredAvailabilityZone": "us-west-1b",
                "PreferredMaintenanceWindow": "thu:02:30-thu:03:30",
                "CacheSubnetGroupName": {
                  "Ref": "cachesubnetecssampleredissubnetgroup"
                },
                "VpcSecurityGroupIds": [
                  {
                    "Fn::GetAtt": [
                      "sgecssampleredis",
                      "GroupId"
                    ]
                  }
                ]
              }
            },
        ```
This template is preconfigured to `us-west-2` region.

## Migrate docker images to ECR

Before setting up the infrastructure I will move docker images to `elastic container registry`. If you want to still use docker hub or other private docker image repository, you can skip this part

- ecr setup

    [Getting started with Amazon ECR using the AWS CLI](https://docs.aws.amazon.com/AmazonECR/latest/userguide/getting-started-cli.html)

    log into ecr

    ```python
    $ eval $(aws ecr get-login-password --region us-east-2 --no-include-email)
    > Login Succeeded
    ```

    Create repository with cli 

    ```python
    $ aws ecr create-repository --repository-name ecs-sample-nginx --region us-east-2
    $ aws ecr create-repository --repository-name ecs-sample-api --region us-east-2
    $ aws ecr create-repository --repository-name ecs-sample-worker --region us-east-2
    ```

    or create repository on the console 

![ecr creation](https://dev-to-uploads.s3.amazonaws.com/i/2y5uz0ojxspz7zipbm67.png)

    Build docker image if you have not build the image from previoud post

    ```bash
    $ docker build -f config/app/Docker_base -t ecs-fargate-sample_base:latest .
    $ docker build -f config/app/Docker_app -t ecs-fargate-sample_app:latest .
    $ docker build -f config/app/Docker_worker -f ecs-fargate-sample_app:latest .
    ```

    Tag and push images 

    ```bash
    $ docker tag ecs-fargate-sample_app:base <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-api:base
    $ docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-api:base
    $ docker tag ecs-fargate-sample_app:latest <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-api:latest
    $ docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-api:latest
    $ docker tag ecs-fargate-sample_worker:base <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-worker:latest
    $ docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-worker:latest
    ```

    Check the console for the images 
![ecr image pushed](https://dev-to-uploads.s3.amazonaws.com/i/uu5yul3ky21gtebmlqe8.png)
   


## IAM role

You need access to ECR from your ec2 instances, instead of embedding AWS accesskey in the instance I will attach an iam role.

- ecs-sample-ec2_role (optional)
    - I will give permission to read from ECR

    [Amazon ECR Managed Policies](https://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr_managed_policies.html)


## Network

### VPC

I have created VPC with CIDR 172.10.0.0/16. refer to the images attached for reference. For production I advise you to make more subnets in multi-AZ for better availability. For the sake of the post, I have just used two available zones (serverless mysql needs at least two AZs)

- **Public subnet**

![public subnet](https://dev-to-uploads.s3.amazonaws.com/i/fksef5dwu7ew8pl33wku.png)

    CIDR - 172.10.1.0/24, 172.10.2.0/24

    Components

    - ec2 instance for `api`
    - NAT GATEWAY for `worker` instance

- **Private subnet**

![private subnet](https://dev-to-uploads.s3.amazonaws.com/i/xfqz8mnsue76ctsxqc6a.png)

    CIDR - 172.10.11.0/24, 172.10.12.0/24

    Components

    - mysql
    - elastic cache
    - ec2 instance for `worker`

### Security group

I have added minial security measure for the sake of the post. This is just sample you should not use this for production. 

- Setup

    **ecs-sample-VPC**

    - Opened all ports to public (not safe)

    **ecs-sample-loadbalancer**

    - 80 and 443 to public
    - 80 with ec2-api instance

    **ecs-sample-mysql**

    - 3306 with ec2

    **ecs-sample-ec2-api** and **ecs-sample-ec2-worker** (they should be different but I just used single sg for both ec2, should block 80 access for particular ec2 instance)

    - 3306 mysql
    - 6379 redis,
    - 80 for ALB and public ip access
    - 22 for ssh

    **ecs-sample-redis**

    - 6379 with ec2
### ALB

Created Alb with simple setup attached to public subnet. We don't have instance yet so just created target group with no instance attached yet. 

- configuration
    - name: ecs-sanple-lb
    - Listener: port 80
    - Security group: ecs-sample-lb
    - VPC and subnet - public subnet created above

![alb config 1](https://dev-to-uploads.s3.amazonaws.com/i/wjkor7dbv8a4sdfyvp2z.png)

![alb result 1](https://dev-to-uploads.s3.amazonaws.com/i/b6xt3xedwhntkuwezben.png)


## Database

### RDS - Mysql

I will use serverless mysql instead of traditional mysql instance to minimize cost. (serverless mysql only support mysql engine version up to 5.6)

- configuration
    - database - `sample`
    - username - `admin`
    - network - VPC and private subnet created above

![mysql db](https://dev-to-uploads.s3.amazonaws.com/i/y8uqfzejh18n9novasxh.png)

Take a note on your mysql endpoint and password, will be using it later for env variable setup.

### Elasticcache - Redis

Take a note on your primary endpoint, will be using it later for env variable setup.

## Compute instances

We will start two ec2 instances. One for api in public subnet and another for worker in private sunet.

### EC2 instance - API

- configurations
    - AMI : Ubuntu image 20.04 LTS
    - Network: public subnet
    - auto-assign ip : enabled

![ec2 conf1](https://dev-to-uploads.s3.amazonaws.com/i/j27obf0rlo0d6kuc62ut.png)

![ec2 conf2](https://dev-to-uploads.s3.amazonaws.com/i/h8ihw4gb3hxgztwmkfrc.png)

![ec2 conf3](https://dev-to-uploads.s3.amazonaws.com/i/vpu3ljoka1u6ojsfwpad.png)

### Ec2 instance - worker

This instance is basically same with `api` instance but it needs to in private subnet. I advise making AMI from `api` instance and fire up on private instance. Since there will be no public ip attached, you need to tunnel through `api` or `bastion` instance to connect the instance with private ip. (You can use [following method](https://www.notion.so/kokospapa/How-to-deploy-django-app-to-ECS-Fargate-part2-93a372ebd8b941cea99d03de8ca98863#f8064c59d13c42b78a576173a979c34b) to connect directly since we have NAT gateway deployed)


# Deployment

Once the instance is up and running, we are ready to deploy docker containers.

Let's ssh into to ec2 instance.

```bash
ssh -i "/path/to/keyfile/" ubuntu@<public_ip>
```

### Docker setup

Next, you need to install docker engine and docker-compose. refer to following [link](https://docs.docker.com/engine/install/ubuntu/)

- install steps

    [Install Docker Engine on Ubuntu](https://docs.docker.com/engine/install/ubuntu/)

    [Install Docker Compose](https://docs.docker.com/compose/install/)

    [Post-installation steps for Linux](https://docs.docker.com/engine/install/linux-postinstall/)

    ```bash
    # docker engine
    $ sudo apt-get update
    $ sudo apt-get install \
        apt-transport-https \
        ca-certificates \
        curl \
        gnupg-agent \
        software-properties-common
    $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
    $ sudo add-apt-repository \
       "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
       $(lsb_release -cs) \
       stable"
    $ sudo apt-get update
    $ sudo apt-get install docker-ce docker-ce-cli containerd.io
    # docker-compose
    $ sudo curl -L  \ 
"https://github.com/docker/compose/releases/download/1.26.0/docker-compose-$(uname -s)-$(uname -m)" \ 
-o /usr/local/bin/docker-compose

    ```

### Environment variables

You need to set couple env for staging since we are using redis and mysql.

Please take a closer look at `settings/staging.py` for `DB_USER` and `DB_NAME`. If they are different, update the file accordingly.

```bash
export SECRET_KEY=""
export DB_HOST=""
export DB_PASSWORD=""
export REDIS_HOST=""
```

### Run containers

Check out sources from git and run the docker-compose

```bash
$ git clone https://github.com/kokospapa8/ecs-fargate-sample-app.git
$ cd ecs-fargate-sample-app
$ docker-compose -f docker-compose-staging-api.yml up --build
# add -d option for detach mode I explicitly left it out to see access logs
# if you get error running docker-compose or docker, you probably don't have previlige to run docker as user, 
# refer to https://docs.docker.com/engine/install/linux-postinstall/

# instead of building the whole images you can pull the image from ECR - https://docs.docker.com/compose/reference/pull/
# in order to pull the image from repository make sure you add images filed to each docker-compose file - https://docs.docker.com/compose/compose-file/#image
# 
$ docker-compose -f docker-compose-staging-api.yml pull
$ docker-compose -f docker-compose-staging-api.yml up
```

Once docker containers are up and running and no error shown, access public ip of your app instance.

[![asciicast](https://asciinema.org/a/tNnXTRKWU6u0572aS3GSNNo00.svg)](https://asciinema.org/a/tNnXTRKWU6u0572aS3GSNNo00)


### Attach instace behind ALB

Attach your instance to ALB's target group then you will start seeing ALB's health check on the log.

![target group](https://dev-to-uploads.s3.amazonaws.com/i/qacmn2ijjtxz6yl2eva6.png)

### For worker instance

Follow same step as previous step on docker setup or create another AMI image from api instance and start the instance in private subnet.

```bash
$ git clone https://github.com/kokospapa8/ecs-fargate-sample-app.git
$ cd ecs-fargate-sample-app
$ docker-compose -f docker-compose-staging-worker.yml up --build
```

# Caveats

Everything checked out; in order to use this setup in production level, there are couple problems we need to address. 

- Scaling services
- Updating sources for further development

## Scaling services

We can create launch config and attach it to autoscaling group for automatic scaling. 

[What Is AWS Auto Scaling?](https://docs.aws.amazon.com/autoscaling/plans/userguide/what-is-aws-auto-scaling.html)

We need to prepare couple things for this to work

- Each instances must have environment variable injected on startup. There are couple ways to do this
- Pull new source
- Run docker-compose on start

There are various way to accomplish this, you can search for other methods

[Aws Ec2 run script program at startup](https://stackoverflow.com/questions/49594391/aws-ec2-run-script-program-at-startup)

## Updating sources for further development

- Configuration changes - environment
- pulling new src from git

### Brute force

Login into each instance and update the source and build image manualy

- ssh into each ec2 instance

```bash
> ssh -i "/path/to/keyfile/" ubuntu@<public_ip>
$ git pull
$ docker-compose -f docker-compose-staging-api.yml up --build 
```

Obviously this is not a great idea if you have multiple instances of ec2 running. Also ip address change if instances are running behind autoscaling group.

### Fabric

You can use `fabric` to alleviate such painful process. It's library designed to execute shell commands remotely over SSH. 

[Fabric](http://www.fabfile.org/)

I have provided sample fab file for accessing ec2 instances in public subnets.

```python
def _get_ec2_instances():
    instances = []
    connection = connect_to_region(
        region_name = "ap-northeast-2", #TODO beta env
        aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
        aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY")
    )

    try:
        reservations = connection.get_all_reservations(filters= {'tag:Name':'ecs-sample-api','tag:env':'staging'})
        for r in reservations:
            for instance in r.instances:
                instances.append(instance)

    except boto.exception.EC2ResponseError as e:
        print(e)
        return

    instances=filter(None, instances)
    return instances
```

[kokospapa8/ecs-fargate-sample-app](https://github.com/kokospapa8/ecs-fargate-sample-app/blob/master/fabfile.py)

When you run fabfile, it uses boto library to traverse ec2 instance with tag named as `ecs-sample-api`. Therefore you need to add  `ecs-sample-api` for Name tag for each ec2 instances.

```bash
# you need to install fab and boto library
$ pip install fabric, boto
#edit fabfile.py 
# SSH_KEY= "PATH/TO/YOUR/SSH_KEY"
# set env for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with ec2 describe policies

$ fab --list
$ fab gitpull
$ fab docker-restart
```

# Wrap up

We are done for staging environment setup. We now have a docker containers running on cloud evironment. However there are couple concerns I woud like to address.

- Accessing each instances manually to update the source is not secure job
- I would like to address immutability to container deployment.
- I would like to fully utilize my compute node for containers.


Let's move to part3
{% link https://dev.to/kokospapa8/how-to-deploy-django-app-to-ecs-fargate-part3-3i7p %}


# Reference

[Save yourself a lot of pain (and money) by choosing your AWS Region wisely](https://www.concurrencylabs.com/blog/choose-your-aws-region-wisely/)

[VPC with public and private subnets (NAT)](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Scenario2.html)

[Fabric](http://www.fabfile.org/)

[Securely Connect to Linux Instances Running in a Private Amazon VPC | Amazon Web Services](https://aws.amazon.com/ko/blogs/security/securely-connect-to-linux-instances-running-in-a-private-amazon-vpc/)

[What Is Amazon EC2 Auto Scaling?](https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html)