Migrate python async worker to asynchrounous Lambda
Tags: aws, serverless, lambda, django
Published: 2020-10-10 06:44:56

[Original blog post](https://blog.kokospapa.com/Migrate-python-async-worker-to-asynchrounous-Lambda-36a0551c835b4bd1b6559b0d9450bb56)

# Intro

When you build a web service, there is a time that you need to implement simple async worker for background process. I am big fan of Django and whenever I need a async worker, I used `celery` or `rq` for convenience. `celery` or other async framework provides a lot of cool features but most of the time all I need is a simple background process that would not interrupt user experiences.

Overtime, as paradigm shifts to cloud, I have been thinking lambda as new async worker solution due to deployment complexity, scaling and cost constraint. Here are pros and cons for using lambda instead of `celery` on cloud environment(non-local environment).

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/9to8z3plvgbfouf085ea.png)

### Advantage

- FasS advantage
    - scaling benefit - cost, ops, etc
    - Resiliency
    - faster development
- No need to maintain Broker (e.g elastic search redis)
- Smaller side effect on code change
- Fully-managed

### Disadvantage

- FasS disadvantage
    - limited state
- Hard to reuse app code
    - Django ORM or settings module
    - You can still use them with django package installed but inefficient
    - can't reuse code from django app
    - Django package is too heavy to use on lambda (still usable but too complex to setup
- Latency on startup
- Json formatted Invocation parameter
    - If you need to pass binary or big size paras, need to consider S3 or other medium

### Limitation

Most of early stage limitation are lifted as lambda is in mature state (e.g conccurent executions, layers, etc)

[AWS Lambda quotas](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html)

- timeout - 15min
    - If your async worker take more than 5min, you need to break up codes in to multiple lambda functions or use other distributed system such as EMR, etc.
- Invocation payload
    - asynchronous - **256KB**

### Consideration

- VPC, Subnet, Security Group
    - If you are accessing Resources in private VPC, you need to consider placing lambda in same subnet.
    - If you are lambda needs to talk to outside world, private subnets needs NAT gateway
- Maintenance

# Setup

AWS recently came up with SAM(Serverless Application Model) which utilize cloudformation for easier deployment of lambda application. SAM provides functionality such as API gateway and statemachine, but we will only be using `AWS::Serverless::Function` resource for our purpose. You can consider using `Zappa` for other option.

## Github

refer to this repo for details

[kokospapa8/async-lambda-sample-app](https://github.com/kokospapa8/async-lambda-sample-app)

## Prerequisite

You need following AWS resource created in order to invoke asynchronous lambda.

### DLQ

In order use `asynchronous lambda`, you need to provide DLQ in case lambda fails and retry. Click link for details. I will create SNS topic in this post.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/yun3xmo8yu0m3wius0nw.png)

[Asynchronous invocation](https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html#dlq)

### SECRET

Let's assume that you have mysql db in public VPC or s3 and need to access them in order to process some data, you can add secrets to lambda env, but it is not a good idea to put secret in plaintext for env variable. You can use `aws secret manger` , `ssm parameterstore` or `vault` . I will use `ssm parameterstore` for this post. Refer to previous post for creating secrets.(parameter store section)

[How to deploy django app to ECS Fargate part3](https://www.notion.so/How-to-deploy-django-app-to-ECS-Fargate-part3-13875ffa0c7e4e1da3d1c46cad0bae94)

`arn:aws:ssm:<region>:<account_id>:parameter/SAMPLE/BUCKET`

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/o3do07z9o149wb2i24l7.png)

### IAM ROLE

You need lambda execution role with following permission and polcies attached.

```
Policies
- AWSLambdaExecute
Permission 
- SNS – sns:Publish
- kms - kms:Decrypt
- ssm - ssm:GetParameters, ssm:DescribeParameters, ssm:GetParameter
You need following permission if you want to place your lambda function in a designated VPC
- ec2 - ec2:CreateNetworkInterface, ec2:DeleteNetworkInterface, ec2:DescribeSecurityGroups
```

## SAM

[What is the AWS Serverless Application Model (AWS SAM)?](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html)

### Install (on macOS)

[Installing the AWS SAM CLI on macOS](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install-mac.html)

```bash
$ aws configure
AWS Access Key ID [None]: your_access_key_id
AWS Secret Access Key [None]: your_secret_access_key
Default region name [None]: 
Default output format [None]:

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
$ brew tap aws/tap
$ brew install aws-sam-cli
```
### Development

```bash
$ sam init
```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/f856gvse9rpaspn55wgt.png)

This code will create following structure 

```bash
sam-app/
   ├── README.md
   ├── events/
   │   └── event.json
   ├── hello_world/
   │   ├── __init__.py
   │   ├── app.py            #Contains your AWS Lambda handler logic.
   │   └── requirements.txt  #Contains any Python dependencies the application requires, used for sam build
   ├── template.yaml         #Contains the AWS SAM template defining your application's AWS resources.
   └── tests/
       └── unit/
           ├── __init__.py
           └── test_handler.py
```
### template.yaml

Update template file if you need more info on syntax, refer to the link.


[AWS::Serverless::Function](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html)
- template.yaml
```
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sample-app
  Sample SAM Template for sample-app

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  SampleAppdFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: sample_app/
      DeadLetterQueue:
        Type: SNS
        TargetArn: arn:aws:sns:<region>:<account_id>:sample-dlq
      Handler: app.lambda_handler
      Runtime: python3.8
      Description: sample lambda
      EventInvokeConfig:
        MaximumEventAgeInSeconds: 60
        MaximumRetryAttempts: 2
      FunctionName: SampleApp
      Role: arn:aws:iam::<account_id>:role/sample_lambda_execution_role
      Environment:
        Variables:
          S3_URL: ""

Outputs:
  SampleAppdFunction:
    Description: "Sample app Function ARN"
    Value: !GetAtt SampleAppdFunction.Arn
```
- requirements.txt
```
boto3
```
- app.py
This sample app receives image_url as parameter and puts the image on s3bucket name provided on s3 bucket. (AWSLambdaExecute policy has s3 access permission)
```
import boto3
import os
import requests

def lambda_handler(event, context):
    image_url = event['image_url']
    S3_BUCKET_PARAM = os.environ['S3_BUCKET_PARAM']

    # download

    image = requests.get(image_url).content


    ssm_client = boto3.client('ssm', region_name="ap-northeast-2")
    response = ssm_client.get_parameter(
        Name=S3_BUCKET_PARAM,
        WithDecryption=True
    )
    bucket_name = response['Parameter']['Value']

    s3_client = boto3.client('s3', region_name="ap-northeast-2")
    response = s3_client.put_object(
        Bucket=bucket_name,
        Key="image.png",
        Body=image
    )

    return response
```

## Build

Once your logic code is ready, you can build it using following command 

```yaml
sam build -t template.yaml --region <region_name>
```

You can test your function with following command

```bash
$ sam local invoke
# if you need to provide environment variable or paramters, use following command
$ sam local invoke -e events/event.json --env-vars env.json
```

```json
#env.json
{
  "SampleAppdFunction": {
	    "S3_BUCKET_PARAM_ARN": "parameterstore_arn",
  }
}
#event.json
{
  "image_url": "https://picsum.photos/200/300"
}
```

## Deploy

Once your function is tested locally, let's deploy the function to cloud. You have two options. Enter appropriate response to prompt

```bash
$ sam deploy --guided

Configuring SAM deploy
======================

	Looking for samconfig.toml :  Found
	Reading default arguments  :  Success

	Setting default arguments for 'sam deploy'
	=========================================
	Stack Name [sample-app]: y
	AWS Region [ap-northeast-2]: y
	#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
	Confirm changes before deploy [Y/n]: y
	#SAM needs permission to be able to create roles to connect to the resources in your template
	Allow SAM CLI IAM role creation [Y/n]: y
	Save arguments to samconfig.toml [Y/n]: y

```
output should look like this
```

	Looking for resources needed for deployment: Found!

		Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
		A different default S3 bucket can be set in samconfig.toml

	Deploying with following values
	===============================
	Stack name                 : sample-app
	Region                     : ap-northeast-2
	Confirm changeset          : True
	Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
	Capabilities               : ["CAPABILITY_IAM"]
	Parameter overrides        : {}

Initiating deployment
=====================

	Saved arguments to config file
	Running 'sam deploy' for future deployments will use the parameters saved above.
	The above parameters can be changed by modifying samconfig.toml
	Learn more about samconfig.toml syntax at 
	https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

	Deploying with following values
	===============================
	Stack name                 : sample-app
	Region                     : ap-northeast-2
	Confirm changeset          : True
	Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
	Capabilities               : ["CAPABILITY_IAM"]
	Parameter overrides        : {}

Initiating deployment
=====================

Waiting for changeset to be created..

CloudFormation stack changeset
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation                                                    LogicalResourceId                                            ResourceType                                               
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add                                                        SampleAppdFunctionEventInvokeConfig                          AWS::Lambda::EventInvokeConfig                             
+ Add                                                        SampleAppdFunction                                           AWS::Lambda::Function                                      
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:ap-northeast-2:982947632035:changeSet/samcli-deploy1602235720/1bb1b8a3-da32-4de9-9a14-49aaae3f066f


Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y

2020-10-09 18:29:04 - Waiting for stack create/update to complete

CloudFormation events from changeset
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                                ResourceType                                  LogicalResourceId                             ResourceStatusReason                        
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS                            AWS::Lambda::Function                         SampleAppdFunction                            -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::Function                         SampleAppdFunction                            Resource creation Initiated                 
CREATE_COMPLETE                               AWS::Lambda::Function                         SampleAppdFunction                            -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           Resource creation Initiated                 
CREATE_COMPLETE                               AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           -                                           
CREATE_COMPLETE                               AWS::CloudFormation::Stack                    sample-app                                    -                                           
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs                                                                                                                                                                              
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key                 SampleAppdFunction                                                                                                                                               
Description         Hello World Lambda Function ARN                                                                                                                                  
Value               arn:aws:lambda:ap-northeast-2:982947632035:function:SampleApp                                                                                                    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - sample-app in ap-northeast-2
```
Check cloudformation and lambda console for successful deployment.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/h4rcy28q0eaflxwf1s61.png)
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/m6swx0h8t5f3r9v9dvzc.png)

## Invoke function in lambda console
Once Lambda is deployed, you can create test event in console.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/bqdo1i2rbf6b8b1yoy8g.png)
You will get successful log data.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/3qbsa46a5g8a0p47wo4c.png)

### Invoke from python app

Now you need to invoke lambda function from your app code. Here are examples you can use.

```json
# invoke_from_app.py
import boto3
import json

SETTINGS = {
    "AWS_DEFAULT_REGION": "ap-northeast-2",
    "ENV": "prod"
}
def main():
    lambda_client = boto3.client('lambda', region_name=SETTINGS['AWS_DEFAULT_REGION'])

    payload = {
        "image_url": "https://picsum.photos/200/300"
    }

    ret = lambda_client.invoke(
        FunctionName="SampleApp",
        InvocationType="DryRun" if SETTINGS['ENV'] == "test" else "Event",
        Payload=json.dumps(payload)
    )
    print(ret)

if __name__ == "__main__":
    main()
```
# Monitoring

Celery or rq provides native or 3rd party too for monitoring such as `sentry` . There are some options for monitoring lambda functions but SAM application also provides minimal monitoring  environment. Go to lambda service and `application` menu. Select `Monitoring` tab to dashboard and cloudwatch logs. You can also configure `x-ray` for tracing.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/0e5hcdaio3yixo6p228j.png)
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/c84lrxlh4gfi1ecua5m7.png)

## CD

Let's talk about CD pipeline, I will use conditional trigger on github workflow and run the 

### Github workflow

following workflow file will build and deploy to lambda. You need to set secret key with AWS credential.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/1ax8k81yxhib2a5x2xqc.png)

```yaml
on:
  push:
    branches:
      - master

name: Lambda deployment
jobs:
  deploy:
    name: Lambda deploy
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ap-northeast-2

    - name: Build, and deploy
      run: |
        /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
        test -d ~/.linuxbrew && eval $(~/.linuxbrew/bin/brew shellenv)
        test -d /home/linuxbrew/.linuxbrew && eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)
        test -r ~/.bash_profile && echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.bash_profile
        brew --version
        brew tap aws/tap
        brew install aws-sam-cli
        sam --version

        sam build
        sam deploy --stack-name sample-app --region ap-northeast-2 -t .aws-sam/build/template.yaml --capabilities CAPABILITY_IAM --no-confirm-changeset --s3-bucket aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
```

If you want to implement canary or linear deployment using codedeploy, refer to the following link for more detail.

[Deploying serverless applications gradually](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/automating-updates-to-serverless-apps.html)

# Conclusion

Once lambda is deployed, I removed 20% of my app code related to RQ. I also removed broker(Elasticsearch Redis) service anymore. Moreover since I was using ECS and EKS, I can remove additional task and pod for more api tasks and pods. I am pretty happy with the result that I don't need to manage another container on production. If you have simple workers running on RQ or Celery, I recommend that you try this setup. Thank you for reading.