Creating Python AWS Lambda Layers with Docker
Tags: aws, python, serverless
Published: 2020-05-25 13:59:43

---
title: Creating Python AWS Lambda Layers with Docker
published: true
description: Create your own library of AWS Lambda Layers using Docker!
tags: aws, python, serverless
---
By default your AWS Lambda functions include only core Python functionality.  Lambda layers are a fantastic feature allowing you to add external dependencies to your Lambda functions.  In this post I'll walk you through how you can create your own library of Lambda layers using Docker.

## What are Lambda Layers?

If you're unfamiliar with Lambda layers, a layer is additional code you can include at runtime with your Lambda functions to incorporate external libraries into your Lambda function.  By default Lambda functions only include core Python libraries.  Layers are a way for you to include other external libraries into your Lambda functions.  A useful feature of layers is that they exist outside of the Lambda function itself.  Once you create a layer you can use that layer across all your Lambda functions.  You can read more about [layers](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html) from the AWS documentation.

# Creating a Lambda Layer with Docker

For this post we're going to create a layer using Flask as an example.  Creating a Lambda Layer with Docket has four main steps:

1. Setup the a local directory for the layer
2. Use Docker to install the layer packages
3. Create a zip file of your directory
4. Upload to AWS using the AWS CLI

The next few sections will describe each of these steps in more detail.  The last section will show how these steps can be combined into a single bash script to make deployment even easier.

## Setting Up Your Layer Directory

The first step is to build the local directory structure that will contain all the artifacts for our layer.  In the terminal enter the commands below.  A new directory called `flask-layer` is created along with a subdirectory `python/lib.python3.8/site-packages` where we will install our Python packages to.

```commandline
mkdir flask-layer
cd flask-layer
mkdir -pv python/lib/python3.8/site-packages
```

If you want to create your Layer for multiple Python versions you can repeat the last command and change "python3.8" to whatever other version of Python you want to include.

Within the `flask-layer` directory create a `requirements.txt` file that includes the following.

```text
flask==1.1.1
```

The `requirements.txt` file defines the packages that will be included in your layer.  If you have additional dependencies you want to include in your layer you can add those to this `requirements.txt` file.
  
One thing to be aware of with layers is that when they are loaded into your Lambda function they cannot exceed 250MB in size.  A Lambda function can include up to 5 different layers.  Given these limitations, having smaller layers you can mix-and-match across your functions is preferable rather than building large, single-purpose, layers.

## Installing the Requirements with Docker

Docker will install the dependencies you specify in `requirements.txt`.  Docker has [images](https://hub.docker.com/r/lambci/lambda/) to replicate the AWS Lambda environment.  Using those images Docker will install the libraries you specified in `requirements.txt` that are compatible with the Lambda runtime.  In the terminal enter the following code.  Make sure you have Docker running before entering this command!

```commandline
docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"
```

After running the command you should see the `site-packages` directory you created in the previous step populated with all your dependencies.

If you have additional versions you want to create your Layer for make sure you substitute the occurrences of "python3.8" with the correct version you want to use.

## Uploading Your Layer to AWS

AWS requires all the layer code to be in a zip archive, so we need to zip everything in the `python` directory.

```commandline
zip -r flask-layer.zip python
```

All the contents of the `python` directory are included in the zip archive called `flask-layer.zip`.

Now the layer can be uploaded to AWS using the [AWS CLI](https://aws.amazon.com/cli/).  You need to provide a few parameters in this step:

- `layer-name` is the name your want to give your layer
- `description` to briefly summarize the layer
- `zip-file` is the path to the zip archive you created in the previous step
- `compatible-runtimes` details the Python versions your layer is compatible with

Enter the following into the terminal to publish our layer to AWS:

```commandline
aws lambda publish-layer-version \
    --layer-name "flask-layer" \
    --description "Lambda Layer for Flask 1.1.1" \
    --zip-file "fileb://flask-layer.zip" \
    --compatible-runtimes "python3.8"
```

And with that your Flask layer is published to AWS!  Now let's simplify this process a little more by packaging some of these steps into a single bash script.

# Creating Your Own Lambda Layer Directory

To manage my layers I created a single bash script packaging most of the steps above together to make deploying layers easier.  I have a directory where I manage all my layers.  Below is a sample of those layers.  Each layer has its own directory and the `create-lambda-layer.sh` script handles installing the layer dependencies and uploading them to AWS.

![Screenshot of Lambda layer directory](https://dev-to-uploads.s3.amazonaws.com/i/widyhy8qtq2ks6wdopbf.png)


Here is what `create-lambda-layer.sh` looks like:

```bash
#!/usr/bin/env bash


while [[ "$#" -gt 0 ]]; do case $1 in
  -v|--version) pythonEnvs+=("$2"); shift;;
  -n|--layer-name) layerName="$2"; shift;;
  -d|--desc) layerDescription="$2"; shift;;
  *) echo "Unknown parameter passed: $1"; exit 1;;
esac; shift; done


# Create and install requirements to directory.
for penv in ${pythonEnvs[@]}; do
    mkdir -pv python/lib/${penv}/site-packages
    docker run -v "$PWD":/var/task "lambci/lambda:build-${penv}" /bin/sh -c "pip install -r requirements.txt -t python/lib/${penv}/site-packages/; exit"
done


# Create zip file of environments.
zip -r ${layerName}.zip python


# Publish Layer to Lambda.
aws lambda publish-layer-version \
    --layer-name "${layerName}" \
    --description "${layerDescription}" \
    --zip-file "fileb://${layerName}.zip" \
    --compatible-runtimes ${pythonEnvs[@]}
```

The script accepts parameters for Python versions (you can provide multiple!), a name and description for your layer.  From those parameters the script will generate the file structure, run the docker command for each Python version, create the zip archive, and then update the layer to AWS.

We can test this by installing the [Twilio](https://github.com/twilio/twilio-python) helper library.  In the Terminal, create a directory where you're going to store your Lambda layers.

```commandline
mkdir my-lambda-layers
cd my-lambda-layers
```

Create a file inside that directory, `create-lambda-layer.sh`, and paste the code above into the file.

Each layer needs to have its own directory, run the following code in the Terminal to create a directory `twilio-layer`.

```commandline
mkdir twilio-layer
cd twilio-layer
```

The last bit of setup is to create the `requirements.txt` file, which should include:

```text
twilio==6.40.0
```

Now we can call our bash script to finish creating and deploying our layer (again, make sure you have Docker running before executing this code!)

```commandline
bash ../create-lambda-layer.sh -v "python3.8" -n "twilio-layer" -d "Twilio 6.40.0 Layer"
```

And your layer is deployed to AWS!

If you want to create your layer for multiple different Python versions all you have to do is specify additional `-v` parameters with those versions.  The example below creates a layer for the Python 3.6, 3.7 and 3.8 runtimes.

```commandline
bash ../create-lambda-layer.sh -v "python3.8" -v "python3.7" -v "python3.6" -n "twilio-layer" -d "Twilio 6.40.0 Layer"
```

Easy Peasy!

# Wrapping Up

In this post I showed you how to deploy a Lambda Layer through the commandline using Docker.  The bash script provided isn't fully featured, but in its current form provides a convenient way deploy your Lambda layers.  Layers are a neat tool that can help you add functionality to your Lambdas.  Docker simplifies creating layers so you can manage your own library of layers that you can share with your friends and coworkers!

Happy Building!