Using external Python packages with AWS Lambda layers
Tags: python, aws, tutorial
Published: 2020-12-06 17:56:00

[Amazon Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is an awesome service that lets you run code serverlessly: rather than keeping a server running to execute your code anytime, a server is spun up only when necessary in response to some event. This event could be from an HTTP request, a message posted to a queueing service like [Amazon SQS](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html), or even when a file is uploaded to an S3 bucket! 

Lambda supplies different [runtimes](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html) depending on what language you want to write your function in. For Python programming, this runtime includes the Python standard library, but what if you want to use external packages from [PyPI](https://pypi.org/) or elsewhere? This is something you can do with Lambda layers! Layers provide dependencies (or even a custom runtime) for your function, and in this quick tutorial we'll walk through how to use them!

## What you'll need

- An AWS account and a little experience using Lambda functions. If you haven't used them before, Amazon has [great tutorials](https://aws.amazon.com/getting-started/hands-on/run-serverless-code/)!
- [Docker](https://www.docker.com) installed on your computer, as we'll use that to install dependencies in a similar environment to the AWS Lambda Python runtime

Our demo project will be a very simple Lambda function that accesses the [PokéAPI](https://pokeapi.co/)'s `/api/v2/pokemon` endpoint to return information about a Pokémon, given a name passed in the event that triggers the function.

## Setting up our project locally

First we'll create a [virtual environment](https://docs.python.org/3/tutorial/venv.html) for our project and work in there, to keep any project dependencies separate from others on your computer:

```shell
$ mkdir poke-lambda 
$ cd poke-lambda
$ python -m venv venv
$ source venv/bin/activate
```

We're using the [requests](https://requests.readthedocs.io/en/master/) library to do our API calls, so install that with `pip install requests`. Our one file, `lambda_function.py` looks like this:

```python
import json
import requests

def lambda_handler(event, context):
    # Make sure a Pokemon name (or ID) was passed in the event object:
    pokemon = event.get('pokemon')
    if pokemon is None:
        return {
            'statusCode': 400,
            'body': json.dumps('Missing pokemon attribute in the event object')
        }

    # If we have a pokemon name/ID passed in, try to get info for it:
    r = requests.get(f"https://pokeapi.co/api/v2/pokemon/{pokemon}")

    if r.status_code == 200:
        status_code = 200
        body = json.dumps(r.json())
    else:
        status_code = r.status_code
        body = json.dumps(f"Could not load information for Pokemon {pokemon}")

    # Return what we got from the API:
    return {
        'statusCode': status_code,
        'body': body
    }
```

What this code does is check for the `pokemon` attribute in the `event` data the Lambda function receives, queries the PokéAPI for information about the Pokémon passed in, and returns the API's response back (with basic error handling). At this point our project structure looks like this:

```
poke-lambda
|   lambda_function.py
└───venv
```

## Building our Lambda layer

In order to put this code on AWS Lambda, we need a way to include the `requests` library first! Lambda layers are .zip files containing libraries (and a custom runtime if you need) your function requires. For Python lambda layers, AWS [requires](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html) libraries to be placed in either the `python` or `python/lib/python-3.x/site-packages` folders. 

1. While the virtual environment is activated for your project, save a list of its dependencies to a `requirements.txt` file: `pip freeze > requirements.txt`. 

2. Create a `python` folder. This will house all the libraries to go in our layer. At this point, your folder structure should be:

    ```
    poke-lambda
    | lambda_function.py
    | requirements.txt
    └───venv
    └───python
    ```

3. From the project root, run the following command to build your dependencies in a container similar to the Lambda execution environment Amazon provides (courtesy of [lambci](https://hub.docker.com/u/lambci)):

    ```shell
    $ docker run --rm \
    --volume=$(pwd):/lambda-build \
    -w=/lambda-build \
    lambci/lambda:build-python3.8 \
    pip install -r requirements.txt --target python
    ```

    Let's break down what this command does, argument-by-argument:
    - `--rm`: Makes sure that once the Docker container exits, it's removed (keeps things clean)
    - `--volume`: [Bind mounts](https://docs.docker.com/storage/bind-mounts/) the current working directory to the `/lambda-build` directory on the container (this allows the container to access the requirements.txt file we've just generated, and add files to the `python` directory). If you're on Windows, you can also paste in a full path to your project root instead of `$(pwd)`.
    - `-w`: Sets the working directory in the container (in this case, to our project root)
    - `lambci/lambda:build-python3.8`: The docker image to run this container from -- make sure it matches the Python version you're working with! For reference, Amazon currently provides runtimes for Python 2.7, 3.6, 3.7 and 3.8 (I'm using 3.8 here)
    - `pip install -r requirements.txt --target python`: Installs the project dependencies as normal, from the requirements.txt file to the `python` folder (more on that [here](https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-t))

4. You should see some output from `pip`. This is running in the Docker container. If successful, you should have dependencies installed to the `python` folder you created earlier! Mine looks like this:

    ```
    poke-lambda
    | lambda_function.py
    | requirements.txt
    └───venv
    └───python
        └───bin
        └───certifi
        └───certifi-2020.11.8.dist-info
        └───chardet
        └───chardet-3.0.4.dist-info
        └───idna
        └───...
    ```

5. At this point, all we have to do is zip our `python` folder: `zip -r layer python/`. This will create a `layer.zip` file in your project's root directory. 

6. Next, upload the .zip file to Lambda! Sign into your AWS Console and head over to Services > Lambda > Layers (it should be under "Additional resources"). Click "Create layer" and give your layer a name! Pick the runtime that corresponds to the version of Python you're trying to use in your function, and upload the .zip file you created earlier:

![Filling out the name, runtime and layer .zip file in the Create Layer form](https://dev-to-uploads.s3.amazonaws.com/i/cgv94fjghy0tfy3vf7y0.png)

## Adding the layer to our Lambda function

We'll walk through this using the front-end console, but you can totally do this all over the AWS CLI!

1. While in your AWS Lambda console, click "Create function". Set it up however you want (pick a Python runtime though)

2. In the Lambda function designer, click on the "Layers" button:

    ![Clicking on the Layers button in the designer](https://dev-to-uploads.s3.amazonaws.com/i/gecow286pxsz8eqebxi0.png)

3. A table full of layers your function is using should appear below the designer. There aren't any so far, so lets add ours! Click "Add a layer" and choose "Custom layers". You should be able to see your layer you created earlier in the dropdown. Select and add it!

4. Once added, you'll be taken to the Designer again. Scroll down and paste your function code in (you can alternatively upload a .zip file containing just the `lambda_function.py` file since that's all we need here)

    ![Pasting in our function code from before](https://dev-to-uploads.s3.amazonaws.com/i/6inl9dz4julagqrpe8ei.png)

5. Save and click "Deploy" to update your function's code. Let's send our code a test event! At the top of the page, click on "Configure test events" under "Select a test event". We'll create an event payload to fetch details about Charizard:

    ![Configuring the test event for the Lambda function. This is a JSON payload with only one key (pokemon) equal to Charizard](https://dev-to-uploads.s3.amazonaws.com/i/4fl59jxnbpzol9uj8aki.png)

6. Save your test event, pick it from the dropdown and click "Test". You can now see your function returning results!

    ![Testing the Lambda function with the test event configured in step 5 should result in a success](https://dev-to-uploads.s3.amazonaws.com/i/w1pzxxndhl85kwxl5dtn.png)

That's all there is to it! Keep in mind functions can have multiple Lambda layers (up to 5 right now) and functions can share layers too.

In this tutorial we:

- Wrote a simple Lambda function that calls an external API and returns the results
- Created a Lambda layer containing all dependencies for our project that we can reuse in other projects later, using the [lambci/lambda](https://hub.docker.com/r/lambci/lambda) images and the Lambda GUI

Keep in mind all of this can also be accomplished over the AWS CLI (uploading the function, creating the layer and so on) if you prefer this method. There are lots of [AWS docs](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-using) which document this pretty well!

Hope you enjoyed this tutorial! Let me know if it was helpful in your project 😄