Manage Python dependencies in Serverless projects with serverless-layers plugin
Tags: aws, serverless, python
Published: 2020-12-31 12:44:18

---
title: Manage Python dependencies in Serverless projects with serverless-layers plugin
published: true
date: 2020-12-27 00:00:00 UTC
tags: aws,serverless,python
canonical_url: https://jurriaan.cloud/manage-python-dependencies-in-serverless-projects-with-serverless-layers-plugin/
---

Struggling to properly package your Python dependencies into your functions? In this post I’ll take a look at `serverless-layers`, a Serverless Framework plugin that can do it for you!

## Sample project

Given a simple project that has the following directory structure:

```
serverless-layers-example/
├── requirements.txt
├── serverless.yml
└── src
    ├── __init__.py
    └── functions
        ├── __init__.py
        └── example_function
            ├── __init__.py
            └── handler.py
```

Where `serverless.yml` contains the following:

```yaml
service: serverless-layers-example

provider:
  name: aws
  runtime: python3.8

functions:
  example_function:
    handler: src/functions/example_function/handler.handle_event
```

The `example_function` is a simple function that calls an external API ([cat-facts](https://alexwohlbruck.github.io/cat-facts/) 🐱) using the `requests` library and returns the response in the body.

```python
import json

import requests

FACTS_URL = "https://cat-fact.herokuapp.com/facts"

def get_fact():
    r = requests.get(FACTS_URL, timeout=5)

    return r.json()[0]["text"]

def handle_event(event, context):
    fact = get_fact()

    body = {
        "message": fact
    }

    response = {
        "statusCode": 200,
        "body": json.dumps(body)
    }

    return response

```

The `requirements.txt` file contains the following (generated by running `pip freeze > requirements.txt` after running `pip install requests`):

```
certifi==2020.12.5
chardet==4.0.0
idna==2.10
requests==2.25.1
urllib3==1.26.2
```

After deploying the function with `sls deploy` and invoking the function with `sls invoke -f example_function`, you’ll be greeted with the following error:

```
{
    "errorMessage": "Unable to import module 'src/functions/example_function/handler': No module named 'requests'",
    "errorType": "Runtime.ImportModuleError"
}
```

Let’s see how `serverless-layers` can help us solve the above.

## serverless-layers

One of the first hits when searching for solutions to manage Python dependencies in Serverless projects is the popular [serverless-python-requirements](https://github.com/UnitedIncome/serverless-python-requirements) plugin. However [serverless-layers](https://github.com/agutoli/serverless-layers) is a plugin that can work just as well.

As the name suggests, `serverless-layers` utilizes [Lambda layers](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html) to manage functions dependencies. It automatically attaches a layer to each function defined in your `serverless.yaml` containing the dependencies.

Let’s give it a try!

### Install

- Create a `package.json` by running `npm init` (If not already exists)
- Install the plugin `npm i -D serverless-layers`
- Add the plugin to the `plugins` section in `serverless.yml`
- Because we haven’t specified the `deploymentBucket` in the `provider` section, we have to configure the `layersDeploymentBucket` in the plugin configuration in the `custom` section and make sure the bucket exists.

```yaml
service: serverless-layers-example

provider:
  name: aws
  runtime: python3.8

plugins:
  - serverless-layers

custom:
  serverless-layers:
    layersDeploymentBucket: <bucketName>

functions:
  example_function:
    handler: src/functions/example_function/handler.handle_event
```

Redeploy the function

![serverless-layers deploy](https://jurriaan.cloud/assets/images/serverless-layers-deploy-1.png)

Invoke the function again with `sls invoke -f example_function`. You should now see the response as expected.

![serverless-layers invoke](https://jurriaan.cloud/assets/images/serverless-layers-invoke-1.png)

### What happened

1. The plugin first checks if the layer for this project and stage already exists by checking if a `requirements.txt` exists at `<bucketName>/serverless/<projectName>/<stage>/layers/requirements.txt`.

2. If not, it downloads the dependencies as specified in `requirements.txt` to `.serverless/layers/python/`

3. This directory is zipped and uploaded to the bucket. The resulting zipfile can be found the `.serverless` directory and with the example config as above has the name `<projectName>-<stage>-<runtime>-default.zip`. You could inspect the contents of this zipfile by running `zipinfo <zipfileName>`.

4. A Lambda layer is created
![Lambda layer in AWS console](https://jurriaan.cloud/assets/images/serverless-layers-1.png)

5. The `requirements.txt` that the layer is based on is uploaded to the bucket
![Zipped layer and requirements.txt in AWS console](https://jurriaan.cloud/assets/images/serverless-layers-bucket-1.png)

6. The layer gets attached to the function. This is visible in the `cloudformation-template-update-stack.json` file in the `.serverless` directory.

```json
"ExampleUnderscorefunctionLambdaFunction": {
  "Type": "AWS::Lambda::Function",
  "Properties": {
    "Handler": "src/functions/example_function/handler.handle_event",
    "Runtime": "python3.8",
    "FunctionName": "serverless-layers-example-dev-example_function",
    "Layers": [
      "arn:aws:lambda:us-east-1:************:layer:serverless-layers-example-dev-python-default:1"
    ]
  },      
}
```

### Adding/updating dependencies

When adding or updating the dependencies the same process as described above gets triggered, but only if the `requirements.txt` has actually changed, compared to the one stored in the bucket next to the active layer.

Example of the output when changes are detected: 
![Changes detected](https://jurriaan.cloud/assets/images/serverless-layers-2.png)

Example of the output when no changes are detected: 
![No changes](https://jurriaan.cloud/assets/images/serverless-layers-3.png)

### Caveats

- At the time of writing the plugin does only perform a simple check to see if dependencies changed. Changing the order of the listed dependencies already triggers an update of the Lambda layer.

## Conclusion

`serverless-layers` is a handy plugin that can help you with managing Python dependencies using Lambda layers. It takes away the operational overhead of managing Lambda layers and because the dependencies are no longer packaged with the Lambda function itself it drastically reduces its size and time it takes to deploy new versions of your code.

It might even improve cold-start times (check out [This is all you need to know about Lambda cold starts#Where the dependency is loaded from matters](https://lumigo.io/blog/this-is-all-you-need-to-know-about-lambda-cold-starts/)), but I have not tested that.

`serverless-layers` currently only supports dependencies defined through `requirements.txt`. There is no support for other Python dependency management tools such as Pipenv/Pipfile or Poetry.