How to Package Dependency for AWS Lambda with Docker
Published: 2024-05-13 18:21:17

## Introduction:

This guide explains how to package dependencies for your AWS Lambda function. The tutorial demonstrates automating the process of packaging Python dependencies for AWS Lambda layers using Docker.

By the end of this guide, you will have created a `Dockerfile`. Building this file produces a Docker Image. From this image, you can run a Docker container to generate a `.zip` file. You can then upload this file to Lambda as a dependency package for your Lambda function.


We will explore two methods:

   1. **Manual method:** involving setting up an EC2 instance

   2. **Automated method:** using Docker containers.

So, let's dive right in! We'll walk you through each step for both the manual and automated approaches in the sections below. Let's get started!

## Manual Approach:

This method requires setting up the following:

- **Launching:** an EC2 instance with Amazon Linux OS.
- **Installing:** the following:
   - The desired version of Python,
   - Pip, and
   - The external Python library into a target folder.
- **Compressing:** the library folder.
- **Using:** scp (secure copy) to transfer it from the remote machine to your local computer.
- **Uploading:** the compressed folder from step 3 to the Lambda function.

This approach could be ideal if you're fond of hands-on processes. However, for those in search of automation and a method that's free of cloud charge, kindly opt for the second approach.

## Automated Approach:
In this method, we'll use Docker containers to streamline the process. We'll explore the files involved in the project and the functions they perform.

**1. Dockerfile**

This is the `Dockerfile`, serving as the entry point into our code. We use it to configure the Amazon Linux image as the OS within our container. It orchestrates the setup of all supporting code files, such as `install_python.sh` and `requirements.txt`. Each line in the code snippet has a comment to explain its function.

```Dockerfile
# Use the official Amazon Linux Docker image
FROM amazonlinux:latest

# Update package repositories and install necessary packages
RUN yum update -y

# Set the working directory
WORKDIR /usr/src

# Copy the bash script and requirements file into the container
COPY install_python.sh .
COPY requirements.txt .

# Make the bash script executable
RUN chmod +x install_python.sh

# Run the bash script to install Python
RUN ./install_python.sh

# Install Python dependencies into 'modules' directory
RUN pip3.12 install -r requirements.txt -t /usr/src/modules/

# Zip up the installed dependencies
RUN zip -9r /usr/src/modules.zip /usr/src/modules

# Declare a volume to persist data
VOLUME /usr/src/data

# Move modules.zip to the volume
CMD ["mv", "/usr/src/modules.zip", "/usr/src/data/"]
```

**2. Install_python.sh**
This script builds and installs Python from source, along with installing pip. It is called from the `Dockerfile`.

```Python
#!/bin/bash

# Install development tools
yum groupinstall "Development Tools" -y

# Install necessary dependencies
yum install -y gcc bzip2-devel libffi-devel openssl openssl-devel wget

# Set Python version
PYTHON_VERSION="3.12.2"

# Download Python source
wget "https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz"

# Extract Python source
tar xzf "Python-${PYTHON_VERSION}.tgz"

# Navigate into the extracted directory
cd "Python-${PYTHON_VERSION}"

# Configure Python build with OpenSSL 1.1
./configure --with-openssl=/usr/lib64/openssl11 --enable-optimizations

# Build and install Python
make && make install

# Download get-pip.py
curl -O "https://bootstrap.pypa.io/get-pip.py"

# Install pip for Python 3.12
python3.12 get-pip.py

# Check pip version
pip3.12 --version
```

**3. Requirements.txt**
Our `requirements.txt` file lists the libraries needed for packaging the project, following a standard practice in Python projects. The content of the file is as follows:

```txt
requests
```

## Compile Python Version
To compile a different Python version, you can obtain the link to your preferred version from [here](https://www.python.org/ftp/python/). You have to ensure that the link points to a `.tgz` file.  If you select a different version from the one in this post, adjust the following lines in the `Dockerfile` and `install_python.sh`.

**1. Install_python.py:**

```python
# Set Python version
PYTHON_VERSION="3.12.2"

# Install pip for Python 3.12
python3.12 get-pip.py

# Check pip version
pip3.12 --version
```

The third digit in the version number, 3.12.2, is included in the `PYTHON_VERSION`, but Python ignores it when running `get-pip.py` and in `pip3.12`. This applies to any version of Python we decide to use.

**2. Dockerfile**

```bash
# Install Python dependencies into 'modules' directory
RUN pip3.12 install -r requirements.txt -t /usr/src/modules/
```

## Result:
When we build our Docker image and convert it to a container, the output is a zip file named modules.zip in the same folder as our `Dockerfile`. Below is the terminal log of successfully runing our `Dockerfile`.

![output of successfully running the docker file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0vtajk9hulgmx0y2wq18.png)


## Code:
Our code is hosted on a GitHub repository. You can access it: [here](https://github.com/ichdamola/package-lambda-dependency).

## Conclusion:
In this guide, we covered two approaches to packaging Python dependencies for AWS Lambda. The first involves using an AWS EC2 instance, installing Amazon Linux OS on it, and building Python from source, along with installing pip, which is then used to install dependencies listed in the requirements.txt file. The second approach streamlines the process using Docker, with the `Dockerfile` as the entry file.

You can read more in the AWS documentation [here](https://docs.aws.amazon.com/lambda/latest/dg/python-package.html).