Using Docker on Lambda for Postgres to S3 Backups 
Tags: lambda, docker, postgres, aws
Published: 2021-01-07 23:22:48

---
title: Using Docker on Lambda for Postgres to S3 Backups 
published: true
description: Exploring AWS Lambda's new container image support by implementing a cron job that backs up my Postgres database to S3.
tags: lambda, docker, postgres, aws
cover_image: https://i.postimg.cc/N0c38Ns6/dia3.png
---

### Intro

If you don't care about any context you can just skip down to the [Ship it](#ship-it) section.

I've never written a tech blog post before, but I figured out how to leverage AWS Lambda's [newly announced container image support (Dec 1, 2020)](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/) to back up a database I'm maintaining and it wasn't as straightforward as I'd hoped, so I figured I'd write something on it.

For context, as a cost-cutting measure™, I have just a single EC2 instance with some docker containers running my application, and that same EC2 instance houses my database! Sorry if you hate it!

If you *also* didn't feel like dealing with the costs and complexities of RDS & snapshotting and just want a nice way to back up your data, you're in the right place!

### My reasons

1. I have 2- count 'em- 2 users for an app I made and I want to make sure their data is less ephemeral than the reliability of a database-on-EC2-instance setup would suggest. I imagine this scales to databases much larger than what I have, but I haven't tested it out personally. I suppose you're up against the max timeout (15 minutes), and the max memory capacity (10GB) of a Lambda execution.
1. It's super cheap- for S3, all you're paying for is the storage, since data transfer *into* S3 is free, and you shouldn't be exporting from S3 very often. And of course Lambda is very cheap- I've calculated that for my setup, backing up an admittedly very small amount of data once every hour, it will cost approximately $0.05/mo.
1. It's super configurable- you can snapshot as frequently or infrequently as you'd like, and I guess if you wanted to you could backup only certain tables or something- I don't know I'm not super familiar with RDS' snapshotting capabilities, maybe they're similar.
1. I can pull my production data down for local testing/manipulating very easily! It's just a single docker command!

### High-Level

1. For the actual database backups I adapted a Docker-based script I found on GitHub [here](https://github.com/schickling/dockerfiles/tree/master/postgres-backup-s3).
1. For running that script I'm using AWS Lambda's new "Container Image" option. I borrowed the Dockerfile from the **Building a Custom Image for Python** section of [Amazon's announcement](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/) to help configure things in a way that'd make sense.
1. For triggering the Lambda on a cron I'm using [Amazon EventBridge](https://console.aws.amazon.com/events/home). I've never heard of it before but it was really easy to create a rule that just says, "run this Lambda once an hour," so I'm recommending it.
1. I'm storing the SQL dump files in a private S3 bucket with a retention policy of 14 days.

### Let's go lower now

So in my head I was like, "Oh cool I can just chuck this Dockerfile on a Lambda and run a command on an image with some environment variables once an hour and we're golden."

It doesn't really work that way though.

Lambda requires that your Docker image's entrypoint be a function, in some programming language, that gets called when triggered.

So rather than just being able to trigger a docker run command (which could in my case run a script, `backup.sh`) like so:

```bash
docker run schickling/postgres-backup-s3
```

It's more that you're setting up a Docker environment for a program (in my case a Python program), that'll have an entrypoint function, that'll run whatever you need to run (again, in my case `backup.sh`).

What's that entrypoint function look like? Pretty simple:

```python
import json
import subprocess

def handler(event, context):
    print("Received event: " + json.dumps(event, indent=2))
    subprocess.run("sh backup.sh".split(" "))
    print("Process complete.")
    return 0
```

The mangling of the Dockerfile they provide as an example in the **Building a Custom Image for Python** section of [Amazon's announcement](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/), to look more like [the Dockerfile of the Postgres to S3 backup script](https://github.com/schickling/dockerfiles/blob/master/postgres-backup-s3/Dockerfile) was the more complicated part. I'll let you take a look at that what that final Dockerfile looks like [here](https://github.com/cd17822/lambda-s3-pg-backup/blob/main/db-backup.Dockerfile).

### Some other gotchas

#### AWS Environment variables

By far the most annoying part of this whole thing was finding some [GitHub issue comment](https://github.com/jschneier/django-storages/issues/606#issuecomment-426136721) that mentioned that Lambdas automatically set the `AWS_SESSION_TOKEN` and `AWS_SECURITY_TOKEN` environment variables, and it turns out it was causing a hard-to-track-down error in the backup script's invocation of the aws client along the lines of:

```
An error occurred (InvalidToken) when calling the PutObject operation: The provided token is malformed or otherwise invalid.
```

If all this article accomplishes is that someone stumbles upon this section after Googling that error, I will consider it astoudingly successful.

Anyway, I just had to edit the `backup.sh` file to add these two lines and the complaining stopped:

```bash
unset AWS_SECURITY_TOKEN
unset AWS_SESSION_TOKEN
```

#### Writing to files

For some reason Lambda didn't like that the `backup.sh` script was writing to a file. After a couple minutes of researching with no luck, I decided to change:

```bash
echo "Creating dump of ${POSTGRES_DATABASE} database from ${POSTGRES_HOST}..."
pg_dump $POSTGRES_HOST_OPTS $POSTGRES_DATABASE | gzip > dump.sql.gz

echo "Uploading dump to $S3_BUCKET"
cat dump.sql.gz | aws $AWS_ARGS s3 cp - s3://$S3_BUCKET/$S3_PREFIX/${POSTGRES_DATABASE}_$(date +"%Y-%m-%dT%H:%M:%SZ").sql.gz || exit 2
```

to:

```bash
echo "Creating dump of ${POSTGRES_DATABASE} database from ${POSTGRES_HOST} and uploading dump to ${S3_BUCKET}..."

pg_dump $POSTGRES_HOST_OPTS $POSTGRES_DATABASE | gzip | aws $AWS_ARGS s3 cp - s3://$S3_BUCKET/$S3_PREFIX/${POSTGRES_DATABASE}_$(date +"%Y-%m-%dT%H:%M:%SZ").sql.gz || exit 2
```

There might be a better way around this but I couldn't find one, so here we are, just piping away.

#### Lambda timeouts

The Lamba I had was timing out by default after 3 seconds. Make sure you jack that up in the function configuration's **General Configuration**.

### Test it out locally

Testing this out locally is really easy because the example Dockerfile that Amazon provided in their announcement has "Lambda Runtime Interface Emulator" support [built in](https://github.com/cd17822/lambda-s3-pg-backup/blob/main/db-backup.Dockerfile#L55-L57).

Build the image:
```bash
docker build -t db-backup -f db-backup.Dockerfile .
```

Run it in Terminal Window 1:
```bash
docker run \
    -e POSTGRES_DATABASE=<POSTGRES_DATABASE>ms
    -e POSTGRES_HOST=<POSTGRES_HOST> \
    -e POSTGRES_PASSWORD=<POSTGRES_PASSWORD> \
    -e POSTGRES_USER=<POSTGRES_USER> \
    -e S3_ACCESS_KEY_ID=<S3_ACCESS_KEY_ID> \
    -e S3_BUCKET=<S3_BUCKET> \
    -e S3_REGION=<S3_REGION> \
    -e S3_PREFIX=<S3_PREFIX> \
    -e S3_SECRET_ACCESS_KEY=<S3_SECRET_ACCESS_KEY> \
    -p 9000:8080 \
    db-backup:latest
```

Trigger it in Terminal Window 2:
```bash
curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'
```

### Ship it

Sure, the right™ way to ship it can be debated by whoever, but the simplest way is probably:

1. Clone the code:
	```bash
	git clone https://github.com/cd17822/lambda-s3-pg-backup.git
	```
1. Build the Docker image:
	```bash
	docker build -t db-backup -f db-backup.Dockerfile
	```
1. Tag the built image to be pushed to a private [ECR](https://aws.amazon.com/ecr/) repository:
	```bash
	docker tag db-backup <AWS_ACCOUNT_ID>.dkr.ecr.<ECR_REGION>.amazonaws.com/<ECR_REPO_NAME>:latest
	```
1. Push the image up to ECR:
	```bash
	docker push <AWS_ACCOUNT_ID>.dkr.ecr.<ECR_REGION>.amazonaws.com/<ECR_REPO_NAME>:latest
	```
1. Create a private S3 bucket that we'll be storing the backups in (I'd also recommend setting up a retention policy unless you want to keep around these files forever).
1. Create a Lambda function by selecting `Create Function` in the [Lambda Console](https://console.aws.amazon.com/lambda/home).
1. Select `Container Image`, name it whatever you want, find the Docker image in ECR in `Browse Images`, leave everything else as default and finally select `Create Function`.
1. Scroll down to **Environment variables** and set values for the following environment variables:
	```
	POSTGRES_DATABASE
	POSTGRES_HOST
	POSTGRES_PASSWORD
	POSTGRES_USER
	S3_ACCESS_KEY_ID
	S3_BUCKET
	S3_PREFIX
	S3_SECRET_ACCESS_KEY
	```
1. Scroll down further and make sure you edit the **General Configuration** such that the Timeout is bumped up to something like 5 minutes.
1. At this point you can select `Test` on the top right and check to make sure your function's working.
1. Finally, you can set up a scheduled trigger by selecting `Add trigger` in the **Designer**. I'd recommend setting up a simple EventBridge trigger that runs on a cron (`cron(13 * * * *)`) or with a set frequency (`rate(1 hour)`).

### Restoring from backup

You could set up a Lambda to restore your database from backup that's triggered by emailing a photo of yourself crying to an unsolicited email address using AWS Computer Vision, but for the sake of this article I figured I'd just include the easy way to do it.

In the same repo that the backup script is in lies a [restore script](https://github.com/schickling/dockerfiles/tree/master/postgres-restore-s3). It's hosted on [DockerHub](https://registry.hub.docker.com/r/schickling/postgres-restore-s3) making it really easy to pull and run locally:

```bash
docker pull schickling/postgres-restore-s3
docker run -e S3_ACCESS_KEY_ID=key -e S3_SECRET_ACCESS_KEY=secret -e S3_BUCKET=my-bucket -e S3_PREFIX=backup -e POSTGRES_DATABASE=dbname -e POSTGRES_USER=user -e POSTGRES_PASSWORD=password -e POSTGRES_HOST=localhost schickling/postgres-restore-s3
```