Serverless file storage with AWS Lambda
Tags: serverless, lambda, bugout, devops
Published: 2021-09-28 21:30:57

Hello! Today we will deploy a serverless infrastructure based on AWS Lambda for uploading images (or any other files) with private storage in an AWS S3 bucket. We will be using terraform scripts that are uploaded and available at my [kompotkot/hatchery](https://github.com/kompotkot/hatchery/tree/main/files_distributor) GitHub repository.

This approach has the following advantages:
- lambda is called on request and therefore allows to save on server maintenance costs if this functionality is not key for your application
- lambda functions have an isolated runtime environment, which is ideal for processing uploaded files. Should malicious code be uploaded, the attacker will not be able to leave the sandbox, and the sandbox session will be forcibly terminated after some time
- storing files in an S3 bucket is very cheap

## Project structure

As an example, we'll be using an abstract app for journal entries with an API.
We can upload an image in each entry, and the structure is similar to a file directory:

```
- journal_1
  - entry_1
    - image_1
  - entry_2
    - image_1
    - image_n
  - entry_n
- journal_n
```

Our hypothetical API has an endpoint for receiving an entry in a journal:

```
curl \
  --request GET \
  --url 'https://api.example.com/journals/{journal_id}/entries/{entries_id}'
  --header 'Authorization: {token_id}'
```

If in response to this endpoint `status_code` is equal to 200, it means the user is authorized and has access to the journal. Accordingly, we will let them store images for this entry.

## Registering the app on Bugout.dev

To avoid adding an extra table to the database, which we would need for storing which image belongs to which entry, we will use `resources` from [Bugout.dev](https://bugout.dev/). This approach is used to simplify our infrastructure, but, if required, this step can be substituted for creating a new table in your database and writing an API for creating, modifying, and deleting data about the stored images. Bugout.dev is open source and you can review the API documentation at the GitHub [repository](https://github.com/bugout-dev).

We will need an account and a team called `myapp` (you can use any name in relation to your project) at the [Bugout.dev Teams](https://bugout.dev/account/teams) page, you should save this team’s ID for the next step (in our case it’s `e6006d97-0551-4ec9-aabd-da51ee437909`):

![2](https://miro.medium.com/max/700/1*5Lk9gSuzQguv5XdXjvYOnw.png)

Next, let’s create a Bugout.dev Application for our team myapp through a curl request (the token can be generated at the [Bugout.dev Tokens](https://bugout.dev/account/tokens) page) and save it in the `BUGOUT_ACCESS_TOKEN` variable:

```
curl \
  --request POST \
  --url 'https://auth.bugout.dev/applications' \
  --header "Authorization: Bearer $BUGOUT_ACCESS_TOKEN" \
  --form 'group_id=e6006d97-0551-4ec9-aabd-da51ee437909' \
  --form 'name=myapp-images' \
  --form 'description=Image uploader for myapp notes' \
  | jq .
```

In response we will get confirmation of a successfully created application:

```json
{
    "id": "f0a1672d-4659-49f6-bc51-8a0aad17e979",
    "group_id": "e6006d97-0551-4ec9-aabd-da51ee437909",
    "name": "myapp-images",
    "description": "Image uploader for myapp notes"
}
```

The ID `f0a1672d-4659-49f6-bc51-8a0aad17e979` will be used for storing `resources`, where every resource is the uploaded image’s metadata. The structure is set in any form depending on the required keys, in our case, it will look as follows:

```json
{
    "id": "a6423cd1-317b-4f71-a756-dc92eead185c",
    "application_id": "f0a1672d-4659-49f6-bc51-8a0aad17e979",
    "resource_data": {
        "id": "d573fab2-beb1-4915-91ce-c356236768a4",
        "name": "random-image-name",
        "entry_id": "51113e7d-39eb-4f68-bf99-54de5892314b",
        "extension": "png",
        "created_at": "2021-09-19 15:15:00.437163",
        "journal_id": "2821951d-70a4-419b-a968-14e056b49b71"
    },
    "created_at": "2021-09-19T15:15:00.957809+00:00",
    "updated_at": "2021-09-19T15:15:00.957809+00:00"
}
```

As a result, we have a remote database, where every time we upload an image to an S3 bucket, we’ll be writing which journal(`journal_id`) and which entry(`entry_id`) the image was added to under which ID, name, and extension.

## Preparing the AWS project environment

AWS will store images in an S3 bucket and function as a server on Lambda for image manipulation. We will need an AWS account and a configured IAM user for terraform. It is an account with `Programmatic access` to all resources without having access to the web console:

![3](https://miro.medium.com/max/700/1*zvEBy6syc1LRM2_jOMk2bw.png)

Get the access keys and add these variables to your environment:

```
export AWS_ACCESS_KEY_ID=<your_aws_terraform_account_access_key>
export AWS_SECRET_ACCESS_KEY=<your_aws_terraform_account_secret_key>
```

Let’s also deploy a VPC with the subnets:
- 2 subnets with private access
- 2 subnets with public access

They will be useful for configuring the AWS Load Balancer. The code for this module can be found under [files_distributor/network](https://github.com/kompotkot/hatchery/tree/main/files_distributor/network). Let’s edit the variables in the `variables.tf` file and launch the script:

```
terraform apply
```

From the output, add environment variables with values `AWS_HATCHERY_VPC_ID`, `AWS_HATCHERY_SUBNET_PUBLIC_A_ID` and `AWS_HATCHERY_SUBNET_PUBLIC_B_ID`.

## Server code

In our project, we’ll be using a simple AWS Lambda function. In my experience, I’ve noticed that as the packet with code surpasses 10MB, the upload speed to AWS drops dramatically. Even if we upload it to the S3 bucket in advance and then make a lambda from it, AWS can freeze for a long time. Therefore, if you are using third-party libraries it can make sense to use lambda layers, whereas if you aren’t planning to use any libraries with lightweight code on CloudFront, consider looking into lambda@edge.

The full code can be found in the [lambda_function.py] (https://github.com/kompotkot/hatchery/blob/main/files_distributor/bucket/modules/s3_bucket/files/lambda_function.py) file in the repository. In my opinion, it’s more effective to work with nodejs, but to facilitate in-depth file processing we’ll use python. The code consists of main blocks:

```
MY_APP_JOURNALS_URL = "https://api.example.com"     # API endpoint for accessing out journal entries app
BUGOUT_AUTH_URL = "https://auth.bugout.dev"         # Bugout.dev endpoint for writing resources (image metadata)
FILES_S3_BUCKET_NAME = "hatchery-files"      # S3 bucket name, where we’ll store the images
FILES_S3_BUCKET_PREFIX = "dev"               # S3 bucket prefix, where we’ll store the images
BUGOUT_APPLICATION_ID = os.environ.get("BUGOUT_FILES_APPLICATION_ID")   # Bugout.dev application ID that we created prior
```

Let’s expand the default exception to proxy the response from Bugout.dev Resources. E.g., if the image does not exist, when we request the resource, we’ll receive error 404, which we’ll in turn return to the client as a reply to the request for the missing image.

```python
class BugoutResponseException(Exception):
    def __init__(self, message, status_code, detail=None) -> None:
        super().__init__(message)
        self.status_code = status_code
        if detail is not None:
            self.detail = detail
```

To save an image in an S3 bucket we’ll use the `cgi` standard library that’ll let us parse the request’s body that was sent in `multipart/<image_type>` format. We’ll save images under the path `{journal_id}/entries/{entry_id}/images/{image_id}` without specifying the file’s name and extension.

```python
def put_image_to_bucket(
    journal_id: str,
    entry_id: str,
    image_id: UUID,
    content_type: str,
    content_length: int,
    decoded_body: bytes,
) -> None:
    _, c_data = parse_header(content_type)
    c_data["boundary"] = bytes(c_data["boundary"], "utf-8")
    c_data["CONTENT-LENGTH"] = content_length

    form_data = parse_multipart(BytesIO(decoded_body), c_data)

    for image_str in form_data["file"]:
        image_path = f"{FILES_S3_BUCKET_PREFIX}/{journal_id}/entries/{entry_id}/images/{str(image_id)}"
        s3.put_object(
            Body=image_str, Bucket=FILES_S3_BUCKET_NAME, Key=image_path
        )
```

When we extract an image from the S3 bucket we’ll need to encode it into base64 for correct transmission.

```python
def get_image_from_bucket(journal_id: str, entry_id: str, image_id: str) -> bytes:
    image_path = f"{FILES_S3_BUCKET_PREFIX}/{journal_id}/entries/{entry_id}/images/{image_id}"
    response = s3.get_object(Bucket=FILES_S3_BUCKET_NAME, Key=image_path)
    image = response["Body"].read()
    encoded_image = base64.b64encode(image)
    return encoded_image
```

The `lambda_handler(event,context)` function’s implementation is available at this GitHub [link](https://github.com/kompotkot/hatchery/blob/main/files_distributor/bucket/modules/s3_bucket/files/lambda_function.py), to sum up:
- Firstly, we assert that the request is formatted correctly and contains `journal_id` and `entry_id`
- Then we call our hypothetical app’s API `https://api.example.com/journals/{journal_id}/entries/{entry_id}` 
- Depending on the request method: `GET`, `POST` or `DELETE` we read, upload or delete an image from the journal’s entry
- When we’re uploading to the S3 bucket, we check the extension and the file’s size. This can be expanded into hash verification to avoid uploading the same file, etc.

Next, we’ll need to package `requests` into the Lambda library. Luckily, `boto3`  for work with AWS functionality is ready out of the box. Let’s create an empty python environment, install the library and package the contents of `site-packages`:

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install requests
cd .venv/lib/python3.8/site-packages
zip -r9 "lambda_function.zip" .
```

Place the created archive `lambda_function.zip`  into the `files_distributor/bucket/modules/s3_bucket/files` directory and add the Lambda function itself:

```bash
zip -g lambda_function.zip -r lambda_function.py
```

Our server is ready, now we can upload code to AWS and deploy Lambda server, to do so use the script in `files_distributor/bucket`:

```bash
terraform apply
```

We’re left with:
- A private AWS S3 bucket `hatchery-sources` that stores the Lambda function code
- A private AWS S3 bucket `hatchery-files` that we’ll store our images into with the prefix `dev`
- AWS Lambda function with working server code
- An IAM role for the Lambda that allows writing into a specific S3 bucket and logs

The IAM role rules are in `files_distributor/bucket/modules/iam/files/iam_role_lambda_inline_policy.json`. The other file `iam_role_lambda_policy.json`  is needed for the Lambda to function works correctly.

To debug Lambda you can just `print` the required values or use the standard `logging` module for python. The output for every Lambda function call is available at AWS CloudWatch:

![4](https://miro.medium.com/max/700/1*Skrr3o73JHKmTStJ2SFJgA.png)

After creating the function, add a variable `BUGOUT_FILES_APPLICATION_ID` from our code to the Lambda environment, which you can do in the tab `Configuration/Environment variables`. 

As the last step, save the AWS Lambda arn into the environment variable `AWS_HATCHERY_LAMBDA_ARN`.

## Configuring the AWS Load Balancer and open ports

The only step left now is to create AWS Security Group where we’ll set a port the AWS Load Balancer will listen to for subsequent data transmit into the Lambda function (in our case it’s 80 and 443).

```bash
terraform apply \
    -var hatchery_vpc_id=$AWS_HATCHERY_VPC_ID \
    -var hatchery_sbn_public_a_id=$AWS_HATCHERY_SUBNET_PUBLIC_A_ID \
    -var hatchery_sbn_public_b_id=$AWS_HATCHERY_SUBNET_PUBLIC_B_ID \
    -var hatchery_lambda_arn=$AWS_HATCHERY_LAMBDA_ARN
```

Congratulations, our AWS Lambda function is open to the world and ready to upload and return images for our journal entries app!