Processing large payloads with Amazon API Gateway asynchronously
Tags: aws, serverless, apigw, sfn
Published: 2022-04-27 06:09:09

---
title: Processing large payloads with Amazon API Gateway asynchronously
published: true
description: The Amazon API Gateway payload size limit is 10MB. How could you process bigger payloads? – via asynchronous workflows. This article will show you how.
tags: aws, serverless, apigw, sfn
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dpj0tlctfynb5uzsq136.jpeg
---

In my previous article, I've talked about [_Synchronous AWS Lambda & Amazon API Gateway limits and what to do about them_](https://dev.to/aws-builders/synchronous-aws-lambda-amazon-api-gateway-limits-and-what-to-do-about-them-2oec).

As stated in the blog post, the ultimate solution for a "big payload" problem is making the architecture asynchronous. Let us then zoom in on the aspect of asynchronous communication in the [Amazon API Gateway](https://aws.amazon.com/api-gateway/) service context and build a serverless architecture based on the notion of "pending work".

## Problem refresher

The following illustrates the Amazon API Gateway & [AWS Lambda](https://aws.amazon.com/lambda/) payload size problem I've touched about in the previous article.

![Problem refresher](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fz6tq6h3zchsg27s2v95.jpeg)

No matter how hard you try, you **unable** to **synchronously send more than 6 MB of data** to your AWS Lambda function. The limit can seriously mess with your significant data-processing needs.

Luckily there are ways to process much more data via AWS Lambda using asynchronous workflows.

## The starting line - pushing the data to the storage layer

Suppose you were to ask the community about the potential solution to this problem. In that case, I wager that the most common answer would be to use [Amazon S3](https://aws.amazon.com/s3/) as the data storage layer and utilize _[presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html)_ to push the data to Amazon S3 storage.

![Presigned URL flow with S3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0d6wvy9xm2d0tkd5t7qr.jpeg)

While very universal, the flow of _presigned URL to Amazon S3_ can be nuanced, especially since one can create the _presigned URL_ in two ways.

Describing these would make this article a bit too long for my liking, so I'm going to defer you to [this great article](https://zaccharles.medium.com/s3-uploads-proxies-vs-presigned-urls-vs-presigned-posts-9661e2b37932) by my colleague [Zac Charles](https://twitter.com/zaccharles) which did the topic much more justice than I could ever do.

## I have the data in Amazon S3. Now what?

Before processing the data, our system must know whether the client used the _presigned URL_ to push the data to the storage layer. To my best knowledge, there are two options we can pursue here (please let me know if there are other ways to go about it, I'm very keen to learn!)

> I omitted the [AWS CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html) to [Amazon EventBridge](https://aws.amazon.com/eventbridge/) flow on purpose as I think engineers should favor the direct integration with AWS EventBridge instead.

### S3 Event Notifications

The [Amazon S3 event notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html), till recently, was a de facto way of knowing whether data landed into Amazon S3.

![S3 Event Notifications flow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sr8taqny6r2jdaro6oxu.jpeg)

While sufficient for most use-cases, the feature is not without its problems, the biggest of which, I would argue, are the misunderstandings around event filtering and IaC implementation.

The most essential thing to have in mind when it comes to **event filtering** is that you **cannot use wildcards for prefix or suffix** matching in your filtering rules. There are more things to consider, though. If you plan to use the filtering feature of S3 event notifications, I strongly encourage you to read [this documentation page](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-filtering.html) thoroughly.

On the **IaC side of things**, know that, in creating the [AWS CloudFormation](https://aws.amazon.com/cloudformation/) template, you might [end up with circular dependency problems](https://aws.amazon.com/blogs/mt/resolving-circular-dependency-in-provisioning-of-amazon-s3-buckets-with-aws-lambda-event-notifications/). Deployment frameworks like [AWS CDK](https://aws.amazon.com/cdk/) will make sure that should never happen. However, I still think you should be aware of this potential problem, even if you use deployment frameworks.

### EventBridge events

In late 2021, AWS announced [Amazon EventBridge support for S3 Event Notifications](https://aws.amazon.com/blogs/aws/new-use-amazon-s3-event-notifications-with-amazon-eventbridge/). The announcement had a warm welcome in the serverless community as EventBridge integration solves most of the "native" S3 event notifications problems.

![S3 event to EventBridge flow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/34gautwqzttxb92tktvd.jpeg)

Utilizing [Amazon EventBridge](https://aws.amazon.com/eventbridge/) for S3 events gives you **more extensive integration surface area** (you can forward the event to more targets) and **better filtering capabilities** (one of the strong points of Amazon EventBridge). The integration is not without its problems, though.

For starters, the events are sent to the "default" bus, and using EventBridge might be more costly for high event volumes. To learn more about different caveats, consider giving [this great article](https://bitesizedserverless.com/bite/monitor-events-from-multiple-s3-buckets-with-eventbridge/) a read.

## Munching on the data

You have the data in the Amazon S3 storage, and you have a way to notify your system about that fact. Now what? How could you process the data and yield the result back to the user?

Given the nature of AWS, for better or worse, there are multiple scenarios one can move forward. We will start from the "simplest" architecture and move our way up to deploying an orchestrator with shared, high-speed data access.

### Processing with a dedicated AWS Lambda

The [AWS Lambda](https://aws.amazon.com/lambda/) service is often called a "swiss knife" of serverless. In most cases, processing the data in-memory within the AWS Lambda is sufficient.

![Processing the S3 upload event via AWS Lambda](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5dgovmp4f1jix4t7c505.jpeg)

The only limitations are your imagination and the AWS Lambda service timeout – the 15-minute maximum function runtime. Please note that **in this setup, your AWS Lambda function must spend some of that time downloading the object**.

I'm vague about saving the results of the performed work on purpose as it is very use-case dependant. You might want to keep the outcome of your work back on Amazon S3 or add an entry to a database. Up to you.

But what if that 15-minute timeout limitation is a thorn at your side? What if the process within the compute layer of the solution is complex and would benefit from splitting it into multiple chunks? If that is the case, keep on reading, we are going to be talking about [AWS Step Functions](https://aws.amazon.com/step-functions/) next.

### Processing with Amazon StepFunctions

If the compute process within the AWS Lambda we looked at previously is complex or takes more time than the hard timeout limit, the AWS Step Functions might be just the service you need.

![Processing the S3 upload event via AWS Step Functions](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/88dvip4lft2ovfx1zyyh.jpeg)

Take a note of the number of times the code within your Step Function definition needs to download the object from Amazon S3 storage (the illustration being an example, of course). Depending on the workload, that number may vary, but no matter the workflow, after a certain number threshold, it does feel "wasteful" for me to have to download the object again and again.

> Keep in mind that AWS Step Functions maximum [payload size is 256KB](https://docs.aws.amazon.com/step-functions/latest/dg/avoid-exec-failures.html). That is why you have to download the object repeatedly whenever you need access to the object.

You could implement [in-memory caching in your AWS Lambda function](https://www.jambit.com/en/latest-info/toilet-papers/easy-in-memory-caching-for-aws-lambda/), but that technique only applies to a single AWS Lambda and is tied to its container lifecycle.

Depending on the requirements and constraints, I like to use [Amazon EFS](https://aws.amazon.com/efs/) integration with AWS Lambda in such situations. Amazon EFS allows me to have a storage layer shared by all AWS Lambda functions that partake in the workflow. Let us look into that next.

#### Shared AWS Lambda storage

Before starting, understand that **using Amazon EFS with AWS Lambda functions requires [Amazon VPC](https://aws.amazon.com/vpc/)**. For some workflows, this fact does not change anything. For others, it does. I strongly advocate keeping an open mind (I know some people from serverless community despise VPCs) and evaluating the architecture according to your business needs.

![Processing the S3 upload event via AWS Step Functions and Amazon EFS](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6cqtab15g92x17kofsqn.jpeg)

This architecture is, arguably, quite complex. You have to consider some networking concerns. If your object is quite large and downloading it takes a lot of time, I would advise you to look into this architecture – VPCs do not bite!

Suppose you yearn for a practical example, [here is a sample architecture I've built](https://github.com/WojciechMatuszewski/serverless-video-transcribe-fun) for parsing media files. It utilizes Amazon EFS for fast access to that video file across all AWS Lambda functions involved in the AWS Step Function workflow.

## Yielding back to the user

With the compute part behind us, it is time to see how we might notify the user that our system processed the object and that the results are available.

No matter what kind of solution you choose to notify the user about the result, the state of the work has to be saved somewhere. Is the work pending? Is it finished? Maybe an error occurred?

### Keeping track of the work status

The following is a diagram of an example architecture that keeps track of the status of the performed work.

![Generic diagram depicting how to keep track of the work state via CDC](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mxapk7yv5ue3cuw41djl.jpeg)

The database sends CDC events to the system. We could use these CDC events to notify the user about the work progress in real-time! (although that might not be necessary, in most scenarios, polling for the results by the client is sufficient).

In my humble opinion, the `key:value` nature of the "work progress" data makes the [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) a perfect choice for the database component (nothing is stopping you from using RDS, which also supports CDC events, which themselves are optional here).

### Notifying the user about the results

The last piece of the puzzle is making sure the user has a way to be notified (or retrieve) the status of the request.

Usually, in such situations, we have two options to consider – either we implement an API in which the user is going to pool for the updates, or we push the status directly to the user.

The first option, pooling, would look similar to the following diagram in terms of architecture.

![Polling for changes](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o5imk29vemne93gm6om9.jpeg)

This architecture is sufficient up to a particular scale. The architecture must scale according to the pollers hitting the API. The more pollers actively engage with the API, the higher the chance of throttling or other issues (refer to the [_Thundering herd problem_](https://en.wikipedia.org/wiki/Thundering_herd_problem)).

In higher throughput scenarios, one might want to replace the polling behavior with a **push model based on [_WebSockets_](https://en.wikipedia.org/wiki/WebSocket)**. For AWS-specific implementations, I would recommend looking at [Amazon API Gateway WebSocket support](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-websocket-api.html) or [AWS IoT Core WebSockets](https://docs.aws.amazon.com/iot/latest/developerguide/mqtt.html) for broadcast-like use-cases.

> I've written an article solely focused on serverless WebSockets on AWS. You can find [the blog post here](https://dev.to/aws-builders/serverless-websockets-on-aws-3nm9).

## Closing words

And that is the end of our journey. We have looked at how to get the large payload into our system, process it, and respond to the user. Implementing this architecture is not a small feat, and I hope you found the walkthrough helpful.

For more AWS serverless content, consider following me on Twitter - [@wm_matuszewski](https://twitter.com/wm_matuszewski)

Thank you for your precious time.