Lambda Unleashed: Mastering Massive Data Responses
Tags: aws, middleware, serverless, node
Published: 2023-12-27 15:35:52

In the realm of serverless architecture, AWS Lambda has emerged as a cornerstone, offering scalability, flexibility, and efficiency. However, even in this advanced cloud environment, we encounter certain constraints — one being the limit on Lambda response sizes, traditionally capped at 6MB. At first glance, this seems reasonable. After all, APIs are generally not designed to handle or return voluminous amounts of data. The principle of keeping responses lean and efficient underlines much of our best practices in API design.

But what happens when the unexpected occurs? When, despite our best efforts and designs, the need arises for Lambdas to return larger payloads? It's not always about wanting to push the boundaries; sometimes, it's about adapting to them. In this blog post, I aim to delve into the nuances of scenarios where larger Lambda responses are not just beneficial but necessary.

## Large Response Middleware

This middleware allows a service to log and save large responses to an S3 bucket, enabling developers to investigate the causes of such large responses. Furthermore, this middleware accepts a special header that allows the rewriting of the response with a `$payload_ref` pointing to the large payload stored in S3, enabling clients to recover gracefully.

### Use Case

An internal application of this middleware at epilot involves a service that constructs a Programmatic Runtime containing a large `{} variable` context. This context is then used to generate sophisticated Document Templates.  Our clients leverage these templates to dispatch Emails or Attachments to their end-customers. Although the resulting context may be substantial, its necessity is unquestionable for thorough exploration and real-time evaluation of the compiled template against the context.

### Limitations

This work is a great all-in-one solution for one specific lambda use case:

- API Gateway with Lambda Proxy Integration

Although, while we at @epilot use this pattern a lot, it's by no means the only way Lambda is used.

The same 6MB response limit applies in all Lambda use cases. Just to give some examples that are not covered by this middleware:

- Lambda function URLs
- Lambda in Step Functions
- Direct Invoke via Lambda API
- API Gateway Lambda service integration
- Lambda@Edge

Since the 6MB limit applies anywhere, we could build and publish something more generic that works for all Lambda use cases (not just this one).

The good news for the community is that, we've an open issue for that [aws-lambda-utility-middlewares/issues/1](https://github.com/epilot-dev/aws-lambda-utility-middlewares/issues/1). And we may put some more work on it during the near future. Contributions are also more than welcomed 🙌

### How it works

![Sequence Diagram](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mv9q7b4eg1hq106suqr5.png)

For handling a Large Response, clients must request with the HTTP Header `Accept: application/large-response.vnd+json`. This custom MIME type signals consent to receive a large response payload when required. The format of the response body for this MIME type is:

```json
{
  "$payload_ref": "http://<s3 file reference link>"
}
```

Should the client specify this MIME type, the Lambda bypasses logging an error with `Log.error`. Instead, it restructures the original response, incorporating a reference to the substantial payload now in S3. The modified response also carries the HTTP header `Content-Type` set to `application/large-response.vnd+json`.

Absent this MIME type, the Lambda logs an error using `Log.error`, leading to a response failure due to the oversized body.

### Configuration Options

Supported Parameters:

| Parameter       | Type                | Description                                                                  |
| --------------- | ------------------- | ---------------------------------------------------------------------------- |
| thresholdWarn   | `number`            | Warning threshold level (percentage of `sizeLimitInMB`), e.g: 0.80           |
| thresholdError  | `number`            | Error threshold level (percentage of `sizeLimitInMB`), e.g: 0.90             |
| sizeLimitInMB   | `number`            | Maximum allowed size limit in MB, e.g 6                                      |
| outputBucket    | `string`            | Identifier or name of the output S3 bucket                                   |
| groupRequestsBy | `function - mapper` | Function to group requests, based on API Gateway event V2. Defaults to 'all' |

Example Usage:

```ts
withLargeResponseHandler({
  thresholdWarn: 0.85, // 85% of the limit = 5.1MB
  thresholdError: 0.9, // 90% of the limit = 5.4MB
  sizeLimitInMB: 6,
}),
```

### Repository

Explore the repository and npm package here:
- [github.com/epilot-dev/aws-lambda-utility-middlewares](https://github.com/epilot-dev/aws-lambda-utility-middlewares)
- [npmjs.com/package/@epilot/large-response-middleware](https://www.npmjs.com/package/@epilot/large-response-middleware)

Thank you for reading! If you found this post insightful, please don't hesitate to show your reaction and support.

[#stay-hard #keep-pushing-the-limits](https://dev.to/epilot)

Wishing you a Happy New Year's Eve! May the year ahead be filled with innovation and success.