Returning large objects from Lambda functions using stream responses
Tags: aws, serverless, lambda, apigateway
Published: 2023-07-13 08:59:58

The new response stream invocation mode in Lambda gives another great tool in our hand. With its help, we can return larger payloads than the limit from our Lambda functions.

## 1. The problem

Everything works well for Bob in his application, which consists of an <a href="https://aws.amazon.com/api-gateway/" class="link" target="_blank" rel="noopener noreferrer">API Gateway</a> and some <a href="https://aws.amazon.com/lambda/" class="link" target="_blank" rel="noopener noreferrer">Lambda</a> functions.

One day, a response to an endpoint request doesn't come, and Bob sees the following error in the logs:

```bash
LAMBDA_RUNTIME Failed to post handler success response.
Http response code: 413.
```

It turned out that Lambda throws this error when the response size exceeds the maximum limit. API Gateway invokes the Lambda integration synchronously, so the response payload cannot exceed **6 MB**. But the returned object's or array's size might exceed the limit. Some examples can include array responses from a database or a large object from <a href="https://aws.amazon.com/s3/" class="link" target="_blank" rel="noopener noreferrer">S3</a> that the function needs to return to the client.

Either way, we need a solution!

## 2. Some options

We can solve this problem in multiple ways depending on our use case. Some options are the following:

<ul class="list">
  <li class="list__item">Classic pagination.</li>
  <li class="list__item">The function compresses the response, and the client unzips it.</li>
  <li class="list__item">The backend uploads the large object to S3, returns the object key, and the client downloads it.</li>
  <li class="list__item">Replace Lambda with a different technology, e.g., <a href="https://aws.amazon.com/ecs/" class="link" target="_blank" rel="noopener noreferrer">ECS</a> containers.</li>
  <li class="list__item">Use Lambda stream response.</li>
</ul>

All options have advantages and disadvantages and are suitable for specific use cases.

## 3. Using streams in Lambda function response

Node.js <a href="https://nodejs.org/docs/latest-v18.x/api/stream.html#stream" class="link" target="_blank" rel="noopener noreferrer">streams</a> are a great way to return response objects that are **larger than the payload limit** from a Lambda function. This way, the response won't be one single object. Instead, the client will receive it in multiple pieces.

Currently, we must configure a <a href="https://docs.aws.amazon.com/lambda/latest/dg/lambda-urls.html" class="link" target="_blank" rel="noopener noreferrer">function URL</a> that is a dedicated endpoint for the Lambda function if we want to return streamed responses.

The example shows logic that downloads an object (~ 8 MB in size) from S3, but the principle will be the same for responses from a database or 3rd party call.

Let's see how it works.

### 3.1. Function URL

First, we configure a URL for the function. We can <a href="https://dev.to/aws-builders/controlling-access-to-lambda-function-urls-idd" class="link" target="_blank" rel="noopener noreferrer">control access to the endpoint</a> with `AWS_IAM` authorization or can choose to make it public.

Everything in AWS is about **permissions**, so don't forget to add a statement like this to the function's **resource-based policy** (not the execution role):

```json
{
  "Effect": "Allow",
  "Principal": "*",
  "Action": "lambda:InvokeFunctionUrl",
  "Resource": "arn:aws:lambda:eu-central-1:123456789012:function:FUNCTION_NAME"
  "Condition": {
    "StringEquals": {
      "lambda:FunctionUrlAuthType": "AWS_IAM"
    }
  }
}
```

If we make the endpoint public, we should replace `AWS_IAM` with `NONE` in the `Condition` block.

### 3.2. Code

Enabling streamed responses requires some changes in the function's handler code. Lambda provides the global `awslambda` object that contains the `streamifyResponse` wrapper method. We must add the original handler function as an argument to this method.

By wrapping the handler in `streamifyResponse`, a new `responseStream` argument will be available. This way, the `context` object will become the third input to the handler.

A sample code can look like this:

```js
import { S3Client, GetObjectCommand } from '@aws-sdk/client-s3';
import { pipeline } from 'stream/promises';
import { Readable } from 'stream';

const client = new S3Client();

const { BUCKET_NAME } = process.env;

async function streamHandler(event, responseStream, context) {
  const commonMetadata = {
    headers: {
      'x-custom-header': 'some value',
    },
  };

  const input = {
    Bucket: BUCKET_NAME,
    Key: 'large-response.json',
  };
  const command = new GetObjectCommand(input);

  let responseReadableStream;
  let metadata;

  try {
    const s3Response = await client.send(command);
    // (1) S3 response body is a readable stream
    responseReadableStream = s3Response.Body;

    metadata = {
      statusCode: 200,
      ...commonMetadata,
    };

    console.log('SUCCESS!');
  } catch (error) {
    // (2) Create a readable stream from the error object
    responseReadableStream = Readable.from(Buffer.from(JSON.stringify(error)));

    metadata = {
      // (3) Get the status code from the error object
      statusCode: error.$metadata.httpStatusCode,
      ...commonMetadata,
    };

    console.error(error.message);
  }

  // (4) Add metadata, e.g., custom headers to the response
  responseStream = awslambda.HttpResponseStream.from(responseStream, metadata);

  // (5) A promisified version of the pipeline method is also available
  await pipeline(responseReadableStream, responseStream);
}

// (6) Wrap the handler
exports.lambdaHandler = awslambda.streamifyResponse(streamHandler);
```

Let's discuss some key points.

**Using `pipeline`**

The Node.js `stream` module provides the `pipeline` method, which automatically handles errors, deals with backpressure, and properly ends the stream. It requires streams as parameters and adds the previous stream as input to the next one. It's a good idea to use `pipeline` because we'll have a shorter code and less work to do.

We can use the **promisified** version of `pipeline` from `stream/promises` to avoid working with callbacks (5).

**`responseStream` argument**

`responseStream` from `awslambda` is a **writable stream**, and we can add metadata like status code or custom headers to the response. We will also add the actual return value to `responseStream`.

First, we can add the metadata to the response object with the `HttpResponseStream.from` method (4). Next, we add the downloaded large object to the response using `pipeline` (5).

The `Body` property of the S3 response object is a **readable stream** (1), so we can add it to `pipeline` as is.

**Wrapping the handler**

The last step is to wrap the `streamHandler` function into `streamifyResponse` (6).

If we now invoke the function URL in <a href="https://www.postman.com/" class="link" target="_blank" rel="noopener noreferrer">Postman</a>, we'll see that the first bytes of the response arrive very soon. The entire object will be available after a few seconds. The good news is that we can now return payload whose size is up to 20 MB, which is a soft limit.

**Error handling**

It's still a good idea to wrap the S3, database, or 3rd party API calls inside a `try/catch` block to handle any errors.

We can create a **readable stream** from the `error` object (2), and get the error status code (3) from it inside `catch`.

After we have made a stream from the error, we can use the same process described above to return it to the client.

### 3.3. With API Gateway

Bob's original problem includes an API Gateway. How can we incorporate it into the architecture without losing the ability to stream the Lambda response?

**Reduced response size**

API Gateway has a maximum response size of 10 MB, and this is a hard limit. So if we want to use an API Gateway, we might face the same size issue again. If it's not a cause for concern, the following solution can be a good workaround. Otherwise, we'll need to change the architecture and move the Lambda backend behind a bare function URL.

**Using HTTP_PROXY integration**

If we use an API Gateway, we are <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html" class="link" target="_blank" rel="noopener noreferrer">more flexible</a> how we control access to the endpoint. It's a good idea to move authorization to the API Gateway.

We'll still need the function URL for the Lambda function, but we should configure an `HTTP_PROXY` integration instead of the classic `LAMBDA_PROXY`. It makes sense because the function's logic is now available via an HTTP endpoint (the function URL).

**Protecting the function URL**

We should also change the function URL's authorization type to `NONE` from `AWS_IAM`. But this way, we'll leave the endpoint unprotected, so we might want to implement something to filter out illegitimate requests.

A potential solution can be to check for the existence of a **secret header**:

```js
const { BUCKET_NAME, SECRET_HEADER } = process.env;

async function streamHandler(event, responseStream, context) {
  const commonMetadata = {
    headers: {
      'x-custom-header': 'some value',
    },
  };

  if (!(event.headers && event.headers['x-secret-header'] === SECRET_HEADER)) {
    const noHeaderReadableStream = Readable.from(Buffer.from(JSON.stringify({
      body: 'Unauthorized request',
    })));
    const metadata = {
      statusCode: 403,
      ...commonMetadata,
    };
    responseStream = awslambda.HttpResponseStream.from(responseStream, metadata);

    await pipeline(noHeaderReadableStream, responseStream);

    console.error('Unauthorized request');
    return;
  }

  // code continues
}
```

If the value of the `x-secret-header` header is correct, we'll let the function complete its logic. If not, we can use the already discussed way to stream a custom `403` response.

But if we do so, we must instruct API Gateway to add the header and its value to each integration request it makes to the function URL. We can easily achieve it by adding parameter mapping and configuring the value of the `x-secret-header` as a **static value**:

![Secret header in the integration request](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8dbf6plekegqoscwo2ok.png)

It's easy to miss wrapping the value inside **single quotes**, so don't forget to do so.

## 4. Summary

Lambda functions have a 6 MB limit of the response payload size for synchronous invocations. Sometimes we need to return responses that are larger than the size limit.

We can implement stream responses to return large objects from our functions, but this feature is only available with function URLs.

## 5. Further reading

<a href="https://docs.aws.amazon.com/lambda/latest/operatorguide/invocation-modes.html" class="link" target="_blank" rel="noopener noreferrer">Comparing Lambda invocation modes</a> - The three ways we can call a Lambda function

<a href="https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html" class="link" target="_blank" rel="noopener noreferrer">Getting started with Lambda</a> - How to create a Lambda function

<a href="https://docs.aws.amazon.com/lambda/latest/dg/urls-configuration.html" class="link" target="_blank" rel="noopener noreferrer">Creating and managing Lambda function URLs</a> - How to create Lambda function URLs

<a href="https://docs.aws.amazon.com/lambda/latest/dg/configuration-response-streaming.html" class="link" target="_blank" rel="noopener noreferrer">Configuring a Lambda function to stream responses</a> - Short official documentation on stream responses