Using streams when getting objects from S3
Tags: aws, serverless, node, s3
Published: 2022-08-25 06:39:59

We should change how we process the `getObject` requests in our Node.js AWS applications. Readable stream techniques come in handy when we handle the S3 response.

## 1. The classic problem

Say we are facing the classic problem: We have a <a href="https://aws.amazon.com/lambda" class="link" target="_blank" rel="noopener noreferrer">Lambda</a> function, which programatically receives objects from <a href="https://aws.amazon.com/s3" class="link" target="_blank" rel="noopener noreferrer">S3</a> with the <a href="https://aws.amazon.com/sdk-for-javascript/" class="link" target="_blank" rel="noopener noreferrer">AWS SDK</a> in <a href="https://nodejs.org/en/" class="link" target="_blank" rel="noopener noreferrer">Node.js</a>.

The application uses the `getObject` method to receive the object from the bucket.

## 2. Changes

But when we upgrade to version 3 of the SDK (or write a new application with that version), we will experience some changes in the method signature.

Version 3 is modular, so we only have to install what we need in the application. It will reduce the package size, which improves deployment time, so everything sounds good.

We should only install the `@aws-sdk/client-s3` module instead of the whole `aws-sdk` package. The module contains the `getObject` method that helps us receive the objects from the bucket.

The `S3` constructor is still available in the module, so it's nothing new up to this point.

### 2.1. No promise() method

The first change is that the `getObject` method will return a <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise" class="link" target="_blank" rel="noopener noreferrer">`Promise`</a>.

In version 2, the `getObject` method returns an object, and we had to call the `promise()` method, which resolves to the S3 response. Because we always want to use the `async/await` syntax instead of callbacks, the `promise()` method has been part of our development life.

The good news is that AWS has simplified the signature in version 3, and the `getObject` method already returns a `Promise`. Therefore we don't have to call the `promise()` method if we want to `await` to get the resolved value.

### 2.2 Readable streams instead of Buffer

The promise the S3 `getObject` method resolves to an object, which extends the `GetObjectOutput` type. This object has the same properties as in SDK v2 but contains a breaking change.

In version 3 the `Body` property of the resolved S3 response object is a <a href="https://nodejs.org/dist/latest-v16.x/docs/api/stream.html#readable-streams" class="link" target="_blank" rel="noopener noreferrer">readable stream</a> instead of `Buffer`. The modification implies that we should change how the application handles the object.

## 3. Some TypeScript code

Readable streams implement the <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol/asyncIterator" class="link" target="_blank" rel="noopener noreferrer">Symbol.asyncIterator</a> method, so the streams are also **async iterables**.

So we can use the `for...of` construct to iterate over the readable stream and get the chunks the stream provides.

In the following example, we will return the object we have downloaded from S3. The code example that handles the `getObject` requests can look like this:

```ts
async function getObject(params) {
  const s3ResponseStream = (await s3.getObject(params)).Body
  const chunks = []

  for await (const chunk of s3ResponseStream) {
    chunks.push(chunk)
  }

  const responseBuffer = Buffer.concat(chunks)
  return JSON.parse(responseBuffer.toString())
}
```

Each `chunk` is a `Buffer`. After we have received the last chunk of the S3 object, we can concatenate and convert them to a string, then finally to a JavaScript object.

The Lambda handler can look like this:

```ts
import { S3 } from '@aws-sdk/client-s3'

const s3 = new S3({ region: 'us-east-1' })

export async function handler(event) {
  try {
    const s3Object = await getObject({
      Bucket: 'ARN OF THE BUCKET',
      Key: 'NAME OF THE OBJECT TO FETCH',
    })

    return s3Object
  } catch (error) {
    console.error('Error while downloading object from S3', error.message)
    throw error
  }
}
```

We can wrap the stream handling logic to a function called `getObject`, and use it in a `try/catch` block as we usually do in the Lambda handler.

Please note that we still store the S3 object in memory in the above example. The real benefit of streams is that we process the chunk as they arrive. These use cases like transforming the data, saving it to a database, or returning the response as a stream are not part of this post, and I might cover them another time.

## 4. Summary

The `getObject` method's signature has changed in SDK version 3. The `Body` property of the response is now a readable stream instead of `Buffer`.

We can use the core Node.js stream logic to handle the return value in our Lambda functions.

## 5. References

<a href="https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/index.html" class="link" target="_blank" rel="noopener noreferrer">AWS SDK for JavaScript v3</a>