Generating thumbnails via S3 events using aws-cdk (TypeScript)
Tags: aws, typescript, cdk, lambda
Published: 2020-09-16 18:48:12

This is another post on CDK after a while that's a bit different. You can check out the previous two posts that I had written here:

{% link https://dev.to/ryands17/deploying-a-spa-using-aws-cdk-typescript-4ibf %}

{% link https://dev.to/ryands17/deploying-a-node-app-to-beanstalk-using-aws-cdk-typescript-3b8d %}

### Let's see something today that we have all done manually in the past, creating thumbnails for images for profile pic uploads in S3.

Here we shall walk through how we can react events when we upload a file to S3 and send that to SQS. We shall also attach a Lambda handler that processes the message from the SQS queue and generates our thumbnail in the same bucket.

__*Note*__: The code for the entire repository is here if you want to jump right in :)

{% github ryands17/s3-thumbnail-generator %}

In this repo, all the code that we write to create resources will be in the __s3_thumbnail-stack.ts__ file situated in the  __lib__ folder.

Let's start by initializing a couple of constants:

```ts
const bucketName = 'all-images-bucket'
const prefix = 'photos/'
```

These constants are the name of the bucket that we will be creating and the path where we will be storing our photos respectively.

Then, in the constructor of our `S3ThumbnailStack` class, we shall start by creating the Queue that will handle all the object creation events coming from S3.

```ts
const queue = new sqs.Queue(this, 'thumbnailQueue', {
  queueName: 'thumbnailPayload',
  receiveMessageWaitTime: cdk.Duration.seconds(20),
  visibilityTimeout: cdk.Duration.seconds(60),
  retentionPeriod: cdk.Duration.days(2),
})
```

This code creates an SQS Queue named _thumbnailPayload_ and sets some parameters:

- _receiveMessageWaitTime_: This is the time at what interval should one poll for messages. We have kept it at _20 seconds_ as that is the longest we can have. This method is also known as __long polling__.

- _visibilityTimeout_: This is the time for a message to be visible again for other handlers to process it. Here we are assuming that the Lambda will complete the thumbnail generation within 60 seconds. This time should be set accordingly so that the same messages are not consumed twice.

- _retentionPeriod_: The amount of days the message will remain in the queue if unprocessed.

Moving on, lets create the lambda handler.

```ts
const handler = new lambda.Function(this, 'resizeImage', {
  runtime: lambda.Runtime.NODEJS_12_X,
  code: lambda.Code.fromAsset('resources'),
  handler: 'index.handler',
  timeout: cdk.Duration.seconds(20),
  memorySize: 256,
  logRetention: logs.RetentionDays.ONE_WEEK,
  environment: {
    QUEUE_URL: queue.queueUrl,
    PREFIX: prefix,
  },
})
```

Most of these options are familiar so lets discuss about the _code_ and _environment_ parameters respectively.

- _code_: This is where our Lambda code will be taken from. We have specified a `fromAsset('resources')`, meaning CDK will take this from a folder named _resources_ in our code. In this folder, there's a simple NodeJS Lambda function that resizes images using the [sharp](https://prisma.slack.com/archives/CA491RJH0/p1600254914446100) library.

- _environment_: These contain two values. One for the queue URL which we will be needing to delete the message after we have processed it and the other is the S3 bucket prefix from where we need to fetch the image uploaded.

The Lambda handler looks something like this that is in the _resources_ folder.

```ts
const AWS = require('aws-sdk')
const sharp = require('sharp')

const prefix = process.env.PREFIX
const s3 = new AWS.S3()
const sqs = new AWS.SQS({
  region: process.env.AWS_REGION,
})

exports.handler = async event => {
  try {
    for (let message of event.Records) {
      const body = JSON.parse(message.body)

      if (Array.isArray(body.Records) && body.Records[0].s3) {
        const objectName = decodeURIComponent(
          body.Records[0].s3.object.key.replace(/\+/g, ' ')
        )
        const bucketName = body.Records[0].s3.bucket.name
        console.log({ objectName, bucketName })
        try {
          await resizeAndSaveImage({ objectName, bucketName })
          await deleteMessageFromQueue(message.receiptHandle)
        } catch (e) {
          console.log('an error occured!')
          console.error(e)
        }
      } else {
        await deleteMessageFromQueue(message.receiptHandle)
      }
    }

    return {
      statusCode: 200,
      body: JSON.stringify('success!'),
    }
  } catch (e) {
    console.error(e)
    return {
      statusCode: 500,
      error: JSON.stringify('Server error!'),
    }
  }
}

async function resizeAndSaveImage({ objectName, bucketName }) {
  const objectWithoutPrefix = objectName.replace(prefix, '')
  const typeMatch = objectWithoutPrefix.match(/\.([^.]*)$/)
  if (!typeMatch) {
    console.log('Could not determine the image type.')
    return
  }

  // Check that the image type is supported
  const imageType = typeMatch[1].toLowerCase()
  if (imageType !== 'jpg' && imageType !== 'png') {
    console.log(`Unsupported image type: ${imageType}`)
    return
  }

  // Download the image
  const image = await s3
    .getObject({
      Bucket: bucketName,
      Key: objectName,
    })
    .promise()

  // Transform the image via sharp
  const width = 300
  const buffer = await sharp(image.Body).resize(width).toBuffer()

  const key = `thumbnails/${objectWithoutPrefix}`
  return s3
    .putObject({
      Bucket: bucketName,
      Key: key,
      Body: buffer,
      ContentType: 'image',
    })
    .promise()
}

function deleteMessageFromQueue(receiptHandle) {
  if (receiptHandle) {
    return sqs
      .deleteMessage({
        QueueUrl: process.env.QUEUE_URL,
        ReceiptHandle: receiptHandle,
      })
      .promise()
  }
}
```

This handler performs two functions, resize and save the image to S3, and delete the message from the queue once processed **successfully**.

And the third and final resource that we will be creating is our S3 bucket as follows:

```ts
const imagesBucket = new s3.Bucket(this, 'allImagesBucket', {
  bucketName,
  removalPolicy: cdk.RemovalPolicy.DESTROY,
})
```

This will just create our bucket and destroy it once we delete our Cloudformation stack.

There are three things needed to make this entire workflow running. Lets go through them and perform the final touches.

- _Send object creation events from S3 to SQS_: This is so that S3 sends an event to SQS whenever an object, in our case an profile picture is uploaded in the **photos** folder. This is how we can add that:

```ts
imagesBucket.addObjectCreatedNotification(new s3Notif.SqsDestination(queue), {
  prefix,
})
```

Here, we are adding an _ObjectCreatedNotification_ in which we are setting the destination to be an SQS Queue. Note the `prefix` paramter. This means only events for objects added in the _photos_ folder or prefix in the case of S3 will be sent to SQS.

- _Allow Lambda access to S3_: This is important as we need to fetch the image, generate the thumbnail, and put the thumbnail back in the bucket. Lets attach a policy to the handler:

```ts
handler.addToRolePolicy(
  new iam.PolicyStatement({
    effect: iam.Effect.ALLOW,
    resources: [`${imagesBucket.bucketArn}/*`],
    actions: ['s3:PutObject', 's3:GetObject'],
  })
)
```

This tells Lambda "You have access just to `GetObject` and `PutObject` methods on S3 on the created bucket and nothing else".

- _Allow Lambda to process messages from SQS_: Finally, we tell Lambda to execute when a mesasge is added in the SQS Queue from the S3 object creation event.

```ts
handler.addEventSource(new eventSource.SqsEventSource(queue))
```

This basically says, "Lambda, consider this SQS Queue as a source and consume any messages that you get from this queue".

And we're done! Let's deploy this stack by firing `yarn cdk deploy` or `npm run cdk deploy` if you're using NPM.

__*Note*__: I'm assuming you have setup the AWS CLI and set your access and secret keys in the **default** profile. You can read more about that [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html).

After successful stack creation, you will get a bucket like this:

![S3 Dashboard](https://dev-to-uploads.s3.amazonaws.com/i/lws31qc70tf6mt5soh8a.png)

As you can see I have already created a folder named _photos_. Let's upload a profile picture for our user that's a cat, and see if the event fires to SQS and the Lambda function is triggered.

After uploading, we can see two folders here after some time. You can refresh it after a while if you don't see it.

![After the Lambda has run](https://dev-to-uploads.s3.amazonaws.com/i/04ithbr7t8uig8b3v8ls.png)

Awesome! Our Lambda has successfully run and we can check the  difference between the two pictures as well to see whether it has worked!

Original profile pic:

![The original profile pic of the cat](https://dev-to-uploads.s3.amazonaws.com/i/merv3jxndijl5impaog7.png)

Resized thumbnail:

![Shrunk down cat](https://dev-to-uploads.s3.amazonaws.com/i/6f6j1ihe6mroy15liuzi.png)


This is how we successfully deployed a stack with S3, Lambda and SQS to generate thumbnails automatically on upload.

Thank you all for reading! Do like and share this post if you've enjoyed it :)