💥 AWS CDK 101 - 🐣 Event source mapping with Cfn property override
Tags: serverless, aws, cdk, typescript
Published: 2022-05-24 04:21:13

🔰  Beginners new to AWS CDK, please do look at my previous articles one by one in this series.

If in case missed my previous article, do find it with the below links.

🔁 Original previous post at 🔗 [Dev Post](https://devpost.hashnode.dev/aws-cdk-101-dynamodb-streams-triggering-batch-deleteobjects-s3)

🔁 Reposted the previous post at 🔗 [dev to @aravindvcyber](https://dev.to/aravindvcyber/aws-cdk-101-dynamodb-streams-triggering-batch-deleteobjects-s3-2abn)

In this article, we will be making a simple change in our CDK stack which eventually helps us with more optimization and control in processing streams. The high-level configuration which we have made to connect dynamodb streams event source with lambda in the last two articles will be optimized using event source mapping L2 construct with simple override with its cfn L1 construct as well.

## Why we need this refactoring :beetle:

In our last article, you can find that we are using dynamodb streams from `stgMessages` to delete objects from S3. The current configuration does not filter or distinguish the event based on the event name at the source, instead, we invoke the lambda unnecessary even for events other than `REMOVE` like `INSERT` and we end up, making unwanted batch invocations, which could have otherwise been avoided.

## Benefits we achieve :volcano:

* In a previous couple of articles we directly interfaced with our dynamodb streams without any filter and our lambda is processing every single one of them and optionally choosing business logic based on the CRUD operation which leads to more number of stream records processed by the lambda eventually.
* The above scenario can be avoided by filtering using the `eventName` in the dynamodb stream record and selectively using only the most appropriate record needed to be processed in our specific lambda handler.
* Here we not only get a chance to filter with CRUD operation, but also you can filter with the other fields which are part of the record object, this will help us filter these messages to the fullest.
* One more thing we achieve here would be that we can offload the filtering from inside the lambda to the invocation source mapping itself and hence time and cost are saved.

## Necessary imports in the stack :orange_book:

Here we will be making use of the below imports in our stack file.

```ts

import {
  CfnEventSourceMapping,
  EventSourceMapping,
  StartingPosition,
} from "aws-cdk-lib/aws-lambda";

```

## Previous event source mappings :tulip:

Earlier we have used something like the below to configure the source to the lambda. This does not have the filtering enabled and all the streams are used to invoke the lambda, which leads to unwanted executions.

The below L2 construct does not yet provide a provision to add the filter and hopefully, we can expect this soon.

```ts

const stgTableSourceProps: DynamoEventSourceProps = {
    startingPosition: lambda.StartingPosition.LATEST,
    batchSize:100,
    maxBatchingWindow: Duration.seconds(60),
}
stgMessageStreamFunc.addEventSource(new DynamoEventSource(stgMessages, stgTableSourceProps));

```

## Generic event source mapping construct :chestnut:

Due to the shortcomings discussed above, we are going to try another generic L2 construct.

We will replace this with a more generic event source mapping construct as shown below for both the lambda handlers used in the previous 2 articles.
Here we have added a few changes besides filtering.

* `startingPosition` is changed to `StartingPosition.TRIM_HORIZON` to get the oldest item first.
* `bisectBatchOnError` will split the record chunk into half and retry each half so that we can identify and isolate the problematic record from the other records while processing them inside the handler.

```ts
const msgDeleteEventSourceMap = new EventSourceMapping(this, "msgDeleteEventSourceMap", {
      target: messageStreamFunc,
      eventSourceArn: messages.tableStreamArn,
      batchSize: 100,
      maxBatchingWindow: Duration.seconds(60),
      startingPosition: StartingPosition.TRIM_HORIZON,
      bisectBatchOnError: true, 
});

msgDeleteEventSourceMap.applyRemovalPolicy(RemovalPolicy.DESTROY);

```

```ts

const stgMsgDeleteEventSourceMap = new EventSourceMapping(this, "stgMsgDeleteEventSourceMap", {
      target: stgMessageStreamFunc,
      eventSourceArn: stgMessages.tableStreamArn,
      batchSize: 100,
      maxBatchingWindow: Duration.seconds(60),
      startingPosition: StartingPosition.TRIM_HORIZON,
      bisectBatchOnError: true, 
});

stgMsgDeleteEventSourceMap.applyRemovalPolicy(RemovalPolicy.DESTROY);

```

## Granting stream read to handler functions :mushroom:

In the last two articles, we have not performed this explicitly, since the dynamodb event source addition to the lambda construct, automatically applied for the necessary permissions. Whereas in this case, we may have to grant the privileges as shown below.

```ts

messages.grantStreamRead(messageStreamFunc);

```

```ts

stgMessages.grantStreamRead(stgMessageStreamFunc);

```

## Cloudformation property override :blossom:

Currently, we could not add the filter criteria as this `EventSourceMapping` L2 construct directly, so here we have to access its L1 construct and add a property override as shown below for both the lambda handlers.

Once do note that there is much scope in the filtering pattern used here where we could include more fields to achieve the desired effect and it is not only limited to `eventName` only.

```ts
const cfnMsgDeleteEventSourceMap = msgDeleteEventSourceMap.node.defaultChild as CfnEventSourceMapping;

cfnMsgDeleteEventSourceMap.addPropertyOverride("FilterCriteria", {
      Filters: [
        {
          Pattern: JSON.stringify({
            eventName: ["INSERT"],
          }),
        },
      ],
});
```

![insert map](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rdu8v2avufyg99dj6zpe.png)

```ts

const cfnStgMsgDeleteEventSourceMap = stgMsgDeleteEventSourceMap.node.defaultChild as CfnEventSourceMapping;

cfnStgMsgDeleteEventSourceMap.addPropertyOverride("FilterCriteria", {
      Filters: [
        {
          Pattern: JSON.stringify({
            eventName: ["REMOVE"],
          }),
        },
      ],
});
```

![remove map](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2rh1sq13czv8s0jpnczv.png)

## Issue while deploying the stack :lollipop:

You may occasionally get an error failure to deploy the stack, by then it is necessary for us to manually delete the event source mapping in the lambda either with the AWS console or using the AWS CLI as shown below.

```sh
The stack named CommonEventStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "The event source arn (" arn:aws:dynamodb:ap-south-1:*****:table/stgMessagesTable/stream/2022-05-21T03:39:54.488 ") and function (" CommonEventStack-stgMessageStreamFuncF95CB7BF-877BwMFlctoc ") provided mapping already exists. Please update or delete the existing mapping with UUID 43eccb79-7a6f-4fd0-82f1-c176cff317da
```

### AWS console to delete the mapping :flags:

![delete aws console](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jehxe8w6z2a9uij2jq2z.png)

### AWS cli to delete mapping :sushi:

Verify the event source mapping if it is existing as shown below

`aws lambda delete-event-source-mapping --uuid ********`

![get 2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3ujsmtt8v0j03vkkgzta.png)

Then we can delete it by executing the below command

`aws lambda delete-event-source-mapping --uuid ********`

![del 2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iwvse9x9pqkx4hc1lip4.png)

## Conclusion :fried_shrimp:

Here we have demonstrated two things as shown below.

* Ability to filter streams inside the generic event source mapping which is not only limited to dynamodb stream, this has a lot of other applications to be used similarly for kinesis streams, SQS source, and a lot more sources.
* Additionally we have learned how we can use the L1 construct for an L2 resource and override certain properties, this will have a lot of use cases beyond this example.

We will be adding more connections to our stack and making it more usable in the upcoming articles by creating new constructs, so do consider following and subscribing to my newsletter.

⏭ We have our next article in serverless, do check out

https://dev.to/aravindvcyber/aws-cdk-101-graphql-using-appsync-with-dynamodb-16dd

🎉 Thanks for supporting! 🙏

Would be great if you like to [☕ Buy Me a Coffee](https://www.buymeacoffee.com/AravindVCyber), to help boost my efforts 😍.

<a href="https://www.buymeacoffee.com/AravindVCyber"><img src="https://img.buymeacoffee.com/button-api/?text=Buy me a coffee&emoji=&slug=AravindVCyber&button_colour=BD5FFF&font_colour=ffffff&font_family=Cookie&outline_colour=000000&coffee_colour=FFDD00" /></a>

<a href='https://ko-fi.com/X8X0CITDJ' target='_blank'><img height='36' style='border:0px;height:36px;' src='https://cdn.ko-fi.com/cdn/kofi4.png?v=3' border='0' alt='Buy Me a Coffee at ko-fi.com' /></a>

🔁 Original post at 🔗 [Dev Post](https://devpost.hashnode.dev/aws-cdk-101-event-source-mapping-with-cfn-property-override)

🔁 Reposted at 🔗 [dev to @aravindvcyber](https://dev.to/aravindvcyber/aws-cdk-101-event-source-mapping-with-cfn-property-override-3laa)

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">🏄 AWS CDK 101 - 🐲 GraphQL using AppSync with dynamodb<br>{ by <a href="https://twitter.com/Aravind_V7?ref_src=twsrc%5Etfw">@Aravind_V7</a> } <br><br>Checkout more such in my page<a href="https://t.co/CuYxnKr0Ig">https://t.co/CuYxnKr0Ig</a><a href="https://t.co/phzOKFlXXO">https://t.co/phzOKFlXXO</a><a href="https://twitter.com/hashtag/aws?src=hash&amp;ref_src=twsrc%5Etfw">#aws</a> <a href="https://twitter.com/hashtag/typescript?src=hash&amp;ref_src=twsrc%5Etfw">#typescript</a> <a href="https://twitter.com/hashtag/graphql?src=hash&amp;ref_src=twsrc%5Etfw">#graphql</a> <a href="https://twitter.com/hashtag/dynamodb?src=hash&amp;ref_src=twsrc%5Etfw">#dynamodb</a> <a href="https://twitter.com/hashtag/awscdk?src=hash&amp;ref_src=twsrc%5Etfw">#awscdk</a> <a href="https://t.co/yeqeLZQt8A">https://t.co/yeqeLZQt8A</a></p>&mdash; Aravind V (@Aravind_V7) <a href="https://twitter.com/Aravind_V7/status/1531153165298839552?ref_src=twsrc%5Etfw">May 30, 2022</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>