NEW: DynamoDB Streams Filtering in Serverless Framework
Tags: aws, cloudnative, dynamodb, serverless
Published: 2021-12-06 14:19:12

From this article, you will learn how to utilize recently released functionality of Streams Filtering with DynamoDB and Lambda. 

We will move deeper than a basic sample of DynamoDB event action filtering. You will learn how to combine it with your business logic. I will be using DynamoDB _single-table_ design setup for that.

## What's new?
If you haven't heard, just before #reInvent2021 AWS dropped [this huge update](https://aws.amazon.com/about-aws/whats-new/2021/11/aws-lambda-event-filtering-amazon-sqs-dynamodb-kinesis-sources/).

## What's changed?
### Before the update
Every action made in a DynamoDB table (`INSERT`, `MODIFY`, `REMOVE`) triggered an event that was sent over DynamoDB Streams to a Lambda function. Regardless of the action type, a Lambda function was always invoked. That had two repercussions:
- You had to implement filter logic inside your Lambda code (`if` conditions) before executing your business logic (i.e. filter `INSERT` actions to send welcome email whenever new `User` was added into the table). 
- You paid for every Lambda run, even though in most cases you were interested only in _some_ events.

That situation was multiplied in _single-table_ design, where you store multiple types in a single table, so in reality you have many `INSERT`s with subtypes (ie. new user, new address, new order etc.)

### After the update
Now, **you can filter out events that are not relevant to your business logic**. By defining _filter criteria_, you control which events can invoke a Lambda function. Filtering evaluates events based on values that are in the message.

This solves above-mentioned problems: 
- Logic evaluation is pushed on AWS (no more `if`s in Lambda code)
- No more needless Lambda execution.

All of that thanks to the small JSON snippet defining _filter criteria_.

## Refactoring to the Streams Filtering

Since you're reading this article, it's safe to assume you're like me, already using DynamoDB Streams to invoke your Lambda functions.

Therefor, let me take you through the refactoring process. It's a simplified version of the code that I use on production. 

In my DynamoDB table, I store two types of entities: `Order` and `Invoice`. My _business logic_ requires me to _do something_ only when `Invoice` is modified. 
![Business logic conditions](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yj72f9jku4yw9r5rppu4.png)
As you can see, it's just the single case out of six. Imagine what happens when you have more types in your table, and your business logic requires you to perform other actions as well.


### Old event filtering
Let's start from those ugly `if` statements that I had before the update because I had to manually filter events. 

My Lambda's handler started with execution of `parseEvent` method:

```JavaScript
const parseEvent = (event) => {
  const e = event.Records[0] // batch size = 1
  const isInsert = e.eventName === 'INSERT'
  const isModify = e.eventName === 'MODIFY'

  const isOrder = e.dynamodb.NewImage?.Type?.S === 'Order'
  const isInvoice = e.dynamodb.NewImage?.Type?.S === 'Invoice'

  const newItemData = e.dynamodb.NewImage
  const oldItemData = e.dynamodb.OldImage

  return {
    isInsert, isModify, isOrder, isInvoice, newItemData, oldItemData
  }
}
```
Next step, I had to evaluate the condition in my handler:
```JavaScript
const {
    isInsert, isModify, isOrder, isInvoice, newItemData, oldItemData
  } = parseEvent(event)


if (isModify && isInvoice) {
  // perform business logic
  // uses newItemData & oldItemData values
}
```
### New event filtering
New functionality allows us to significantly simplify that code by pushing condition evaluation on AWS.

Just to recap, my business logic requires me to let in only `MODIFY` events that was performed on `Invoice` entities. Fortunately, I keep `Type` value on my entities in DynamoDB Table (thanks Alex 🤝).

The DynamoDB event structure is well-defined, so basically what I need to do is make sure that:
- `eventName` equals to `MODIFY`, and
- `dynamodb.NewImage.Type.S` equals to `Invoice`.

All of that is defined in `filterPatterns` section of Lambda configuration. Below is a snippet from Serverless Framework `serverless.yml` config file. Support for `filterPatterns` was introduced in version 2.68.0 - make sure you are using it  or newer.
```YAML
    functionName:
      handler: src/functionName/function.handler
      # other properties
      events:
      - stream:
          type: dynamodb
          arn: !GetAtt DynamoDbTable.StreamArn
          maximumRetryAttempts: 1
          batchSize: 1
          filterPatterns:
            - eventName: [MODIFY]
              dynamodb:
                 NewImage:
                   Type:
                     S: [Invoice]
```
And that's all you need to do to filter your DynamoDB Stream.
 
Amazing, isn't it?

## Gotchas
Bear in mind that there can be several filters on a single source. In such case, each filter works independently of the other. Simply put, there is `OR` not `AND` logic between them.

I learned that the hard way by mistakenly creating two filters:
```YAML
          filterPatterns:
            - eventName: [MODIFY]
            - dynamodb:
                 NewImage:
                   Type:
                     S: [Invoice]
```
by adding `-` in front of `dynamodb:`. It resulted in the wrong filter:
```JSON
{
  "filters": [
    {
      "pattern": "{\"eventName\":[\"MODIFY\"]}"
    },
    {
      "pattern": "{\"dynamodb\":{\"NewImage\":{\"Type\":{\"S\":[\"Invoice\"]}}}}"
    }
  ]
}
```
That one catches all `MODIFY` actions OR anything that has `Invoice` as `Type` in `NewImage` object, so DynamoDB `INSERT` actions as well!

Correct filter:
```JSON
{
  "filters": [
    {
      "pattern": "{\"eventName\":[\"MODIFY\"],\"dynamodb\":{\"NewImage\":{\"Type\":{\"S\":[\"Invoice\"]}}}}"
    }
  ]
}
```
You can view filter in Lambda console, under _Configuration->Triggers_ section.

### Global Tables
As [kolektiv](https://dev.to/droizman) mentioned in the comments below, this functionality does not work with Global Tables.

> One more catch, you can't use filtering with global tables, your filter will not be evaluated and your function will not be called. Confirmed with aws support.

Thanks for pointing that out.

## How much does it cost?

Nothing. 

There is no information about any additional pricing. Also, Jeremy Daly confirmed that during re:Invent 2021.

{% twitter 1466137340565278721 %}

{% twitter 1466171465733062657 %}

In reality, **this functionality saves you money on maintenance because it's easier to write & debug Lambda code, and on operations, as functions are executed only responding to business relevant events**.

## Low coupling
Before the update, people implemented event filtering logic in a single Lambda function. Thus, struggling from high coupling (unless they utilized some kind of dispatcher pattern).

Now, we can have several independent Lambda functions, each with its _filter criteria_, attached to the same DynamoDB Stream. That results in lower coupling between code that handles different event types. This will be very much appreciated by all _single-table_ design practitioners.

## Update

I forgot to mention that you can do more than just evaluate string equals condition in the filter. There are more possibilities, delivered by several comparison operators.

Here is a table ~~stolen~~ borrowed from [AWS Docs](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html#filtering-syntax) (If it's not OK to included it here please let me know.):

| Comparison operator | Example                                             | Rule syntax                                       |
|---------------------|-----------------------------------------------------|---------------------------------------------------|
| Null                | UserID is null                                      | "UserID": [ null ]                                |
| Empty               | LastName is empty                                   | "LastName": [""]                                  |
| Equals              | Name is "Alice"                                     | "Name": [ "Alice" ]                               |
| And                 | Location is "New York" and Day is "Monday"          | "Location": [ "New York" ], "Day": ["Monday"]     |
| Or                  | PaymentType is "Credit" or "Debit"                  | "PaymentType": [ "Credit", "Debit"]               |
| Not                 | Weather is anything but "Raining"                   | "Weather": [ { "anything-but": [ "Raining" ] } ]  |
| Numeric (equals)    | Price is 100                                        | "Price": [ { "numeric": [ "=", 100 ] } ]          |
| Numeric (range)     | Price is more than 10, and less than or equal to 20 | "Price": [ { "numeric": [ ">", 10, "<=", 20 ] } ] |
| Exists              | ProductName exists                                  | "ProductName": [ { "exists": true } ]             |
| Does not exist      | ProductName does not exist                          | "ProductName": [ { "exists": false } ]            |
| Begins with         | Region is in the US                                 | "Region": [ {"prefix": "us-" } ]                  |

## Summary
I hope this short article convinced you to refactor your Lambda functions that are invoked by DynamoDB Streams. It's really simple and makes a huge difference in terms of code clarity and costs.