Guest Post: Developing a Serverless Scheduler Using DynamoDB TTL and Filtered Streams
Tags: dynamodb, tutorial, aws, serverless
Published: 2022-03-04 00:06:52

Originally posted at [Serverless](https://www.serverless.com/blog/guest-post-developing-a-serverless-scheduler-using-dynamodb-ttl-and-filtered-streams)

This guest post is from Frédéric Barthelet. He has led [Theodo](http://www.theoodo.fr)’s serverless initiative since 2017 and contributes to the Serverless Framework and is building the expertise with his team. He is part of the AWS Community Builder program and helps the Typescript community with an improved developer-experience. Frédéric also is a DIY-er lover and an IoT enthusiastic. Learn more about him on [Github](https://github.com/fredericbarthelet) and [Twitter](https://twitter.com/bartheletf)!

Let us know if you’d like to be a [guest author](https://bit.ly/3mphVRE) on the Serverless blog!

AWS announced [Lambda event filtering](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html) during re:Invent 2021. This new feature helps reduce the quantity of Lambda’s invocation to exactly match your requirements and possibly save on costs. We’ll deep dive into a specific example in this article.

We’re happy to announce that Serverless Framework now supports event filtering for both **f** and **sqs** event types via the new **filterPatterns** option:
{% gist https://gist.github.com/richardgrantserverless/89cf042b82eecdf5a64a493e403eb609.js %}
[**Upgrade to v2.68.0 or greater**](https://www.serverless.com/framework/docs/getting-started#upgrade) to start using it. In case you are [**trying out Serverless Framework v3 beta**](https://www.serverless.com/blog/serverless-framework-v3-beta), make sure to update the v3 beta with “npm -g i serverless@pre-3”.

## Scheduler Pattern

Applications that need to perform a specific action at a certain time (sending out a marketing communication regarding a new menu just before lunch time i.e.) usually rely on a polling strategy, periodically checking in a database if any upcoming tasks are due. Polling however unnecessarily requires compute resources to not perform anything the majority of the time.

An alternative is the scheduler pattern. It leverages DynamoDB TTL (Time To Live) and DynamoDB stream features to be much more efficient in resource consumption.

Scheduled tasks are inserted in DynamoDB in the form of an item with a timestamp attribute mapped to the DynamoDB TTL attribute. This timestamp corresponds to the moment you’d like this task to be processed. Anything can schedule a task as long as it can write a new item in the DynamoDB table.

When the task scheduled time is reached, DynamoDB deletes the corresponding item. A **REMOVE** event is then published in the table stream, event that can be processed by Lambda to execute the actions corresponding to the task.

![](https://cdn-images-1.medium.com/max/3540/0*mSnBGc4xOojE3G64.png)

Additional information can be found related to this pattern, its advantages and drawbacks in [Yan Cui’s article](https://theburningmonk.com/2019/03/dynamodb-ttl-as-an-ad-hoc-scheduling-mechanism/).

It is critical that the task processor logic filters out events corresponding to **INSERT** or **MODIFY** operations within the table. Only **REMOVE** events correspond to scheduled tasks and must be processed accordingly. Newly inserted scheduled task will result in a **INSERT** event being published in DynamoDB stream, triggering the processor as well. Those events must be discarded. A common way to achieve this is the following:
{% gist https://gist.github.com/richardgrantserverless/d049c48442cedfdc7a752d9f9742291a.js %}
## Improving scheduler pattern with the event filtering feature

Using the new event filtering feature, you can ensure only **REMOVE** operations actually trigger the Lambda. The following corresponding configuration leverage the new **filterPatterns** property:
{% gist https://gist.github.com/richardgrantserverless/338ace0fec5b8bff1451c3400e7ea7df.js %}
![](https://cdn-images-1.medium.com/max/3320/0*pwPY_85dk7W5zUcz.png)

The **scheduledTasksProcessor** can then be simplified, removing the need to check for the **eventName**:
{% gist https://gist.github.com/richardgrantserverless/7c6e046f8f67df0a2d1e9d029eafcb8d.js %}
This improved pattern both reduces processor code base and improves readability. It also reduces the quantity of invocation for this processor, thus reducing overall costs of the pattern.

If you want to learn more, [**check out the official AWS documentation**](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html) on Lambda event filters as well as the [**Framework documentation**](https://www.serverless.com/framework/docs/providers/aws/events/streams#setting-filter-patterns).