The Unsung Hero of Serverless apps - Event Source Mapping
Tags: aws, serverless, architecture
Published: 2021-09-05 15:37:38

---
title: The Unsung Hero of Serverless apps - Event Source Mapping
published: true
description: What is the Event Source Mapping, and what it has to do with the Circuit Breaker pattern.
tags: aws, serverless, architecture
cover_image: https://images.unsplash.com/photo-1611329857570-f02f340e7378?ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&ixlib=rb-1.2.1&auto=format&fit=crop&w=2850&q=80
---
With the rise of serverless infrastructure tools like [_AWS CDK_](https://aws.amazon.com/cdk/), [_AWS SAM_](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html) or [_Serverless Framework_](https://www.serverless.com/) more and more developers have immense power at their hands. With the robust abstraction layer over the _AWS CloudFormation_ at our disposal, we can prototype, develop and ship serverless apps faster.

Unfortunately, in using those abstractions, the knowledge of how the underlying service operates might be forgotten or never acquired in the first place. While I believe this precedent is not harmful in a grand scheme of things, on some occasions, a deep understanding of how pieces fit together can help us built more robust and reliable services.

Today, let us focus on one such topic that might get lost as it's most likely entirely abstracted away by the framework you are using - the _Event Source Mapping_.

## What is the _Event Source Mapping_

Every time I learn something new about an AWS service, especially a service that I used for a long time, I'm met with the realization that, in reality, I know very little about it.

For a long time, while working with _AWS Lambda_ and _AWS SQS_, integrating the services in various ways, I never thought about what makes those two work together - I did not have to.

My perspective shifted drastically after building (and augmenting) the popular [circuit breaker](https://www.jeremydaly.com/the-circuit-breaker/) pattern.

Here is how my version of the pattern looked like:

![Initial design of the pattern](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ktakb9fferea5yc5aya7.png)

To make it all work, I had to dynamically disable the integration between _Amazon SQS_ and _AWS Lambda_ to make my architecture work. There is no point in invoking the _Lambda function_ if the 3rd party endpoint is unavailable.

![How do I turn off the SQS integration?](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9vaf0ny1q5a4e8lomcss.png)

After a bit of searching, I stumbled upon [this AWS CLI command](https://docs.aws.amazon.com/cli/latest/reference/lambda/update-event-source-mapping.html) and [this AWS documentation page](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html).

Reading the documentation, my mental model on how _AWS Lambda_ integrates with _Amazon SQS_ greatly improved. **In the context of _AWS Lambda_, the _Event Source Mapping_ refers to an _AWS Lambda_ resource that reads from an event source and invokes a Lambda function**.

Suddenly, what [Anahit](https://twitter.com/anahit_fi) and [Jeremy](https://twitter.com/jeremy_daly) were talking about in [episode #92 of Serverless Chats](https://www.serverlesschats.com/92/) made sense, how tracking the iterator when reading from _Amazon DynamoDB_ and _Amazon Kinesis_ streams works made sense. You get the idea.

The _Event Source Mapping_ is the gluing piece for many integrations. If you are using _DynamoDB_ streams with Lambda, the _Event Source Mapping_ is involved. The same rule applies for _Kinesis_ integration with Lambda. Check out the [documentation](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html) for the complete list of services and integrations.

## Back to the Circuit Breaker pattern

With my newfound knowledge, here are the adjustments I made to my architecture.

![With toggling the integration on and off](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vvlukpsm85mx91haya41.png)

The first change(1) is the addition of a Lambda function, which responsibility is to toggle the _Event Source Mapping_ depending on the state of the state machine.

The second(2) is probing the 3rd party endpoint to check if it's available. This is done by invoking the "main" Lambda function directly.
Depending on the response I get back, I will either close the circuit or wait a bit and try again.

Looking back, it would even be possible to skip the "invoker" Lambda function altogether by using a direct integration between _AWS Step Functions_ and _AWS Lambda_.

While the architecture is not perfect, and I do not feel comfortable recommending it for production usage, I believe it's a viable alternative to what [Jeremy](https://twitter.com/jeremy_daly) proposed. Maybe it could be used as a starting point for your architecture?

## Summary

AWS landscape is vast. I sometimes find it hard to keep track of all the changes to the AWS services relevant to my work. That is why I'm a big advocate for "just in time learning" if possible and diving deep from time to time.

I hope that this article was just that, a nudge to dive a bit deeper into the architecture of your serverless app. You never know what kind of new service or resource you might discover.

If you are hungry for more, I post some of my serverless discoveries on Twitter - [@wm_matuszewski](https://twitter.com/wm_matuszewski).

Thank you for your time.