AWS EventBridge Replays
Tags: eventbridge, serverless, aws, lambda
Published: 2020-11-09 23:57:23

AWS recently [announced native replay functionality](https://aws.amazon.com/blogs/aws/new-archive-and-replay-events-with-amazon-eventbridge/) for [EventBridge](https://aws.amazon.com/eventbridge/)! In my opinion, replayability and event archiving in event-driven architectures are often overlooked and a resilient, cost-effective solution is often needlessly complicated - so this is fantastic news! 🚀

I was eager to try it out so I decided to build a small demo - come join me!

## Project setup

`npm init -y`, install [Serverless framework](https://www.serverless.com/), [serverless-pseudo-parameters](https://www.serverless.com/plugins/serverless-pseudo-parameters) and the [AWS SDK](https://www.npmjs.com/package/aws-sdk)

```bash
npm install serverless serverless-pseudo-parameters --dev
```

Let's start by creating a fake order producer that we later can pretend is some important service that lives somewhere else. The "createOrder" service will put an event on our message bus containing an order based on the `orderId` we pass.

```js
// createOrder.js
const AWS = require('aws-sdk');

const eventBridge = new AWS.EventBridge({ region: 'eu-north-1' });

const handler = async (event) => {
  const order = {
    id: event.orderId,
  };
  
  return eventBridge.putEvents({
    Entries: [{
      EventBusName: 'orders',
      Source: 'elk.orders',
      DetailType: 'order',
      Detail: JSON.stringify(order)
    }]
  }).promise();
};

module.exports.handler = handler;
```

next, let's create our "orderCreated" function which will listen for any order events on the bus and work its magic on them.

```js
// orderCreated.js
const handler = async (event) => {
  console.log(`handled order ${event.detail.id}`);
};

module.exports.handler = handler;
```

and finally, let's create our `serverless.yml` where we'll define all the infrastructure resources our service will need.

```yml
# serverless.yml
service: order-service

plugins:
  - serverless-pseudo-parameters

provider:
  name: aws
  region: eu-north-1
  runtime: nodejs12.x
  iamRoleStatements:
    - Effect: Allow
      Action: events:PutEvents
      Resource: ${self:custom.eventBusArn}

custom:
  eventBusArn: arn:aws:events:${self:provider.region}:#{AWS::AccountId}:event-bus/orders

functions:
  createOrder:
    handler: createOrder.handler
  orderCreated:
    handler: orderCreated.handler
    events:
      - eventBridge:
          eventBus: ${self:custom.eventBusArn}
          pattern:
            source:
              - elk.orders

resources:
  Resources:
    EventBus:
      Type: AWS::Events::EventBus
      Properties:
        Name: orders
    EventArchive:
      Type: AWS::Events::Archive
      Properties:
        ArchiveName: orders-archive
        SourceArn: ${self:custom.eventBusArn}
```

There's quite a lot going on here, so let's break it down a bit, shall we? We shall.

We're specifying that the `orderCreated` function should be triggered by all events on the "orders" EventBus. The EventBus itself is defined as a CloudFormation resource.
However, at the time of writing this, there's a bug in Serverless framework that prevents us from referencing the ARN of the CF resource in the event definition so we reference the "calculated" ARN in a custom property, `eventBusArn`, instead.

In addition to the `EventBus` resource, there's an `Archive` resource as well. This is the resource that defines our archive, in which all events put on the bus will be archived (and thus available to be replayed). We also set up the Lambda EventBridge trigger and specified that we're interested in all events coming from the "elk.orders" source.  

We've also added an IAM role statement that allows our service to put events on the bus since we're hosting our event producer in the same service.

Alright, let's try it out.

From your nearest terminal, deploy the service by running `npx sls deploy`. We can create our first event by invoking the producer with an order

```bash
npx sls invoke --function createOrder -d '{ "orderId": "1" }'
```

if we now look at the logs of our `orderCreated` function, we should see that that it's received and "processed" the order.

```bash
npx sls logs --function orderCreated
[...]
START RequestId: d8069d8e-ed1a-4837-8e3f-b44f12e23773 Version: $LATEST
2020-11-09 00:18:41.268 (+01:00)	d8069d8e-ed1a-4837-8e3f-b44f12e23773	INFO	handled order 1
END RequestId: d8069d8e-ed1a-4837-8e3f-b44f12e23773
```

Great, it works - let's destroy something! If our `orderCreated` Lambda were to fail, the events would automatically be retried with an automatic exponential backoff - but what happens if we don't *actually* "fail", we just fail to process the event correctly? Unfortunately, it would be lost.

Let's simulate something like that happening by changing the name of our order's id field from `id` to `orderId` in the event.

```js
// createOrder.js
[...]
const handler = async (event) => {
  const order = {
    orderId: event.orderId,
  };
[...]
```

deploy our service again, and then create another event, order 2

```bash
npx sls invoke --function createOrder -d '{ "orderId": "2" }'
```

The logs from the `orderCreated` function will now say

```bash
INFO	handled order undefined
```

Alright, I think we can figure out what the problem is, let's fix the bug

```js
// orderCreated.js
const handler = async (event) => {
  console.log(`handled order ${event.detail.orderId}`);
};

module.exports.handler = handler;
```

and deploy again.

## Replays to the rescue

We botched the handling of that second order but luckily we created an archive for our events when we created the event bus.

[@mhlabs/evb-cli](https://www.npmjs.com/package/@mhlabs/evb-cli) is a neat little CLI tool to manage EventBridge, so let's use that to trigger our replay.

```bash
npm i @mhlabs/evb-cli -g
```

Run the tool and specify the "orders" bus, `evb replay -b orders`, then adjust the time span as needed.

![gif](https://media.giphy.com/media/7qzNgUdqRM8lSZY2tD/giphy.gif)

It'll take a minute or so for EventBridge to fetch and re-send your events from the archive, but in a moment we should be able to see that the second order has now been processed by our `orderCreated` handler! 🎉

```sh
INFO	handled order 2
```

Isn't that amazing? With just a handful of lines of code, we've set up an EventBus that durably stores all events for as long as we need them from which we can replay events sent within a given time span to any consumers that need to (re)process the events as well as a service that reads and writes to the bus.

Do keep in mind that when designing systems capable of replaying events, it's crucial to make sure your services are idempotent. If (when) the same order event is sent for the second time, your customer likely didn't buy two of the same Die Hard 2 DVD off your site at the same time.

The code for this demo can be found [here](https://github.com/TastefulElk/eventbridge-replay-demo).

PS.

Did you know EventBridge can automatically generate schemas and code bindings for Typescript, Python or Java? Using those would've prevented that whole id->orderId fiasco from before... Oh well.


If you enjoyed this guide and want to see more, follow me on Twitter at [@TastefulElk](https://twitter.com/tastefulelk) where I frequently write about serverless tech, AWS and developer productivity!