Maximize your EventBridge productivity with evb-cli
Tags: aws, serverless, eventbridge
Published: 2020-12-29 07:46:56

The aim of this article is to demonstrate how [evb-cli](https://www.npmjs.com/package/@mhlabs/evb-cli) can be used to boost your productivity when working with [Amazon EventBridge](https://aws.amazon.com/eventbridge/). I will present use cases and how you can use the tool to solve them.

I assume that you are familiar with or a user of EventBridge. If not, there are some [great videos](https://www.youtube.com/watch?v=0gPZx9ex9gY) and [blog posts](https://aws.amazon.com/blogs/compute/tag/amazon-eventbridge/) available to get started.

This is part one of two covering this tool. Some commands require a back-end deployed to the target account and for the sake of digestability I will cover them in a separate post at a later date.

## Table of contents
* [Background](#background)
* [Prerequisites](#prerequisites)
* [Installation](#installation)
* [Schema Registry basics](#schema-registry-basics)
* [Commands](#commands)
 * [evb pattern](#pattern) - generate event patterns from schemas in the Schema Registry
 * [evb input](#input) - generate InputTransformer based on schemas in the Schema Registry
 * [evb code-binding](#code-binding) - generate code bindings for you Lambda consumers
 * [evb browse](#browse) - browse where and how events from a given source/detail-type are consumed 
 * [evb diagram](#diagram) - Visualise your EventBridge architecture
 * [evb extract-sam-event](#extract-sam-event) - convert SAM notation to `AWS::Events::Rule` for more complex use cases
 * [evb test-event](#test-event) - test event pattern against an event payload

Commands covered in part two:
 * evb replay - replay archived events against select target(s)
 * evb replay-dead-letter - replay dead letter events
 * evb local - local debugging

## Background <a name="background"><a>
EventBridge was released in July 2019, but it took about six months before we started using it heavily at [MatHem](https://mathem.se). Up until then we were leveraging SNS and SQS for our events, but as we started playing with EventBridge we quickly noticed how it improved the decoupling of our services and decreased friction between teams. 

Prior to EventBridge, when someone from team A wanted to subscribe to messages using filtering from an SNS topic owned by team B, they had to submit a pull request to team B to add the message attribute they wanted to filter on. We found this being backwards and inefficient.

With EventBridge, the producing team doesn't need to know its consumers. The consumers have access to perform filtering on any part of the payload, but they still need to know the contract of the data they can filter on. The release of the [EventBridge Schema Registry](https://aws.amazon.com/about-aws/whats-new/2019/12/introducing-amazon-eventbridge-schema-registry-now-in-preview/) at re:Invent 2019 meant that the decoupling got even more optimized.

As we started composing patterns we found that although we had removed team friction we still spent a lot of time on getting the patterns right. Also, writing complex patterns is error prone and requires multiple deploys to test and get right, so we began automating the bridge between the Schema Registry and the pattern composition.

The first version of `evb-cli` had that sole purpose - to generate event patterns to be pasted into the template. As we got really fast at that, we quickly found further bottlenecks, so we added commands for composing `InputTransformer`s, architecture visualization, code bindings, etc. The sections below describe each command in depth.

## Prerequisites <a name="prerequisites"></a>
Much of the functionality is based on content in the EventBridge Schema Registry. Make sure you have [enabled schema discovery](https://aws.amazon.com/blogs/compute/introducing-amazon-eventbridge-schema-registry-and-discovery-in-preview/#:~:text=To%20discover%20the%20schema%2C%20in,bus%20and%20choose%20Start%20discovery.) on your event buses.

To get the most of this tool you need to have the [aws-cli](https://aws.amazon.com/cli/) preconfigured with an IAM or SSO user with permissions covering `events:Get*`, `events:List*`, `schemas:Get*`, `schemas:List*` and `events:List*`. 

You will also need [NPM package manager](https://www.npmjs.com/) and NodeJS 12+ installed on your system.

## Installation <a name="installation"></a>
`npm install -g @mhlabs/evb-cli`

## Schema Registry basics <a name="schema-registry-basics"></a>
To make the most of this tool it's important to understand the basics and the limitations of the EventBridge Schema Registry.

There are three types of registries;
* AWS event schema registry - the schemas of the events the AWS services produce. These are the events that live on the default event bus
* Discovered schema registry - on a custom eventbus you can enable schema discovery. These schemas for these events end up here. They can be from an SaaS-provider or your own custom events
* Custom schema registry - here you can upload your own schemas

The discoverer ingests a sampling of the real events and analyses the values. An annoying thing here is that it can't handle null very well and seem to create new versions from 2 events where a value is null in one and non-null in the other. This means when we run commands like `evb pattern` it only knows the structure of the _last_ event it ingested. If that contained many null values it simply won't know about them.

Another important thing to know is that what constitutes a schema is the combination of `source` and `detail-type`. For example, the following event will get an entry in the registry under the name `my-source@my-detail-type`
```
{
  "source": "my-source",
  "detail-type": "my-detail-type
  ...
}
```

It's therefore a good practice to stick to the same `detail` structure for all events sharing that combination.

## Commands <a name="commands"></a>
The commands of the CLI have come to life organically as we identified the need for them. 

## `evb pattern` <a name="pattern"></a>
*Purpose:* Quickly and accurately build event patterns to match events from the EventBridge Schema Registry

*Example use case*: You are tasked with sending a Slack message to a channel each time a CodePipeline execution in any US region fails.

The events from CodePipeline look like this (taken from [here](https://docs.aws.amazon.com/eventbridge/latest/userguide/event-types.html#codepipeline-event-type)):
```
{
  "version": "0",
  "id": "CWE-event-id",
  "detail-type": "CodePipeline Pipeline Execution State Change",
  "source": "aws.codepipeline",
  "account": "123456789012",
  "time": "2017-04-22T03:31:47Z",
  "region": "us-east-1",
  "resources": [
    "arn:aws:codepipeline:us-east-1:123456789012:pipeline:myPipeline"
  ],
  "detail": {
    "pipeline": "myPipeline",
    "version": "1",
    "state": "STARTED",
    "execution-id": "01234567-0123-0123-0123-012345678901"
  }
}
```

The pattern you're after looks like this:
```
source:
  - "aws.codepipeline"
detail-type:
  - "CodePipeline Pipeline Execution State Change"
region:
  - prefix: "us-"
detail:
  state:
    - "FAILED"
```
Even though this is a very simple pattern it still requires googling the event structure if you want to compose it manually.

Using `evb pattern` you can solve this in seconds:
{% youtube K48-gfii9p8 %}
Note that I'm passing `-t template.yaml` to the command. This will prompt you with the option to attach the generated pattern to existing resources in your template. These can either be of type `AWS::Serverless::Function` or `AWS::Events::Rule`. If you skip the `-t` flag, the event will be written to the console.

Template injection works for both YAML and JSON, but note that any YAML comments will be stripped during serialization.

## `evb input`
*Purpose:* Quickly and accurately build `InputTransformer` CloudFormation objects.

*Example use case*: Building on the use case above we now want to invoke a Lambda function when events from CodePipeline match our pattern. To make the function code simpler, we only want to forward the properties we actually need from the event payload. Let's say we want to let the Slack users know the name of the pipeline and the region it was running in.

To achieve this we need to compose an `InputTransformer` block for the `Target` under the `AWS::Events::Rule` resource that maps JSON paths form the original payload into a new template:
```
InputTransformer:
  InputPathsMap:
    region: "$.region"
    pipeline: "$.detail.pipeline"
  InputTemplate: "{\"region\": <region>, \"pipeline\": <pipeline>}" 
```

This is tedious copy/paste work and, at least for us, the `InputTemplate` is often a straight off representation of the `InputPathsMap`. A nice feature would be if the CloudFormation team made `InputTemplate` optional and default it to mirror the `InputPathsMap` if omitted.

`evb input` is very similar to evb-pattern, but the output is different. Also, at the time of writing it doesn't yet support template injection, so a copy/paste from the console is required:
{% youtube oa23XcZPJmk %}

Note that I pass in `-f yaml`. This is telling the tool to output the result in YAML. Default is JSON.

## `evb code-binding`
*Purpose:* Generate code bindings in your preferred language for your Lambda function to use.

*Example use case*: Given the above `InputTemplate` we now want to receive this as a strongly typed object. For this example I will use C#, but a fill list of supported languages can be found in [quicktype's documentation](https://github.com/quicktype/quicktype#target-languages)

In the following video I generate two types of bindings; one for the `InputTemplate` and one for the entire original event.

{% youtube nbuY-HS0VAU %}

To get the optimized code binding for my `InputTemplate` (FromTemplate.cs), I run `evb code-binding` and point it at my template. At first I get prompted which language I want to generate for. Next it asks me in which registry the schema can be found. Since we're working with AWS produced events, it's under `aws.events`. If it's from a custom event bus, then you'd select `discovered-schemas`. Lastly, I select which `Rule` and `Target` combination I want to generate for. Here we only have one choice, so I select that one.

For the sake of demo I also generate a class for the entire original untransformed event (FromSchema.cs). I do this by skipping to provide the `-t` flag. The tool now takes me through prompts to select language, registry, source and detail-type. As you can see, the output is very verbose and the majority of properties will not be of any interest to the function.

## `evb browse` <a name="browse"></a>
*Purpose:* Quickly find usages of events conforming to a given schema.

*Example use cases:* 
* I could be a producer of an event and I've sent broken data for a period of time and I need to alert downstream teams
* I'm about to start working on a feature and I want to check if something similar already exists in the system

{% youtube C_qrJd6GIiI %}
Here we can see our Lambda target that consumes events belonging to the `aws.codepipeline@CodePipelinePipelineExecutionStateChange` schema along with some other pre-existing consumers. I can view the event pattern and any input transformations.

Another way to browse events is to visualize them using `evb diagram` as described below.

## `evb diagram`
*Purpose:* Visualize how events are flowing within an event bus

*Example use cases:* 
* To have a discussion around during project planning meetings
* To quickly get a mental view of how things hang together

{% youtube vFd0E0Rh7qk %}

In my next post I will go through setting up the `evb-local` back-end. With that deployed on the target account you are able to see the events as JSON in real-time as they happen.

## `evb extract-sam-event` <a name="extract-sam-event"></a>
*Purpose:* To convert a SAM [EventBridgeRule](https://github.com/aws/serverless-application-model/blob/master/versions/2016-10-31.md#eventbridgerule) to an `AWS::Event::Rule` to get more control over the rule.

*Example use case:* Often when you need an EventBridge to Lambda integration it's tempting to start with the [EventBridgeRule](https://github.com/aws/serverless-application-model/blob/master/versions/2016-10-31.md#eventbridgerule) SAM shorthand. Under the hood, the SAM macro inflates this into a full `AWS::Events::Rule` and an `AWS::Lambda::Permission`, but using this disables you from using the powerful `InputTransformer`. I've opened an [issue](https://github.com/aws/serverless-application-model/issues/1535) about this, so please upvote. 

I've found myself refactoring a SAM `EventBridgeRule` into the real thing enough times to automate it with a simple command. Again, note that if you have comments in your YAML, then these will be stripped using this command.

{% youtube qzv0VPBGoC8 %}

## `evb test-event` <a name="test-event"></a>
*Purpose:* Tests an event payload against rules on an event bus.

*Example use cases:* 
* You have been given an example payload to consume and you want to test your rule.
* You are a producer of events and want to see which rules match it.
* Integration tests

{% youtube EGrE_MNylx8 %}

## Conclusion
In this post I've gone through how we optimize many of the tasks developers working with EventBridge encounter - from building patterns, input transformers, code bindings to gaining insights into the system by browsing schema usage or visualizing the architecture.

In my next post I will focus on local debugging and replaying of archived events.