Evaluating AWS EventBridge Schema Discovery
Tags: aws, eventdriven
Published: 2021-07-24 17:31:54

This article is going to experiment with the [AWS Schema Registry with Event Discovery](https://aws.amazon.com/blogs/compute/introducing-amazon-eventbridge-schema-registry-and-discovery-in-preview/) and evaluate [EventBridge Atlas](https://github.com/boyney123/eventbridge-atlas) which is a 3rd party tool that generates documentation pages for it.

This is part 1 of 2.  Part 2 will include the code repo.

## Getting Started (for me, not you)

Architecture-wise I don't need much for evaluating things.  I'll need an event bus with discovery turned on... and really thats it.  I ❤️ CDK so I use that to generate my CloudFormation and to make an event bus and turn on schema discovery it's as simple as:

```typescript
    const bus = new EventBus(this, 'inquisitorBus', {
      eventBusName: 'inquisitorBus',
    });
    new CfnDiscoverer(this, 'inquisitorDisco', {
      sourceArn: bus.eventBusArn,
      description: 'Inquisitor Discoverer',
    });
```

From there we need to send some events to see if they're discovered...

```bash
aws events put-events --entries '[{"Source": "mystore","DetailType": "Review Created","EventBusName":"inquisitorBus","Detail": "{\"star_rating\": 5,  \"description\": \"The size and length fit me well and the design is fun. I felt very secure wearing this tshirt. \",  \"helpful_count\": 34,  \"unhelpful_count\": 1,  \"pros\": [\"lightweight\",\"fits well\"  ],  \"cons\": [],  \"customer\": {\"name\": \"Julian Wood\",\"email\": \"julianreview@amazon.com\",\"phone\": \"+1 604 123 1234\"  },  \"product\": {\"product_id\": 788032119674292922,\"title\": \"Encrypt Everything Tshirt\",\"sku\": \"encrypt-everything-tshirt\",\"inventory_id\": 23190823132,\"size\": \"medium\",\"taxable\": true,\"image_url\": \"https://img.mystore.test/encrypt-tshirt.jpg\",\"weight\": 200.0}}"}]'
```

After sending the first event I checked the Discovered Schema Registry in AWS Console to find.... nothing.


![Screen Shot 2021-07-24 at 10.08.00 AM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1627144473125/hrVNF2vA6.png)

It does say "It could take several minutes for schemas to appear once discovery has been enabled."

Ultimately it took around 5 minutes for the schema to be discovered...

![Screen Shot 2021-07-24 at 10.11.39 AM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1627144506304/GDY-uNRgO.png)

I'd like to see if the schema discovery is smart enough to infer optional properties of payloads, so let's send some variations of the event:

```bash
aws events put-events --entries '[{"Source": "mystore","DetailType": "Review Updated","EventBusName":"inquisitorBus","Detail": "{\"star_rating\": 5,  \"description\": \"The size and length fit me well and the design is fun. I felt very secure wearing this tshirt. \",  \"helpful_count\": 34,  \"unhelpful_count\": 1,  \"pros\": [\"lightweight\",\"fits well\"  ],  \"cons\": [],  \"customer\": {\"name\": \"Julian Wood\",\"email\": \"julianreview@amazon.com\",\"phone\": \"+1 604 123 1234\"  },  \"product\": {\"product_id\": 788032119674292922,\"title\": \"Encrypt Everything Tshirt\",\"sku\": \"encrypt-everything-tshirt\",\"inventory_id\": 23190823132,\"size\": \"medium\",\"taxable\": true}}"}]'
```

```bash
aws events put-events --entries '[{"Source": "myOtherStore","DetailType": "Review Created","EventBusName":"inquisitorBus","Detail": "{\"star_rating\": 5,  \"description\": \"The size and length fit me well and the design is fun. I felt very secure wearing this tshirt. \",  \"helpful_count\": 34,  \"unhelpful_count\": 1,  \"pros\": [\"lightweight\",\"fits well\"  ],  \"cons\": [],  \"customer\": {\"name\": \"Julian Wood\",\"email\": \"julianreview@amazon.com\",\"phone\": \"+1 604 123 1234\"  },  \"product\": {\"product_id\": 788032119674292922,\"title\": \"Encrypt Everything Tshirt\",\"sku\": \"encrypt-everything-tshirt\",\"inventory_id\": 23190823132,\"size\": \"medium\",\"taxable\": true,\"image_url\": \"https://img.mystore.test/encrypt-tshirt.jpg\",\"weight\": 200.0}}"}]'
```

```bash
aws events put-events --entries '[{"Source": "mystore","DetailType": "Review Created","EventBusName":"inquisitorBus","Detail": "{\"star_rating\": 5,  \"helpful_count\": 34,  \"unhelpful_count\": 1,  \"pros\": [\"lightweight\",\"fits well\"  ],  \"cons\": [],  \"customer\": {\"name\": \"Julian Wood\",\"email\": \"julianreview@amazon.com\",\"phone\": \"+1 604 123 1234\"  },  \"product\": {\"product_id\": 788032119674292922,\"title\": \"Encrypt Everything Tshirt\",\"sku\": \"encrypt-everything-tshirt\",\"inventory_id\": 23190823132,\"size\": \"medium\",\"taxable\": true,\"image_url\": \"https://img.mystore.test/encrypt-tshirt.jpg\"}}"}]'
```

```bash
aws events put-events --entries '[{"Source": "mystore","DetailType": "Review Created","EventBusName":"inquisitorBus","Detail": "{\"helpful_count\": 34,  \"unhelpful_count\": 1, \"customer\": {\"name\": \"Julian Wood\",\"email\": \"julianreview@amazon.com\",\"phone\": \"+1 604 123 1234\"  },  \"product\": {\"product_id\": 788032119674292922,\"sku\": \"encrypt-everything-tshirt\",\"inventory_id\": 23190823132,\"taxable\": true,\"image_url\": \"https://img.mystore.test/encrypt-tshirt.jpg\"}}"}]'
```

```bash
aws events put-events --entries '[{"Source": "mystore","DetailType": "Review Created","EventBusName":"inquisitorBus","Detail": "{\"helpful_count\": 34}"}]'
```

There is a fairly consistent (several minute) lag between events and discovery.

It does NOT infer optional properties and only uses the most recent payload type for the schema, which is a little disappointing.  It DOES store multiple versions of the schemas though... so you could re-combine them.  I suppose the best practice in this case would be to have a consistent event structure and if you need optional parameters, have different event detail types?

## Is there a schema registry change event?

AWS has two provided events `aws.schemas@SchemaCreated` and `aws.schemas@SchemaVersionCreated`.  The source of these events is `aws.schemas` and the detail-types are `SchemaCreated` and `SchemaVersionCreated`... so if I subscribe to just the source using the default event bus you can get these notifications.  The event payload looks like this:

```json
{
    "version": "0",
    "id": "f480b317-a12f-5ccc-89ac-2446862c329b",
    "detail-type": "Schema Version Created",
    "source": "aws.schemas",
    "account": "359317520455",
    "time": "2021-07-24T15:01:29Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:schemas:us-east-1:359317520455:schema/discovered-schemas/mystore@ReviewCreated"
    ],
    "detail": {
        "SchemaName": "mystore@ReviewCreated",
        "SchemaType": "OpenApi3",
        "RegistryName": "discovered-schemas",
        "CreationDate": "2021-07-24T15:01:19Z",
        "Version": "5"
    }
}
```

This will be useful in part 2 😈

At this point I think we're ready to kick the tires on EventBridge Atlas.

## EventBridge Atlas

EventBridge Atlas isn't an npm library.  Setup for it wants you to clone the repo, install and then locally run / configure everything.  It doesn't auto-update.  It would be pretty easy to create a pipeline to re-generate the docs on a Schema Event.  Part 2 will deal with improving that, but for now let's just use the default.

[EventBridge Atlas Getting Started](https://eventbridge-atlas.netlify.app/docs/getting-started/installation)

It's a pretty simple project that downloads the schema and then processes it through some parsers to support 4 different documentation engines.

1. Slate - pretty bare bones...
2. asyncapi - I like how this one split out the messages separate from the schemas
3. docuowl - similar to slate
4. flow (node diagram) - nice for visualization and would be great combined with x-ray or something, but for dev purposes I don't think it's _that_ useful


![Screen Shot 2021-07-24 at 10.11.45 AM.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1627145007632/10Y9MzX9s.png)

asyncapi looks to be the most useful.

## So what does the EventBridge Atlas project actually do?

First it has you run `npm run generate-metadata-templates` command, **but this step is optional**.  This uses the aws sdk to list all the schemas, retrieves them all as JSON, and extends them a bit.  Nothing too special here... could be done as part of a lambda execution and saving to s3 and/or dynamodb.  This step is so that you can take these templates and add your own notes manually.

Next you have your choice of documentation engines... Using your choice you run `npm run build:<engine>` which in turn runs `npm run generate`... which does similar stuff to generating the metadata templates.  It also retrieves the schemas and extends them with some OpenAPI functionality.  After that it runs everything through a parser to get it in the preferred engine's format.  The project in general requires docker, but it looks like it's only for the slate and docuowl engines... which I don't want to use anyways.

## Next Steps

This concludes part 1.  I played with Schema Discovery and kicked the tires on EventBridge Atlas and learned a lot in the process.  But I can make this better.

Part 2 will involve:

* Having a lambda subscribe to the Schema Change Events
* Use the lambda to generate and store metadata for the events (and infer optional properties)
* Generate the TypeScript Interfaces to be published via an npm package
* Store the automated documentation in DynamoDB (for merging / extending)
* Host the documentation

Stay tuned...