Amazon EventBridge API Destinations with parametrized endpoint URL and Mailchimp integration
Tags: serverless, eventdriven, mailchimp, eventbridge
Published: 2021-04-06 10:23:47

I love ❤️ working with Amazon EventBridge service. This is not a big surprise as it is beloved by the community. However, for me, it's not just the pure technical excellence of it.

Whenever I think about EventBridge I hear in my mind famous words from the TRON movie. They were even [reused at the very beginning of the sequel](https://youtu.be/QBYr0k8dOtw?t=26) TRON: Legacy.

> I tried to picture clusters of information as they moved through the computer.
What do they look like? Ships? Motorcycles?
Were the circuits like freeways?
I kept dreaming of a world I thought I'd never see.
And then, one day...
I got in.

Even though those words were said about computer hardware not message bus *per se*, in my opinion they apply perfectly to it. 

Whenever I work with EventBridge I have this *sci-fi* excitement that origins from that movie. 

I hope you will get that *vibe* too!

## What you will learn from the article
This article is not just about Mailchimp integration. You should read it if you want to learn how to:
* setup Amazon EventBridge API Destinations to send REST requests to any 3rd party endpoint on the Internet
* **use HTTP Path Parameters** to trigger different endpoints based on message content - **this is huge** as it is almost not described in the documentation, and there are no examples explaining that functionality.
* setup and deploy sample project using Serverless Framework to **learn by doing** in your AWS account
* integrate it with Mailchimp to tag users based on messages on your Event Bus
* and finally, what to do when something doesn't work (**how to debug** API Destinations integrations)?

## Business case
This tutorial is based on my experiences while implementing API Destinations integration with Mailchimp. 

I have a small e-commerce platform that sells online courses. Whenever a customer buys any of the products I want to mark that information in my *Email Service Provider* (ESP). My ESP of choice is Mailchimp and I want to give a specific **tag** associated with the product to my customer, so I will not send him promotional emails about products that he already has. Unfortunately, my e-commerce platform is unable to tag users in Mailchimp, so I had to implement that myself.

Of course, I utilized serverless and as always it was fun & a lot of learning 😃

### High-level architecture
Take a look at high-level architecture of the solution.

![High-level architecture](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9tnzmtazqalytfli7y77.png)

Getting Orders from e-commerce platform is beyond the scope of this article. We're going to focus on EventBridge and integration with Mailichimp by using API Destinations. The assumption is that you know how to send a new event to a Custom Event Bus.

## What is API Destinations?
API Destinations is relatively new functionality of Amazon EventBridge which allows us to integrate with 3rd party REST endpoints on the Internet directly without any Lambda functions!

All you need to do is to configure the service instead of writing your own code. As you're going to see, there is plenty of CloudFormation. However, you will get much better result compared to your own implementation. 

This update follows the essential AWS principle stated by Chriss Munns during re:Invent 2019 **use functions to transform, not transport**.
![Chriss Munns during re:Invent 2019](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/83qtcftz1jmjzw99u71c.jpg)
And that explains the whole idea behind it. Let's leave the mundane task of calling 3rd party endpoint (data transport) to AWS. We trade code (custom implementation) for configuration of API destinations. In return, we get a reliable solution that:  
* always works
* is someone's else problem / responsibility
* is being developed by best engineers there are. 😃

## Implementation
In real life there is no place for clicking and defining infrastructure manually. Professional projects require automation, repeatability, reliability, and reusability.

Therefore, in this article instead of AWS Console screenshots I will present code excerpts following *Infrastructure as Code* principle.

If you're impatient, you can check this sample [project on GitHub](https://github.com/serverlesspolska/eventbridge-api-destinations-mailchimp), that contains all the sources.

### Solution components

#### Event Bus
To configure API Destinations we need our *custom Event Bus*, which is a backbone of our solution. It is simple to define, using CloudFormation.
```YAML
    EventBus:
      Type: AWS::Events::EventBus
      Properties:
        Name: ${self:service}-${self:provider.stage}
```
Both variables `${self:service}` and `${self:provider.stage}` come from **Serverless Framework** (SF). Let me explain, for those of you who are not fluent with SF. The first one resolves to the name of the project, and the second one to  the name of a stage to which the service is deployed. That way there is **no name clashing** when different stages of the same project are deployed to the same region (or at all, as some resources are global i.e. S3 buckets, IAM Roles). Personally, I use `dev`, `test` and `prod` stage names in my projects.

#### Connection
Next we need `AWS::Events::Connection` object that contains authentication credentials (such as username and password) to the 3rd party endpoint.

```YAML
    MailchimpConnection:
      Type: AWS::Events::Connection
      Properties:
        Name: MailchimpConnection-${self:provider.stage}
        AuthorizationType: BASIC
        AuthParameters:
            BasicAuthParameters:
                Username: "randomUser"
                Password: ${ssm:mailchimp-api-key~true}
```
To login to Mailchimp we will use *BASIC* authorization type.   In case of the `Username` parameter Mailchimp allows any string and is only interested in the `Password` part. From Mailchimp perspective `Password` value is an *Api Key* that can be generated under Account->Extras-> API Keys.

Basic Authorization Type configuration will result in the `authorization` header added to the request created by *API Destinations* when calling Mailchimp REST endpoint. It looks something like that:
```
authorization	Basic cmFuZG9tVXNlcjpZlPkmY2hpbXBIcGlQYXNzd5CyZA==
```
Value of the `Password` parameter is set to `${ssm:mailchimp-api-key~true}` which is again *Serverless Framework* specific. It retrieves secret value from Systems Manager Parameter Store (`ssm` prefix) during deployment. Parameter is of name `mailchimp-api-key` and `~true` means it must be decrypted. More on this in [Serverless Framework documentation](https://www.serverless.com/framework/docs/providers/aws/guide/variables#reference-variables-using-the-ssm-parameter-store).

#### ApiDestination
What could be strange for you is that we didn't define endpoint URL in the connection object. This is done in `AWS::Events::ApiDestination`.
```YAML
    MailchimpApiDestination:
      Type: AWS::Events::ApiDestination
      Properties:
        Name: mailchimp-tag-${self:provider.stage}
        ConnectionArn: !GetAtt MailchimpConnection.Arn
        InvocationEndpoint: ${self:custom.mailchimp.endpoint}/lists/${self:custom.mailchimp.list}/segments/*/members
        HttpMethod: POST
        InvocationRateLimitPerSecond: 20
```
Here we have three interesting elements:
* `InvocationEndpoint`
* `HttpMethod`
* `InvocationRateLimitPerSecond`

The `InvocationEndpoint` contains two custom variables that contain my specific Mailchimp URL and mailing list Id. This particular URL allows adding a user to the `tag` (named as *segment* in Mailchimp). However, what's fascinating from API Destinations perspective is the `*` character in URL. It works as a placeholder which will be replaced by the **HTTP Path Parameter** value, taken from the `Order` event. We will define that parameter later.

Right now I want you to fully grasp the meaning of that. The endpoint URL can be parametrized based on event *payload* sent to the EventBus. This is great, as allows us to reuse single `ApiDestination` configuration to send data to complex endpoints.

The `InvocationRateLimitPerSecond` is another **great treat** of API Destinations. It allows us to **limit rate** of requests fired by API Destinations to the 3rd party. Obviously, it's a crucial thing to consider during integrations, as not all systems are as scalable as AWS serverless solutions. Here we get that *for free* and without any additional coding.

> I hope this incredibly usefull setting will be added to other contstructs in EventBridge family. I could imagine rate limit added to existing Lambda targets or Simple Email Service targets (which dear AWS would be awesome to have 🤞) 😉 

#### Rule

The last EventBridge specific element of the puzzle is `AWS::Events::Rule` that includes so called `Target`. The `Rule` object acts as a router or dispatcher. It tells our *custom EventBus* which events send where. The `Target` defines the *where* part and ties together `Rule` with new API Destinations functionality. 
```YAML
    ApiDestinationDeliveryRule:
      Type: AWS::Events::Rule
      Properties:
        EventBusName: !Ref EventBus
        Name: SendOrderEventsToApiDestinations
        EventPattern:
          detail-type:
            - ORDER_COURSE_SERVERLESS
            - ORDER_COURSE_DATALAKE
        State: "ENABLED"
        Targets:
          - Id: MailchimpApiDestination
            Arn: !GetAtt MailchimpApiDestination.Arn
            RoleArn: !GetAtt ApiDestinationsTargetRole.Arn
            InputTransformer:
              InputPathsMap:
                mcTagId: $.detail.mcTagId
                email: $.detail.email
              InputTemplate: >
                {
                  "email_address": <email>
                }
            HttpParameters:
              PathParameterValues:
                - $.detail.mcTagId
```
In the `EventPattern` section it is defined to which *events* this rule applies to. In my case, it is limited only to *Order events*.

Now let's talk about *targets*. A single `Rule` can define many of them, here we have just one, named `MailchimpApiDestination`. Target refers to defined above `ApiDestination`, it requires a dedicated IAM Role. 

It also defines `InputTransformer`, that allows us to modify event *payload* and tailor it to the 3rd party API specification. In my case, it's Mailchimp's [*Add member to segment*](https://mailchimp.com/developer/marketing/api/list-segment-members/add-member-to-segment/) method, which expect single value under `email_address` variable in JSON.

Event values can be access by `$.detail` and used in the `InputTemplate` section to construct expected object structure. 

The last section allows us to parametrize endpoint URL defined in `ApiDestination` object with values from the *event*. `PathParameterValues` takes an array of values, which means that we can define more than one placeholder in the URL. They are resolved in order, the first element in the array refers to the first `*` in the URL, and so on.

#### IAM Role
We also need to define a `Role` that can be assumed by a `events.amazonaws.com` service. This role must grant `events:InvokeApiDestination` on `MailchimpApiDestination` resource. The full definition of the role can be found in the sample [project on GitHub](https://github.com/serverlesspolska/eventbridge-api-destinations-mailchimp) in `serverless.yml` file.

## Sample solution deployment
I have prepared a sample project using Serverless Framework, so you can deploy it on your AWS account and play with it.

You need to clone the code, and install dependencies:
```
git clone https://github.com/serverlesspolska/eventbridge-api-destinations-mailchimp.git
cd eventbridge-api-destinations-mailchimp
npm i
```
This requires `node.js` and `npm` be installed on your machine.

As not everyone uses Mailchimp, in the project I defined a second target that sends REST requests to `webhook.site` service. This is a free, easy to use web application that will work as our 3rd party endpoint.

Please go to the [webhook.site](https://webhook.site) and copy **Your unique URL**. Paste that URL into `serverless.yml` config file in line `38`.  

Now we're ready to deploy the project to the `dev` stage using command:
```
sls deploy
```
(This assumes that you have your default AWS profile defined under `~/.aws/credentials`.)

After successful deployment (can take several minutes) you can invoke a Lambda function, that will send a sample `Order` event to the *EventBus*. 
```
sls invoke -f sendOrderEvent -l
```
As a result, you should see the new request on the `webhook.site` website. (If don't refresh the page)

That means that API Destinations just called the endpoint responding to the new message on the *EventBus*.

![Request received by WebHook Site](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vmwo6daqzpkfqhik0zdl.png)

## How to debug EventBridge API Destinations?
If you misconfigured something, or perhaps your credentials are wrong **you will not get any error messages, there are no logs** that you can read and try to determine what happened. All you can do is to check 3rd party system and see if the request came to it - that's why we used `webhook.site`. 

### What can we do to find out what's the root cause if the request doesn't hit the endpoint at all?

Fortunately, **API Destinations allows the use of Dead Letter Queues** (DLQ). For each `Target` defined under `AWS::Events::Rule` we can specify SQS queue that will contain messages that couldn't be successfully processed by the `Target` API Destination. Each message that ends up in the DLQ is annotated with information explaining the problem.

Here are two samples of error messages that I managed to get. 
![Wrong Mailchimp API Key](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i32h844ij3078752k9l8.png) *This one was returned when I set wrong Mailchimp API Key.*

![Lack of privileges](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aen3o461779toygq6hql.png) *This one was caused by the wrong IAM Role definition*.

## Summary
I hope you found that article informative, and now you know how to use Amazon EventBridge API Destinations in your projects. Should you have any questions ask them below or on [GitHub project site](https://github.com/serverlesspolska/eventbridge-api-destinations-mailchimp).

I'm convinced that this is a great functionality, even if it took me several hours to understand and implement it correctly for the first time. In the long run, it will save me much more on implementing and debugging Lambda functions that call 3rd party endpoints. They seem to be trivial to implement, but in reality they are not (*How would you implement `InvocationRateLimitPerSecond` functionality in Lambda in a way that you don't pay for idle time?*) and I will not miss them. Remember **use functions to transform, not transport**. 

What is really cool and worth stressing out is that this functionality is fully developed and comes with all the parts. Most importantly there is full CloudFormation support which I must sadly admit was not always the case. Kudos to the CloudFormation team! 🙏

Amazon EventBridge is a central point of many serverless solutions, and I'm glad it is still being developed. API Destinations is a great addition to that service and hopefully indicator of many more to come.