Manage webhooks with MongoDB Functions and Triggers
Tags: atlashackathon, showdev, mongodb
Published: 2022-01-13 22:46:31

In this post I explain how I built a webhook management system with MongoDB triggers and functions. This is part of mongodb atlas hackathon here in dev.to

### Overview of My Submission
Webhooks are still a preferred choice of many organizations to communicate with 3rd party services. Since webhook calls are event-driven, if we use MongoDB Atlas as our database solution, we can use the features like [triggers](https://docs.mongodb.com/realm/triggers/trigger-types/) and [functions](https://docs.mongodb.com/realm/functions/) to build a low code webhook management system. In this project, I showcase how to build such system with MongoDB Realm.

### Submission Category: 

Action Star

### Link to Code

You can find the source code of the application at [https://github.com/pubudusj/webhook-management-with-mongodb/](https://github.com/pubudusj/webhook-management-with-mongodb/)

Setup instructions are available further in this post.

### Additional Resources / Info

#### MongoDB features/tools used
1. [MongoDB Atlas](https://www.mongodb.com/cloud/atlas/lp/try2)
2. [Realm Triggers](https://docs.mongodb.com/realm/triggers/trigger-types/)
3. [Realm Functions](https://docs.mongodb.com/realm/functions/)
4. [Realm External Dependancies](https://docs.mongodb.com/realm/functions/add-external-dependencies)
5. [Realm CLI](https://docs.mongodb.com/realm/cli/)
6. [Realm static hosting](https://docs.mongodb.com/realm/hosting/)
7. [MongoDB NodeJS SDK](https://docs.mongodb.com/realm/sdk/node/)


#### Demo

You can see a fully functional demo at
[https://webhookmanagement-gfbga.mongodbstitch.com/](https://webhookmanagement-gfbga.mongodbstitch.com/)

#### Architecture

![System architecture](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a7t4neex5pjj11czk51y.png)

#### How it works

There are 3 parts to this implementation.

1. Webhook registration.
2. Webhook call for create candidate/user event.
3. View webhook call history.

**Webhook Registration**

1. There is an API exposed in the AWS API gateway which proxy to a Lambda function (_RegisterWebhookFunction_).
2. In this function, MongoDB NodeJS SDK is included, where it generate a webhook signing token and save the companyId, webhookUrl, event type as ‘candidate.created’ and the signing token to MongoDB collection - webhooks.
3. Generated webhook signing token will be returned from the API end point. 
4. This token can be used to authenticate the incoming webhook.

**Create user**
1. There is an API exposed in the AWS API gateway which proxy to a Lambda function (_CreateUserFunction_).
2. Again, in this function, MongoDB NodeJS SDK is included, where it save user data including the companyId to a MongoDB collection - users.
3. In MongoDB Realm, I have created a application - WebhookManagement.
4. There, I have created a trigger (_CandidateCreated_), which will trigger when a new user is added to the users collection.
5. Also, there is a function created as _CandidateCreated_ attached to this trigger, so this function will be executed upon the trigger.
6. Within this _CandidateCreated_ function, there are few steps involved to call the webhook.
7. First it calls another function, _GetWebhook_ to check if there are webhooks registered for this companyId and event type ‘candidate.created’.
8. If so, _GetUserObject_ function will be called to fetch the respective user data from users collection.
9. With those information, there is another function - _CreateWebhookPayload_ - called to create the payload for the webhook.
10. In this _CreateWebhookPayload_ function, it generates a token to be added to the payload using signing token. To generate the token, I use SHA256 algorithm as:
`
hmacSHA256(userId + currentTime, signingToken)
`
11. This token will be used to verify the authenticity of the request coming into the webhook. To generate this token, I use JS library 'crypto-js' which can be added easily into the realm app as an external dependency.
12. Once the payload is created, external url will be called including this payload using JS library 'axios'. Here also, I use external dependancy feature of realm app to load 'axios' library into the function.
13. Once the external api is called, the result will be saved into the webhook_calls collection using the function - _CreateWebhookCall_. Here, the url, payload, timestamp, status and response from the webhook call are saved into the webhook_calls collection.

**View webhook call history**
1. To fetch webhook call history, an API exposed in the AWS API gateway which proxy to a Lambda function (_FetchWebhookHistoryFunction_).
2. Given the companyId, using MongoDB NodeJS SDK, it will query the webhook_call collection to fetch the data.

**How to Set up**
1. First, create a new MongoDB Cluster with the name "Testing" at MongoDB Atlas.
2. Create a new user with username+password authentication at the Database Access section with permission read and write to any database.
3. In the Realm section, create a new application. Use name as "WebhookManagement" and choose the option: 'Use and existing MongoDB Atlas Data Source' and choose the cluster created in step 1 - Testing.
4. Under the 'Authentication' section, enable API keys and create a API key. Generated public key and private keys is required for realm cli authentication later on.
4. Install realm cli using `npm install -g mongodb-realm-cli` in your local machine.
5. Clone this [github repository](https://github.com/pubudusj/webhook-management-with-mongodb) to your local machine.
6. This repo contains 3 main directories.
7. Goto the the realm directory, open the Makefile and set the values for parameters: REALM_APP_ID, APP_NAME, REALM_PUBLIC_KEY, REALM_PRIVATE_KEY.
8. Within the realm directory, run `make realm_login`, `make realm_pull_app`, `make setup_functions`, `make setup_triggers`, `make deploy_all` in the same order. This will create all the functions, triggers and dependancies in the Realm app.
9. `backend` directory contains a AWS SAM application to set up the required Lambda functions and API gateway apis which are consumed by the frontend app. Once the backend deployed, the API gateway url is required to set up frontend app. To deploy the backend, you need to have the connection string for the MongoDB database. You can get it from 'Connect' button on the dashboard of your DB cluster.
10. Instead of this AWS backend, you can use Realm endpoints as well. [https://docs.mongodb.com/realm/endpoints/](https://docs.mongodb.com/realm/endpoints/)
11. `frontend` directory contains a VueJS app. Copy the .env.example file into .env and changed the VUE_APP_API_BASE_URL parameter with the one received in the backend deployment.
12. Then, use `npm run serve` for local testing or for production build, use `npm run build --production`.
13. Goto the hosting section of the Realm App and enable hosting. This will create a sample app with a url. ex: webhookmanagement-gfbga.mongodbstitch.com
14. Upload the frontend build files generated in step 12.
15. Confirm the changes and visit the url. You will see the frontend app.

**Sample screenshot**

![Success Webhooks](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jkz36ywl8l2vfin196b6.png)

![Failed Webhooks](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/osuhp318bdsedmyiqdde.png)

#### Conclusion

Overall, Atlas and Realm provides great experience in terms of developer experience and performance. There are few thing I would like to see in future.

1. Retry feature: Currently, if Realm function fails, we cannot automatically retry.
2. Individual function calls in log: Currently if one function is being called within another function, it is not recorded in the logs. Only the calling function has a entry. Better to make all the function executions available in the logs.