Orchestrate AWS Lambdas using MongoDB - Part 1
Tags: orchestration, lambda, statemachine, mongodb
Published: 2023-11-15 01:01:30

> This post assumes you are a developer with working knowledge on AWS, Lambda, EventBridge, MongoDB, NodeJs.

**Usecase**: Let's say you have several lambdas that you need to run some in parallel, in sequential and some in conditional, and their collective goal is to finish a task. To orchestrate these jobs we need a state-machine(https://statecharts.dev/what-is-a-state-machine.html), which ensures the jobs are run based on the given conditions and will terminated itself at the end.

**Solution**: For the similar problem, AWS Lambda provides a solution called step-functions where you could chain these lambdas, I felt its too difficult to establish and manage them, so I created a custom solution using MongoDB Triggers and AWS EventBridge. Now lets work on the process.

**Components**
_1. Statemachine Configuration: A configuration which consists of all the phases and jobs to run in order. This will be saved to a collection `state-machines`.

_2. Status Event_: Whenever a job status changes, i.e, `success`, `failed`, `started` we insert an event to database which will be consumed by other necessary jobs. This events will be saved `statuses` collections. This is the collection where MongoDB trigger is configured.

_3. Statemachine Job_: An additional job apart from the business jobs to update the state machine whenever a job completes or fails. This job only receives `success` events.

_4. Orchestrator Job_: An additional job apart from the business jobs to trigger the next jobs. This job only receives `success` events.

_5. Notification Job_: You could have another lambda to send messages to slack or any other messaging platform. This job only receives all events.

**How it works:**
1. A state machine will be created by either an API or any such equivalent method based on the requirements. And sends an `status` event as the start of the process. Now this event is received by `statemachine_job`, `orchestrator_job` and `notifications_job`

2. `statemachine_job` will update the corresponding statemachine with the finished job. Check if it has to either end the state machine or move onto next phase.

3. `orchestrator_job` will find the next job to run based on the given configuration, until nothing left to run.

4. `notifications_job` will send the current event message to configured messaging medium.

5. This cycle continues until all the phases finished & satisfied with finished jobs.

**Set Up**: 
As mentioned we need to two collections `state-machines`, `statuses`. The model will be explained later. Create a trigger on `statuses` collection, for `insert` event such that whenever an item inserted into `statuses` it will be sent to the AWS Eventbridge. Set up the database trigger for a collection `statuses`. Follow this link to setup events for AWS Eventbridge https://www.mongodb.com/docs/atlas/triggers/.
The above created event bridge will invoke `statemachine_job`, `orchestrator_job` and `notifications_job`

Lets work on the technical implementation on the [next part](https://dev.to/sateeshm/orchestrate-aws-lambdas-using-mongodb-part-2-43h6).