Enhancing Workflow Efficiency with Direct HTTP API Calls in Step Functions
Tags: aws, serverless, stepfunctions, architecture
Published: 2024-01-29 14:14:00

[Background](#background)
[Logic](#logic)
[Calling API Authentication](#calling-api-authentication)
[System Overview](#system-overview)
[Solution](#solution)
[Retry mechanism](#retry-mechanism)
[Limitation](#limitation)
[Outcome](#outcome)



<br/>&nbsp;<br/>&nbsp;

## Background

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m6aw62b7inezkyirha46.png)

Within the FSI (Financial Service Industry) company, there are 2000 traders who rely on real-time interest rate updates from an external third-party API endpoint to make informed decisions. It is crucial to closely monitor for errors and ensure the ability to retry.



<br/>&nbsp;<br/>&nbsp;

## Logic


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8is5xn6dl6zbx3dkf91l.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/78ckxkxyimprwzgwsz9t.PNG)

Whenever the interest rate exceeds the threshold of 5.1, the traders' decision pool is promptly updated, enabling them to make informed choices. Otherwise, no action is taken. As the interest rate value constantly changes, it is essential to have a simple and efficient way to update it.



<br/>&nbsp;<br/>&nbsp;

## Calling API Authentication

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m9rzmplqpvihzs2z2v2m.png)

To authenticate the API calls, we set up Amazon EventBridge and create authentication credentials. The API key is securely stored in AWS Secrets Manager.



<br/>&nbsp;<br/>&nbsp;

## System Overview


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nr830d6j3y1ua2rxnuof.png)


<u>Step Functions</u>
Orchestrate multi-step applications and microservices using visual workflows.
uses "state machines" to represent applications as a series of steps.

<u>Step Functions -> Choice state</u>
Allows a workflow to branch based on conditions or dynamic data. The workflow will continue down one of multiple paths based on the choice rules.

<u>Step Functions -> Choice state -> Rules</u>
Choice rules define the conditions that determine which path the workflow will take after a choice state. Rules can reference the output of previous steps, static conditions, or dynamic data to evaluate which path to follow.

<u>Step Functions -> Error handling</u>
Step Functions has features for retrying steps that fail, catching errors, and continuing or stopping the workflow based on error conditions.

<u>EventBridge</u>
Serverless event bus that connects applications together using data from applications and AWS services.

<u>EventBridge -> Scheduler</u>
Schedule regular events for EventBridge.

<u>EventBridge -> API destinations</u>
Send events to API endpoints.

<u>Secrets Manager</u>
Store and retrieve secrets securely without plaintext secrets.

<u>Simple Notification Service (SNS)</u>
Fully managed messaging service for application and person communication.

<u>Lambda</u>
Serverless compute service to run code without provisioning servers.

<u>DynamoDB</u>
Fully managed NoSQL database with single-digit millisecond performance.



<br/>&nbsp;<br/>&nbsp;

## Solution

<u>Third-Party API Endpoint Retry:</u>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uld6gsn7x8225zevv5h4.png)



- Within the Step Function, we make HTTP calls to the endpoint and implement a retry mechanism. The system retries the call five times, with a 20-second interval between each attempt. In case of errors, a message is sent to an analysis error monitor dashboard, ensuring critical monitoring.

<u>Lambda Concurrency Scaling Rate:</u>

- By scaling the lambda function's concurrency, we can achieve a rate of 1000 execution environment instances every 10 seconds. 
- This ensures that all 2000 traders' decision pools are updated within a 20-second timeframe, meeting critical requirements.

<u>Real-Time Update of Traders' Decision Pool:</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lsmq9ya67ob8uqji8u1l.png)

Leveraging Lambda functions, we implement a system with a currency of 2000 lambda functions.
This allows for real-time updates to be made across all traders' decision pools simultaneously, ensuring timely and accurate information.

<u>Parameter Store Integration in Step Function:</u>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3lpr3voku10gv3h772z4.png)

- To facilitate the ease of updating the threshold value (currently set at 5.1), we utilize choice rules within the Step Function. This enables us to store and modify the threshold value conveniently, ensuring flexibility and adaptability.


<br/>&nbsp;<br/>&nbsp;

## Retry mechanism


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sosiso1dpqja7bkrwg2m.png)



If there is an error in the HTTP call, the system will automatically retry.

- Step Functions offer you the ability to handle errors by retrying, ensuring that your workflow stays on track and doesn't get derailed.
- It allows you to easily restart workflows from the point of failure, skipping the successful steps that have already been completed.
- This simplifies the recovery process by restarting only the necessary workflows.

<u>Benefit</u>
It helps you build resilient workflows, ensuring that your system can handle errors and recover smoothly.
You can configure an initial set of retries for errors that can be automatically resolved, all without any unnecessary delays.


<br/>&nbsp;<br/>&nbsp;

## Limitation

Your workflow execution should have fewer than 24,999 events.
If you exceed this limit, the event history of redriven executions will be appended to the existing event history.


<br/>&nbsp;<br/>&nbsp;

## Outcome

- The implementation of a visualized dashboard enhances the workflow, making it easier, faster, and more user-friendly. 
- By directly calling the HTTP API, we significantly save development time and gain flexibility, enabling us to set up an efficient orchestration flow for the traders' decision pool.