AWS DynamoDB: Comparing Lambda & Step Functions To Scan 1MB Data
Tags: lambda, stepfunctions, serverless, costoptimisation
Published: 2023-12-10 10:46:50

while working In a project i was exploring the **Lambda, DynamoDB, and step function**, while my aim was to fetch data from DDB in both the Lambda and in Step Function, which led me to an idea of comparing both the method with time, memory usage, and cost for each method. to this idea i added another layer of scanning 1MB of data from DynamoDB. now without wasting time let's explore the setup, execution and cost which should be considered.

## Setup

To test both the scenario, let's setup required Lambda & step function, for both method will be sharing code to scan DB records, all the creation of Lambda & step function definition will be done in the aws console.

**stepFunction Definition:**

```
{
  "Comment": "Scan all the DB records",
  "StartAt": "Scan User Data",
  "States": {
    "Scan User Data": {
      "Type": "Task",
      "Parameters": {
        "TableName": "userInfo",
        "Limit": 50
      },
      "Resource": "arn:aws:states:::aws-sdk:dynamodb:scan",
      "End": true
    }
  }
}
```
**Lambda Logic:** Below code aims to scan the data from DDB, in the starting of the process i noted the time and and the completion of the task i again noted the time by the end i can find the difference from start to end, to find the total time taken to complete the task.

```
const { DynamoDBClient, ScanCommand } = require("@aws-sdk/client-dynamodb");

const dynamoDB = new DynamoDBClient({ region: 'us-east-1' });

const scanWholeTable = async (startKey) => {
    let items = [];
    const startTime = new Date().getTime();
    try {
      const params = {
        TableName: 'userInfo',
        ExclusiveStartKey: startKey,
        Limit: 25,
      };
    
      // prepare the params to scan the table
      const command = new ScanCommand(params);
      const data = await dynamoDB.send(command);
   
      items = items.concat(data.Items);
    
      // If there are more items to scan, recursively call the scanDynamoDBTable function with the last evaluated key
      if (data.LastEvaluatedKey) {
        return scanWholeTable(data.LastEvaluatedKey);
      }

      // check the time taken to complete the scan
      const endTime = new Date().getTime();
      const totalTime = endTime - startTime
      console.log('[INFO] time taken to complete the db scan', totalTime)
      return items;
    } catch (error) {
      console.log('[ERROR] Error while scanning the table', error);
      throw error;
    }
}

module.exports.handler = async (event, context) => {
  try {
      const finalUserInfo = await scanWholeTable(null);
      console.log('[INFO] UserInfo', finalUserInfo);
      return {
        statusCode: 200,
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({userData:finalUserInfo, length: finalUserInfo.length}),
      }
    } catch (error) {
      console.info('Error in the handler:', error);
      return error;
    }
};

```

## Time to complete the task.

Once all the setup is done, I triggered both the scenario multiple time to determine amount of time both method takes to scan 1MB of data from DDB.

**Lambda:**
- I aimed trigger the lambda multiple times.
- As I triggered multiple time, Lambda execution time ranged between **500-1100 milliseconds** for scanning 1MB of data. 
- To complete the task Lambda execution took **1-2 seconds**
- And for The first trigger Lambda took **1253 milliseconds**.

![First Trigger](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ieeixz6gn3h14svpd0sv.png)

- As triggering multiple time lambda showed reduced time of 544 milliseconds.

![Optimised Response](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vmfd9yif6ld9sbohz9ry.png)

**Step Function:** 
- Continuously triggering the step function surprisingly took only **266 milliseconds** to complete the task.
- Task initialisation and completion took **400-500 milliseconds**.
- Apart from some lag, both Lambda and Step Function times are worth considering.

![StepFunction time](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y5ikvdv8f3g6gnxwv8pk.png)

## Cost Consideration.

Let's use the [AWS Price Calculator](https://calculator.aws/#/) to estimate the cost of scanning 1MB on DynamoDB.

**Lambda Cost:** 
Let's configure the region (us-east-1), architecture (x84), memory (512), timeout (30), and trigger per day (let's keep it at 50).once all the configuration is set let's break down the cost.

- For 50 requests per day, which total will be 1,520 per month, and memory usage of **608.33** GB for computing.
- AWS provides **400,000** GB as a free tier for compute, resulting in a compute cost of **0 USD**.
- Currently AWS Lambda makes 1 million triggers as part of the free tier, which result in our process to cost **0 USD**.
- Running the Lambda will cost 0 USD until it reaches **400k** compute and **1 million** requests.

![lambda cost](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v9rsvtfostao0ntpb7m7.png)

**Step Function Cost:**
Now configure the region (us-east-1), select standard workflow, and set the workflow requests to 50 per day, 5 state transition of the step function, let's consider there will be 5 states after it to transform data and send back the result.
 
- With 1,520 requests triggered to the Step Function per month.
- AWS provides **4,000** transitions as a free tier for the step function service, and for our step function it will be total of 7,604 transitions per month.
- As a result above point the billable transitions amount will be **3,604** transitions after 7,604 - 4,000.
- The cost for one transition state is **0.000025 USD**. 
- leading to an estimated cost of **0.09 USD**.


![step function cost](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/31ojknzwejuyh8uc2jcd.png)

## Summary


| Scenario | Step Function | Lambda |
|----------|----------|----------|
| Data Size(MB) | 1MB | 1MB |
| Time in Milliseconds | 266 | 500 - 1100 |
| Compute(GB) | 686 | not-specified |
| Cost(USD) | 0.09 | 0 |
| Disadvantage | As the Step Function send input to next state as marshalled data from DDB, and un-marshalling correctly mapping the "Null" type is problematic from the Payload of the step function in next states. | N/A |

As i delve into comparing the Lambda & Step Function to scan 1MB of data from DynamoDb, yield insightful observations, on the time & cost for both methods, where **Lambda** free-tier benefits overshadow the Step Function consistent performance.