Creating an AWS Lambda function to process records in Amazon DynamoDB stream
Tags: lambda, dynamodbstream, dynamodb, aws
Published: 2022-11-29 04:37:28

In this article, I will show you how to create an AWS Lambda function which will process records in an Amazon DynamoDB Streams, **as you do** any data modification to the DynamoDB table - add, delete or update an item.

# What is Lambda?
**AWS Lambda** is a serverless, event-driven compute service that lets you run code for virtually any type of application or back-end service without provisioning or managing servers. You can trigger Lambda from over 200 AWS services and only pay for what you use.

# Lambda Concepts used in this article

**Function** 
You organize your code into Lambda functions. A function is a resource that you can invoke to run your code to process events in Lambda. You can invoke your function using the Lambda API, or you can configure an AWS service or resource like Amazon DynamoDB to invoke your function. Lambda runs your function only when needed and scales automatically, from a few requests per day to thousands per second. You pay only for the compute time that you consume, and there is no charge when your code is not running.


**Trigger**
A trigger is a resource or configuration that invokes a Lambda function. Triggers include AWS services that you can configure to invoke a function and event source mappings. 

**Lambda event source mappings**
An event source mapping is a resource in Lambda that reads items from a stream (DynamoDB) / queue (Amazon MQ or Amazon Simple Queue Service (Amazon SQS)) and invokes a function. 

An event source mapping uses permissions in the function's execution role to read and manage items in the event source. Permissions, event structure, settings, and polling behavior vary by event source. 

**Deployment packages and .zip file archives**
When you create a Lambda function, you package your function code into a deployment package. 

Lambda supports two types of deployment packages: container images and .zip file archives. 

The workflow to create a function depends on the deployment package type. A .zip file archive includes your application code and its dependencies. 

When you create a function defined with a .zip file archive, you choose a code template, the language version, and the execution role for the function. You add your function code after Lambda creates the function.

You can use the Lambda console and the Lambda API to create a function defined with a .zip file archive. You can also upload an updated .zip file to change the function code. 

# What is DynamoDB stream?
**A DynamoDB stream** is a time-ordered flow of information about changes to items in a DynamoDB table. When you enable a stream on a table, DynamoDB captures information about every modification to data items in the table. Whenever an application creates, updates, or deletes items in the table, DynamoDB Streams writes a stream record with the primary key attributes of the items that were modified. A stream record contains information about a data modification to a single item in a DynamoDB table. You can configure the stream so that the stream records capture additional information, such as the "before" and "after" images of modified items in near-real time.

- Each stream record appears exactly once in the stream.
- For each item that is modified in a DynamoDB table, the stream records appear in the same sequence as the actual modifications to the item.

Please visit my [GitHub Repository for DynamoDB articles](https://github.com/awsmine/DynamoDB_projects) on various topics being updated on constant basis.

_**Let’s get started!**_

# Objectives:

- Create a Lambda execution role
- Create the function
- Create a DynamoDB table with a stream enabled
- Add an event source in AWS Lambda
- Test the setup - Verify the logs in Amazon CloudWatch
- Cleanup


# Pre-requisites:

- AWS user account with admin access, not a root account.
- Cloud9 IDE with AWS CLI.



## Create a Lambda execution role

This role grants the function permission to access AWS services and resources. For example, you might create an execution role that has permission to send logs to Amazon CloudWatch.

When you invoke your function, Lambda automatically provides your function with temporary credentials by assuming this role. You don't have to call sts:AssumeRole in your function code.

_**From the Identity and Access Management (IAM) Console**_ , Create a role using these parameters: 
- **Trusted entity type** - Check AWS Service
- **Use Case** - Select Lambda
- **Permissions policies** - Search and select AWS Managed **AWSLambdaDynamoDBExecutionRole**
- **Under Role details** - **Role name** - lambda-dynamodb-role

**AWSLambdaDynamoDBExecutionRole** has the following permissions to list and read access to DynamoDB streams and write permissions to CloudWatch logs.

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "dynamodb:DescribeStream",
                "dynamodb:GetRecords",
                "dynamodb:GetShardIterator",
                "dynamodb:ListStreams",
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j15cdbzx878zyrx0ll7x.png)





![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jgyvxweiy9vi8z9k7l1u.png)





![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ciibgaci0y91qzw0qicq.png)






![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ivshkada5oynzjxf1iq7.png)







## Create the function
To create the function


- Copy the sample code into a file named "index.js"

```
console.log('Loading function');

exports.handler = function(event, context, callback) {
    console.log(JSON.stringify(event, null, 2));
    event.Records.forEach(function(record) {
        console.log(record.eventID);
        console.log(record.eventName);
        console.log('DynamoDB Record: %j', record.dynamodb);
    });
    callback(null, "message");
};

```
- Zip up the sample code (index.js) to create a deployment package.

```
zip function.zip index.js
```

- Create a Lambda function with the create-function command


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jth911xoeb4xx6lq6vlh.png)






![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ixw2w3ea6lonmdg2yl38.png)






## Create a DynamoDB table with a stream enabled

- **To create a DynamoDB table**

_**From the Amazon DynamoDB Console**_

**Create table** using these parameters
- **Table name** – lambda-dynamodb-stream
- **Primary key** – id (string)
- **Create table**
    


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cd65iyq4f2697q5rs1kr.png)



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lzzx56a2yl3h6j7ys7f2.png)



- **Create table**


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6becq4gmg3eh3iv94qw8.png)





- **To enable streams**

_**Open the Amazon DynamoDB Console**_

- Choose **Tables**

- Choose the **lambda-dynamodb-stream table**

- Under **Exports and streams**, choose **DynamoDB stream details**

- Choose **Enable**

- Choose **Enable stream**

Write down the stream ARN. You need this in the next step when you associate the stream with your Lambda function.



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/188wk0fxkmskpfoydf9i.png)




![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vb0k153x8g6y4x2wrgmm.png)




![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d67evzlkl37e152bqxi4.png)



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yzksi2ov98zyostzm949.png)




![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ajvctirrm0jxxl5xaijc.png)




![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ylmfx6tgqkap3uim1k24.png)








## Add an event source in AWS Lambda

This event source mapping associates the DynamoDB stream with your Lambda function. After you create this event source mapping, AWS Lambda starts polling the stream.

Use the AWS CLI to map a function named **ProcessDynamoDBRecords** to a DynamoDB stream that its Amazon Resource Name (ARN) specifies, with a batch size of 100. After the command runs, note down the UUID. 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x8vzwjc3dvcqi6tgwjad.png)




![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6kqrrafnyws023ex44v2.png)



**No records processed** (indicates that AWS Lambda has not started polling or that there are no records in the stream) 



This creates a mapping between the specified DynamoDB stream and the Lambda function.


## Test the setup - Verify the logs in Amazon CloudWatch

As and when you perform table updates, DynamoDB writes event records to the stream. As AWS Lambda polls the stream, it detects new records in the stream and invokes your Lambda function on your behalf by passing events to the function.

- In the DynamoDB console, add, update, and delete items to the table. DynamoDB writes records of these actions to the stream.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j6lw2clsojxuczes7bh4.png)

-- Delete ids 2000 and 2006


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y4z9v1tm41ryfdg1on2n.png)



- AWS Lambda polls the stream and when it detects updates to the stream, it invokes your Lambda function by passing in the event data it finds in the stream.

- Your function runs and creates logs in Amazon CloudWatch. You can verify the logs reported in the Amazon CloudWatch console.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c8hql4jxpjgr667j51g4.png)


- **deleted id 2000**


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/stanm2wvobu6b8hdkouo.png)

- **deleted id 2006**


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g5no7xcx4fc6z3ucjxd3.png)








## Cleanup

- delete the Lambda function
- delete the execution role
- delete the DynamoDB table


# What we have done so far

Successfully created an AWS Lambda function to process records in an Amazon DynamoDB Streams, **as you do** add, delete or update an item in the DynamoDB table.