Storing AWS SQS messages to DynamoDB with AWS Lambda
Tags: dynamodb, lambda, serverless, aws
Published: 2023-06-19 08:25:28

In my last post I covered creating a lambda & API Gateway to accept a POST request. Now lets save that request to a table in DynamoDB and send it to a queue in SQS for processing.

**Creating the Orders Table**

We create a new table by updating the template.yml file. Before defining the new resource, lets define a few common properties that we can leverage for all resources. Add a *Parameters* section to the template under *Description*, with the following values -

![Template parameters section](https://cdn-images-1.medium.com/max/2000/1*H6sjJkW1T7arJbzfAK5Big.png)

*These parameters are used when setting up environment variables for various resources.*

Under Resources, we add a resource for the DynamoDB table -

![Template resources section](https://cdn-images-1.medium.com/max/2000/1*i8Uz4_iIdREu0-M488e8zQ.png)

Rather than defining environment variables for each function, we are going to define them under *Globals* so that they are available to all the lambda functions we create.

![Template globals section](https://cdn-images-1.medium.com/max/2000/1*GjSlkt0uIF7PGr4Sl7w4gA.png)

We also need to update the properties for the Create function so that it has the required permissions to insert records into the dynamodb table. We add an aws defined policy — *AmazonDynamoDBFullAccess*

![Template policies section](https://cdn-images-1.medium.com/max/2000/1*LS-AiZaYrEa8htdimETXqA.png)

**Creating the Orders Queue**

We create an SQS queue for the orders by adding another resource to the template.yaml file -

![Template resource section](https://cdn-images-1.medium.com/max/2000/1*Rky9FoICNYniETBR_XH3OA.png)

We need to tell the lambda function about the queue, so add another environment variable to the Globals: Function: Environment: Variables:

![Template environment variables](https://cdn-images-1.medium.com/max/2000/1*yg1Apshuarcfu-YJmSQCAQ.png)

We also need to give the Create lambda access to add orders to the queue. Add another policy to the Policies list for the function -

![Template policy section](https://cdn-images-1.medium.com/max/2000/1*m3fzANlbsCL-xFZltYkwGw.png)

The template.yaml should look like this -
{% gist https://gist.github.com/karthiks3000/c9e923b0e4f58181b52c8e3766cd4228.js %}
**DB Helper**

Under the src directory, create a new directory called “db” and create the following 4 files under it -

![DB folder structure](https://cdn-images-1.medium.com/max/2000/1*AqSrCxPG1vKa6VOsjiINng.png)

To test database changes locally, we need to spin a local instance of DynamoDB and manually create a table in it.

We can use docker to spin up an instance of DynamoDB. Update docker-compose.yml with the following -

![Docker compose file](https://cdn-images-1.medium.com/max/2000/1*qNSC5xvERhDhf1r-ptPxEA.png)

In the terminal, go to the db folder and spin up a dynamodb instance -

    cd src/db
    docker-compose up -d

The output should look like -

    Creating dynamodb-local ... done

To create the Orders table locally, update the *init_db.py* file with the following code -
{% gist https://gist.github.com/karthiks3000/2a4826918a4ac8c3cb1c34d07854b0e7.js %}
Add boto3 to the requirements.txt file and run pip install again -

![requirements file](https://cdn-images-1.medium.com/max/2000/1*DtNCoJTiJAYhA6MpqhTMZQ.png)

    cd ..
    pip install -r requirements.txt

Run init_db.py to create the table on the local dynamodb instance -

    python db/init_db.py

If the table is created, you should see the following output -

    Table status: ACTIVE

***NOTE**: You can use [NoSQL Workbench](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/workbench.settingup.html) to view the tables in your local dynamodb instance.*

Update db_helper.py with the following code -
{% gist https://gist.github.com/karthiks3000/6adad2662e073c93cc7c8ef982f01157.js %}
**Putting it all together**

Update create.py to send the request to the sqs queue and update the order status in the table -
{% gist https://gist.github.com/karthiks3000/0ec7b4083db93ced87aff3e7a7917518.js %}
For this to work in a container, we need to add another line to the Dockerfile -

![Dockerfile](https://cdn-images-1.medium.com/max/2000/1*ysKXjtXmdMH1jKs0yj4WfQ.png)

**Testing locally**

Build the app locally -

    sam build

Since we have specified environment variables in our template, we need to provide those env vars when running the code locally.

Create a new file under the tests folder called “env.json” with the following values -

![env.json file](https://cdn-images-1.medium.com/max/2000/1*OGlRfk462Fmw-h7Fn8TJLQ.png)

***NOTE**: SQS_QUEUE is empty since we can’t host an sqs queue locally. We will update that value with the deployed queue url once the changes are deployed.*

Run the app locally -

    sam local start-api --env-vars ./tests/env.json

Test the create endpoint from postman like before -

![Postman output](https://cdn-images-1.medium.com/max/4176/1*XEUNZgCjuu0dnZzSdEZu7Q.png)

Since we did not setup the queue locally, an error is expected. We should however, be able to see a new record created in the dynamodb table for the request -

![DynamoDB table](https://cdn-images-1.medium.com/max/3844/1*Ua_akiPZPXvbhUgdLcKj0A.png)

**Deploy the app**

Run the following command to deploy the app to aws

    sam deploy

The output should look like this -

![SAM Deploy output](https://cdn-images-1.medium.com/max/2000/1*DYZwfyLEIVSZlGsiD1rqQQ.png)

Test the changes by making the POST call to create a request from postman like before -

![Postman output](https://cdn-images-1.medium.com/max/4172/1*VD-roYvm89PavTTaHDtYxg.png)

We can verify the request was saved to the table by going to aws console > DynamoDB > Tables > Explore items > serverless-arch-example-orders

![AWS DynamoDB Table](https://cdn-images-1.medium.com/max/2426/1*qVJQAdM9A30GZvQLd7NoWg.png)

The request status is also set as “Queued” indicating that the request was sent to SQS.

We can validate that as well by going to aws console > Amazon SQS and confirming that the value under Messages available is now 1

![AWS SQS Queue](https://cdn-images-1.medium.com/max/5280/1*0bTIbRDQEctQfO4jGZL6xQ.png)

You can view the message that was sent by clicking on the queue name > Send and receive messages > Poll for messages

![SQS Queue](https://cdn-images-1.medium.com/max/5232/1*3hV9QtSnUsM3JfmNWGN0qQ.png)

Click on the message ID to view the message

![SQS Message](https://cdn-images-1.medium.com/max/3372/1*IFKTOu7jl61bvKXRCAOuNQ.png)

**Source Code**

[Here](https://github.com/karthiks3000/serverless-arch-example) is the source code for the project created here.

Next: Part 4: Web Scraping with Selenium & AWS Lambda