Use Golang for data processing with Amazon SNS and AWS Lambda
Tags: serverless, go, tutorial, programming
Published: 2023-05-01 08:24:31

In this blog post, you will be using the [aws-lambda-go](https://github.com/aws/aws-lambda-go) library along with the [AWS Go SDK v2](https://github.com/aws/aws-sdk-go-v2) for an application that will process records from an [Amazon SNS](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) topic and store them in a [DynamoDB](https://docs.aws.amazon.com/dynamodb/index.html) table. 

You will also learn how to use [Go bindings for AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/work-with-cdk-go.html) to implement "Infrastructure-as-code" for the entire solution and deploy it with the AWS CDK CLI.

> The code is available on [GitHub](https://github.com/abhirockzz/sns-lambda-events-golang)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3ni7zgdlv7xu9xnrio6n.png)

**What's covered?**

- [Introduction](#introduction)
- [Pre-requisites](#pre-requisites)
- [Use CDK to deploy the solution](#use-cdk-to-deploy-the-solution)
- [Verify the solution](#verify-the-solution)
- [Don't forget to clean up](#dont-forget-to-clean-up)
- [Code walk through](#code-walk-through)
- [Wrap up](#wrap-up)

## Introduction

Amazon Simple Notification Service (SNS) is a highly available, durable, and scalable messaging service that enables the exchange of messages between applications or microservices. It uses a publish/subscribe model where publishers send messages to topics, and subscribers receive messages from topics they are interested in. Clients can subscribe to the SNS topic and receive published messages using a supported endpoint type, such as Amazon Kinesis Data Firehose, Amazon SQS, AWS Lambda, HTTP, email, mobile push notifications, and mobile text messages (SMS).

[AWS Lambda and Amazon SNS integration](https://docs.aws.amazon.com/lambda/latest/dg/with-sns.html) enable developers to build event-driven architectures that can scale automatically and respond to changes in real-time. When a new message is published to an SNS topic, it can trigger a Lambda function (Amazon SNS invokes your function asynchronously with an event that contains a message and metadata) which can perform a set of actions, such as processing the message, storing data in a database, sending emails or SMS messages, or invoking other AWS services. 

## Pre-requisites

Before you proceed, make sure you have the [Go programming language](https://go.dev/dl/) (**v1.18** or higher) and [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install) installed.

Clone the project and change to the right directory:

```bash
git clone https://github.com/abhirockzz/sns-lambda-events-golang

cd sns-lambda-events-golang
```

## Use CDK to deploy the solution

To start the deployment, simply invoke `cdk deploy` and wait for a bit. You will see a list of resources that will be created and will need to provide your confirmation to proceed.

```bash
cd cdk

cdk deploy

# output

Bundling asset SNSLambdaGolangStack/sns-function/Code/Stage...

✨  Synthesis time: 5.94s

This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening).
Please confirm you intend to make the following modifications:

//.... omitted

Do you wish to deploy these changes (y/n)? y
```

This will start creating the AWS resources required for our application.

> If you want to see the AWS CloudFormation template which will be used behind the scenes, run `cdk synth` and check the `cdk.out` folder

You can keep track of the progress in the terminal or navigate to AWS console: `CloudFormation > Stacks > SNSLambdaGolangStack`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tc2rggzcbi0ckf5fiwka.png)

Once all the resources are created, you can try out the application. You should have:

- A Lambda function
- A SNS topic
- A `DynamoDB` table
- along with a few other components (like IAM roles etc.)

## Verify the solution

You can check the table and SNS info in the stack output (in the terminal or the **Outputs** tab in the AWS CloudFormation console for your Stack):

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hg1ljq74ro86jc9yaitf.png)

Send few messages to the SNS topic. For the purposes of this demo, you can use the AWS CLI:

```bash
export SNS_TOPIC_ARN=<enter the queue url from cloudformation output>

aws sns publish --topic-arn $SNS_TOPIC_ARN --message "user1@foo.com" --message-attributes 'name={DataType=String, StringValue="user1"}, city={DataType=String,StringValue="seattle"}'

aws sns publish --topic-arn $SNS_TOPIC_ARN --message "user2@foo.com" --message-attributes 'name={DataType=String, StringValue="user2"}, city={DataType=String,StringValue="new delhi"}'

aws sns publish --topic-arn $SNS_TOPIC_ARN --message "user3@foo.com" --message-attributes 'name={DataType=String, StringValue="user3"}, city={DataType=String,StringValue="new york"}'
```

> You can also use the [AWS console](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-using-send-messages.html) to send SQS messages


Check the DynamoDB table to confirm that the file metadata has been stored. You can use the AWS console or the AWS CLI `aws dynamodb scan --table-name <enter the table name from cloudformation output>`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yqg4qffk3mxmre19hxfy.png)

### Don't forget to clean up

Once you're done, to delete all the services, simply use:

```bash
cdk destroy

#output prompt (choose 'y' to continue)

Are you sure you want to delete: SQSLambdaGolangStack (y/n)?
```

You were able to setup and try the complete solution. Before we wrap up, let's quickly walk through some of important parts of the code to get a better understanding of what's going the behind the scenes.

## Code walk through

Some of the code (error handling, logging etc.) has been omitted for brevity since we only want to focus on the important parts.

**CDK**

> You can refer to the [CDK code here](https://github.com/abhirockzz/sns-lambda-events-golang/blob/master/cdk/cdk.go)

We start by creating a `DynamoDB` table:

```go
	table := awsdynamodb.NewTable(stack, jsii.String("dynamodb-table"),
		&awsdynamodb.TableProps{
			PartitionKey: &awsdynamodb.Attribute{
				Name: jsii.String("email"),
				Type: awsdynamodb.AttributeType_STRING},
		})

	table.ApplyRemovalPolicy(awscdk.RemovalPolicy_DESTROY)
```

Then, we handle the Lambda function (*CDK will take care of building and deploying the function*) and make sure we provide it appropriate permissions to write to the `DynamoDB` table.

```go
	function := awscdklambdagoalpha.NewGoFunction(stack, jsii.String("sns-function"),
		&awscdklambdagoalpha.GoFunctionProps{
			Runtime:     awslambda.Runtime_GO_1_X(),
			Environment: &map[string]*string{"TABLE_NAME": table.TableName()},
			Entry:       jsii.String(functionDir),
		})

	table.GrantWriteData(function)
```

Then, we create the SNS topic and add that as an event source to the Lambda function.

```go
	snsTopic := awssns.NewTopic(stack, jsii.String("sns-topic"), nil)
	function.AddEventSource(awslambdaeventsources.NewSnsEventSource(snsTopic, nil))
```

Finally, we export the SNS topic and `DynamoDB` table name as CloudFormation outputs. 

```go
	awscdk.NewCfnOutput(stack, jsii.String("sns-topic-name"),
		&awscdk.CfnOutputProps{
			ExportName: jsii.String("sns-topic-name"),
			Value:      snsTopic.TopicName()})

	awscdk.NewCfnOutput(stack, jsii.String("dynamodb-table-name"),
		&awscdk.CfnOutputProps{
			ExportName: jsii.String("dynamodb-table-name"),
			Value:      table.TableName()})
```

**Lambda function**

> You can refer to the [Lambda Function code here](https://github.com/abhirockzz/sns-lambda-events-golang/blob/master/function/main.go)

The Lambda function handler iterates over each SNS topic, and for each of them:

- Stores the message body in the primary key attribute (`email`) of the `DynamoDB` table
- Rest of the message attributes are stored as is.

```go
func handler(ctx context.Context, snsEvent events.SNSEvent) {
	for _, record := range snsEvent.Records {

		snsRecord := record.SNS

		item := make(map[string]types.AttributeValue)
		item["email"] = &types.AttributeValueMemberS{Value: snsRecord.Message}

		for attrName, attrVal := range snsRecord.MessageAttributes {
			fmt.Println(attrName, "=", attrVal)
			attrValMap := attrVal.(map[string]interface{})

			dataType := attrValMap["Type"]

			val := attrValMap["Value"]

			switch dataType.(string) {
			case "String":
				item[attrName] = &types.AttributeValueMemberS{Value: val.(string)}
			}
		}

		_, err := client.PutItem(context.Background(), &dynamodb.PutItemInput{
			TableName: aws.String(table),
			Item:      item,
		})
	}
}
```

## Wrap up

In this blog, you saw an example of how to use Lambda to process messages sent to SNS and storing them in DynamoDB, thanks to the [SNS and Lamdba integration](https://docs.aws.amazon.com/lambda/latest/dg/with-sns.html). The entire infrastructure life-cycle was automated using AWS CDK.

All this was done using the Go programming language, which is well supported in [DynamoDB](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/dynamodb), [AWS Lambda](https://pkg.go.dev/github.com/aws/aws-lambda-go) and [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/work-with-cdk-go.html).

Happy building!