Getting started with MSK Serverless and AWS Lambda using Go
Tags: go, kafka, serverless, tutorial
Published: 2023-01-03 10:06:43

In this blog post you will learn how to deploy a [Go Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html) and trigger it in response to events sent to a topic in a [MSK Serverless](https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html) cluster.

  - [Prerequisites](#prerequisites)
  - [Infrastructure setup](#infrastructure-setup)
  - [Send data to MSK Serverless using producer application](#send-data-to-msk-serverless-using-producer-application)
  - [Configure and deploy the Lambda function](#configure-and-deploy-the-lambda-function)
  - [Verify the integration](#verify-the-integration)
  - [Conclusion](#conclusion)

The following topics have been covered:

- How to use the [franz-go Go Kafka client](https://github.com/twmb/franz-go) to connect to MSK Serverless using [IAM authentication](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html).
- Write a Go Lambda function to process data in MSK topic.
- Create the infrastructure - VPC, subnets, MSK cluster, `Cloud9` etc.
- Configure Lambda and `Cloud9` to access MSK using IAM roles and fine-grained permissions.

## Prerequisites

You will need an AWS account, [install AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) as well a [recent version of Go](https://go.dev/doc/install) (`1.18` or above).

Clone [this GitHub repository](https://github.com/abhirockzz/lambda-msk-serverless-trigger-golang) and change to the right directory:

```bash
git clone https://github.com/abhirockzz/lambda-msk-serverless-trigger-golang
cd lambda-msk-serverless-trigger-golang
```

## Infrastructure setup

**Create VPC and other resources**

Use a [CloudFormation template](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/gettingstarted.templatebasics.html) for this.

```bash
aws cloudformation create-stack --stack-name msk-vpc-stack --template-body file://template.yaml
```

Wait for the stack creation to complete before proceeding to other steps.

**Create MSK Serverless cluster**

Use [AWS Console to create the cluster](https://docs.aws.amazon.com/msk/latest/developerguide/create-cluster.html). 

Configure the `VPC` and *private* subnets created in the previous step.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lu89x8ooihb49zqp04a7.png)

**Create AWS Cloud9 instance**

Make sure its in the *same* `VPC` as the MSK Serverless cluster and choose the *public* subnet that you created earlier.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z0ha9ate3tcgt3od9wlm.png)

**Configure MSK cluster security group**

After the `Cloud9` instance is created, edit the MSK cluster security group to allow access from  the `Cloud9` instance.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uh9xu3t0494uqteagctl.png)

**Configure Cloud9 to send data to MSK Serverless cluster**

The code that we run from `Cloud9` is going to *produce* data to the MSK Serverless cluster. So we need to ensure that it has [the right privileges](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html). For this, we need to create an IAM role and attach required permissions policy.

```bash
aws iam create-role --role-name Cloud9MSKRole --assume-role-policy-document file://ec2-trust-policy.json
```

Before creating the policy, update the `msk-producer-policy.json` file to reflect the required details including MSK cluster ARN etc.

```bash
aws iam put-role-policy --role-name Cloud9MSKRole --policy-name MSKProducerPolicy --policy-document file://msk-producer-policy.json
```

Attach the IAM role to the `Cloud9` EC2 instance:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c780pri1l2e3pzvypcre.png)

## Send data to MSK Serverless using producer application

Log into the `Cloud9` instance and run the producer application (its a Docker image) from a terminal.

```bash
export MSK_BROKER=<enter the MSK Serverless endpoint>
export MSK_TOPIC=test-topic

docker run -p 8080:8080 -e MSK_BROKER=$MSK_BROKER -e MSK_TOPIC=$MSK_TOPIC public.ecr.aws/l0r2y6t0/msk-producer-app
```

The application exposes a REST API endpoint using which you can send data to MSK.

```bash
curl -i -X POST -d 'test event 1' http://localhost:8080
```

This will create the specified topic (since it was *missing* to begin with) and also send the data to MSK.

Now that the cluster and producer application are ready, we can move on to the consumer. Instead of creating a traditional consumer, we will deploy a Lambda function that will be automatically invoked in response to data being sent to the topic in MSK.

## Configure and deploy the Lambda function

**Create Lambda execution IAM role and attach the policy**

```bash
aws iam create-role --role-name LambdaMSKRole --assume-role-policy-document file://lambda-trust-policy.json
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaMSKExecutionRole --role-name LambdaMSKRole
```

Before creating the policy, update the `msk-consumer-policy.json` file to reflect the required details including MSK cluster ARN etc.

```
aws iam put-role-policy --role-name LambdaMSKRole --policy-name MSKConsumerPolicy --policy-document file://msk-consumer-policy.json
```

**Build and deploy the Go function and create a zip file**

Build and zip the function code:

```bash
GOOS=linux go build -o app
zip func.zip app
```

Deploy to Lambda:

```bash
export LAMBDA_ROLE_ARN=<enter the ARN of the LambdaMSKRole created above e.g. arn:aws:iam::<your AWS account ID>:role/LambdaMSKRole>

aws lambda create-function \
    --function-name msk-consumer-function \
    --runtime go1.x \
    --zip-file fileb://func.zip \
    --handler app \
    --role $LAMBDA_ROLE_ARN
```

**Lambda VPC configuration** 

Make sure you choose the same `VPC` and *private* subnets as the MSK cluster. Also, select the *same* security group ID as MSK (for convenience) - if you select a different one, make sure to update MSK security group to add an inbound rule (for port `9098`), just like you did for the `Cloud9` instance in an earlier step.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4a36r10duu9p5tvcwgbf.png)

**Configure the MSK trigger for the function**

Make sure to choose the right MSK Serverless cluster and enter the correct topic name.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ulmtkw9vj1w5sxrwyi3s.png)

## Verify the integration

Go back to the `Cloud9` terminal and send more data using the producer application

> I used a handy json utility called [jo](https://github.com/jpmens/jo) (`sudo yum install jo`)

```bash
APP_URL=http://localhost:8080

for i in {1..5};
    do jo email=user${i}@foo.com name=user${i} | curl -i -X POST -d @- $APP_URL; 
    done
```

In the Lambda function logs, you should see the messages that you sent.

## Conclusion

You were able to setup, configure and deploy a Go Lambda function and trigger it in response to events sent to a topic in a MSK Serverless cluster!