Use CDK to deploy a complete solution with Kafka, App Runner, EKS and DynamoDB
Tags: tutorial, cloud, go, aws
Published: 2023-01-09 04:09:43

[A previous blog post](https://dev.to/aws/getting-started-with-msk-serverless-and-aws-lambda-using-go-1eo2) covered how to deploy a [Go Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html) and trigger it in response to events sent to a topic in a [MSK Serverless](https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html) cluster. 

This blog will take it a notch further. 

- The solution consists of a MSK Serverless cluster, a producer application on [AWS App Runner](https://docs.aws.amazon.com/apprunner/latest/dg/what-is-apprunner.html) and a consumer application in [Kubernetes (EKS)](https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html) persisting data to [DynamoDB](https://docs.aws.amazon.com/dynamodb/index.html).
- The core components (MSK cluster, EKS and DynamoDB) and the producer application will be provisioned using Infrastructure-as-code with [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/home.html).
- Since the consumer application on EKS will interact with both MSK and `DynamoDB`, you will also need to [configure appropriate IAM roles](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).

All the components in this solution have been written in [Go](http://go.dev/).

- The MSK producer and consumer app use the [franz-go](https://github.com/twmb/franz-go) library (it also supports [MSK IAM authentication](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html)).
- The CDK stacks have been written using [CDK Go library](https://docs.aws.amazon.com/cdk/v2/guide/work-with-cdk-go.html).

## Prerequisites

You will need the following:

- An AWS account
- Install [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install), [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), [Docker](https://docs.docker.com/get-docker/), [eksctl](https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html) and [curl](https://everything.curl.dev/get).

## Use CDK to provision MSK, EKS and DynamoDB

All the code and config is present in [this GitHub repo](https://github.com/abhirockzz/msk-cdk-apprunner-eks-dynamodb). Clone the GitHub repo and change to the right directory:

```bash
git clone https://github.com/abhirockzz/msk-cdk-apprunner-eks-dynamodb
cd msk-cdk-apprunner-eks-dynamodb/cdk
```

Deploy the first CDK stack:

```bash
cdk deploy MSKDynamoDBEKSInfraStack
```

Wait for the all the components to get provisioned, including MSK Serverless cluster, EKS cluster and `DynamoDB`.
You can check its progress in the AWS CloudFormation console.

You can take a look at the CDK stack code [here](https://github.com/abhirockzz/msk-cdk-apprunner-eks-dynamodb/blob/master/cdk/main.go#L72).

## Deploy MSK producer application to App Runner using CDK

Deploy the second CDK stack. 

> Note that in addition to deploying the producer application to App Runner, it also builds and uploads the *consumer* application Docker image to an ECR repository.

Make sure to enter the MSK Serverless broker endpoint URL.

```bash
export MSK_BROKER=<enter endpoint>
export MSK_TOPIC=test-topic

cdk deploy AppRunnerServiceStack
```

Wait for the the producer application to get deployed to App Runner. You can check its progress in the AWS CloudFormation console.

You can take a look at the CDK stack code [here](https://github.com/abhirockzz/msk-cdk-apprunner-eks-dynamodb/blob/master/cdk/main.go#L161) and the producer application [here](https://github.com/abhirockzz/msk-cdk-apprunner-eks-dynamodb/blob/master/msk-producer/main.go).

Once complete, make a note of the App Runner application public endpoint as well as the ECR repository for the consumer application. You should see these in the stack output as such:

```bash
Outputs:
AppRunnerServiceStack.AppURL = <app URL>
AppRunnerServiceStack.ConsumerAppDockerImage = <ecr docker image>
....
```

Now, you can verify if the application is functioning properly. Get the publicly accessible URL for the App Runner application and invoke it using `curl`. This will create the MSK topic and send data specified in the `HTTP` `POST` body.

```bash
export APP_RUNNER_URL=<enter app runner URL>

curl -i -X POST -d '{"email":"user1@foo.com","name":"user1"}' $APP_RUNNER_URL
```

Now you can deploy the consumer application to the EKS cluster. Before that, execute the steps to configure appropriate permissions for the application to interact with MSK and `DynamoDB`.

## Configure IRSA for consumer application

Exit the `cdk` directory and change to the root of the project:

```bash
cd ..
```

### Create an IAM OIDC identity provider for your cluster with `eksctl`

```bash
export EKS_CLUSTER_NAME=<EKS cluster name>

oidc_id=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)

aws iam list-open-id-connect-providers | grep $oidc_id

eksctl utils associate-iam-oidc-provider --cluster $EKS_CLUSTER_NAME --approve
```

### Define IAM roles for the application

Configure [IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (also known as **IRSA**).

> Refer to the [documentation for details](https://docs.aws.amazon.com/eks/latest/userguide/create-service-account-iam-policy-and-role.html)

Start by creating a [Kubernetes Service Account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/):

```bash
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eks-app-sa
EOF
```

> To verify - `kubectl get serviceaccount/eks-app-sa -o yaml`

Set your AWS Account ID and [OIDC Identity provider](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) as environment variables:

```bash
ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)

export AWS_REGION=<enter region e.g. us-east-1>

OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")
```

Create a JSON file with *Trusted Entities* for the role:

```bash
read -r -d '' TRUST_RELATIONSHIP <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:default:eks-app-sa"
        }
      }
    }
  ]
}
EOF
echo "${TRUST_RELATIONSHIP}" > trust.json
```

> To verify - `cat trust.json`

Now, create the IAM role:

```bash
export ROLE_NAME=msk-consumer-app-irsa

aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://trust.json --description "IRSA for MSK consumer app on EKS"
```

You will need to create and attach policy to role since we only want the consumer application to consume data from MSK cluster and put data to `DynamoDB` table - this needs to be fine-grained.

In the `policy.json` file, replace values for MSK cluster and `DynamoDB`. Create and attach the policy to the role you just created:

```bash
aws iam create-policy --policy-name msk-consumer-app-policy --policy-document file://policy.json

aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn=arn:aws:iam::$ACCOUNT_ID:policy/msk-consumer-app-policy
```

Finally, associate the IAM role with the Kubernetes Service Account that you created earlier:

```bash
kubectl annotate serviceaccount -n default eks-app-sa eks.amazonaws.com/role-arn=arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME

#confirm
kubectl get serviceaccount/eks-app-sa -o yaml
```

## Deploy MSK consumer application to EKS

> You can refer to the consumer application code [here](https://github.com/abhirockzz/msk-cdk-apprunner-eks-dynamodb/blob/master/msk-consumer/main.go).

Make sure to update consumer application manifest (`app-iam.yaml`) with the MSK cluster endpoint and ECR image (obtained from the stack output).

```bash
kubectl apply -f msk-consumer/app-iam.yaml

# verify Pods
kubectl get pods -l=app=msk-iam-consumer-app
```

## Verify end to end solution

Continue to send records using App Runner producer application:

```bash
export APP_RUNNER_URL=<enter app runner URL>

curl -i -X POST -d '{"email":"user2@foo.com","name":"user2"}' $APP_RUNNER_URL
curl -i -X POST -d '{"email":"user3@foo.com","name":"user3"}' $APP_RUNNER_URL
curl -i -X POST -d '{"email":"user4@foo.com","name":"user4"}' $APP_RUNNER_URL
```

Check consumer app logs on EKS to verify:

```bash
kubectl logs -f $(kubectl get pods -l=app=msk-iam-consumer-app -o jsonpath='{.items[0].metadata.name}')
```

### Scale out consumer app

The MSK topic created by the producer application has three topic partitions, so we can have maximum of three consumer instances.
Scale out to three replicas:

```bash
kubectl scale deployment/msk-iam-consumer-app --replicas=3
```

Verify the number of `Pod`s and check logs for each of them. Notice how the data consumption is being balanced across the three instances.

```bash
kubectl get pods -l=app=msk-iam-consumer-app
```

## Conclusion

You were able to deploy the end to end application using CDK. This comprised of a producer on App Runner sending data to MSK Serverless cluster and a consumer on EKS persisting data to `DynamoDB`. All the components were written using the Go programming language!