Build a Twitter Leaderboard app with Redis and AWS Lambda (part 1)
Tags: tutorial, database, programming, serverless
Published: 2022-05-17 14:21:33

Hello and welcome 👋🏼 to this two-part blog series that uses a practical application to demonstrate how to integrate Redis with AWS Lambda. The first part (this one) covers the application - by the end of this blog, you should have the solution deployed, played around with it and in the process, have a good overview of the solution. 

The second part is about the Infrastructure (IaaC to be specific) aspects - its mostly centred around AWS CDK along with some code walkthrough.

Over the course of this blog series, you will learn:
- How to use Lambda and Redis together - including VPC and other configuration to make things work
- How to use Lambda Function URL
- How to deploy your Lambda function as a Docker container (not a zip file)
- Use AWS CDK to create all the components of the solution - Infrastructure (VPC, subnets etc.), database as well as Lambda functions (this includes multiple stacks in the context of a single CDK app)

I have used [Go](https://go.dev/) for the Lambda functions ([aws-lambda-go](https://github.com/aws/aws-lambda-go)) as well as infrastructure (with [CKD Go support](https://docs.aws.amazon.com/cdk/v2/guide/work-with-cdk-go.html)), but you should be easily able to apply the concepts to the programming language of your choice.

> As always, the code is [available on Github](https://github.com/abhirockzz/twitter-leaderboard-app)

Here is a quick overview of the services involved in the solution:
- [Amazon MemoryDB for Redis](https://docs.aws.amazon.com/memorydb/latest/devguide/what-is-memorydb-for-redis.html) - It is a durable, in-memory database service that is compatible with Redis, thus empowering you to build applications using the same flexible and friendly Redis data structures, APIs, and commands that they already use today.
- [Lambda Function URL](https://docs.aws.amazon.com/lambda/latest/dg/lambda-urls.html) is a relatively new feature (at the time of writing this blog) that provides dedicated HTTP(S) endpoint for your Lambda function. It is really useful when all you need is a single endpoint for your function (e.g. to serve as a webhook) and don't want to setup and configure an API Gateway.
- [AWS Cloud Development Kit](https://docs.aws.amazon.com/cdk/v2/guide/home.html) (CDK) is all about IaaC (Infrastructure-as-code). It is a framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation. You can choose from a list of supported programming languages (at the time of writing - TypeScript, JavaScript, Python, Java, C#/.Net, and Go (in developer preview)) to define your infrastructure components as code, just like you would with any other application!

## The Twitter Hashtag Leaderboard app

Don't worry, it's simpler than it sounds! Here is the high level architecture:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2qxmomj6nb5iw1kbl0wk.png)

The solution can be divided into two logical parts:

- The first part handles tweet ingestion: A Lambda function fetches tweets (from Twitter), extracts hashtags for each tweet, and stores them in MemoryDB (in a Redis Sorted Set). This function gets invoked based on a schedule based on a rule in CloudWatch trigger
- The second part provides the leaderboard functionality: This is yet another Lambda function that provides a HTTP(s) endpoint (thanks to Lambda Function URL) to query the sorted set and extract top 10 hashtags (leaderboard)

I told you, it's quite simple!

Alright, with the intro out of the way, we can move on to the fun part - which is deploying the application. Before that, make sure you have the following ready:

## Requirements

- [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) (if you do not already have one) and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
- Install and configure [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
- Install and bootstrap [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install)
- Setup [Docker](https://docs.docker.com/get-docker/)
- [Install Go](https://go.dev/dl/)
- Get your [Twitter API credentials](https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api)

## Deploy the application - One CDK Stack at a time

We will cover the details in the second part. For now, just know that the infrastructure part of this solution is comprised of three (CDK) [Stack](https://docs.aws.amazon.com/cdk/v2/guide/stacks.html)s (in the context of a single [CDK App](https://docs.aws.amazon.com/cdk/v2/guide/apps.html)). Although its possible to deploy all of them together (with `cdk deploy --all`), we will do it one by one. This way, you can review what's happening at each stage, introspect the  components that have been created and better understand how everything is wired together.

First, clone the Github repo:

```bash
git clone https://github.com/abhirockzz/twitter-leaderboard-app
```

### Starting with the infrastructure...

The first stack deploys a [VPC](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) (and also subnets, NAT gateway etc.), a MemoryDB for Redis cluster and a few security groups.

Choose a password of your choice for MemoryDB and export it as a environment variable (this is just for demonstration purposes - for production, you will have specific processes in place to handle sensitive info) 

Be mindful of the password requirements. From the [documentation](https://docs.aws.amazon.com/memorydb/latest/devguide/clusters.acls.html):

> "In particular, be aware of these user password constraints when using ACLs for MemoryDB:
> 
> - Passwords must be 16–128 printable characters.
> - The following non-alphanumeric characters are not allowed: , "" / @."

Change to the correct folder and kick off the stack deployment:

```bash
export MEMORYDB_PASSWORD=<enter a password e.g. P@ssw0rd12345678>

cd cdk

# stack1 is the name of the stack - used for simplicity
cdk deploy stack1
```

There is lots to be done. While CDK is hard at work for us, you need to wait patiently :) This is probably a good time to navigate to the `CloudFormation` in AWS console and see what's going on behind the scenes.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1pllzgbe8fb4ikmcin59.png)

Once the stack creation is complete, go the AWS console and check out your freshly minted VPC, MemoryDB cluster and other components!

Here is the stack output for your reference:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5jw2nicbmnky35h1nab1.png)

The next two stacks deploy separate Lambda functions. Before moving on though, make sure you build the Go binaries for both these functions.

**Go, build!**

Move to the respective folders and just invoke `go build` for each function:

```bash
cd tweet-ingest && GOOS=linux go build -o app
cd leaderboard-function && GOOS=linux go build -o app
```

> To package the Lambda function as a Docker container, I used the [Go:1.x base image](https://gallery.ecr.aws/lambda/go). But, you can explore [other options](https://docs.aws.amazon.com/lambda/latest/dg/go-image.html) as well. During deployment (via `cdk deploy`), the Docker image is built locally, pushed to a [private ECR registry](https://docs.aws.amazon.com/AmazonECR/latest/userguide/Registries.html) and finally the Lambda function is created - all this, with a few lines of code!


### The second stack - For the first Lambda function

The function requires your Twitter API credentials (along with the MemoryDB password) - seed them as environment variables. Then, initiate stack creation:

```bash
export MEMORYDB_PASSWORD=<enter the password you had previously chosen e.g. P@ssw0rd12345678>
export TWITTER_API_KEY=<enter twitter API key>
export TWITTER_API_SECRET=<enter twitter API secret>
export TWITTER_ACCESS_TOKEN=<enter twitter access token>
export TWITTER_ACCESS_TOKEN_SECRET=<enter twitter API access token secret>

# note the name of the stack is stack2
cdk deploy stack2
```

This one will be faster (compared to `stack1`), I promise!

Once the stack creation is complete, to make things work, there is one manual step required. Go to the AWS console, open the Lambda function (named `tweet-ingest-function`), that was just created, click **Add Trigger** and manually add the CloudWatch trigger configuration.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pj9791pnoxvc7sjl9sdx.png)

Now your function will be automatically triggered once every minute!

To check how things are going, check out the logs for your Lambda function (AWS console > Lambda > Monitor):


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mu4bq7rilwzkw6w2slnz.png)

I would also encourage you to check the VPC configuration for your Lambda function:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/27aaw4i4qaq16mvm864t.png)

### And finally, deploy the Leaderboard function

```bash
export MEMORYDB_PASSWORD=<enter the password you had previously chosen e.g. P@ssw0rd12345678>

# note the name of the stack is stack3
cdk deploy stack3
```

After successful deployment, you should have a Lambda Function URL ready for you to access - you can simply copy it from the stack output!

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dsfpnpn6u0gzrmnhug94.png)

Just access the endpoint (I have used `curl` CLI, but a browser should work just fine): 

```bash
curl -i <enter lambda function URL from the stack output>
```

You should get back a JSON payload with info about top 10 hashtags along with their names and number of times they were mentioned - something similar to this:

```json
[
  {
    "Score": 121,
    "Member": "AWS"
  },
  {
    "Score": 56,
    "Member": "gaming"
  },
  {
    "Score": 56,
    "Member": "RESTOCK"
  },
  {
    "Score": 56,
    "Member": "ALERT"
  },
  {
    "Score": 35,
    "Member": "aws"
  },
  {
    "Score": 26,
    "Member": "rtx3080"
  },
  {
    "Score": 26,
    "Member": "geforce3080"
  },
  {
    "Score": 24,
    "Member": "Infographic"
  },
  {
    "Score": 23,
    "Member": "箱マスク"
  },
  {
    "Score": 23,
    "Member": "startups"
  }
]
```

This concludes the first part. In the second one, we dive into the CDK aspects and look at some Go code - see you there!