DynamoDB, Lambdas and API GW with localstack (or not)
Tags: aws, cdk, localstack
Published: 2020-06-19 12:53:19

*My original goal for this post was to get a local version of DynamoDB, API Gateway and Lambdas up and running using localstack.  Unfortunately localstack doesn't seem to play well with cloudformation + lambdas... At a certain point this tutorial will transition from a localstack to a AWS-native approach.  I'm keeping the localstack related commentary because I think it's useful to people looking to do the same.*

My goal for this post is to get DynamoDB, API Gateway and Lambdas up and running ~~locally~~ on AWS using cloudformation).

This is a very small expansion of this post:
{% link https://dev.to/elthrasher/exploring-aws-cdk-loading-dynamodb-with-custom-resources-jlf %}

I had intended for it to be heavier on the local dev experience but that didn't pan out.

![Alt Text](https://media.giphy.com/media/TXL85EytXEMc8/giphy.gif)

# Table of Contents

* [Getting Started](#started)
* [Creating the DynamoDB table](#dynamo)
  * [DynamoDB + localstack via cloudformation 🙂](#dynamolocal)
* [Creating the Lambdas](#lambdas)
  * [Lambdas + localstack via cloudformation 🤬](#lambdalocal)
* [API Gateway](#apigw)
* [Deploying to AWS](#deploy)
* [Clean-up](#cleanup)
* [Final thoughts...](#thoughts)

# Getting Started <a name="started"></a>

![Alt Text](https://media.giphy.com/media/3og0IQCfsZXScpvKhi/giphy.gif)

The code for this is here:

{% github martzcodes/blog-cdk-localstack no-readme %}

Make sure you have [cdk installed](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html)

Locally, I have this installed:

```bash
$ node --version
v12.13.1
$ npm --version
6.12.1
$ aws --version
aws-cli/2.0.23 Python/3.7.4 Darwin/19.5.0 botocore/2.0.0dev27
$ cdk --version
1.45.0 (build 0cfab15)
```

To initialize a project:

```bash
$ cdk init --language typescript
Applying project template app for typescript
Initializing a new git repository...
Executing npm install...

# Welcome to your CDK TypeScript project!

This is a blank project for TypeScript development with CDK.

The `cdk.json` file tells the CDK Toolkit how to execute your app.

## Useful commands

 * `npm run build`   compile typescript to js
 * `npm run watch`   watch for changes and compile
 * `npm run test`    perform the jest unit tests
 * `cdk deploy`      deploy this stack to your default AWS account/region
 * `cdk diff`        compare deployed stack with current state
 * `cdk synth`       emits the synthesized CloudFormation template
```

There are some dependencies that will need to get installed... let's just take care of all of that now.  Run: `npm i @aws-cdk/aws-apigateway @aws-cdk/aws-dynamodb @aws-cdk/aws-iam @aws-cdk/aws-lambda @aws-cdk/aws-s3 @aws-cdk/aws-s3-assets @aws-cdk/aws-lambda-nodejs --save`.  S3 is needed because that's where the lambda code will actually live. *Ends up... probably could've skipped the S3 dependencies, but they don't hurt anything...*

# Creating the DynamoDB table <a name="dynamo"></a>

Now, let's start to get our feet wet with CDK... Time to create the DynamoDB table.  For this project we're going to make a table that tracks a user's ratings of beer.  In my `lib/blog-cdk-localstack-stack.ts` file I'm going to update the code to:

```ts
import { AttributeType, Table, BillingMode } from "@aws-cdk/aws-dynamodb";
import { Construct, RemovalPolicy, Stack, StackProps } from "@aws-cdk/core";

export class BlogCdkLocalstackStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const modelName = "Beer";

    const dynamoTable = new Table(this, modelName, {
      billingMode: BillingMode.PAY_PER_REQUEST,
      partitionKey: {
        name: `UserId`,
        type: AttributeType.STRING,
      },
      sortKey: {
        name: `${modelName}Id`,
        type: AttributeType.STRING,
      },
      removalPolicy: RemovalPolicy.DESTROY,
      tableName: modelName,
    });
  }
}
```

This creates the dynamoDB table with a partition key as the UserId and a SortKey with the BeerId.  Since this is a user-focused app this will enable me to get all of a user's beer ratings by using the partition key, and if I want to filter down to a specific beer and its ratings I can do that (there's a LOT more you can do with partition / sort keys but it's not needed for this example).

![Alt Text](https://media.giphy.com/media/DohrJX1h2W5RC/giphy.gif)
<figcaption>So far so good...</figcaption>


## DynamoDB + localstack <a name="dynamolocal"></a>

Next, I'll run `cdk synth BlogCdkLocalstackStack --profile personal > BlogCdkLocalstackStack.template.yml` which will output the CloudFormation template to create the DynamoDB table.  Now I need to get localstack somewhat set up...

I'll start with a somewhat standard docker-compose file:

```yml
version: "3.8"
services:
  localstack:
    image: localstack/localstack
    ports:
      - "4566-4599:4566-4599"
      - "8080:8080"
    environment:
      - SERVICES=s3,lambda,cloudformation,apigateway,dynamodb
      - DEBUG=true
      - DATA_DIR=/tmp/localstack/data
      - LAMBDA_EXECUTOR=local
      - LAMBDA_REMOTE_DOCKER=false
      - DOCKER_HOST=unix:///var/run/docker.sock
      - START_WEB=1
    volumes:
      - ./tmp:/tmp/localstack
      - "/var/run/docker.sock:/var/run/docker.sock"
```

And run `docker-compose up -d`.  After waiting (and periodically checking the logs) to make sure it's up and ready... I can now run:

`aws cloudformation create-stack --stack-name BlogCdkLocalstackStack --template-body file://./BlogCdkLocalstackStack.template.yml --endpoint-url http://localhost:4566` which should initialize the DynamoDB table in localstack.  If I check the logs I get some log statements that don't necessarily give me a warm and fuzzy:

```
localstack_1  | 2020-06-18T14:39:51:DEBUG:localstack.services.cloudformation.cloudformation_starter: Currently processing stack resource BlogCdkLocalstackStack/BeerE09E1431: None
localstack_1  | 2020-06-18T14:39:51:DEBUG:localstack.services.cloudformation.cloudformation_starter: Deploying CloudFormation resource (update=False, exists=False, updateable=False): {'Type': 'AWS::DynamoDB::Table', 'Properties': {'KeySchema': [{'AttributeName': 'UserId', 'KeyType': 'HASH'}, {'AttributeName': 'BeerId', 'KeyType': 'RANGE'}], 'AttributeDefinitions': [{'AttributeName': 'UserId', 'AttributeType': 'S'}, {'AttributeName': 'BeerId', 'AttributeType': 'S'}], 'BillingMode': 'PAY_PER_REQUEST', 'TableName': 'Beer'}, 'UpdateReplacePolicy': 'Delete', 'DeletionPolicy': 'Delete', 'Metadata': {'aws:cdk:path': 'BlogCdkLocalstackStack/Beer/Resource'}}
localstack_1  | 2020-06-18T14:39:51:DEBUG:localstack.utils.cloudformation.template_deployer: Running action "create" for resource type "DynamoDB::Table" id "BeerE09E1431"
localstack_1  | 2020-06-18T14:39:51:DEBUG:localstack.utils.cloudformation.template_deployer: Request for resource type "DynamoDB::Table" in region us-east-1: create_table {'TableName': 'Beer', 'AttributeDefinitions': [{'AttributeName': 'UserId', 'AttributeType': 'S'}, {'AttributeName': 'BeerId', 'AttributeType': 'S'}], 'KeySchema': [{'AttributeName': 'UserId', 'KeyType': 'HASH'}, {'AttributeName': 'BeerId', 'KeyType': 'RANGE'}], 'ProvisionedThroughput': {'ReadCapacityUnits': 5, 'WriteCapacityUnits': 5}}
localstack_1  | 2020-06-18T14:39:51:WARNING:localstack.services.cloudformation.cloudformation_starter: Unable to determine physical_resource_id for resource <class 'moto.dynamodb2.models.Table'>
localstack_1  | 2020-06-18T14:39:51:DEBUG:localstack.services.cloudformation.cloudformation_starter: Currently processing stack resource BlogCdkLocalstackStack/CDKMetadata: None
localstack_1  | 2020-06-18T14:39:51:WARNING:moto: No Moto CloudFormation support for AWS::CDK::Metadata
localstack_1  | 2020-06-18T14:39:51:DEBUG:localstack.services.cloudformation.cloudformation_starter: Running CloudFormation stack deployment loop iteration 1
```

But if I run `aws dynamodb list-tables --endpoint-url http://localhost:4566` the table is definitely listed.  From here I can verify the stacks: `aws cloudformation list-stacks --endpoint-url http://localhost:4566`, delete the stack `aws cloudformation delete-stack --stack-name BlogCdkLocalstackStack --endpoint-url http://localhost:4566` and *attempt* to verify that the delete happened... turns out deleting the stack did not delete the table... could be a quirk of localstack or I'm using the wrong command...

![Alt Text](https://media.giphy.com/media/SVIdWguIxWX7DwQ0ws/giphy.gif)
<figcaption>That kinda works...</figcaption>


# Creating the Lambdas <a name="lambdas"></a>

Ultimately, I'm going to want to have two endpoints... GET all user's and their beer ratings... and Update a User's beer ratings.  Clearly we'd want some more fidelity there, but this is a simple demo.

Let's start with the update lambda.  I'll be somewhat basing my code off of this example: https://github.com/aws-samples/aws-cdk-examples/tree/master/typescript/api-cors-lambda-crud-dynamodb but I want to use the `@aws-cdk/aws-lambda-nodejs` cdk module so my lambda can be stored as a typescript file.

```typescript
import { DynamoDB } from "aws-sdk";

const db = new DynamoDB.DocumentClient();

export const handler = async (event: any): Promise<any> => {
  if (!event.body) {
    return {
      statusCode: 400,
      body: "invalid request, you are missing the parameter body",
    };
  }

  const body = JSON.parse(event.body);

  try {
    const review = await db
      .update({
        TableName: 'Beer',
        Key: {
          UserId: event.pathParameters.user,
          BeerId: event.pathParameters.beer,
        },
        UpdateExpression: "set rating = :r",
        ExpressionAttributeValues: {
          ":r": `${body.rating || 0}`
        },
        ReturnValues: "ALL_NEW",
      })
      .promise();
    console.log(`Update complete. ${JSON.stringify(review)}`);
    return {
      statusCode: 200,
      headers: {},
      body: JSON.stringify(review),
    };
  } catch (e) {
    console.error("GET failed! ", e);
    return {
      statusCode: 400,
      headers: {},
      body: `Update failed: ${e}`,
    };
  }
};
```

The GET ALL lambda ends up being very similar except it uses a scan of the DB which **99% of the time you should absolutely NOT do** since it actively reads every item in your table!

Next... the lambdas need to get tied into the main `blog-cdk-localstack-stack.ts` file.  Where we left off we only had the table.  After creating the table we need to create the lambdas.  For that I'm using the experimental NodejsFunction which can directly compile typescript files (it's quite convenient, but could be minimized more).

```typescript
const updateReview = new NodejsFunction(this, "updateReviewFunction", {
      entry: `${lambdaPath}/update-review.ts`,
      handler: "handler",
      runtime: Runtime.NODEJS_12_X,
    });

    const getReviews = new NodejsFunction(this, "readReviewFunction", {
      entry: `${lambdaPath}/get-reviews.ts`,
      handler: "handler",
      runtime: Runtime.NODEJS_12_X,
    });
```

The lambdas also need to have access to dynamodb:

```typescript
    dynamoTable.grantReadWriteData(updateReview);
    dynamoTable.grantReadData(getReviews);
```

Time to deploy this to localstack and invoke the functions manually...

![Alt Text](https://media.giphy.com/media/QsyY2BA6UGGR7gJG3n/giphy.gif)
<figcaption>This should go well, right?</figcaption>

## Lambdas + localstack via cloudformation <a name="lambdalocal"></a>

***At this point I figured out CloudFormation lambdas on localstack STILL do not work***

I tried a number of things to no avail.  The error has to do with where localstack / cloudformation is trying to store the lambda code.  Lambda code is stored on S3... localstack had s3 turned on but the parameter was supposedly missing.  This is the error for anyone curious:

```
localstack_1  | 2020-06-18T16:26:12:DEBUG:localstack.services.cloudformation.cloudformation_listener: Error response for CloudFormation action "CreateStack" (400) POST /: b'<ErrorResponse xmlns="http://cloudformation.amazonaws.com/doc/2010-05-15/">\n  <Error>\n    <Type>Sender</Type>\n    <Code>Missing Parameter</Code>\n    <Message>Missing parameter AssetParameters7708d9acea8c2c3d47875fb4c6499e05f52897d0ff284eb6811ad1fc7d97eeadS3Bucket9261AF10</Message>\n  </Error>\n  <RequestId>cf4c737e-5ae2-11e4-a7c9-ad44eEXAMPLE</RequestId>\n</ErrorResponse>'
```

So, I'm going to skip localstack, for now...

![Alt Text](https://media.giphy.com/media/gF8cfJ1Ml5W3X5aZvB/giphy.gif)

# API Gateway <a name="apigw"></a>

Finally, we need to set up the api gateway to use the lambdas:

```typescript
    const api = new LambdaRestApi(this, 'beer-api',
    {
      handler: getReviews,
      proxy: false
    });

    const users = api.root.addResource('users');
    users.addMethod("GET", new LambdaIntegration(getReviews));
    const user = users.addResource('{user}');
    const userReview = user.addResource('{beer}');
    userReview.addMethod("POST", new LambdaIntegration(updateReview));
```

First we create a lambda-based REST api.  It requires a default handler but with proxy set to false nothing will actually go to it... they need to be manually defined which is what the following lines do.

# Deploying to AWS <a name="deploy"></a>

Since localstack is out of the picture, to get this deployed to AWS I'll need to do `cdk bootstrap --profile personal` first... Which will initialize the stack on my AWS account.  If I follow that by `cdk deploy --profile personal` it will ask if I want to deploy the changeset (I do).  After a minute or two of logs I get the result:

```
Outputs:
BlogCdkLocalstackStack.beerapiEndpointE86AAAC2 = https://1ruxq80qo2.execute-api.us-east-1.amazonaws.com/prod/
```

Which, if I add add my api routes I can hit `https://1ruxq80qo2.execute-api.us-east-1.amazonaws.com/prod/users` to get a 200 response with no items and alternately do a POST to `https://1ruxq80qo2.execute-api.us-east-1.amazonaws.com/prod/users/123/123` with the body `{"rating": 8}` which responds with the updated item:

```json
{
    "Attributes": {
        "BeerId": "123",
        "UserId": "123",
        "rating": "8"
    }
}
```

It's not local, but it WAS easy...

![Alt Text](https://media.giphy.com/media/Kf4scyT0xr4KoU7iX0/giphy.gif)


# Clean-up <a name="cleanup"></a>

To remove everything in AWS it's as simple as running `cdk destroy --profile personal`

```bash
$ cdk destroy --profile personal
Are you sure you want to delete: BlogCdkLocalstackStack (y/n)? y
BlogCdkLocalstackStack: destroying...
   0 | 10:12:39 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::Stack  | BlogCdkLocalstackStack User Initiated
   0 | 10:12:42 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Account    | beer-api/Account (beerapiAccount2152DC9C) 
   0 | 10:12:42 PM | DELETE_IN_PROGRESS   | AWS::CDK::Metadata          | CDKMetadata 
   0 | 10:12:42 PM | DELETE_IN_PROGRESS   | AWS::Lambda::Permission     | beer-api/Default/users/{user}/{beer}/POST/ApiPermission.BlogCdkLocalstackStackbeerapi01021E61.POST..users.{user}.{beer} (beerapiusersuserbeerPOSTApiPermissionBlogCdkLocalstackStackbeerapi01021E61POSTusersuserbeer86A593EA) 
   0 | 10:12:42 PM | DELETE_IN_PROGRESS   | AWS::Lambda::Permission     | beerapiusersuserGETApiPermissionBlogCdkLocalstackStackbeerapi01021E61GETusersuser199DA8EE 
   0 | 10:12:42 PM | DELETE_IN_PROGRESS   | AWS::Lambda::Permission     | beer-api/Default/users/{user}/{beer}/POST/ApiPermission.Test.BlogCdkLocalstackStackbeerapi01021E61.POST..users.{user}.{beer} (beerapiusersuserbeerPOSTApiPermissionTestBlogCdkLocalstackStackbeerapi01021E61POSTusersuserbeer16885B45) 
   1 | 10:12:42 PM | DELETE_COMPLETE      | AWS::ApiGateway::Account    | beer-api/Account (beerapiAccount2152DC9C) 
   1 | 10:12:42 PM | DELETE_IN_PROGRESS   | AWS::Lambda::Permission     | beerapiusersuserGETApiPermissionTestBlogCdkLocalstackStackbeerapi01021E61GETusersuser9933087C 
   1 | 10:12:43 PM | DELETE_IN_PROGRESS   | AWS::IAM::Role              | beer-api/CloudWatchRole (beerapiCloudWatchRole81F996BB) 
   2 | 10:12:43 PM | DELETE_COMPLETE      | AWS::CDK::Metadata          | CDKMetadata 
   3 | 10:12:44 PM | DELETE_COMPLETE      | AWS::IAM::Role              | beer-api/CloudWatchRole (beerapiCloudWatchRole81F996BB) 
   4 | 10:12:52 PM | DELETE_COMPLETE      | AWS::Lambda::Permission     | beerapiusersuserGETApiPermissionBlogCdkLocalstackStackbeerapi01021E61GETusersuser199DA8EE 
   5 | 10:12:52 PM | DELETE_COMPLETE      | AWS::Lambda::Permission     | beer-api/Default/users/{user}/{beer}/POST/ApiPermission.Test.BlogCdkLocalstackStackbeerapi01021E61.POST..users.{user}.{beer} (beerapiusersuserbeerPOSTApiPermissionTestBlogCdkLocalstackStackbeerapi01021E61POSTusersuserbeer16885B45) 
   6 | 10:12:52 PM | DELETE_COMPLETE      | AWS::Lambda::Permission     | beerapiusersuserGETApiPermissionTestBlogCdkLocalstackStackbeerapi01021E61GETusersuser9933087C 
   7 | 10:12:52 PM | DELETE_COMPLETE      | AWS::Lambda::Permission     | beer-api/Default/users/{user}/{beer}/POST/ApiPermission.BlogCdkLocalstackStackbeerapi01021E61.POST..users.{user}.{beer} (beerapiusersuserbeerPOSTApiPermissionBlogCdkLocalstackStackbeerapi01021E61POSTusersuserbeer86A593EA) 
   7 | 10:12:53 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Stage      | beer-api/DeploymentStage.prod (beerapiDeploymentStageprod3F2FBBD3) 
   8 | 10:12:54 PM | DELETE_COMPLETE      | AWS::ApiGateway::Stage      | beer-api/DeploymentStage.prod (beerapiDeploymentStageprod3F2FBBD3) 
   8 | 10:12:55 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Deployment | beerapiDeployment3077C14B26da823256c4ecc7e0f6cf539683f3ef 
   9 | 10:12:56 PM | DELETE_COMPLETE      | AWS::ApiGateway::Deployment | beerapiDeployment3077C14B26da823256c4ecc7e0f6cf539683f3ef 
   9 | 10:12:57 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Method     | beerapiusersuserGETCE81253D 
   9 | 10:12:57 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Method     | beer-api/Default/users/{user}/{beer}/POST (beerapiusersuserbeerPOST2D1FBB1F) 
  10 | 10:12:57 PM | DELETE_COMPLETE      | AWS::ApiGateway::Method     | beerapiusersuserGETCE81253D 
  11 | 10:12:58 PM | DELETE_COMPLETE      | AWS::ApiGateway::Method     | beer-api/Default/users/{user}/{beer}/POST (beerapiusersuserbeerPOST2D1FBB1F) 
  11 | 10:12:58 PM | DELETE_IN_PROGRESS   | AWS::Lambda::Function       | readReviewFunction (readReviewFunction94EC729D) 
  11 | 10:12:58 PM | DELETE_IN_PROGRESS   | AWS::Lambda::Function       | updateReviewFunction (updateReviewFunctionF0FF17E0) 
  11 | 10:12:58 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Resource   | beer-api/Default/users/{user}/{beer} (beerapiusersuserbeerE37FAAD9) 
  12 | 10:12:59 PM | DELETE_COMPLETE      | AWS::Lambda::Function       | readReviewFunction (readReviewFunction94EC729D) 
  13 | 10:12:59 PM | DELETE_COMPLETE      | AWS::ApiGateway::Resource   | beer-api/Default/users/{user}/{beer} (beerapiusersuserbeerE37FAAD9) 
  14 | 10:12:59 PM | DELETE_COMPLETE      | AWS::Lambda::Function       | updateReviewFunction (updateReviewFunctionF0FF17E0) 
  14 | 10:13:00 PM | DELETE_IN_PROGRESS   | AWS::IAM::Policy            | readReviewFunction/ServiceRole/DefaultPolicy (readReviewFunctionServiceRoleDefaultPolicy2543AE62) 
  14 | 10:13:00 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Resource   | beer-api/Default/users/{user} (beerapiusersuser40A97AB3) 
  14 | 10:13:00 PM | DELETE_IN_PROGRESS   | AWS::IAM::Policy            | updateReviewFunction/ServiceRole/DefaultPolicy (updateReviewFunctionServiceRoleDefaultPolicyD907B783) 
  15 | 10:13:00 PM | DELETE_COMPLETE      | AWS::IAM::Policy            | readReviewFunction/ServiceRole/DefaultPolicy (readReviewFunctionServiceRoleDefaultPolicy2543AE62) 
  16 | 10:13:00 PM | DELETE_COMPLETE      | AWS::ApiGateway::Resource   | beer-api/Default/users/{user} (beerapiusersuser40A97AB3) 
  17 | 10:13:01 PM | DELETE_COMPLETE      | AWS::IAM::Policy            | updateReviewFunction/ServiceRole/DefaultPolicy (updateReviewFunctionServiceRoleDefaultPolicyD907B783) 
  17 | 10:13:01 PM | DELETE_IN_PROGRESS   | AWS::IAM::Role              | readReviewFunction/ServiceRole (readReviewFunctionServiceRole77A9B597) 
  17 | 10:13:01 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::Resource   | beer-api/Default/users (beerapiusers7623AC8A) 
  17 | 10:13:02 PM | DELETE_IN_PROGRESS   | AWS::DynamoDB::Table        | Beer (BeerE09E1431) 
  17 | 10:13:02 PM | DELETE_IN_PROGRESS   | AWS::IAM::Role              | updateReviewFunction/ServiceRole (updateReviewFunctionServiceRole57FA81FA) 
  18 | 10:13:02 PM | DELETE_COMPLETE      | AWS::ApiGateway::Resource   | beer-api/Default/users (beerapiusers7623AC8A) 
  19 | 10:13:03 PM | DELETE_COMPLETE      | AWS::IAM::Role              | readReviewFunction/ServiceRole (readReviewFunctionServiceRole77A9B597) 
  19 | 10:13:04 PM | DELETE_IN_PROGRESS   | AWS::ApiGateway::RestApi    | beer-api (beerapi67AE41CF) 
  20 | 10:13:04 PM | DELETE_COMPLETE      | AWS::IAM::Role              | updateReviewFunction/ServiceRole (updateReviewFunctionServiceRole57FA81FA) 
  21 | 10:13:04 PM | DELETE_COMPLETE      | AWS::ApiGateway::RestApi    | beer-api (beerapi67AE41CF) 

 ✅  BlogCdkLocalstackStack: destroyed
```

# Final thoughts... <a name="thoughts"></a>

It's definitely possible to run everything in localstack by manually running all the commands... but that ultimately defeats the purpose of using CDK.

Do you have any tips for doing local development across API Gateway -> Lambdas -> DynamoDB?  With or without localstack?

![Alt Text](https://media.giphy.com/media/W3a0zO282fuBpsqqyD/giphy.gif)