Using API Gateway mapping templates for direct DynamoDB integrations
Tags: aws, serverless, apigateway, dynamodb
Published: 2024-04-12 13:15:10

Integrating API Gateway directly with DynamoDB can significantly enhance speed by reducing overall response time. Leveraging request and response mapping templates for various DynamoDB operations allows us to remove intermediary Lambda functions in the backend.

## 1. The scenario

I'm developing a <a href="https://dev.to/aws-builders/refactoring-a-serverless-application-to-use-step-functions-third-party-api-call-integration-4ogl" class="link" target="_blank" rel="noopener noreferrer">weather application</a> that periodically fetches current temperature data from the <a href="https://openweathermap.org/" class="link" target="_blank" rel="noopener noreferrer">OpenWeather API</a>. The application incorporates specific business logic to protect my plum, apple, and peach fruits from pests like plum moths, <a href="https://en.wikipedia.org/wiki/Codling_moth" class="link" target="_blank" rel="noopener noreferrer">codling moths</a>, and peach twig borers. It achieves this by monitoring growing degree units (GDUs) for each species and notifying me of the likely emergence day for larvae. Consequently, I can apply organic protection measures before they infest my fruits.

I store temperature and pest data in a <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html" class="link" target="_blank" rel="noopener noreferrer">DynamoDB</a> table, adhering to the principles of the <a href="https://aws.amazon.com/blogs/compute/creating-a-single-table-design-with-amazon-dynamodb/" class="link" target="_blank" rel="noopener noreferrer">single table design</a>. A REST-type <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html" class="link" target="_blank" rel="noopener noreferrer">API Gateway</a> sits in front of the backend.

## 2. Eliminating transport functions

I've been exploring architectures that utilize native integrations between AWS services lately. In this context, I aim to avoid provisioning <a href="https://docs.aws.amazon.com/lambda/latest/dg/welcome.html" class="link" target="_blank" rel="noopener noreferrer">Lambda functions</a> solely for data transmission, following the **use Lambda to transform and not transport** principle.

Consequently, I've opted not to implement Lambda functions between the API Gateway and DynamoDB. My objective is to construct this application with direct service integrations wherever feasible.

This adjustment has notably reduced existing endpoint response times to under 200ms on average, sometimes dipping below 100ms.

Additionally, utilizing direct integration translates to cost savings by bypassing Lambda function invocations. It also mitigates concerns related to managing Lambda function concurrencies.

## 3. Mapping templates

By removing the functions from the architecture, it's imperative to instruct API Gateway on processing client input data and configuring its behaviour during data retrieval from the database. This is accomplished using **mapping templates**.

Mapping templates are composed in the <a href="https://velocity.apache.org/engine/2.0/vtl-reference.html" class="link" target="_blank" rel="noopener noreferrer">Apache Velocity Template Language (VTL)</a> format. While initially challenging to work with, especially for developers accustomed to Lambda functions, they can provide granular control over data flow.

However, API Gateway's VTL support is not exhaustive. Successful local tests of a template do not guarantee identical processing by API Gateway. Consequently, ensuring proper template functionality may require additional effort.

## 4. TransactWriteItems operation from API Gateway

Let's examine the request mapping template for a `POST` endpoint.

In this scenario, I aim to add a new moth with various properties to the database. Because I'd like to retrieve all monitored moths via a `GET /moths` request later, I want to store their names in a dedicated item in the database.

To achieve this, I leverage DynamoDB's <a href="https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TransactWriteItems.html" class="link" target="_blank" rel="noopener noreferrer">TransactWriteItems</a> API.

### 4.1. Input

The client input data will be similar to the following:

```jsonc
{
  "newMoth": {
    "PK": "MOTH#Cydia pomonella",
    "SK": "MOTH#Cydia pomonella",
    "MothName": "Cydia pomonella",
    "BaseTemperature": "10",
    "EmergingGDU1": "104", // heat accumulation when larvae emerge from the eggs - 1st wave
    "MothEng": "Codling moth",
    // other properties
  },
  "toMothList": {
    "MothName": "Cydia pomonella"
  }
}
```

The `newMoth` object contains moth data, necessitating a `PutItem` operation. Simultaneously, the `toMothList` object specifies the moth's name for the `UpdateItem` operation, appending it to the `MothsMonitored` list in a dedicated **singleton** item.

### 4.2. TransactWriteItems request template

To execute both operations within a single database request without Lambda functions, we can create an **integration request mapping template** in API Gateway:

```
{
  "TransactItems": [
    {
      "Put": {
        "TableName": "Moths",
        "Item": {
          #set($inputRoot = $input.path('$.newMoth'))
          #foreach($key in $inputRoot.keySet())
              #set($value = $inputRoot.get($key))
              "$key": { "S": "$value" }#if($foreach.hasNext()),#end
          #end
        },
        "ConditionExpression": "attribute_not_exists(PK) AND attribute_not_exists(SK)"
      }
    },
    {
      "Update": {
        "TableName": "Moths",
        "Key": {
          "PK": { "S": "MOTHS" },
          "SK": { "S": "MOTHS" }
        },
        "UpdateExpression": "SET #attrName = list_append(if_not_exists(#attrName, :emptyList), :val)",
        "ExpressionAttributeNames": {
          "#attrName": "MothsMonitored"
        },
        "ExpressionAttributeValues": {
          ":val": { "L": [{ "S": "$input.path('$.toMothList.MothName')" }] },
          ":emptyList": { "L": [] }
        }
      }
    }
  ]
}
```

This template predominantly comprises typical DynamoDB `PutItem` and `UpdateItem` command codes.

However, let's emphasize a few key lines of the VTL part.

`#set($inputRoot = $input.path('$.newMoth'))` extracts the object from the input's `newMoth` property and stores it in the `inputRoot` variable. We then go over the keys using `#foreach` and wrap their values inside another object with the `S` DynamoDB data type.

The last thing in the `Put` part is that we must add **commas** after each item in the map, except the last one. API Gateway doesn't support trailing commas!

That's why the `#if($foreach.hasNext()),#end` part is in the code. `hasNext` returns `true` if there are more object properties left to iterate over, so in this case, we add the comma. When the logic processes the last item in the collection, `hasNext` will be `false`, and the conditional block's context (the `,`) will not run.

The `Update` part only contains one VTL syntax: `"$input.path('$.toMothList.MothName')"`. We access the second input object here and get the value for `MothName`. We add the moth name to the end of the dedicated `MOTHS` item's `MothsMonitored` list.

### 4.3. Permissions

Ensuring appropriate permissions is crucial. API Gateway must be granted permission to execute `PutItem` and `UpdateItem` actions on the designated DynamoDB table.

## 5. GetItem

For the `GET /moth?mothName=<MOTHNAME>` endpoint, I've created an **integration response mapping template** in API Gateway to transform DynamoDB's moth data into the desired client format.

Since a single item is fetched, a `GetItem` operation is performed on DynamoDB, necessitating the API Gateway role having a `GetItem` permission.

### 5.1. Request template

A straightforward request template is crafted to add the string data type descriptor to the DynamoDB payload:

```json
{
"TableName": "Moths",
  "Key": {
    "PK": { "S": "MOTH#$input.params('mothName')" },
    "SK": { "S": "MOTH#$input.params('mothName')" }
  }
}
```

The moth name, extracted from query parameters, will be part of the primary and sort keys.

### 5.2. Response template

DynamoDB responds to the `GetItem` operation with a payload similar to this:

```jsonc
{
  "Item": {
    "PK": {
      "S": "Value1"
    },
    "SK": {
      "S": "Value2"
    },
    "Attribute1": {
      "S": "Value1"
    },
    "Attribute2": {
      "S": "Value2"
    },
    // other attributes
  },
  "ConsumedCapacity": {
    // ...
  }
}
```

However, the desired client format slightly differs from the database output as I want it to be an object like this:

```jsonc
{
  "Attribute1": "Value1",
  "Attribute2": "Value2",
  // other attributes
}
```

So I want a flat object without any `Item` or other wrappers. I also remove the `S` data type descriptor from the database response. Thirdly, I don't want to return `PK` and `SK` or any other local or global secondary index attributes to the client. I only need data attributes in the client. Lastly, I'll return a short message if the requested item is not saved to the database.

To provide the response the client needs, I created the following transformation:

```
#set($inputRoot = $input.path('$'))
#if($inputRoot.Item && !$inputRoot.Item.isEmpty())
  #set($excludeKeys = ["PK", "SK"])
  #set($resultMap = {})
  #foreach($key in $inputRoot.Item.keySet())
    #if(!$excludeKeys.contains($key))
      #set($dummy = $resultMap.put($key, $inputRoot.Item.get($key).S))
    #end
  #end
  {
    #foreach($key in $resultMap.keySet())
      "$key": "$resultMap.get($key)"
      #if($foreach.hasNext()),#end
    #end
  }
#else
  {"messsage": "No item found"}
#end
```

We create a temporary map called `resultMap` that stores the item's key/value pairs without the `S` data type descriptor. The `#if(!$excludeKeys.contains($key))` condition checks if the key is not an index attribute, i.e., not `PK` or `SK`. If it's not, we store the key/value pair in `resultMap`. This way, we can remove `PK` and `SK`, which don't carry data, from the database response object.

The next challenge is to create a JSON object with no comma after the last property. We can achieve it by creating a second #foreach block to iterate over `resultMap`'s keys. Then, we apply the same condition technique to add commas everywhere except after the last element.

That's it! We now have two fast endpoints, a write and a read one, which work without provisioning and invoking any Lambda functions!

## 6. Considerations

Handling different data types (e.g., numbers, lists, booleans) may necessitate more intricate VTL code. Here, I've opted to represent everything as strings to simplify VTL composition.

While writing VTL may initially be time-consuming compared to traditional programming languages, the benefits of expedited endpoints, reduced management overhead, and cost savings make it a compelling choice.

## 7. Summary

Implementing direct integrations between API Gateway and DynamoDB offers expedited performance, decreased costs, and less management overhead. Utilizing VTL syntax enables seamless mapping of payloads to and from the database.

## 8. Further reading

<a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/create-api-resources-methods.html" class="link" target="_blank" rel="noopener noreferrer">Initialize REST API setup in API Gateway</a> - API Gateway setup guide

<a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-integration-settings.html" class="link" target="_blank" rel="noopener noreferrer">Setting up REST API integrations</a> - Relevant documentation section

<a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStartedDynamoDB.html" class="link" target="_blank" rel="noopener noreferrer">Getting started with DynamoDB</a> - DynamoDB basics and table creation