7 practices I learned while developing applications with API Gateway service integrations
Tags: aws, apigateway, dynamodb, serverless
Published: 2024-07-11 08:16:29

Integrating API Gateway with other AWS services offers several benefits, including improved performance and reduced costs. Let's explore some practices I've adopted when developing application backends, with a focus on DynamoDB.

## 1. Background

I've been involved in the development of multiple web applications using both <a href="https://docs.aws.amazon.com/cdk/v2/guide/home.html" class="link" target="_blank" rel="noopener noreferrer">AWS CDK</a> and other infrastructure frameworks. Additionally, I've created several applications on my own, ranging from small to large projects, where I had the freedom to choose the tech stack (typically <a href="https://www.typescriptlang.org/" class="link" target="_blank" rel="noopener noreferrer">TypeScript</a>) and architecture (serverless when possible).

In this post, I'll share seven practices I learned while developing application backends.

## 2. Disclaimer

Your situation might differ from mine, and you might disagree with the practices I'm about to share, which is perfectly fine.

I prefer applying the <a href="https://aws.amazon.com/blogs/compute/creating-a-single-table-design-with-amazon-dynamodb/" class="link" target="_blank" rel="noopener noreferrer">single-table design</a> principles in DynamoDB. You might not agree with this approach, and that's okay too.

The practices listed below are based on my experience. They have worked well for me, but that doesn't guarantee they will work for you.

## 3. Some practices for DynamoDB integration

You might already be familiar with many of the strategies below. Some I learned early on, while others I discovered more recently.

The examples provided illustrate direct integrations between <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-rest-api.html" class="link" target="_blank" rel="noopener noreferrer">API Gateway</a> and <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html" class="link" target="_blank" rel="noopener noreferrer">DynamoDB</a>. This means using <a href="https://velocity.apache.org/engine/2.0/vtl-reference.html" class="link" target="_blank" rel="noopener noreferrer">VTL</a> in API Gateway's <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/models-mappings.html" class="link" target="_blank" rel="noopener noreferrer">request and/or response templates</a> to define DynamoDB operations, and request and response payloads. The index keys (table and any local/global secondary index partition keys and sort keys) presented here were designed for a single table, which explains their unusual structure.

All examples are written in TypeScript, the language originally used to create CDK.

### 3.1. Use JSON.stringify() in CDK

Instead of writing template strings in the CDK code, we can use `JSON.stringify()` to create mapping templates.

For instance, if we want to request a **specific user** item and assume  the username is `johndoe`, we might have a **partition key** `PK` and **sort key** `SK` like this:

```json
{
  "PK": "USER#johndoe",
  "SK": "USER#johndoe"
}
```
The CDK integration code can be written as follows:

```ts
const integration = new apigateway.AwsIntegration({
  service: 'dynamodb',
  action: 'GetItem',
  // the GetItem action is actually a POST request to the DynamoDB API
  integrationHttpMethod: 'POST',
  options: {
    // API Gateway needs GetItem permission to the table
    credentialsRole: myRole,
    passthroughBehavior: apigateway.PassthroughBehavior.WHEN_NO_TEMPLATES,
    requestTemplates: {
      'application/json': JSON.stringify({
        TableName: tableName,
        Key: {
          PK: { S: `USER#$input.params('userName')` },
          SK: { S: `USER#$input.params('userName')` },
        },
      }),
    },
    // ...other properties
  }
})
```

The `requestTemplates` value is the same `GetItemCommandInput` object we would define in a Lambda function if we used one.

The `$input.params('userName')` expression extracts the `userName` query (or path) parameter's value, enabling us to generate dynamic user requests. For example, the request template above will map the `/user?userName=johndoe` query string to the item with `PK` and `SK` values of `USER#johndoe` in the DynamoDB table.

### 3.2. Use if/else to send a response if the requested item doesn't exist

Even if the requested item doesn't exist in DynamoDB, I want to ensure a response is sent back to API Gateway and the client.

For example, if the item with `PK = USER#johndoe` and `SK = USER#johndoe` doesn't exist in the database, the **responseTemplates** part of `AwsIntegration` might look like this:

```ts
const integration = new apigateway.AwsIntegration({
  action: 'GetItem',
  options: {
    integrationResponses: [
      {
        statusCode: '200',
        responseTemplates: {
          'application/json': `
#set($inputRoot = $input.path('$'))
#if($inputRoot.Item && !$inputRoot.Item.isEmpty())
  #set($user = $inputRoot.Item)
  {
    "status": "success",
    "data": {
      "userName": "$user.UserName.S",
    }
  }
#else
  #set($context.responseOverride.status = 404)
  {
    "status": "error",
    "error": {
      "message": "Item not found"
    }
  }
#end
          `
        }
      }
    ]
    // ...other properties
  },
  // ...other properties
})
```

The `#if/#else/#end` block works similarly to other programming languages.

If the item with the required `PK` and `SK` exists, we'll transform the DynamoDB response by removing the `S` type descriptor and adding the `UserName` to the `userName` property.

If it doesn't exist, DynamoDB will still **return 200 OK** to API Gateway because the request was valid and well-formatted. It's not DynamoDB's fault that a non-existing item was requested. Thus, we override the `200` status code from DynamoDB to `404` and add a descriptive message. (Some properties are omitted for brevity.)

Previously, I used Lambda functions to create these responses and error messages. Now, I try to avoid Lambdas whenever possible, opting for direct integrations with mapping templates. This approach eliminates cold starts and usually keeps response times under 100 ms.

### 3.3. Don't generalize templates - most are unique anyway

I used to spend a lot of time studying VTL syntax and trying to create a generic mapping template that would work for most requests and responses. However, I realized that API Gateway doesn't support the entire VTL ecosystem, and the supported VTL syntax is not well documented. This led to a lot of trial and error, consuming hours without significant progress.

Instead of creating a single, all-encompassing mapping template, I now create separate templates for each access pattern.

While this approach might seem contrary to clean code principles, it has proven more practical for me. If all table items only have string attributes, creating a generic template is feasible. I even have an example of such a template <a href="https://dev.to/aws-builders/using-api-gateway-mapping-templates-for-direct-dynamodb-integrations-6il" class="link" target="_blank" rel="noopener noreferrer">here</a>.

However, as the application's access patterns grow more complex, most mapping templates end up being different. Handling multiple data types, lists (arrays), maps, and nested attributes in a single template proved slow and unproductive. Instead, I now do something like this:

```
#set($user = $inputRoot.Item)
{
  "userName": "$user.UserName.S",
  "address": "$user.Address.L",
  "age": "$user.Age.N",
}
```

I can also create nested response objects by iterating over lists and maps, and change key names to be different from the table attribute names (more on that below). This approach is quick, flexible, and easier to debug in case of errors.

I'm not against extracting repeated code into its own function. If I have a specific response or error format used in multiple templates, I write a simple function with a config argument to return customized messages while maintaining the same format. This is one reason why CDK is great!

### 3.4. Use specific data attributes

When designing an application with a single table, it's common to have multiple different entities, such as users, messages, events, and tasks. To avoid confusion, especially as the business grows and wants to identify individual entities with unique IDs, it's important to use specific data attributes.

I avoid using generic attributes like `Id` or `Name` because, with multiple entities in the table, it becomes difficult to remember which entity an `Id` refers to – is it a user or a task ID?

Instead, I use specific attributes, like `UserId`, `UserName`, `TaskId` and `MessageId`. This approach is straightforward and makes the code easier to read and work with.

### 3.5. Use ExpressionAttributeNames

DynamoDB has a list of <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ReservedWords.html" class="link" target="_blank" rel="noopener noreferrer">reserved words</a> such as `Date` (not case sensitive), which commonly appears as a database attribute.

Using reserved words directly in request input **expressions** will result in errors. For example, the following `GetItem` request template will **not** work:

```ts
requestTemplates: {
  'application/json': JSON.stringify({
    TableName: 'MyTable',
    Key: {
      PK: { S: `TASK#$input.params('taskId')` },
      SK: { S: `TASK#$input.params('taskId')` },
    },
    ProjectionExpression: 'TaskId, StartTime, EndTime, Date',
  }),
},
```

Here, `ProjectionExpression` includes `TaskId`, `StartTime`, `EndTime` and `Date`, which are attributes on the `Task` item we want to return to the client. We don't need the entire large item for this specific access pattern, only these properties.

To make the code work, we can use `ExpressionAttributeNames`:

```ts
requestTemplates: {
  'application/json': JSON.stringify({
    TableName: 'MyTable',
    Key: {
      PK: { S: `TASK#$input.params('taskId')` },
      SK: { S: `TASK#$input.params('taskId')` },
    },
    ProjectionExpression: '#taskid, #starttime, #endtime, #date',
    ExpressionAttributeNames: {
      '#taskid': 'TaskId',
      '#starttime': 'StartTime',
      '#endtime': 'EndTime',
      '#date': 'Date',
    },
  }),
},
```

Now, the template validator will accept it as valid code, and our stack will deploy. In this specific case, it's enough to create an expression attribute name for `Date` since it's the only reserved word among the projected attributes.

### 3.6. Payload and database attributes are the same

Keeping the payload and database attributes the same simplifies development.

Let's say we want to create a new `Task` with the following payload:

```json
{
  "taskId": "3ecd0d50-6ece-468e-a6ed-66c58c9c6525",
  "startTime": "10:00",
  "endTime": "14:00",
  "date": "2024-07-08"
}
```

In this case, I want the `Task` item data attributes in the DynamoDB table to be `taskId`, `startTime`, `endTime` and `date`. Alternatively, I might make minor conversions, like `taskId` to `TaskId`, which is straightforward to implement.

When using Lambda functions for additional logic or input validation during create and update operations, I write a small function to convert payload keys to attributes. This conversion logic is always straightforward, and a generic function covers all use cases without exceptions.

### 3.7. Enable Amazon Q Developer

I found <a href="https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/what-is.html" class="link" target="_blank" rel="noopener noreferrer">Amazon Q Developer</a> (formerly known as CodeWhisperer) to be an excellent tool that helps me write code faster. It effectively predicts the code I'm about to write based on my previous code in the project folder. Q Developer can also recommend VTL mapping template code, and I'm satisfied with its accuracy.

I use <a href="https://code.visualstudio.com/" class="link" target="_blank" rel="noopener noreferrer">VS Code</a> as my code editor, and Q Developer seamlessly integrates with it. It also supports multiple programming languages, including TypeScript. Amazon Q works well in the command line, though this feature is currently available only for macOS.

## 4. Summary

Direct integrations between API Gateway and various AWS services can often eliminate the need for Lambda functions. Writing VTL request and response mapping templates can be daunting, especially for those with limited experience.
However, by applying some tricks and techniques, we can become more productive and create fast web applications more efficiently.

## 5. Further reading

<a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/create-api-resources-methods.html" class="link" target="_blank" rel="noopener noreferrer">Initialize REST API setup in API Gateway</a> - API Gateway setup guide

<a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-integration-settings.html" class="link" target="_blank" rel="noopener noreferrer">Setting up REST API integrations</a> - Relevant documentation section

<a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStartedDynamoDB.html" class="link" target="_blank" rel="noopener noreferrer">Getting started with DynamoDB</a> - DynamoDB basics and table creation

<a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-integration-settings.html" class="link" target="_blank" rel="noopener noreferrer">Integrations for REST APIs in API Gateway</a> - The different integration types