Amazon DynamoDB single-table design with AWS Step Functions
Tags: aws, serverless, cdk, ddb
Published: 2022-01-13 16:11:12

---
title: Amazon DynamoDB single-table design with AWS Step Functions
published: true
description: Let us explore the world of Lambda-less Amazon DynamoDB single-table design.
tags: aws,serverless,cdk,ddb
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sfphkfz36mwqnnigj6as.jpeg
---

In my latest blog post, I've talked about performing [_Lambda-less outbound HTTP requests on AWS_](https://dev.to/aws-builders/lambda-less-outbound-http-requests-on-aws-serverless-41mf).

Continuing the topic of _Lambda-less_ approach, in this article, I would like to explore how one might implement some of the _single-table_ design data schemes on AWS utilizing _AWS DynamoDB_ as the persistence layer and _AWS Step Functions_ as the orchestration layer.

> This blog post is based on the code [from this GitHub repository](https://github.com/WojciechMatuszewski/ddb-single-table-step-functions)

Let us dive in.

## A note about _Amazon DynamoDB_ single-table design

This blog post assumes that you are familiar with the concept of a _single-table_ design and _Amazon DynamoDB_. If you are not, do not fret!

There are **great** resources on the subject matter. In my personal opinion, [this article](https://www.alexdebrie.com/posts/dynamodb-single-table/) by none other than [Alex DeBrie](https://twitter.com/alexbdebrie) is an excellent introduction to the concept.

## Composite Keys

One of the heavily used concepts in the context of _Amazon DynamoDB single-table_ design schemes is the concept of a [_composite key_](https://www.alexdebrie.com/posts/dynamodb-one-to-many/#composite-primary-key--the-query-api-action). The _composite_ primary/sort key is created by concatenating values together, like an entity ID with its type.

The following is an example of utilizing the _composite_ primary and sort key to retrieve an entity from _Amazon DynamoDB_ using _AWS SDK_ (javascript).

```js
const { DynamoDBClient, GetItemCommand } = require("@aws-sdk/client-dynamodb");

const client = new DynamoDBClient({ region: "us-west-2" });

const command = new GetItemCommand({
  TableName: "YOUR_TABLE_NAME",
  Key: {
    // Composite primary (PK) and sort (SK) keys
    PK: "USER#<USER_ID>",
    SK: "METADATA#<USER_ID>"
  }
});

const result = await client.send(command);
```

Creating values such as `USER#<USER_ID>` is usually not a problem in an _AWS Lambda_ environment – we have the power of a programming language at our disposal. But what about _Lambda-less_ environments?

### Composite Keys in _AWS Step Functions_ world

To re-create the example from above, without using the _AWS Lambda_ function, one can use the _AWS Step Functions_ integration with _Amazon DynamoDB_ and the `States.Format` [intrinsic function](https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-intrinsic-functions.html).

The following is an example of defining an _AWS Step Functions_ task that integrates with _Amazon DynamoDB_ and uses composite primary and secondary keys to retrieve user entity.

```ts
const dataTable = ...

new aws_stepfunctions_tasks.DynamoGetItem(this, "GetUserMetadataTask", {
  table: dataTable,
  key: {
    PK: aws_stepfunctions_tasks.DynamoAttributeValue.fromString(
      aws_stepfunctions.JsonPath.stringAt("States.Format('USER#{}', $.userId)")
    ),
    SK: aws_stepfunctions_tasks.DynamoAttributeValue.fromString(
      aws_stepfunctions.JsonPath.stringAt(
        "States.Format('METADATA{}', $.userId)"
      )
    )
  }
});
```

The `States.Format` intrinsic function is excellent for our use case. Sadly, going the other way around, from a composite key to a "regular" ID, is not as simple as utilizing some built-in function. Let us explore that topic next.

### From Composite Key to ID

The first hurdle that we encounter is that, to my best knowledge, **there is no way to split strings using the _AWS Step Functions_ language**. I also could not find a way to retrofit the `States.Format` function to perform the operation we need.

The solution to our problem lies not in some additional functionality that the _AWS Step Functions_ language supports but in structuring the data in a certain way. By duplicating the parts used to create the composite key, we should return only the part that interests us. In our case, that would be the ID of the user entity.

> All credit goes to my [my colleague Jason Legler](https://twitter.com/smashedtoatoms) who mentioned this solution [on Twitter](https://twitter.com/smashedtoatoms/status/1480188681969410053).

```ts
// The same as the previous example
const getUserMetadataTask = ..

const transformUserMetadata = new aws_stepfunctions.Pass(
  this,
  "FormatUserMetadataTask",
  {
    inputPath: "$",
    parameters: {
      /**
       * The `userId` is duplicated. It exists as a separate attribute and is also embedded within the `PK` composite key.
       * The `getUserMetadataTask` returns the data in _AWS DynamoDB_ format, thus the reference to `userId.S`.
       */
      userId: aws_stepfunctions.JsonPath.stringAt("$.Item.userId.S")
    }
  }
);

const getUserTask = getUserMetadataTask.next(transformUserMetadata);
```

If the solution I presented does not fit your requirements, you might be interested in [Stedi Mappings](https://www.stedi.com/docs/mappings) – a powerful data transformation API. 
Full disclosure: I'm part of a team responsible for that service.

## Retrieving item collections

_Amazon DynamoDB_ item collection is a concept not necessarily specific to the idea of single-design, but it still plays a vital role in that context. To learn more about item collections, refer to [this official AWS documentation page](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LSI.html#LSI.ItemCollections).

Retrieving an item collection from _, Amazon DynamoDB_ used not to be possible – the _AWS Step Functions_ ["optimized integrations"](https://docs.aws.amazon.com/step-functions/latest/dg/connect-ddb.html) for _Amazon DynamoDB_ did not include the `Query` operation. All has changed with the introduction of [AWS SDK service integrations](https://docs.aws.amazon.com/step-functions/latest/dg/supported-services-awssdk.html).

With AWS SDK integrations available, we can perform the `Query` operation invoking the _Amazon DynamoDB_ service directly!
The following is an example of performing the `Query` operation, retrieving an item collection of cats belonging to a given user.

```ts
const dataTable = ...

new aws_stepfunctions_tasks.CallAwsService(this, "GetCatsFromDataTableTask", {
  service: "dynamodb",
  action: "query",
  parameters: {
    TableName: dataTable.tableName,
    KeyConditionExpression: "PK = :PK AND begins_with(SK, :SK)",
    /**
     * We cannot use `DynamoAttributeValue` here.
     * The `CallAwsService` construct is incompatible with the `DynamoAttributeValue` class.
     */
    ExpressionAttributeValues: {
      ":PK": {
        "S.$": "States.Format('USER#{}', $.userId)"
      },
      ":SK": {
        S: "CAT#"
      }
    },
    ScanIndexForward: false
  },
  iamResources: [dataTable.tableArn]
});
```

Remember that the data returned from _Amazon DynamoDB_ [will have a particular format](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#API_Query_ResponseSyntax). I would argue that in most cases, another step to transform the data is needed. If that is the case, you might find the `Pass` state we have used in the _From Composite Key to ID_ section handy.

## Data integrity

The serverless community wrote many great articles on the concept of _Lambda-less_ and related topics. [Paul Swail](https://twitter.com/paulswail) in his piece [_The trade-offs with functionless integration patterns in serverless architectures_](https://serverlessfirst.com/functionless-integration-trade-offs/) mentions "Risk to data integrity" as one of the downsides of such architecture.

> For _Amazon DynamoDB_ specific points about data integrity, refer to [this excellent blog post](https://www.jeremydaly.com/ensuring-data-integrity-with-dynamodb/) by [Jeremy Daly](https://twitter.com/jeremy_daly).

**In my opinion, in our case**, the best thing one can do, in the **context of our use case**, to ensure data integrity at the **code-level** is to consolidate all the code that deals with _Amazon DynamoDB_ operations for a particular entity into a single file, like so.

```ts
// user.ts
import { aws_dynamodb, aws_stepfunctions_tasks } from "aws-cdk-lib";

export function getSaveUserTask(dataTable: aws_dynamodb.Table) {
  return new aws_stepfunctions_tasks.DynamoPutItem(
    this,
    "SaveUserToDataTableTask",
    {
      /** Parameters... */
    }
  ).next(transformSaveOwnerResponse);
}

const transformSaveUserToDataTableTask = new aws_stepfunctions.Pass(
  this,
  "TransformSaveUserToDataTableTask",
  {
    /** Parameters... */
  }
);

// user-machine.ts
import { getSaveUserTask } from "./user";

getSaveUserTask().next(anotherTask);
```

Not a great solution, but I could not come up with a better way to compose the areas of concern in the same way that one might do it in an _AWS Lambda_ environment (usually an entity class with public/static methods). Another solution might be to use [_AWS CDK State Machine Fragments_](https://docs.aws.amazon.com/cdk/api/v1/docs/aws-stepfunctions-readme.html#state-machine-fragments).

The ultimate answer would be to **thoroughly test the deployed state machines**. But that opens another can of worms which I might touch on in another article.

## Summary

In closing – utilizing _Amazon DynamoDB_ single-table designs using _AWS Step Functions_ is possible, though it might be hard for more complex architectures. The main issues are limited capabilities for data transformations and issues when it comes to data integrity (some of them can be addressed at _Amazon DynamoDB_ API level).

I encourage you to give this way of building APIs a shot. I had a blast playing around with all the concepts listed in this article.

Consider following me on Twitter for more serverless content - [@wm_matuszewski](https://twitter.com/wm_matuszewski).

Thank you for your time.