GraphQL pagination with DynamoDB - DynamoDB pagination

Tags: graphql, aws, webdev, node
Published: 2021-04-28 10:14:13

Just like the aforementioned GraphQL pagination, DynamoDB also uses cursor based pagination. 

That being said, there are distinct differences that need to be taken into account 🔔.

## 🏎️ Traversal

For any given query to DynamoDB, the following takes place:
 1. In-order traversal takes place on the target table (order specified by the sort key)
 2. The provided _key condition expression_ is evaluated to find matching documents
 3. Optionally provided _filter expressions_ are used to additionally constrain the matching documents


### Inverting traversal order

Traversal order can be inverted using the `ScanIndexForward` attribute.

![Inverting traversal order](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/inzh49h085ym9c87wu0z.gif)

This (logically) inverted collection is traversed in reverse order, and the resulting items are also returned in reverse order.


## 📚 Paging

As matches are found, DynamoDB adds them to a result set - a "page". If a `Limit` is provided, DynamoDB will suspend traversal when the number of matches for the given _key condition expression_ reaches the limit.

![Page size request](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vz8lihpkq0djl8ylkljw.gif)
 

Unfortunately, the limit is applied prior to the filter expression evaluation; meaning a result set will never exceed the limit value, but can have a size smaller than the limit, while also having subsequent pages to follow.


## 🔑 Cursors

Upon the return of a page, assuming the collection hasn't been exhausted, DynamoDB provides a cursor in the form of a `LastEvaluatedKey`. 

![Fetching next page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3mg2jy6ifz0qrg0ur2pb.gif)
 

Providing this value in subsequent queries via the `ExclusiveStartKey` allows DynamoDB to continue where it left off.

### Exhausting a query

In a similar fashion, this pattern can be used to retrieve all items in a query.

```ts
const exhaustQuery = async <T>(
  query: AWS.DynamoDB.DocumentClient.QueryInput,
  agg: T[] = [],
): Promise<T[]> => {
  const res = await dynamodb.query(query).promise();
  const newAgg = [...agg, ...(res.Items || [])];

  if (!res.LastEvaluatedKey || !res.Items) {
    return newAgg;
  }

  return exhaustQuery(
    {
      ...query,
      ExclusiveStartKey: res.LastEvaluatedKey,
    },
    newAgg,
  );
};
```

### Cursor construction

DynamoDB responses only provide a cursor for the position of the last evaluated element in the collection. There doesn't look to be official documentation on how cursors are constructed, but I've found the following thanks to some experimentation.

It looks like DynamoDB needs two things to be able to continue where it left off:
 - The unique identity of the last visited item (i.e. primary key)
 - The position in the index where said element exists (i.e. primary key of index/table)

In the case of querying a table, the primary key typically consists of a partition (hash) and optionally a sort (range) key.

```js
{
  productId: 1,  // partition key (pk)
  type: 'Book'   // sort key (sk)
}
```

For querying an index, the same rule applies. We'll still need the attributes required to uniquely identify the element (primary key), but we also need the partition and (optional) sort attributes to get back to the previous position in the query.

```js
{
  productId: 1,          // pk
  type: 'Book',          // sk, index pk
  createdAt: 1618496921, // index sk
}
```

---

Thanks for reading! 

If you enjoyed this post, be sure to react 🦄  or drop a comment below with any thoughts 🤔. 

You can also hit me up on twitter - [@andyrichardsonn](https://twitter.com/andyrichardsonn)

Disclaimer: All thoughts and opinions expressed in this article are my own.