Things you should consider before using DynamoDb
Tags: aws, dynamodb, nosql
Published: 2020-03-04 09:35:30

Lately, I'm been working on a project with DynamoDb, and this is my first time to meet this AWS service. It is just not a regular NoSQL database like `MongoDB`, I have encountered a few issues (or features?) of it, I will share them along with the way I dealt with them(if I do know how to).

**No empty strings**

DynamoDB is a NoSQL database, but it cannot store empty strings. Not in any levels of your JSON object. If you have a single value which is an empty string in your object, DynamoDB throws an error.

The workaround of this is, in `aws-sdk` (for node developers like me, you can find it on `npm`), it provides an option `convertEmptyValues`, if you set it true, the empty strings will be converted to `null` before being written into DB.

**Query operation size limit**

DynamoDB can maximum return 1MB of data in one query/scan operation. If your query contains more data, the results would include a key `last_evaluated_key` that you need to include in the next query, in order to retrieve "next page" of data.

**Batch write limit**

In every write requests, you can maximum include 25 rows only, which is very inconvenient. To deal with this issue and handle the possible write failure due to the exceed of throughput in concurrent write operations in multiple nodes, I have to do it like this:

```javascript
const R = require('ramda')
const aws = require('aws-sdk')
const batchWriteToDb = async (tableName, data) => {
  const dynamoDb = new aws.DynamoDB({
    maxRetries: 999
  })
  const docClient = new aws.DynamoDB.DocumentClient({
    service: dynamoDb,
    convertEmptyValues: true
  })
  const dataSegments = R.splitEvery(25, data)
  // dynamoDb can only write 25 items once
  for (let i = 0; i < dataSegments.length; i++) {
    const segment = dataSegments[i]
    const params = {
      RequestItems: {
        [tableName]: segment.map(item => ({
          PutRequest: {
            Item: item
          }
        }))
      }
    }
    try {
      let response = await docClient.batchWrite(params).promise()
      while (!R.isEmpty(response.UnprocessedItems)) {
        const count = response.UnprocessedItems[tableName].length
        console.log(`${count} unprocessed item(s) left, retrying...`)
        const params = {
          RequestItems: response.UnprocessedItems
        }
        response = await docClient.batchWrite(params)
      }
    } catch (error) {
      console.log(error, error.stack)
    }
  }
}

```

Very clumsy, right?

**Counting number of rows in a table**

In other databases, there should be simple APIs to count the number of rows in a table, but not in the case of DynamoDb. There are two ways to get the number of rows in a table:

1\. Use `DescribeTable` API

It returns information about a table, including the number of rows in it. But it only updates approximately every six hours, I think this is unacceptable.

2\. Do a full table scan

Yet if your data exceeds 1MB, you have to do the scan operation recursively to get all pages of data…

**Conclusion**

DynamoDB is powerful, but it definitely got some trade-offs. If you come from the world of other NoSQL databases like `MongoDB`, make sure you take the above points into consideration before switching.