Step by step: Store Amazon DynamoDB records in AWS S3 with Amazon Kinesis Data Stream / Firehose
Tags: aws, serverless
Published: 2023-03-05 10:49:18

Sometimes I want to store Amazon DynamoDB records in AWS S3 in order to use Amazon Athena.

We can do it simply with Amazon Kinesis Data Stream and Amazon Kinesis Data Firehose.

I am writing this article on how to set them up.

_**Table Of Contents**_
1. [Set up Amazon DynamoDB with Amplify Studio](#c1)
  1. [ Open AWS Console](#c1s1)
  2. [Create Amplify Project and Amazon DynamoDB](#c1s2)
  3. [ Create Amazon DynamoDB table](#c1s3)
2. [Create Amazon Kinesis in Amazon DynamoDB configuration](#c2)
  1. [Show Amazon DynamoDB configuration](#c2s1)
  2. [Create Amazon Kinesis Data Stream](#c2s2)
  3. [Create Amazon Kinesis Data Firehose](#c2s3)
  4. [Connect Amazon DynamoDB to Amazon Kinesis Data Stream](#c2s4)
3. [Insert data to Amazon DynamoDB and check Amazon S3](#c3)

<a name="c1"></a>
## 1. Set up Amazon DynamoDB with Amplify Studio
I recommend using Amplify Studio for setting up Amazon DynamoDB because of the helpful Data Manager.

<a name="c1s1"></a>
### 1.1. Open AWS Console
Open AWS Console and choose "AWS Amplify".
![Open AWS Console and choose "AWS Amplify"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d84xtxsg9k3en4lqulkx.png)

<a name="c1s2"></a>
### 1.2. Create Amplify Project and Amazon DynamoDB
Create a new Amplify Project.

Click "New app" and "Build an app".
![Click "New app" and "Build an app"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/27sax9lvb4l9btb4yq3m.png)

Input "App name". This time I input "SampleKinesis".
Click "Confirm deployment" and wait a few minutes.

To finish the deployment, you can see "Launch Studio" and click it.
![Click "Launch Studio"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/arhd7wq1nzn70xqmwowa.png)

And then, you can see the "Amplify Studio" page.
![You can see the "Amplify Studio" page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xv2xghtgfeguhm0gbjlm.png)

<a name="c1s3"></a>
### 1.3. Create Amazon DynamoDB table
Click "Data" in the left-side menu.
Then, you can see the "Data modeling" page.
![Click "Data" in the left-side menu](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/urli6g54gtfqcpipb1we.png)

Next, click "Add model".
Input table name and add fields.

For example, I input the table name and added the below fields.

**Table name**
DeviceLog

**Fields**

| Field name | Type |
|:--:|:--:|
| deviceId | String |
| temperature | Float |
| humidity | Float |

![Input table name and add fields](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zwf0koa60za71w89tbux.png)

Click "Save and Deploy" and wait some minutes.

Then, you can see "Successfully deployed data model" and return to the "AWS Amplify console" browser tab.
![Return to the "AWS Amplify console"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e5m0cu2t5qotibh17ejg.png)

<a name="c2"></a>
## 2. Create Amazon Kinesis in Amazon DynamoDB configuration

<a name="c2s1"></a>
### 2.1. Show Amazon DynamoDB configuration
Go back to the AWS Amplify console, and click the link labeled "staging".
![Click the link labeled "staging"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vo8qfm6bal29pnxwj5vi.png)

Click the "API" tab and the "View" link.
![Click the "API" tab and the "View" link](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8ofjz98xlpaw9po2ipql.png)

And then, You can see Amazon DynamoDB console.

<a name="c2s2"></a>
### 2.2. Create Amazon Kinesis Data Stream
Next, click the "Exports and streams" tab.
![Click the "Exports and streams" tab](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/950neuy4861uz5yrs4no.png)

And click "Turn on" in Amazon Kinesis data stream details.
![Click "Turn on" in Amazon Kinesis data stream details](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7y4lnwhsz6iqtr29zvy2.png)

You can see the "Stream to an Amazon Kinesis data stream" page, then click "Create new".
![Click "Create new"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8baun4hqxq25l8vxrmth.png)

You can see the "Create data stream" page and input "Data stream name".
I input "DeviceLogKinesisStream".
![Input "Data stream name"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qwjpikmr5689i0qrn60z.png)

Then, click "Create data stream" bottom of the page.

You can see the "DeviceLogKinesisStream" page.
!["DeviceLogKinesisStream" page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/50v1dcd8vwo2bp1bz067.png)

<a name="c2s3"></a>
### 2.3. Create Amazon Kinesis Data Firehose
Scroll to the bottom of the page, and click the "Process with delivery stream" under the "Amazon Kinesis Data Firehose".
![Click the "Process with delivery stream"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3q1i06ygledi617xrrpx.png)

You can see the "Create delivery stream" page.
Change to "Amazon S3" in the "Destination".
![Change to "Amazon S3" in the "Destination"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lcad785rp6lbd937muck.png)

Change the "Delivery stream name" to "DeviceLogKinesisStreamToS3".
![Change the "Delivery stream name"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gckctn7kvdzjgkcy2iea.png)

Next, create an Amazon S3 bucket in the "Destination settings" category.
Click "Create".
![Click "Create"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ytlis58jydgtfjah5vt5.png)

Input "Bucket name" and click "Create bucket" bottom of the page.
I input "device-log-bucket".
![Input "Bucket name"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y4wapoimhjpf9ji4oxii.png)

Return to the "Create delivery stream" page, and click "Browse".
Click reload button and select the bucket created above.
![Select the bucket](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j5ykbmc75599w5res56p.png)

These configurations are for Amazon Athena's partitioning.
Next, click "Enabled" in "Dynamic partitioning".
![Click "Enabled" in "Dynamic partitioning"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m4fkrzo8vwgwff3rsime.png)

In "Dynamic partitioning keys", insert the following Key names and JQ Expressions.

| Key name | JQ expression |
|:--|:--:|
| deviceId | .dynamodb.NewImage.deviceId.S |
| year | .dynamodb.NewImage.createdAt.S[:4] |
| month | .dynamodb.NewImage.createdAt.S[5:7] |
| day | .dynamodb.NewImage.createdAt.S[8:10] |
| hour | .dynamodb.NewImage.createdAt.S[11:13] |
| minute | .dynamodb.NewImage.createdAt.S[14:16] |

![Insert Key names and JQ Expressions](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i1qzy7pxzy0ortdp7kih.png)

Input "S3 bucket prefix" and "S3 bucket error output prefix".
I input follow configurations.

**S3 bucket prefix**
```
!{partitionKeyFromQuery:deviceId}/!{partitionKeyFromQuery:year}/!{partitionKeyFromQuery:month}/!{partitionKeyFromQuery:day}/!{partitionKeyFromQuery:hour}/!{partitionKeyFromQuery:minute}/
```

**S3 bucket error output prefix**
```
error/
```

![Input "S3 bucket prefix" and "S3 bucket error output prefix"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e52qcnvz1atvj60dpjtj.png)

Then, click "Buffer hints, compression, and encryption" and click "GZIP" in the "Compression for data records" category.
![Click "GZIP" in the "Compression for data records" category](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hgomfwit63hw15vkx25o.png)

Finally, click "Create delivery stream" bottom of the page.
![Click "Create delivery stream"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/axnla0dp6y8oloavwpxn.png)

<a name="c2s4"></a>
### 2.4. Connect Amazon DynamoDB to Amazon Kinesis Data Stream
Return to the "Stream to an Amazon Kinesis data stream" page (appeared in section 2.2.).

Click reload button, and select "DeviceLogKinesisStream" in the "Destination Kinesis data stream".
Then click "Turn on stream" bottom of the page.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tns03a44v4orsd0mnamy.png)

It's a long journey.
Now you finished connecting Amazon DynamoDB to Amazon Kinesis Data Stream.
![Finished connecting Amazon DynamoDB to Amazon Kinesis Data Stream](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hd51fk8n3w4ucgw39gl8.png)

<a name="c3"></a>
## 3. Insert data to Amazon DynamoDB and check Amazon S3
Go back to Amplify Studio, click "Content" in the left-side menu.
Then click "Create DeviceLog".
![Click "Content" in the left-side menu and click "Create DeviceLog"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ztv5lw336ahb8hofdp0z.png)

Input fields and click "Submit".
![Input fields and click "Submit"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/62ci9zasq8hm43lqofus.png)

Go to the Amazon S3 page and check objects.
You should wait 5 or more minutes, and you can see stored data from Amazon DynamoDB.

![Device ID folder](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g5jmon8k5x8ma48fxbgm.png)

![Stored data from Amazon DynamoDB](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v6lrvkgeqr0q0i03rtwt.png)

You can check the stored data when you download it.
```json
{
  "awsRegion": "ap-northeast-1",
  "eventID": "cce6ab95-2efd-4b24-b7a6-acf119a8a7b1",
  "eventName": "INSERT",
  "userIdentity": null,
  "recordFormat": "application/json",
  "tableName": "DeviceLog-2ixgwyd2nbfk5hran4x3k64iw4-staging",
  "dynamodb": {
    "ApproximateCreationDateTime": 1677996227900,
    "Keys": { "id": { "S": "c09657d6-54b0-4777-b274-6cf3036849d1" } },
    "NewImage": {
      "__typename": { "S": "DeviceLog" },
      "_lastChangedAt": { "N": "1677996227878" },
      "deviceId": { "S": "abcd1234" },
      "_version": { "N": "1" },
      "updatedAt": { "S": "2023-03-05T06:03:47.846Z" },
      "createdAt": { "S": "2023-03-05T06:03:47.846Z" },
      "humidity": { "N": "60" },
      "id": { "S": "c09657d6-54b0-4777-b274-6cf3036849d1" },
      "temperature": { "N": "20.4" }
    },
    "SizeBytes": 233
  },
  "eventSource": "aws:dynamodb"
}
```

Now you can store Amazon DynamoDB records in AWS S3 with Amazon Kinesis Data Stream / Firehose.

You can use Amazon S3 folders for Amazon Athena's partitions.

The next step is setting up Amazon Athena with partitions.