Import S3 Data to DynamoDB [ New AWESome feature from AWS ]
Published: 2022-09-03 09:51:05


## DynamoDB (New Feature for Bulk Imports) : Import Table API

AWS recently announced an AWSome feature in DynamoDB by providing the ability to load bulk data into DynamoDB Table using the new Import Options

### This can be achieved using following ways:

 1. CLI

 2. Console

 3. API

### This new feature supports following formats:

 1. CSV

 2. JSON

 3. Amazon ION

Note: You can load the data from S3 to any account or region.

using the below example using API, I will try to explain this cool new feature.

Note : In this example, I am trying to import CSV data to DynamoDB

### Prepare your source and execute

STEP 1: Create a new S3 bucket where you want to upload your data files

Note: For this example, I generated the CSV files using [Online CSV generator — Generate random SQL data (extendsclass.com)](https://extendsclass.com/csv-generator.html)

STEP 2: Using the Visual Code, wrote the following code for importing the data. As mentioned earlier, you can use CLI, Console or use the API code in Lambda.
>  const AWS = require(‘aws-sdk’)
>  AWS.config.update({
>  region: “us-east-1” //Here add you region
>  });
>  const dynamodb = new AWS.DynamoDB();
>  var params = {
>  InputFormat: ‘CSV’, /* required */
>  S3BucketSource: { /* required */
>  S3Bucket: ‘ddbimport-iamvijakishan’ /* required */
>  },
>  TableCreationParameters: { /* required */
>  AttributeDefinitions: [ /* required */
>  {
>  AttributeName: ‘id’, /* required */
>  AttributeType: ’N’ /* required */
>  }
>  /* more items */
>  ],
>  KeySchema: [ /* required */
>  {
>  AttributeName: ‘id’, /* required */
>  KeyType: ‘HASH’ /* required */
>  },
>  /* more items */
>  ],
>  TableName: ‘DDB-Import’,
>  BillingMode: ‘PAY_PER_REQUEST’
>  }
>  };
>  dynamodb.importTable(params, function(err, data) {
>  if (err) console.log(err, err.stack); // an error occurred
>  else console.log(data); // successful response
>  })

### Scenario 1:

When the code gets executed, first thing this API does is create the table you have mentioned in DynamoDB. Please note if the table already exists, then the **import job fails.**

### Scenario 2:

Lets say the table doesn't exist already but the job fails, **still the table gets created**. So next time when you run the job, please double check and act accordingly.

Irrespective of the way you executed the import job ( CLI / API / Console ) By going to the DynamoDB console, you can see the status of the import job.

Go to the DynamoDB console, click the menu “ Imports from S3” from the left side menu.

![](https://cdn-images-1.medium.com/max/3564/1*a6HLBSFUlhfSnBt4RCbyJg.png)

Click on the respective import id to get details as shown in the below screen.

![](https://cdn-images-1.medium.com/max/3642/1*Q-0Y0FQNwI06tt-GFfKNAQ.png)

You can see the cloudwatch log group automatically created which will provide your more insights on the reason for failure.

**Note :**

 1. You cannot import the data to existing table

 2. Somehow the API response on succesful import doesnot give you the clear picture. for example, if you see the response of the above code execution
>  PS D:\Personal\Learnings\DDBImport> node index.js
{
 ImportTableDescription: {
 ImportArn: ‘arn:aws:dynamodb:us-east-1:255683865591:table/DDB-Import/import/01662188069456-fd5eea54’,
 ImportStatus: ‘IN_PROGRESS’,
 TableArn: ‘arn:aws:dynamodb:us-east-1:255683865591:table/DDB-Import’,
 TableId: ‘6744a921–40a7–4357-b360–16277f00caf6’,
 ClientToken: ‘2166fc41-a830–41a9-bd6c-6b72ec9d9aba’,
 S3BucketSource: {
 S3BucketOwner: ‘255683865591’,
 S3Bucket: ‘ddbimport-iamvijakishan’
 },
 ErrorCount: 0,
 InputFormat: ‘CSV’,
 InputCompressionType: ‘NONE’,
 TableCreationParameters: {
 TableName: ‘DDB-Import’,
 AttributeDefinitions: [Array],
 KeySchema: [Array],
 BillingMode: ‘PAY_PER_REQUEST’
 },
 StartTime: 2022–09–03T06:54:29.456Z,
 ProcessedItemCount: 0,
 ImportedItemCount: 0
 }
}

## Observations

**ImportARN **is the Import Job ID you can see in the console and in log groups.

It says ProcessedItemCount : 0 and ImportedItemCount : 0. Because you get the response as soon as the import job is initiated.

At this juncture, couldn’t find a way to get the response after the job execution is completed.

Also when purposefully removed the partition key for one of the row ( removed the HASH column along the delimeter, I got the following error but It doesnot pinpoint to the error row at this juncture.
>  { “itemS3Pointer”: { “bucket”: “ddbimport-iamvijakishan”, “key”: “myFile2.csv” }, “importArn”: “arn:aws:dynamodb:us-east-1:255683865591:table/DDB-Import/import/01662190284205-aa94decf”, “errorMessages”: [ “Expected a valid numeric. Remainder of the file will not be processed.” ] }

Also when purposefully removed the partition key for one of the row ( removed the HASH column but delimeter is left out , I got the following error with itemindex.
>  { “itemS3Pointer”: { “bucket”: “ddbimport-iamvijakishan”, “key”: “myFile2.csv”, “itemIndex”: 69 }, “importArn”: “arn:aws:dynamodb:us-east-1:255683865591:table/DDB-Import/import/01662190890023–4a845996”, “errorMessages”: [ “One or more parameter values were invalid: Missing the key id in the item” ] }

I will add more details to this as soon as I dig more.

**Git link goes here : [iamvijaykishan/ddbimporttable: New AWS Feature : DynamoDB Import (github.com)](https://github.com/iamvijaykishan/ddbimporttable)**

**Hope this helps.**