RDS-Migration-s3-dynamoDB
Published: 2023-12-09 06:37:06


youtube: https://youtu.be/73HSRWiBVsE?si=m73C7bj25Nweuvvc 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ufdxjrz8d6upwe5f4h4k.png)


1.Create DMS(Replication Instance) with specific SG then t3.micro type instance.
2.Create RDS with same SG of DMS & t3.micro rest all are same.
3.We have to create targets. Here my targets are S3 & DynamoDB
4.Create a Iam roles.
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.S3.html#CHAP_Target.S3.Prerequisites 
5.To create IAM we can refer this link and copy paste and create custom IAM policy.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xnz6ut1ow5uqwkcf5vtk.png)


Here while creating S3 Role IAM we have to choose the DMS.
Then click on create a custom policy then copy paste the code from the above link then create role.

**Creation of Target END-POINT:**

In the DMS paze we can click on create end point, choose target end point & target identifier we can give any name & then choose the target engine as s3-bucket & we have to provide ARN of the bucket.
Then enter name of the bucket & then bucket folder we can give anything & we can leave end-point settings & use the same VPC what ever we used in the DMS replication & test the connection whether this endpoint is able to communicate with s3 or not & then create the end point.


**Creation of Source END-POINT:**

Click on DMS then choose endpoint then choose source endpoint then select the rds and choose the which rds we want to migrate.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qyf9z83q2dvbvw6eyl6f.png)


Here we have to choose provide access information Manually.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cq49asj1p26f7blokvnv.png)


We have to test the connection, then create the end point.
So finally we created 2 end points.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qtug3ftl77sxetpz58qf.png)



RDS:



Note down the DB end point.


IAM for Dynamo-Db To DMS:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nuihrg7087jtsxt9as04.png)


Choose DMS then create custom policy then attach that policy and create role.
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.DynamoDB.html 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/md6czvolignkrvcynpyn.png)


Upload only these 3 settings which region & Account-ID & dynamodb Name ‘employee’



Endpoint For DyanamoDb:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w8drzx553hcb6n0qadow.png)

Click on DMS click on end point choose target endpoint then Name of the target endpoint we can enter anything  &  target engine is DynamoDb and ARN of the role.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/svjzuku6h8pghmmtiffp.png)


Test the connection then create the end-point.
So finally 3 end points are ther.



**Create ec2 instance for connecting rds:**

Create a ec2 instance with same vpc and SG(inbound-mysql) what ever we used while creating RDS.
If we used different SG while creating ec2 instance we can attach that Sg in the rds by modifying the rds


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kuntnagqls5q7a59g3zr.png)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/plh6vm658qn0hmh3hbxc.png)

Add data into databases. 
Here 1st we are creating one schema if not exist then we are creating databases.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/284pgha2dsqvl4oxr8e7.png)



Here we are creating the table and adding the data into table.




**Create the parameter groups:**



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6a7xwboirc0m5d9c64lj.png)

Ofter creating choose that and click on edit then modify the checksm & format like this.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/674xhw3p6vbfl178pvpg.png)


Open rds then click on modify under parameter groups add this parameter what ever we created.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9gdyb5kw1r17aajgximt.png)


DB Migration Task:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/da1qyd7rhkk86efcq0v8.png)


We have enter name as anything then select the source and target of the DB


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/trjy9lovhtpypijk0syo.png)
Here we have to select the only migration or ongoing migration.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6oj6qx8ac3xl5x9b4fru.png)


In the database what ever we mention that schema and table 
Same like that for dynamo db and s3 2 migration tasks should create.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zotutimrwigff1iuwvqb.png)


We can check the data is migrated or not.
Her we have to choose ;Query with s3 Select’ 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z92w3gbg8rlules4pxje.png)

Copy data into Dynamodb little fast compared to s3.-


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fnoa6uqaerbwh4hrwybm.png)


How many times we do modification that many files will create in s3 in .csv file format.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w20elvl3mwpl6kc8bnlv.png)