Migrate Data into S3 using AWS Transfer Family
Tags: aws, migration, s3, opensource
Published: 2022-07-19 14:16:00

## What is AWS Transfer Family?
AWS Transfer Family is a fully managed AWS service that you can use to transfer files into and out of Amazon Simple Storage Service (Amazon S3) storage or Amazon Elastic File System (Amazon EFS) file systems over the following protocols:

* Secure Shell (SSH) 
* File Transfer Protocol (SFTP)
* File Transfer Protocol (SFTP)
*  File Transfer Protocol Secure (FTPS)
*  File Transfer Protocol (FTP)

*Read more:* [AWS Transfer Family](https://aws.amazon.com/aws-transfer-family/)

## Benefits Of AWS Transfer Family?
-	It is a fully managed, serverless File Transfer Workflow service that makes it simple to set up, execute, automate, and monitor file processing utilizing AWS Transfer Family.
-	With your data safely stored in Amazon S3, you can utilize native AWS services for processing, analytics, reporting, auditing, and archiving.
-	There are no upfront costs, and you pay only for the use of the service.
-	You don't need to modify your applications or run any file transfer protocol infrastructure.

## How it works?

### AWS Transfer Family Architecture


![AWS Transfer Family Architecture](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ciz0nu8papy1z946tff5.png)

### AWS Transfer Family (SFTP, FTP, FTPS) – Step by Step Implementation Process

#### Setup the prerequisites for AWS transfer for SFTP

```
 - S3 bucket
 - EC2 instances (Linux and Windows)
```
   

#### Create an AWS IAM role and policy

```
- Edit the policy to provide the S3 access

```
	 
####	Create the SFTP server

```
 - Create users
 - Create and assign public keys
```
####	Test the file transfer from SFTP clients

```
- For Linux
- For Windows
```
## What is S3 Bucket? 

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.

*Read more:* [Amazon Simple Storage Service ](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html)

## Create a S3 Bucket

```
- Open AWS Console Click on Services
- In the Storage Section, Choose S3
```

![Choose Storage](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w7x9xfm8qf6oops68igq.jpg)

```
- Click Create Bucket and Create Your Bucket
- Give Bucket Name (Should be unique Name use 
small alphabets Without Space. hyphens and Periods}
- Choose your Region. (Where user wants to store their data)
 rest settings choose default.
```

![Create Bucket](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/or9xba4xw190g44v0vzx.png)

![Bucket Created](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bmucrhdurk8b4gb9mv90.png)

## Create a VPC

```
- Open AWS Console Click on services
- In the Networking & Content Delivery Section, Click on VPC
```

![VpC](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/362678z24rs6uj5k93yh.png)

![VPC configuration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fyd2ag2xrg89brvbsbcf.png)
![VPC Created](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j8mqea7o3cklfl9lagbs.png)

## Create Subnets
![create subnets](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r5ys51p6tjsjtacys5qp.png)

![VPc Settings](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vir2j4ugc5wl5t8v0523.png)

## Create Two EC2 Instances (Linux & Windows)

```
- Open AWS Console Click on services 
- In the Compute section, click on EC2
- Click Launch instance

```

![launch instance](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rvuxxe5lio3oblnncvbm.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/714a87043599hfhj8stf.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/id68gbw74wwna6biaa55.png)
## Security Group rules for Linux Instance
![Security Group Rule Linux](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4qod3mfazgwf4yw84hix.png)

## Security Group rules for Windows Instance
![Security Group Rule Windows](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uqzv6ctogm6kj260djza.png)

## Create an IAM Role

```
- Open AWS Console Click on services 
- In the Security, Identity & Compliance section, click on IAM 
```
![Services](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5g90muoeg070gz58r5bs.png)

## Create IAM roles and policies from the IAM console (Choose AWS service as Transfer)

### The policy to access the S3 bucket
![S3 Bucket Policy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5d3zqfa7y92f4bddk9nw.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3tzt2dai255rx59wutst.png)

## Policy to access CloudWatch Logs

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nqbzsih6ygjy8ienowlz.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hppsjwazw06rg6p5xidk.png)

## Create two roles and attached the policies with each role (Choose AWS service as Transfer)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/if1uqz5gc6jeiyk9flua.png)

## Create the SFTP server, Users, Public Keys and Assign the keys
```
- Open AWS Console Click on services 
- In the Migration & Transfer section, click on AWS Transfer Family
```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gmwvijx6v97cxzy4ncad.png)
## Click on create Server

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hjv0kmwguu345osdv0d2.png)
## Choose Protocols

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fhx29q9c5hwly6s5qpxq.png)
## Choose an Identity Provider

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r9okmnppnpi2bq633ip8.png)
## Choose an Endpoint

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/om8010to46fcvyd6jap4.png)
## Choose a Domain

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dubn2lii6k76huiwj2ne.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qrvvl3fofln2xs92q78u.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/deyv8638mc3qarg530tt.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cnzp5xpm4sw8sudlbcvd.png)

## Add User to the Server

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yu3qgscgj69pgjz62bko.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/65aqh852nqiy2ep4jqc3.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rr13hcapqqt4b3atoiar.png)

## Generate the public key from the Linux machine? 
Log in to your EC2 machine connect via Putty 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jkmkd4w8m056sv0uvuoq.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7b449gq9lv8njtoma4hg.png)

## Generate the key with the below command
```
ssh-keygen -P "" -f key 
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tvevt0kbozo1n715wyhf.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i1ozhbrdkop5gwk99rnj.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jy5sq6f1o1tys8hmfepg.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zrg5a0x9vsvxrta01mju.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rossz3lgvyedil0mdu8r.png)
## Transfer the files using WinSCP for Linux

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zks6anazqrt9ecnw2kyb.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sg0qtvltt06hvpg60x1b.png)
## Test the file transfer from the Linux SFTP client
 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nwkxxgzw0f45c9s4r3in.png)

## Connect to the server
```
- sftp -i key rm-transfer-family-user@
s-c0ca408efe3d4c3e8.server.transfer.ap-southeast-1.amazonaws.com

- put AWS-Transfer-Family-Linux.txt rm-transfer-family-2022 
AWS-Transfer-Family-Linux.txt 
to/rm-transfer-family-2022/rm-transfer-family-user/AWS-Transfer-Family-Linux.txt 
```
## Successfully transfer File from Server to Bucket 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g2jvz6rlsa31k55855p8.png)

## Now do the same using the Windows Machine 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f41z1si0qyvbzch5hrxf.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/abzvd73fhf1mwu909wfh.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7zv06ttomdw2ulu0pgfp.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9hbd0nu22ctqzqug0i71.png)
## Generate Public key for Windows user
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qnnrh7kuzhshchcw7jhe.png)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mryyl1ri4pv5h439kzxp.png)
## Two Public keys are added 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qsy0l8vjnl8y4ccqulkb.png)

## Now transfer the files using WinSCP for Windows

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5miaj4u1qb4748mc1xke.png)

## Connection established using WinSCP for Windows

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pc4jzhzj70viutacimbm.png)

## Test the file transfer from SFTP client for Windows

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/of2331a8tehkqr9t8pn8.png)
## Successfully Tested the file transfer from SFTP client and added to the Bucket

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/emf9ch7faz81r4bomuu2.png)