An Overview of AWS Cloud Data Migration Services
Tags: aws, migration, cloud
Published: 2023-01-10 15:10:22

[The Original Whitepaper](https://docs.aws.amazon.com/whitepapers/latest/overview-aws-cloud-data-migration-services/overview-aws-cloud-data-migration-services.html)

# Introduction

![The migration process](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0lla4xce2n6v8wfykgxr.png)


One of the most challenging steps required to deploy an application infrastructure in the cloud is moving data into and out of the cloud **securely**, As There are many different ways to lift-and-shift data to the cloud such as 
- One-time large batches
- Constant device streams
- Intermittent updates
- Hybrid data storage combining the AWS Cloud and on-premises data stores

## Choosing your migration method
Before you start your migration to the cloud journey you need to answer the following questions to choose the best migration method

### What is the time allocated to perform data transfers?
### What is the volume of data?
### What is the network speeds/bandwidth available?

to calculate number of days required to migrate a given amount of data you can use the following formula.
```
Number of Days = (Terabytes * 8 bites per Byte)/(CIRCUIT gigabits per second *
      NETWORK_UTILIZATION percent * 3600 seconds per hour * AVAILABLE_HOURS) 
```
for example : if you have an Gigabit Ethernet connection (1 Gbps) to the internet and 100 TB of data to move to AWS with 80% utilization it will take around 28 days
```
(100,000,000,000,000 Bytes * 8 bits per byte) /(1,000,000,000 bps * 80 percent * 3600
      seconds per hour * 10 hours per day) = 27.77 days 
```
![ Recommended migration methods ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4qj7jop0ys3997hfodrx.png)

### What is number of repetitive steps required to transfer data?
### What is the existing investments in custom tooling and automation in your organization.
### will you use the self-managed/managed tools for migration ?
use the following diagram to have more insights.

![Migration Methods](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7cqbkwx08rv8qoosxty9.png)


# Migration Tools
### Self-managed migration methods
| Tool      | Use Case |Limits|comments
| ----------- | ----------- |--------------|------------------|
| AWS S3 CLI      |  move small data to s3 buckets.      | upload objects <= 5 GB in a single operation | User Multipart Uploading for objects > 5 GB
| AWS Glacier CLI   |move small data to AWS Glacier|Upload Archieve <= 4 GB| Use Multipart Uploading for Archieves > 100 MB
|Storage partner solutions| | |http://aws.amazon.com/backup-recovery/partner-solutions/


### AWS managed migration tools

-  **AWS Direct Connect**
    - **Use Case** : Establishing a dedicated network connection from your premises to AWS
    - **Limits** : Network speed from 50 Mbps and up to 100 Gbps
    - **Comments** : It is not a data transfer service itself
    - **Security Features** : 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/unfh79va1wku3s14rahv.png)


-  **AWS Snow Family (Snowcone)**
    - **Use Case** : Migrate your data fast and secure offline or when loading your data over the Internet would take a week or more
    - **Limits** : 8 TB HHD && 14 TB SSD
    - **Comments** : https://aws.amazon.com/snow/
-  **AWS Snow Family(Snowball)**
    - **Use Case** : frequently large backlogs of data -physically isolated environment - No available high-speed Internet connections- Disaster Recovery - DataCenter decommission
    - **Limits** : 80TB HDD && 1TB SSD
    - **Comments** : https://aws.amazon.com/snow/
-  **AWS Snow Family (Snowmobile)**
    - **Use Case** :  Massive amount of data up to exabyte
    - **Limits** :  100PB HDD
    - **Comments** : https://aws.amazon.com/snow/
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/guwivd2mehsr4tfkhzjp.png)
    - **Security Features** : 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g8zvzvw3ea66es1qu6sm.png)



-  **AWS Storage Gateway**
     
    - **Use Case** : File sharing, enabling existing on-premises backup applications, disaster recovery,mirroring and archiving data
    - **Limits** : it supports storage interfaces file, volume and tape inrerfaces and if supports  NFS, SMB, iSCSI and iSCSI VTL Protocols.
    - **Comments** : 
        - [AWS File Gateway Demo](https://www.youtube.com/watch?v=tP5edaxBEEI&t=5s)
        - [AWS Volume Gateway Demo](https://www.youtube.com/watch?v=MSJrse6jwq0)
        - [AWS Tape Gateway Demo](https://www.youtube.com/watch?v=0HeyFzBjUNs)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ddpknok2tiedylesxsin.png)
    - **Security Features** :
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6pj1193t9gzrgpb546l0.png)


-  **Amazon S3 Transfer Acceleration (S3TA)**
    - **Use Case** : upload to a centralized bucket from all over the world,transfer gigabytes to terabytes of data on a regular basis across continents, having underutilized the available bandwidth over the Internet when uploading to Amazon S3.
    You can use the online speed [comparison tool](https://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html) to get the preview of the performance benefit from uploading data from your location to Amazon S3 buckets in different AWS Regions using Transfer Acceleration. 
   - **Security Features** :
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7kf2ysbu6x7fwkjhe7ui.png)


-  **AWS Kinesis Data Firehose**
    - **Use Case** : Easiest way to load streaming data and store it in Amazon S3, Amazon Redshift, Amazon OpenSearch Service, or Splunk.
    - **Limits** :  Record size (before base64-encoding) as large as 1000 KiB, Buffer Size (1 MiB to 128 MiB) and Buffer Interval (60 to 900 seconds)
    - **Comments** : 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ft9aahlo40yoar0gk55j.png)
 - **Security Features** :
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0xu9v5xgl4t1e6i1wo7r.png)



-  **AWS Transfer Family**
    - **Use Case** : Secure file sharing between an organization and third parties, provide a central location where users can download and globally access your data securely, facilitate data ingestion for a data lake
    - **Limits** : It supports SFTP, FTPS, FTP, and AS2 protocols
    - **Comments** : [ AWS Transfer Family - Demo ](https://www.youtube.com/watch?v=AQtTG2N_QCg)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ifwdww22z95sg3hzk2vx.png)
    - **Security Features** :
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rvcun7wd521k98lazt88.png)

-  **Data Sync**
    - **Use Case** : Transferring files in opposite directions, Using multiple tasks to write to the same Amazon S3 bucket and Allowing DataSync to access a restricted Amazon S3 bucket
    - **Limits** : To access your self-managed on-premises or cloud storage, you need an AWS DataSync agent that's associated with your AWS account.
    - **Comments** : [AWS Datasync Demo](shorturl.at/aqIUW)
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/srtcwmj968i7ho4hjhva.png)
    - **Security Features** :
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q77z22v7v2xsy2k50nst.png)

-  **Third-Party Connectors**
[Read More](https://docs.aws.amazon.com/whitepapers/latest/overview-aws-cloud-data-migration-services/aws-managed-migration-tools.html#third-party-connectors)



## Conclusion
Summary goes through different AWS Managed/self-managed migration options, Additionally it covered different use cases and security features for each service.