Use S3 Batch Replication to replicate objects to another account and encrypt with AWS KMS
Tags: aws, s3, migration, kms
Published: 2023-02-17 10:55:29

## Our scenario?

We have an existing S3 bucket in an **AWS account (A)** and we have to somehow move the existing data onto **AWS account (B)** because of internal requirements or compliance decisions. Before the data landing onto the destination S3 bucket, it must be encrypted with an **AWS KMS key** which belongs to the destination account.

For this post, I am going to use **S3 Batch Replication** and show steps by steps from AWS console.

## When to use S3 Batch Replication?

As per [AWS official documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html#batch-replication-scenario), here are the main points:
1. Existing objects
2. Objects that failed to replicate previously
3. Objects that were already replicated
4. Replicas that were created from a replication rule

The 1st point is related to our scenario.

## Prerequisites

1. Enable *bucket versioning* on both source and destination S3 buckets.
2. The service, which is **Amazon S3**, must have *permissions* to perform the tasks.
3. Destination account (B) must allow Source acccount (A) to use its **AWS KMS** encryption key.

## Preparation

### Enable Versioning

> To be applied on both Source and Destination AWS accounts

1. Go to your S3 bucket and locate *Properties* tab from AWS console.
2. From *Bucket Versioning*, ensure the value is `Enabled`.

### Source Account A
#### Create a new S3 bucket

Let's quickly create a new bucket for storing replication reports or you can use any existing ones.
```bash
aws s3api create-bucket \
    --bucket <S3_BUCKET_REPORT_A> \
    --region ap-southeast-1 \
    --create-bucket-configuration LocationConstraint=ap-southeast-1
```

#### IAM role for S3 replication

Let's quickly create an **IAM role** for our later usage of S3 replication rule.
```bash
aws iam create-role \
    --role-name <IAM_ROLE_REPLICATION_A> \
    --assume-role-policy-document file://trust.json
aws iam put-role-policy \
    --role-name <IAM_ROLE_REPLICATION_A> \
    --policy-name replication-policy \
    --policy-document file://policy.json
```

Here is the **IAM policy** (`policy.json`) to be used.
```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:InitiateReplication",
                "s3:GetReplicationConfiguration",
                "s3:PutInventoryConfiguration",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectTagging",
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::<S3_BUCKET_A>",
                "arn:aws:s3:::<S3_BUCKET_A>/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:ReplicateObject",
                "s3:ReplicateTags",
                "s3:ReplicateDelete",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectVersionAcl",
                "s3:PutObjectTagging",
                "s3:PutObjectVersionTagging",
                "s3:GetObjectVersionForReplication",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Resource": [
                "arn:aws:s3:::<S3_BUCKET_B>",
                "arn:aws:s3:::<S3_BUCKET_B>/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::<S3_BUCKET_REPORT_A>/*"
        },
        {
            "Effect": "Allow",
            "Action": "kms:*",
            "Resource": "*"
        }
    ]
}
```

Here is the **trust relationship** (`trust.json`) to be used.
```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "batchoperations.s3.amazonaws.com",
                    "s3.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
```

### Destination Account B
#### Update S3 bucket policy
Ensure the **S3 bucket policy** to be updated.
```json
{
    "Version": "2012-10-17",
    "Id": "PolicyForDestinationBucket",
    "Statement": [
        {
            "Sid": "S3PolicyStmt-DO-NOT-MODIFY-1234567890123",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<AWS_ID_A>:root",
                    "arn:aws:iam::<AWS_ID_A>:role/service-role/<IAM_ROLE_REPLICATION_A>"
                ]
            },
            "Action": [
                "s3:GetBucketVersioning",
                "s3:GetObjectAcl",
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:PutObjectAcl",
                "s3:PutObjectVersionAcl",
                "s3:PutBucketVersioning",
                "s3:ObjectOwnerOverrideToBucketOwner",
                "s3:Put*"
            ],
            "Resource": [
                "arn:aws:s3:::<S3_BUCKET_B>",
                "arn:aws:s3:::<S3_BUCKET_B>/*"
            ]
        }
    ]
}
```

#### Update KMS key policy
Ensure the **KMS key policy** to be updated.
```json
{
    "Version": "2012-10-17",
    "Id": "my-key-policy",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<AWS_ID_B>:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow an external account to use this KMS key",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<AWS_ID_A>:root",
                    "arn:aws:iam::<AWS_ID_A>:role/service-role/<IAM_ROLE_REPLICATION_A>"
                ]
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        }
    ]
}
```

## Create a replication rule
From the source S3 bucket, create an **S3 replication rule**.
![create-s3-replication-rule](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ddq6la3ah76ozon8zopy.png)

Choose the *rule scope to all objects* and specify destination bucket with AWS account ID.
![specify-destination-bucket](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zq4j5dp05ys2rcknbjet.png)

Choose the option to change the bucket ownership and specify the **IAM role** that we have created earlier.
![specify-replication-role](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nujzv08552zecipn346p.png)

Enter the *ARN* of the destination **AWS KMS** encryption key to be used.
![specify-encryption-key](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4i97b0a4no96zon4xxk7.png)

For destination classes and replication options, keep them as optional if you don't have any related requirements. Then, choose option to *replicate existing objects*. This will direct you to a new page which is the next step below.

## Create Batch Operations job
For *Job run options*, we choose **Automatically run the job when it's ready** or it is up to you.

In order to gain visibility of replication process, we enable reports generation option and store them onto S3 bucket.
![generate-replication-reports](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y5ucu8gnelig4i8u6bdj.png)

For *Permissions*, you can reuse the earlier **IAM Role**.

Once you clicked the `Save` button, it will create a new **Job ID** and based on my job run option earlier, it automatically runs so we will wait for the result.
![new-batch-operations-job](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ixkdtrxs84fn1cz7pgbr.png)

> If it is `failed`, you can check the job status or report for further details. In my case, my replication **IAM role** does not have enough permissions to generate reports to the specified bucket.

![job-status-failed](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n6t1pj8xrvibclbagaba.png)

> If it is `progresses`, you will observe the % of objects that have been replicated or objects have been failed to replicate.

## References
1. https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html
2. https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-batch-replication-batch.html
3. https://aws.amazon.com/blogs/aws/new-replicate-existing-objects-with-amazon-s3-batch-replication/