AWS: S3 Cross-Region Replication configuration and objects DeleteMarkers replication
Tags: aws, replication
Published: 2019-07-17 12:30:17

---
title: AWS: S3 Cross-Region Replication configuration and objects DeleteMarkers replication
published: true
tags: AWS,replication
canonical_url: https://rtfm.co.ua/en/aws-s3-cross-region-replication-configuration-and-objects-deletemarkers-replication/
---

![](https://rtfm.co.ua/wp-content/uploads/2014/11/aws-logo-square-02-e1417012834176.png)At this moment I’m configuring a new CDN for our project.

Will use CloudFront and Cloudflare here so need to create two dedicated buckets with different names – _cdn.cfr.example.com_ => CloudFront and _cdn.cfl.example.com_ => Cloudflare.

To avoid coping data each time to both buckets – an AWS S3 Cross-Region Replication can be used, so data from a _bucket-1_ will be copied to a _bucket-2_.

Documentation – [Cross-Region Replication](https://docs.aws.amazon.com/en_us/AmazonS3/latest/dev/crr.html).

CRR prerequisites:

1. both buckets must have S3 Versioning enabled (see. [Using Versioning](https://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html))
2. both buckets must be created in different regions
3. both buckets must have permissions for replication between them

CRR limitations:

1. already existing files in a source-bucket will not be copied after CRR will be configured (i.e. only new files will be replicated)
2. can not create a replication-chain, i.e. _Bucket-A => Bucket-B => Bucket-C_

### AWS S3 Cross-Region Replication set up

Create two buckets:

[![](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_152839.png "AWS: настройка S3 Cross-Region Replication и удаление файлов при репликации")](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_152839.png)

For both enable _Versioning_:

[![](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_152929.png "AWS: настройка S3 Cross-Region Replication и удаление файлов при репликации")](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_152929.png)

In a source-bucket _bttrm-crr-source_ go to the _Management > Replication_, click on the _Add rule_:

[![](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_153125.png "AWS: настройка S3 Cross-Region Replication и удаление файлов при репликации")](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_153125.png)

Set replicate all from this bucket:

[![](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_153214.png "AWS: настройка S3 Cross-Region Replication и удаление файлов при репликации")](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_153214.png)

Click _Next_, set the receiver-bucket name:

[![](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_153256.png "AWS: настройка S3 Cross-Region Replication и удаление файлов при репликации")](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_153256.png)

Next – permissions and IAM role.

Chose the _Create new IAM role_, set its name:

[![](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_153352.png "AWS: настройка S3 Cross-Region Replication и удаление файлов при репликации")](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_153352.png)

Save:

[![](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_153424.png "AWS: настройка S3 Cross-Region Replication и удаление файлов при репликации")](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_153424.png)

[![](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_153448.png "AWS: настройка S3 Cross-Region Replication и удаление файлов при репликации")](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190715_153448.png)

#### Replication testing

Upload a test file to the source _bttrm-crr-source_ bucket:

```
$ touch testfile.txt
$ aws --profile bm-backend --region us-east-2 s3 cp testfile.txt s3://bttrm-crr-source
$ upload: ./testfile.txt to s3://bttrm-crr-source/testfile.txt
```

Check if it is present:

```
$ aws --profile bm-backend --region us-east-2 s3 ls s3://bttrm-crr-source
2019-07-15 15:36:38          0 testfile.txt
```

And check the destination bucket:

```
$ aws --profile bm-backend --region us-west-1 s3 ls s3://bttrm-crr-dest
2019-07-15 15:36:38          0 testfile.txt
```

The file is present, all good.

### Deleting files from S3 Cross-Region Replication and DeleteMarkers

if you’ll just delete a file from the current source-bucket – it will be deleted there, but not in the destination bucket.

Remove it in the source:

```
$ aws --profile bm-backend --region us-east-2 s3 rm s3://bttrm-crr-source/testfile.txt
delete: s3://bttrm-crr-source/testfile.txt
```

Check it in the destination:

```
$ aws --profile bm-backend --region us-west-1 s3 ls s3://bttrm-crr-dest
2019-07-15 15:36:38          0 testfile.txt
```

The file is still here.

To delete objects from under S3 Versioning – AWS adds a special marker called `DeleteMarker` (see. [Working with Delete Markers](https://docs.aws.amazon.com/en_us/AmazonS3/latest/dev/DeleteMarker.html)):

```
$ aws --profile bm-backend --region us-west-1 s3api list-object-versions --bucket bttrm-crr-source
{
  "Versions": [
    {
      "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
      "Size": 0,
      "StorageClass": "STANDARD",
      "Key": "testfile.txt",
      "VersionId": "Qc5.Z2XumG6uilJ99XcHJ1vIfzMFcx6C",
      "IsLatest": false,
      "LastModified": "2019-07-15T12:36:38.000Z",
      "Owner": {
      "DisplayName": "example.aws",
      "ID": "9365528c1a92cd01abba171eb7b95c352a330a40ceb28b420c9462fac5891bd2"
      }
    }
  ],
  "DeleteMarkers": [
    {
      "Owner": {
      "DisplayName": "example.aws",
      "ID": "9365528c1a92cd01abba171eb7b95c352a330a40ceb28b420c9462fac5891bd2"
},
      "Key": "testfile.txt",
      "VersionId": "PIr6ZOPcdi9oFP8y0rMXkRhrFuKpDQ4P",
     "IsLatest": true,
     "LastModified": "2019-07-15T12:39:24.000Z"
    }
  ]
}
```

But this marker will not be copied to the destination bucket, see the [What Does Amazon S3 Replicate](https://docs.aws.amazon.com/en_us/AmazonS3/latest/dev/crr-what-is-isnot-replicated.html):

- If you make a DELETE request without specifying an object version ID, Amazon S3 adds a delete marker. Amazon S3 deals with the delete marker as follows:

  - If you are using the latest version of the replication configuration, that is, you specify the `Filter` element in a replication configuration rule, Amazon S3 does not replicate the delete marker.
  - If you don’t specify the `Filter` element, Amazon S3 assumes that the replication configuration is a prior version V1. In the earlier version, Amazon S3 handled replication of delete markers differently. For more information, see [Backward Compatibility ](https://docs.aws.amazon.com/en_us/AmazonS3/latest/dev/crr-add-config.html#crr-backward-compat-considerations).

- If you specify an object version ID to delete in a DELETE request, Amazon S3 deletes that object version in the source bucket, but it doesn’t replicate the deletion in the destination bucket. In other words, it doesn’t delete the same object version from the destination bucket. This protects data from malicious deletions.

Check the file in the destination bucket:

```
$ aws --profile bm-backend --region us-west-1 s3api list-object-versions --bucket bttrm-crr-dest
{
  "Versions": [
    {
      "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
      ...
      "Key": "testfile.txt",
      "VersionId": "Qc5.Z2XumG6uilJ99XcHJ1vIfzMFcx6C",
      ...
    }
  ]
}
```

No `DeleteMarkers` present.

Check the replication settings:

```
$ aws --profile bm-backend --region us-west-1 s3api get-bucket-replication --bucket bttrm-crr-source  --query 'ReplicationConfiguration.Rules[*].DeleteMarkerReplication' --output text

Disabled
```

`DeleteMarkerReplication` – `"Status": "Disabled"`, okay.

#### `IAM s3:ReplicateDelete`

At first – check the IAM role used (see the [Setting Up Permissions for Cross-Region Replication](https://docs.aws.amazon.com/en_us/AmazonS3/latest/dev/setting-repl-config-perm-overview.html)).

Find its name:

[![](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190717_110828.png "AWS: настройка S3 Cross-Region Replication и удаление файлов при репликации")](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190717_110828.png)

And find this role in the IAM:

[![](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190717_110922.png "AWS: настройка S3 Cross-Region Replication и удаление файлов при репликации")](https://rtfm.co.ua/wp-content/uploads/2019/07/Screenshot_20190717_110922.png)

Check it – the `"s3:ReplicateDelete"` in the _Actions_ must be present:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:Get*",
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::bttrm-crr-source",
                "arn:aws:s3:::bttrm-crr-source/*"
            ]
        },
        {
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags",
                "s3:GetObjectVersionTagging"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::bttrm-crr-dest/*"
        }
    ]
}
```

#### S3 Bucket Policy

Now – export existing S3 policy:

```
$ aws --profile bm-backend --region us-west-1 s3api get-bucket-replication --bucket bttrm-crr-source > bttrm-crr-source-origin.json
```

Check its content:

```
{
    "ReplicationConfiguration": {
        "Role": "arn:aws:iam::534***385:role/service-role/s3crr_role_for_bttrm-crr-source_to_bttrm-crr-dest",
        "Rules": [
            {
                "ID": "bttrm-s3-crr-replica",
                "Priority": 1,
                "Filter": {},
                "Status": "Enabled",
                "Destination": {
                    "Bucket": "arn:aws:s3:::bttrm-crr-dest"
                },
                "DeleteMarkerReplication": {
                    "Status": "Disabled"
                }
            }
        ]
    }
}
```

Now remove `ReplicationConfiguration`, `"Filter": {}`, `"Priority": 1` `DeleteMarkerReplication` parameters and add `"Prefix": ""`, so this policy will be like Version 1:

```
{
        "Role": "arn:aws:iam::534***385:role/service-role/s3crr_role_for_bttrm-crr-source_to_bttrm-crr-dest",
        "Rules": [
            {
                "ID": "bttrm-s3-crr-replica",
                "Prefix": "",
                "Status": "Enabled",
                "Destination": {
                    "Bucket": "arn:aws:s3:::bttrm-crr-dest"
                }
            }
        ]
}
```

The problem is that CRR config version 2 still can't do `DeleteMarkers` replication at this moment, the AWS Support reply was:

> I did have a look at your blog and the suggestions provided by you seem correct. In fact, this would have been my suggestion as well. The internal team is working on bringing feature of Delete replication in V2 as well but it may take some time for that to happen. It is possible that the way V2 has been implemented in back-end is little different from V1 and using this feature directly may have been causing issues. Unfortunately, I don't have any ETA on when this would finish. Till then, I'm sure your blog would be able to help AWS Users :)

Apply this config:

```
$ aws --profile bm-backend --region us-west-1 s3api put-bucket-replication --bucket bttrm-crr-source --replication-configuration file://bttrm-crr-source.json
```

Upload some new file to the source bucket:

```
$ aws --profile bm-backend --region us-east-2 s3 cp test.file s3://bttrm-crr-source
```

Delete it:

```
$ aws --profile bm-backend --region us-east-2 s3 rm s3://bttrm-crr-source/test.file
```

Check the source bucket now:

```
$ aws --profile bm-backend --region us-west-1 s3api list-object-versions --bucket bttrm-crr-source
{
  "Versions": [
    {
      "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
      ...
      "Key": "test.file",
      "VersionId": "0mkfsrIRdkGCgL.zKgMyMXjo\_UydigIt",
      ...
    },
    {
      "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
      ...
      "Key": "testfile.txt",
      "VersionId": "Qc5.Z2XumG6uilJ99XcHJ1vIfzMFcx6C",
      ...
    }
  ],
  "DeleteMarkers": [
    {
      ...
      "Key": "test.file",
      "VersionId": "1kpKJlgOM7xpFnH4qsEp0EzqhCE1HFMX",
      ...
    },
    {
      ...
      "Key": "testfile.txt",
      "VersionId": "PIr6ZOPcdi9oFP8y0rMXkRhrFuKpDQ4P",
      ...
    }
  ]
}
```

Great – both files found, both have `DeleteMarkers`.

And check the destination bucket:

```
$ aws --profile bm-backend --region us-west-1 s3api list-object-versions --bucket bttrm-crr-dest
{
  "Versions": [
    {
      "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
      ...
      "Key": "test.file",
      "VersionId": "0mkfsrIRdkGCgL.zKgMyMXjo\_UydigIt",
      ...
    },
    {
      "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
      ...
      "Key": "testfile.txt",
      "VersionId": "Qc5.Z2XumG6uilJ99XcHJ1vIfzMFcx6C",
     ...
    }
    ],
    "DeleteMarkers": [
      {
        ...
        "Key": "test.file",
        "VersionId": "1kpKJlgOM7xpFnH4qsEp0EzqhCE1HFMX",
        ...
      }
   ]
}
```

There is still no `DeleteMarker` for the _testfile.txt_ – it’s OK, as it was deleted before we made changes in the `DeleteMarkers` behavior.

But for the _test.file_ – `DeleteMarker` already present, it was replicated from the source, so this file is “deleted” now as well:

```
$ aws --paws --profile bm-backend --region us-west-1 s3 ls s3://bttrm-crr-dest
2019-07-15 15:36:38          0 testfile.txt
```

No _test.file_ found.

Done.

### Similar posts
 
- <small>03/07/2018</small> [AWS: CloudFormation – пример создания S3 корзины, IAM пользователя, Glacier Lifecycle и SNS Notification](https://rtfm.co.ua/aws-cloudformation-primer-sozdaniya-s3-korziny-iam-polzovatelya-glacier-lifecycle-i-sns-notification/) 
- <small>11/27/2018</small> [AWS: CloudFormation – S3 корзина для логов Application Load Balancer](https://rtfm.co.ua/aws-cloudformation-s3-korzina-dlya-logov-application-load-balancer/)
- <small>10/13/2017</small> [AWS: IAM и bash скрипт бекапа MySQL/MariaDB баз в AWS S3](https://rtfm.co.ua/aws-iam-i-bash-skript-bekapa-mysqlmariadb-baz-v-aws-s3/)