Journey from Elastic Cloud to AWS Elastic Search
Tags: elasticsearch, aws
Published: 2021-10-26 12:19:50

This article demonstrates the steps that we should follow when we migrate Elastic search data from Elastic cloud to AWS. 

Let's get started.

> The Elastic Cloud supports **Major.Minor.Patch** (eg: x.y.z) version of Elastic Search, where AWS supports only **Major. Minor** (eg: x.y).
 
When you build the target cluster on AWS, ensure you created the cluster using the next latest Minor version of the source cluster. (If the source cluster version is 7.7.1, the target cluster version must be 7.8).
 
Elastic Search doesn’t allow migrating from the upper version to the lower version.

```
{"error":{"root_cause":[{"type":"snapshot_restore_exception","reason":"[test:cloud-snapshot-2021.06.01-ucssrgkdq9oyci-vbbfyfw/QYwmbA6TTJOXG2T83AtTTw] the snapshot was created with Elasticsearch version
 [7.7.1] which is higher than the version of this node [7.7.0]"}],"type":"snapshot_restore_exception","reason":"[test:cloud-snapshot-2021.06.01-ucssrgkdq9oyci-vbbfyfw/QYwmbA6TTJOXG2T83AtTTw] the snapsh
ot was created with Elasticsearch version [7.7.1] which is higher than the version of this node [7.7.0]"},"status":500}
```
![Elasticsearch to AWS Opensearch](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vfmx8bmumcs7ahbpqpwt.jpeg)

At first, create an S3 bucket in your AWS account as a custom repository to store ES snapshots/backups.

You could find the steps to set up a custom repository on the Elastic Cloud at their site. Therefore, I am not going to describe them here.

Create an EC2 instance which will be used to execute commands during the data restoration. (Even t2.nano is sufficient).

Create two roles such as **Role-S3**, **Role-ES** and two policies such as **Policy-S3** and **Policy-ES** and attach them to the roles respectively. 

###### Policy-S3
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [ 
 "arn:aws:s3:::<es-snapshot-bucket-name>"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [ 
  "arn:aws:s3::: <es-snapshot-bucket-name>/*”
            ]
        }
    ]
}
```
###### Policy-ES
```
{
  		"Version": "2012-10-17",
  		"Statement": [
    		{
      			"Effect": "Allow",
      			"Action": "iam:PassRole",
      			"Resource": "arn:aws:iam::<account_id>:role/Role-S3"  
    		},
    		{
      			"Effect": "Allow",
      			"Action": "es:ESHttpPut",
      			"Resource": [
                         "arn:aws:es:<region>:<account_id>:domain/<es_domain_name>/*"
                  ]
    		}
  		]
	}
```
Attach the role Role-ES to the EC2 instance and log in to the EC2.

Trigger a curl command to AWS ES endpoint to check if the cluster is reachable.
```
curl <es_endpoint>
```

Install the following python libs in the EC2.

```
$ pip3 install boto3
$ pip3 install requests
$ pip3 install requests_aws4auth
$ pip3 install --upgrade request
```

Copy and Paste the given python script, change the appropriate values and run the script.
 
This script will register the S3 bucket as a snapshot repository for the ES cluster that we created.

```
import boto3
import requests
from requests_aws4auth 
import AWS4Auth

host = <es_domain_endpoint/> 	# Enter the ES domain endpoint and trailing ‘/’
region = <region> 		
service = 'es'

credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# Steps to Register snapshot repository
path = '_snapshot/<repository-name>'   # the ES API endpoint
url = host + path
payload = {
  "type": "s3",
  "settings": {
    "bucket": <es_snapshot_bucket_name>, 
    "region": <region>,  # Specify region for S3 bucket. If the S3 bucket is in the us-east-1 region use endpoint
    "endpoint": "s3.amazonaws.com", 
    "role_arn": "arn:aws:iam::<account_id>:role/Role-S3
  }
}
headers = {"Content-Type": "application/json"}
r = requests.put(url, auth=awsauth, json=payload, headers=headers)
print(r.status_code)
print(r.text)
```

The script would print output as shown below if it is executed successfully.
```
200
{"acknowledged":true}
```

List down the snapshots.
```
curl <es_endpoint>/_snapshot/<repository_name>/_all?pretty
```

Restore a snapshot

```
curl -XPOST <es_endpoint>/_snapshot/<repository-name>/<snapshot_name>/_restore
```

Restore a specific index from a snapshot with settings

```
curl -XPOST <es_endpoint>/_snapshot/<repository-name>/<snapshot_name>/_restore -d '{
"indices": "<index_name>",
"index_settings": {
  "index.routing.allocation.require.data": null
  }
}' -H 'Content-Type: application/json'
```

List down the indices.
```
curl -XGET <es_endpoint>/_cat/indices
```

You'll be able to view the indices, if the restoration is successful.

###### ** Key Points **

> Don't restore the .kibana or system indices. It could throw errors. 
> Don't delete .kibana index in target cluster when restoring.