S3 Snapshots for Elasticsearch via Kibana
Tags: indexing, elasticsearch
Published: 2021-08-29 11:19:49

---
title: S3 Snapshots for Elasticsearch via Kibana
published: true
date: 2021-08-29 10:51:11 UTC
tags: indexing,elasticsearch
canonical_url: https://adithya.dev/s3-snapshots-for-elasticsearch-via-kibana/
---

 ![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/Capture.PNG)

I'm terrified of racking up big bills with AWS so I don't work a lot with it, aside from [Amazon Simple Email Service](https://aws.amazon.com/ses/) (SES). For my ElasticSearch 7.14 cluster that I run alongside with Kibana, I wanted to setup [Snapshot and Restore](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html).

My understanding was that with Snapshots, you obtain a copy of your cluster data at that point and you can restore it when you would like to. I already had an AWS account setup before starting with this. For this, you needed to setup a storage repository and then configure via Kibana.

This is how I went about it.

- Create a S3 Bucket (this is the storage repository we will use with Elasticsearch)
- Create an AWS policy to allow the IAM User to access the S3 Bucket
- Create an IAM User and attach the policy we created
- Install the S3 Repository Plugin
- Register the repository with Kibana
- Setup an automatic policy with Kibana / Elasticsearch so it backs up data via cron

* * *

### Create a S3 Bucket

- Open AWS Console  -> [S3](https://s3.console.aws.amazon.com/s3/) -> Create Bucket
- Enter a bucket name: **elasticsearch-s3-bucket-snapshot** , other defaults are good - make sure you select " **Block all public access**" and then go ahead with creating the bucket (or saving)

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-1.png)<figcaption>Enter a bucket name: <strong>elasticsearch-s3-bucket-snapshot</strong>, other defaults are good - make sure you select "<strong>Block all public access</strong>" </figcaption>

### Create an AWS policy

- Open AWS Console -> [IAM](https://console.aws.amazon.com/iam) -> Policies 
- Click on "Create Policy"
- Switch to JSON editor and paste the following policy (that Elasticsearch [recommends for S3 permissions](https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-s3-repository.html#repository-s3-permissions))

```
{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

```

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-2.png)

- Click on "Next: Tags", tags can be default so click next again
- Enter a policy name **elasticsearch-s3-policy-snapshot** and hit " **Create policy**"  

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-3.png)<figcaption>Enter a policy name <strong>elasticsearch-s3-policy-snapshot</strong> and hit "<strong>Create policy</strong>"</figcaption>

### Create an IAM User

- Open AWS Console -> [IAM](https://console.aws.amazon.com/iam)-> Access Management -> Users
- Click on "Add Users"
- Enter an user name elasticsearch-s3-user (customizable), check **Programmatic access** and click **Next: Permissions**  

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-4.png)<figcaption>Enter an user name elasticsearch-s3-user (customizable), check <strong>Programmatic access </strong>and click Next: Permissions</figcaption>

- Switch to " **Attach existing policies directly**" , search and select " **elasticsearch-s3-policy-snapshot"** 

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-5.png)

- Click " **Next: Tags**", defaults are fine. Review the details and click Create User.

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-6.png)<figcaption>Review screen for Elasticsearch IAM user</figcaption>

- This will generate a "Access Key ID" and "Secret access key" - both of which we will need soon to setup repository in Kibana, so keep them handy. There's a CSV option to download.

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-7.png)<figcaption>Credentials for IAM User</figcaption>

### Install the S3 Repository Plugin

- On your machine or server, go to where the Elasticsearch bin folder is. Mine was at /usr/share/elasticsearch
- Run `sudo bin/elasticsearch-plugin install repository-s3`, answer with y or Y when asked about permissions
- **Restart your cluster!**
- We will [configure the client settins for the plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-s3-client.html). Using [elasticsearch-keystore](https://www.elastic.co/guide/en/elasticsearch/reference/7.14/elasticsearch-keystore.html), set the credentials for your IAM User

```
bin/elasticsearch-keystore add s3.client.default.access_key
# enter your "Access Key ID", hit enter
bin/elasticsearch-keystore add s3.client.default.access_key
# enter your "Secret Access Key", hit enter

```

### Register the repository with Kibana

- Go to your Kibana dashboard -> Management -> Stack Management -> Snapshot & Restore -> Repositories ->  " **Register a repository**"

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-8.png)<figcaption>Snapshot &amp; Restore on Kibana</figcaption>

- Name it as **elasticsearch-repository-snapshot** , select AWS S3 as repository type, click Next. If you don't see AWS S3, make sure you restarted your cluster!  

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-9.png)<figcaption>Repository name &amp; Type</figcaption>

- Client name is **default** (this is part of your keystore secrets s3.client. **default**.access\_key) and your S3 Bucket Name. Review other defaults and hit Register.

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-11.png)<figcaption>Repository client name &amp; S3 Bucket name</figcaption>

- Once this is done, it pops up a details screen for the repository. Click on **Verify repository**

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-12.png)<figcaption>Verify Repository to make sure everything is good</figcaption>

### Setup an automatic policy with Kibana

- On the same screen, switch to **Policies** -> Create Policy
<!--kg-card-begin: markdown-->
- Enter these details
  - **elasticsearch-weekly-snapshot** as the name
  - **<elasticsearch-weekly-snapshot-{now/d}>** as the snapshot name, appends the date at the end
  - Select **elasticsearch-repository-snapshot**
  - Set frequency to every week
  - Click Next
<!--kg-card-end: markdown-->

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-13.png)<figcaption>Details for Policy - Cron</figcaption>

- Click Next, and on review screen, hit Create Policy

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-14.png)<figcaption>Review Policy</figcaption>

- You can run the policy and when it's done, you will see the snapshot

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-15.png)<figcaption>Run now for the policy</figcaption>

- You should see something like this once snapshot is complete  

![S3 Snapshots for Elasticsearch via Kibana](https://adithya.dev/content/images/2021/08/image-16.png)<figcaption>snapshot is complete</figcaption>