Performing CRUD operations with Elasticsearch & Kibana
Tags: beginners, elasticsearch, datascience, node
Published: 2020-12-23 00:52:54

In the [Beginner's Guide to Elasticsearch](https://dev.to/lisahjung/beginner-s-guide-to-elasticsearch-4j2k), we have mastered the basic architecture of Elasticsearch. We also got a chance to download and run Elasticsearch and Kibana on our local machine as well. 

Now that we have mastered the basics of these two products, it is time to get hands-on experience with Elasticsearch and Kibana! 

![image](https://media.giphy.com/media/LNE01Z89j9gis/giphy.gif)

Elasticsearch is a powerful search and analytics engine. It is known as the heart of the Elastic Stack. Elasticsearch stores, searches, and analyzes your data. 

Elasticsearch stores data as documents. This blog will teach you how to perform **CRUD** operations using Elasticsearch and Kibana. 

By the end of this blog, you will be able to **C**reate, **R**ead, **U**pdate, and **D**elete a document from Elasticsearch. 

Let's get started! 
### Resources
If this is your first time learning about Elasticsearch, check out the following resources before getting started with this blog. This blog builds on the content shared in these resources. 

1) [Beginner's Guide to Elasticsearch](https://dev.to/lisahjung/beginner-s-guide-to-elasticsearch-4j2k)

2) [Instructions](https://dev.to/elastic/downloading-elasticsearch-and-kibana-macos-linux-and-windows-1mmo) for downloading and running Elasticsearch and Kibana

3) [Video](https://www.youtube.com/watch?v=gS_nHTWZEJ8&t=8s) of Beginner's Crash Course to the Elastic Stack

For those of you who prefer to learn via video, I have created a recording that covers the content shared in the [Beginner's Guide to Elasticsearch](https://dev.to/lisahjung/beginner-s-guide-to-elasticsearch-4j2k) in detail. 

This video also contains a hands-on lab section where I cover how to perform CRUD operations with Elasticsearch and Kibana(25:03). 

Be sure to check out the description of this video as it includes the timestamps of each sections as well as resources shared during the workshop. 

### Prerequisite Work
Follow these [instructions](https://dev.to/elastic/downloading-elasticsearch-and-kibana-macos-linux-and-windows-1mmo) to complete the following steps.

1. Download Elasticsearch and Kibana. 

2. Run Elasticsearch and Kibana.

3. Load Kibana console on a browser.
You should see the following on your screen. 
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/p61s5ya2d537hi8fx3w0.png)

To follow along, have two windows open side by side. As shown below, you will have the Kibana console open on the left and this blog open on the right. 

![image](https://user-images.githubusercontent.com/60980933/102933003-9c9bc180-445e-11eb-8ad1-8b9649d8a9c0.png)

You are now ready to perform **CRUD** operations! Using Kibana, we will send requests to Elasticsearch to **C**reate, **R**ead, **U**pdate, and **D**elete a document.

### Performing CRUD operations

#### Background info
In Elasticsearch, data is stored as documents. A document is a JSON object that stores whatever data you want to store in Elasticsearch. Each document has a unique ID.

Let's say that you are storing documents about goods sold at an online grocery store. A document for one grocery item would look like the following.

![image](https://user-images.githubusercontent.com/60980933/102928166-80dfed80-4455-11eb-8a0e-35a307374631.png)

In a JSON object, you have a list of fields or key value pairs. For example, it has the name of the product, category it belongs in, its brand and price.

Things are much easier to find when you group them in a logical manner! 

Documents that are logically related to each other are grouped into an index. For example, documents of carrots and clementines would be grouped under the produce index. Documents of Malbec and IPAs would be grouped under the wine and beer index. 

![image](https://user-images.githubusercontent.com/60980933/102933233-221f7180-445f-11eb-9b99-8135f6540760.png)

For this exercise, we are going to perform CRUD operations on documents about our favorite candy. 

Let's start by creating an index!

## C - Create
### Create an index
Syntax:
```javascript
PUT Name-of-the-Index
```
To create an index, we use the HTTP verb PUT followed by the name of the index. 

Since we are indexing documents about our favorite candy, we will name our index `favorite_candy` as shown in the following example. 

Example:
```javascript
PUT favorite_candy
```
Go to your Kibana console. 

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/p61s5ya2d537hi8fx3w0.png)

The console is divided into two panels. The left panel is where we send requests to Elasticsearch. The right panel is where we get a response back from Elasticsearch.

Delete the default query. Copy and paste the following request into the left panel(line 1).

```javascript
PUT favorite_candy
```

![image](https://user-images.githubusercontent.com/60980933/102928599-37dc6900-4456-11eb-9601-08d872058746.png)

In order to send the request, the first line of the request must always be selected. We do that by clicking on the request, which should put a gray bar over the request. Click on the arrow(blue box) to send the request. 

In the right panel of the Kibana console, you will see the following response from Elasticsearch.

![image](https://user-images.githubusercontent.com/60980933/102928763-8c7fe400-4456-11eb-9b43-f3cad77205c1.png)

You will see a `200-OK` response returned to you. When you see `"acknowledged": true`(line 2), that means that an index called `favorite_candy`(line 4) has been successfully created. 

#### Index a document

Now that we have an index, let's index some documents! 

We use the word index as both a noun and a verb. When index is used as a verb, it means that we are storing documents in Elasticsearch. 

When indexing a document, either `POST` or `PUT` can be used. 

**1) POST**
You use POST when you want Elasticsearch to autogenerate an id for your document. 

Syntax:
```javascript
POST Name-of-the-Index/_doc
{
  "field": "value"
}
```
We start the request with a POST HTTP verb, then we specify the name of the index, then the document endpoint(_doc), followed by a JSON object with whatever data you want to store in Elasticsearch. 

In our example, I am going to index a document about my favorite candy. When this syntax is applied to our example, it looks like the following. 

Example:
```javascript
POST favorite_candy/_doc
{
  "first_name": "Lisa",
  "candy": "Sour Skittles"
}
```
This `POST` request directs Elasticsearch to   index the following document(`_doc`) in the `favorite_candy` index. This document has a field named `first_name`, which has a value of `Lisa`. It also has a field named `candy`, which has a value of `Sour Skittles`. Since the verb POST is used, Elasticsearch knows to autogenerate an id for this document.  

Copy and paste this request into the Kibana console(lines 3-7). Make sure the first line(line 3) of the request is selected and send the request. 

![image](https://user-images.githubusercontent.com/60980933/102943043-33bf4400-4474-11eb-9d9e-f49c5c8a9703.png)

You will get the following response from Elasticsearch.

![image](https://user-images.githubusercontent.com/60980933/101933971-2d8ab700-3b9a-11eb-99a4-7d34b9819050.png)

You will see a `201-Created` response(green box). This response states that in our index called `favorite_candy`(line 2), we have a document(`_doc`, line 3) with an `autogenerated id`(line 4) that has been `created`(line 6). 

Great job! You have just indexed your first document!

Earlier, I have mentioned that you can either use POST or PUT verb when indexing a document. 

**2) PUT**
You use the verb PUT when you want to assign a specific id to your document. An instance where you may want to use PUT is when you are indexing data with a natural identifier(i.e. purchase order number, patient id, & etc).

For example, let's say you are indexing patient data where each patient has a unique ID.  At this point, sending POST request is the only way you know how to index documents, which autogenerates document IDs. 

This method will work just fine  until you need to look up patient info by the document ID. 

These are randomly generated IDs(ex. "_id" : "EoZTjHYBnFpZV1kVcyTF") that have no meaning or relations to the data it stores. How will you keep track of these ids of assorted letters and numbers? Will you have to keep track of all of these IDs in addition to the list of patient data containing patient IDs? 
![image](https://media.giphy.com/media/LKTTAzGboJGzC/giphy.gif)

If only there was an easier way to index and find these documents...

There is a better way!

Using PUT, you can assign the patient ID as the ID for the patient's document. This way, you can use the patient ID to find the document that you are looking for! 

If you want to learn more about using POST vs PUT to index documents, check out this [documentation](https://www.elastic.co/guide/en/elasticsearch/guide/current/index-doc.html) from Elastic.

Let's get back to indexing more documents. The following syntax is used when using PUT to index data. 

Syntax:
```javascript
PUT Name-of-the-Index/_doc/id-you-want-to-assign-to-this-document
{
  "field": "value"
}
```
When this syntax is applied to our example, it looks like the following.

Example:
```javascript
PUT favorite_candy/_doc/1
{
  "first_name": "John",
  "candy": "Starburst"
}
```
This request is directing Elasticsearch to `PUT` in `favorite_candy` index the following document(`_doc`), the `JSON object` in the next lines. It also directs Elasticsearch to assign this document an id of `1`. 

Copy and paste this request into the Kibana console(lines 9-13). Make sure to select the first line of the request(line 9) and send the request. 

![image](https://user-images.githubusercontent.com/60980933/102943302-decffd80-4474-11eb-9a9e-360b4d8a4fc9.png)

Elasticsearch will send back the following response.
![image](https://user-images.githubusercontent.com/60980933/102932124-f00d1000-445c-11eb-8354-92718d96cbea.png)

You will see a `201-Created` response(green box). This response states that in the index `favorite_candy`(line 2), a document(`_doc`, line 3) with an assigned id of `1`(line 4) has been `created`(line 6). 

Note the `"_version":1` in line 5. We will go over this in a bit! 

## R - READ
### Read a document 

Now that we have indexed a document, let's send a request to examine the content of the document that has been indexed. 

We use the following syntax to read a document. 

Syntax:
```javascript
GET Name-of-the-Index/_doc/id-of-the-document-you-want-to-retrieve
```
When we apply this syntax to our example, it looks like the following.

Example:
```javascript
GET favorite_candy/_doc/1
```
This request directs Elasticsearch to `GET` from `favorite_candy` index a document(`_doc`) with an id of `1`.

Copy and paste this request into the Kibana console(line 15). Make sure the request is selected and send the request. 

![image](https://user-images.githubusercontent.com/60980933/102943442-46864880-4475-11eb-8f47-312ab321d834.png)

You should see the following response from Elasticsearch.

![image](https://user-images.githubusercontent.com/60980933/101935925-0d102c00-3b9d-11eb-9620-1b642364ef6a.png)

You will see a `200-OK` response(green box). In the JSON object, you will see that in an index called `favorite_candy`(line 2), there is a document(`_doc`, line 3) with an assigned id of `1`(line 4). In the `_source` field(line 9), you will see the content of document 1 (lines 10 and 11). 

As you can see, reading a document is a great way to check whether our CRUD operations have been successfully completed. 

*What do you think would happen if we accidentally index another document with an ID that already exists?*

![image](https://media.giphy.com/media/lKXEBR8m1jWso/giphy.gif)

Let's find out. 

Copy and paste the following into the Kibana console(lines 15-19), right below the request for indexing document 1 that contains John's information.

```javascript
PUT favorite_candy/_doc/1
{
  "first_name": "Sally",
  "candy": "Snickers"
}
```
![image](https://user-images.githubusercontent.com/60980933/102943581-b98fbf00-4475-11eb-81fc-679f6ec85b28.png)

Notice that we are indexing Sally's document with an ID that already exists(red boxes)!

Make sure to select the first line(line 15) of this request and send the request.  

You should get the following response from Elasticsearch. 

![image](https://user-images.githubusercontent.com/60980933/102937066-93aeee00-4466-11eb-9d91-fb29f1f9c62d.png)

Notice that we are getting a `200-OK` response instead of `201-OK` response this time. It tells you that a document(`_doc`, line 3) with an id of `1`(line 4) has been `updated`(line 6). 

Note that `version` number has now been changed to 2 (line 5). Why is that happening?

`_version` number signifies the number of times the document has been created, updated, or deleted. Since document 1 has been originally created with John's favorite candy information then accidentally updated with Sally's info, the version number is now 2. 

Let's double check the result of this request by sending a GET request we have just used(line 21). 

![image](https://user-images.githubusercontent.com/60980933/102943713-0ecbd080-4476-11eb-9344-046f7cb2161c.png)

Select and send the GET request. You will see the following response. 

![image](https://user-images.githubusercontent.com/60980933/102937380-3ebfa780-4467-11eb-8949-75f2d88be4c0.png)

If you look at lines 10-11, you will see that John's information has been overwritten by Sally's information! 

Uh oh... That is not good. We do not want to accidentally overwrite an existing document.

To prevent this from happening, we can use the _create endpoint! 

### _create Endpoint
_create endpoint prevents you from overwriting an existing document. When this endpoint is in use, the request to index a document with an existing ID is denied. Elasticsearch will throw an error and reject the request.  

The following syntax is used when _create endpoint is used.

Syntax:
```javascript
PUT Name-of-the-Index/_create/id-you-want-to-assign-to-this-document
{
  "field": "value"
}
```
When this syntax is applied to our example, it looks like the following.

Example:
```javascript
PUT favorite_candy/_create/1
{
  "first_name": "Finn",
  "candy": "Jolly Ranchers"
}
```
This request directs Elasticsearch to `PUT` in index `favorite_candy` the following `JSON object` and assign it an id of `1`. HOWEVER(`_create`), if a document with ID of 1 exists, then do not index this document. Throw an error message instead. 

Copy and paste the request into the Kibana console(lines 23-27), right below the GET request. Make sure the first line of the request(line 23) is selected and send the request. 

![image](https://user-images.githubusercontent.com/60980933/102943859-78e47580-4476-11eb-9edd-7073b6e1b4ab.png)

You should see the following response from Elasticsearch.

![image](https://user-images.githubusercontent.com/60980933/101937947-cf60d280-3b9f-11eb-8341-316ec4a69b35.png)

Elasticsearch sends a `409-Conflict` error. It states the reason as `document already exists`(lines 6-7).

Let's send a GET request used in the previous example to make sure nothing has been changed. 

Select the GET request we used earlier(line 21) and send the request. 

![image](https://user-images.githubusercontent.com/60980933/102943922-a3363300-4476-11eb-8abe-9df3d5dd21a2.png)

You will see that Sally's information is still there (lines 10-11) and nothing has changed!

![image](https://user-images.githubusercontent.com/60980933/102937380-3ebfa780-4467-11eb-8949-75f2d88be4c0.png)

As you can see, the `_create` endpoint provides a safeguard for you so you do not accidentally overwrite an existing document.

Let's move on to update! 

## U - UPDATE
### Update a document

There will be times when you will want to update an existing document. For example, let's say Sally originally liked Snickers but her favorite candy now is M&Ms.

You will use the following syntax to update a field of a document.

Syntax:
```javascript
POST Name-of-the-Index/_update/id-of-the-document-you-want-to-update
{
  "doc": {
    "field1": "value",
    "field2": "value",
  }
} 
```
In the JSON object, make sure to add `"doc"`. This tells Elasticsearch that only the fields specified in the nested curly brackets are meant to be updated. 
 
When this syntax is applied to our example, it looks like the following.

Example: 
```javascript
POST favorite_candy/_update/1
{
  "doc": {
    "candy": "M&M's"
  }
}
```
This `POST` request directs Elasticsearch to  update(`_update`) a document with an id of `1` in the `favorite_candy` index. By including `"doc"` in the JSON object, it also clarifies that only the field and value specified in nested curly brackets should be updated. The rest of the information in the original document should be kept in tact. 

Copy and paste this request into the Kibana console(lines 21-26), right below the request to index Sally's document. Select the first line(line 21) of the request and send the request. 

![image](https://user-images.githubusercontent.com/60980933/102944253-82221200-4477-11eb-99ef-e5410b30cd40.png)

You should see the following response from Elasticsearch.

![image](https://user-images.githubusercontent.com/60980933/101938690-05528680-3ba1-11eb-8eec-8e2dce678405.png)

You will see a `200-OK` response from Elasticsearch. The response states that a document(`_doc`, line 3) with an id of `1`(line 4) has been `updated`(line 6). Notice that `_version` number has been changed to 3(line 5). 

This makes sense because `_version` keeps track of number of times a document has been created, updated, and deleted. Document 1 has been originally created with John's information(_version: 1). It was accidentally overwritten by Sally's information(_version: 2). Then, we intentionally updated the field candy with M&M's(_version:3). 

Let's send a GET request to check the content of document 1. Select the previous GET request we have used and send the request (line 28). 

![image](https://user-images.githubusercontent.com/60980933/102944344-b8f82800-4477-11eb-8d2d-b1b22c9bb670.png)

You will see that the field candy has been updated with M&M's(neon green box)!

![image](https://user-images.githubusercontent.com/60980933/102939289-3ec1a680-446b-11eb-9f33-86b5908045b8.png)

Last but not least, let's delete a document! 

## D- DELETE
### Delete a document

The following syntax is used to delete a document.

Syntax:
```javascript
DELETE Name-of-the-Index/_doc/id-of-the-document-you-want-to-delete
```
When we apply the syntax to our example, it looks like the following. 

Example:
```javascript
DELETE favorite_candy/_doc/1
```
This request directs Elasticsearch to `DELETE` from `favorite_candy` index a document(`_doc`) with an ID of `1`. 

Copy and paste this request into the Kibana console(line 28), right below the update candy field request. 

![image](https://user-images.githubusercontent.com/60980933/102944430-fceb2d00-4477-11eb-957e-a1e4c209de9c.png)

You will see the following response from Elasticsearch.

![image](https://user-images.githubusercontent.com/60980933/101939174-dab4fd80-3ba1-11eb-93fe-de682853bae4.png)

You will see a `200-OK` response that states that document(`_doc`, line 3) with an id of `1`(line 4) has been `deleted`(line 6). The version number is now `4`(line 5). 

If you send a GET request for document 1 again, you will get a 404 error message because document 1 no longer exists! 

![image](https://user-images.githubusercontent.com/60980933/102939731-0bcbe280-446c-11eb-8cb7-1b895612c61f.png)

There you have it. You are now the master of performing CRUD operations with Elasticsearch and Kibana. 
![image](https://media.giphy.com/media/dmypi9dyBfmOQ/giphy.gif)

Put your newly learned skills to use and try the following exercise on your own! 

### Take Home Assignment
1. Create an index called places.
2. Pick five of the places you want to visit after the pandemic is over. For each place, index a document containing the name and the country. 
3. Read(GET) each document to check the content of the document.
4. Update a field of a document.
5. Read(GET) the updated document to ensure that the field has been updated.
5. Delete a document of one place.
6. Copy and paste the following request to return all documents from the places index. 
This is a great way to check whether all the CRUD operations you have performed thus far have worked!
```
GET places/_search
{
  "query": {
    "match_all": {}
  }
}
```