Part 5: Plan for efficient data storage and search performance in Elasticsearch
Tags: elasticsearch, beginners, node, database
Published: 2022-09-07 15:14:39

[Table of Content](https://dev.to/lisahjung/beginners-guide-to-building-a-full-stack-app-nodejs-react-with-elasticsearch-5347) | [Read Next: Part 6 - Set up Elasticsearch for data transformation and data ingestion](https://dev.to/lisahjung/part-6-set-up-elasticsearch-for-data-transformation-and-data-ingestion-4m2c)

Now that our server is connected to Elasticsearch hosted on Elastic Cloud, it is time to think about the data that we want to ingest into Elasticsearch. 

###Review from [part 1](https://dev.to/lisahjung/part-1-what-are-we-building-and-learning-about-in-season-2-md4)

For our project, we will be retrieving data from the USGS API and ingesting it into Elasticsearch.

![ezgif com-gif-maker (9)](https://user-images.githubusercontent.com/60980933/184019969-c048b95f-1362-4367-9af8-40cfe65f5746.gif)

Before ingesting data into Elasticsearch, assessing the data structure and planning for desired mapping are essential in ensuring efficient storage and search of data.

In this blog, we will assess the data to:
- determine what data we need 
- discover if we need to transform the data to fit our use case
- decide on the desired mapping for efficient storage and search of data
 
### Resources
*Would you rather watch a video to learn this content? Click on the link below!*

- [Episode 5: Plan for efficient data storage and search performance in Elasticsearch](https://www.youtube.com/watch?v=hwnvomBSHPY)

*The following USGS Earthquake API home page contains the link to the API that contains [all earthquake data from the past 30 Days](https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson)(red arrow). We will be retrieving data from this API!* 

- [USGS Earthquake API Home Page](https://earthquake.usgs.gov/earthquakes/feed/v1.0/geojson.php) 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zu1fmq7nm4dwnlkrta52.png)

*The following documentation contains explanations of the terms(field names) included in the API. Refer to this if you need clarifications on the acronyms or want details about certain fields.* 

- [Earthquake Catalog Documentation](https://earthquake.usgs.gov/data/comcat/index.php#nst) 

*The current blog builds upon concepts covered in the blog and video above. Refer to these links if some of the jargons don't make sense to you or if you need a little refresher.*

- [Blog: Understanding mapping with Elasticsearch and Kibana](https://dev.to/lisahjung/beginner-s-guide-understanding-mapping-with-elasticsearch-and-kibana-3646)

- [Video: Beginner's Crash Course to Elastic Stack Part 5](https://ela.st/bcc-part5)

*We will be referring to the following Elasticsearch documentations on field data types and numeric field types while coming up with the desired mapping for our data.* 

- [Elasticsearch documentation on field data types](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html)

- [Elasticsearch documentation on numeric field types](https://www.elastic.co/guide/en/elasticsearch/reference/current/number.html)

You will find these resources helpful when assigning field types to your data. 

Now that you have access to all resources, let's get to work! 

**Step 1: Review the final outcome**

Before we examine the data structure, let's review the final outcome of the app we are building. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ymqw8b0tk1pllmn7yidy.png)
 
Our app allows the user to search for earthquakes using the following criteria. 
 
1. type
2. magnitude 
3. location
4. date range

When the user clicks on the search button, the search results are displayed as cards.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/518nqvd380gv6fzk0uhx.png)
  
Each card displays following information about one earthquake. 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3u3o4a7fnovrn7mc06hu.png)
 
The information outlined above is what we need from the USGS API. 

We will store this information in Elasticsearch in the form of documents. Each document will contain information about one earthquake. 

**Step 2: Examine the data structure of earthquake API**

*Go to the [USGS Earthquake Home Page](https://earthquake.usgs.gov/earthquakes/feed/v1.0/geojson.php) and scroll down to the `Output` section(red box).* 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7l1u28mtk8fo1i3xkusl.png)

The output shows the data structure of a typical earthquake object found in the API. 

*Scroll down on the output to view the field `features`(green box).*

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m7j2pzu3n2hc2dbpfmf0.png)

The `features` field contains an array of objects. Each object contains information about one earthquake. It lists the name and data type of fields contained in a typical earthquake object. 

In step 1, we determined what information we need to store in Elasticsearch. 

You will see that the object `properties`(orange box) and `geometry`(blue box) contain the information we seek(underlined in pink).

If you need clarification on the acronyms or want details about certain fields, refer to the [Earthquake Catalog Documentation](https://earthquake.usgs.gov/data/comcat/index.php#nst).

Take a look at the image below. 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xxdpx1d9gz73w49hti0o.png)
 
You will see that the earthquake object from the API contains more information than we need.

To save storage, we will only index the fields `mag`, `place`, `time`, `url`, `sig`(significance), `type`, and `coordinates` array which includes `longitude`, `latitude`, and `depth` in that order. 

The API fields that correspond to the info on the card are highlighted in same colors.

When you compare the two, for majority of these, the info from the API is identical to the info displayed in the search results.

However, there are a few that are not the same. 

**Step 3: Determine whether we need to transform any data before ingestion**

*Data transformation task 1: time* 
Let's compare the field `time` in the API earthquake object with the field `Time` on our card. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xh0bunmxcwpoflbf24ui.png)
   
You will see that the field `time` in API earthquake object is in Unix epoch time(1651522073266). However, `Time` on the card displays a human readable timestamp(2022-05-02T20:07:53.266Z).

To achieve this outcome, we will convert the Unix epoch time in the API field `time` to human readable timestamp. Then, store transformed information in the field `@timestamp` in Elasticsearch(more on that later). 

*Data transformation task 2: coordinates* 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n5rsw2szavegv9xi2fji.png)
   
You will  see that the search results card has fields called `latitude`, `longitude`, and `depth`(pink boxes).

In the API earthquake object, the values of these fields are contained in an array called `coordinates`(pink box) and are not labeled as such. 

To make it easier to identify this information, we will create fields for `lat`(latitude), `lon`(longitude), and `depth` in Elasticsearch. 

Then, we will store the corresponding info from the API's `coordinates` array into its respective fields. 

Then, we will store `lat` and `lon` into an object called `coordinates` to keep this information together as a pair as shown below. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dw2bibcv3wzgnp74ze9y.png)

*Note that in Elasticsearch, the abbreviation `lat` should be used for latitude and `lon` should be used for longitude.* 
 
**Step 4: Determine the desired mapping**

We just figured out how we should transform the retrieved data from the API before we store it in Elasticsearch. 

Next, we will figure out how to store this data using the smallest disk space while maximizing our search performance. 

This is when customizing your `mapping` come into play! 

`Mapping` defines how a document and its fields are indexed and stored. 

It does that by assigning types to fields being indexed. Depending on the assigned field type, each field is indexed and primed for different types of requests(full text search, exact searches, aggregations, sorting & etc). 

This is why `mapping` plays an important role in how Elasticsearch stores and searches for data.

We will be glossing over a lot of the concepts we have covered in [Understanding mapping with Elasticsearch and Kibana](https://dev.to/lisahjung/beginner-s-guide-understanding-mapping-with-elasticsearch-and-kibana-3646). Check out this resource if you need more in depth review of these concepts! 

Take a look at the table below. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f33sxfadnx4ezsyit2cv.png)
 
To make this process easier, I have created a table that displays(pink box) the name of the field, description of the field, typical values contained in the field, the purpose this field will serve, and the desired mapping I have chosen for the field.
 
Let's go through each field and determine why I chose certain field type for each field.  

When you take a look at the `Typical values` column in the table, you will see that these fields either contain numeric(green box) or string(orange box) values. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eo7v24oquskys2b0n9h4.png)
  
Let's take a look at the fields that contain numeric values first. 
  
###Numeric field types

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e41nl4u60nncae1omayr.png)

There are various field types that can be assigned to numeric fields. The field type you choose will depend on the value type a field contains and for what purpose you will be using the field.  

**`Coordinates`**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8it1yeq2oufabnxk4hsm.png)
 
In step 3, we decided that we want to store the fields `lat`(latitude) and `lon`(longitude) within an object called `coordinates`. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mvn9fa2c4lwpzrilq56j.png)
 
We are planning on using the fields `lat` and `lon` for two tasks:
1. to display this information on a search results card
2. to use these coordinates to mark the location of earthquakes on a heat map([part 10](https://dev.to/lisahjung/part-10-visualize-data-using-kibana-lens-2khe)).

The second task requires running geo-based queries. 

Therefore, the field `coordinates` should be typed as `geo_point` in order for this to work. 

If you want to learn more about the field type `geo_point`, check out this [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html).

**`depth` and `mag`**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hxq0l216fuf7b13kdpj8.png)
 
The typical values for the fields `depth` and `mag` are in decimals. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ue9frvn5qbx9wvb3mw3n.png)
  
As the values of these fields will only be displayed in the search result cards, we will assign the field type `float` for these fields. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jvpl0s4jzp47pqmzx2ho.png)
 
**`sig`**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c4c7u6xcykjxpv3oe5oy.png)
 
When you look at the typical values for the field `sig`, it consists of integers that range between 0 to 1000. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hiy74hke9dw0uxwv48rv.png)

The value of this field will be displayed in the search results card. 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jo2a9lovg0zn80rle4p0.png)

We want to choose the field type that will store integers using the smallest disk space.  

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/37i2udajlte5j0kwamwi.png)
    
If you take a look at the documentation for [numeric field types](https://www.elastic.co/guide/en/elasticsearch/reference/current/number.html), the field type that will allow us to store this data using the smallest disk space is `short`. 

**`time`**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gmbqrmlmsmmrgqoiqqzb.png)

The value of this field will be displayed in the search results card. It will also be used to search for earthquakes that have occurred within a chosen date range.  

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e3f68iviouwrmyvevdml.png)
 
To do so, we will run `range` queries on this field so we will assign the field type `date`.

###Text field types

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z3i6xcuiz5iyv1elyls9.png)
 
Let's go over the fields that contain string data types(`place`, `type`, and `url`). 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/51v7unloxe19cr394tdi.png)
     
By default, every field that contains string data type gets mapped twice as a `text` field and as a `keyword` multi-field. 

Each field type is primed for different types of requests.

`Text` field type is used for full text search.

`keyword` field type is used for aggregations, sorting, and exact searches. 

In scenarios where you do not need both field types, the default setting is wasteful. It will slow down indexing and use up more disk space.

When deciding on a string field type, make sure you know for what purpose this field will be serving so you can choose the correct field type.

**`place`**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/onfjy9qbjkylxcn4pwcz.png)
 
The field `place` will be used for three purposes.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ersl8t87piv4zjwq03wc.png)
 
- The value of this field will be displayed on the search results card. 
- The field will be used for full text search(when a user types in the location, the user input will be searched against this field to retrieve relevant data)
- Aggregation will be performed on this field to yield a table of 10 locations with the highest frequency of earthquakes ([part 10](https://dev.to/lisahjung/part-10-visualize-data-using-kibana-lens-2khe))
 
Since we need to run both full text search and aggregations on the field `place`, we will assign both field types `text` and `keyword`.

**`type`**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z7zrbp7pbpcz6vft7xrl.png)
 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wd19smxhfw6a7eipkozm.png)

The value of this field will be displayed on the search results card. 

This field will also be used for exact searches. 
 
When a user searches for a specific type of quake, the user input is searched against the field `type` to retrieve relevant search results. 

The user is prompted to select a type from a list of options. Therefore, we can perform exact searches on this field and will map this field as `keyword`.

`url`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a7pxpi6ip5d85yx745aa.png)
 
The value of the field `url` is only displayed on a card and it is not used for search. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wk81ly4bd7g8eq0tphcw.png)

Therefore, there is no need to create search data structures(inverted index or doc values) for this field so we will disable this field(`enabled:false`). 

**Summary**
In this blog, we figured out:
1. how we want to transform the retrieved data before ingesting it into Elasticsearch
2. the desired mapping to efficiently store and search data in Elasticsearch

Move on to [Part 6](https://dev.to/lisahjung/part-6-set-up-elasticsearch-for-data-transformation-and-data-ingestion-4m2c) to set up Elasticsearch for data transformation and data ingestion!