OpenSearch for humans
Tags: aws, tutorial, learning, elasticsearch
Published: 2024-02-23 21:34:33

## A bit of history
In 2021 Elasticsearch and Kibana (starting with version 7.1) were being relicensed under ELv2 (Elastic License ) and [SSPL](https://opensource.org/blog/the-sspl-is-not-an-open-source-license) (Server Side Public License).

This change triggered a response from [Amazon Web Services](https://aws.amazon.com/opensearch-service/), which offered [OpenSearch](https://github.com/opensearch-project) (data store and search engine) and [OpenSearch Dashboards](https://opensearch.org/docs/latest/dashboards/) (visualization and user interface) as Apache2.0 licensed open-source projects.

At its core OpenSearch is a distributed search and analytics engine. Distributed means that OpenSearch is backed up by a cluster comprised of a collection of nodes (cluster-manager master and data).


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aiecjgmgznoo7u7yhtkt.png)

Getting acquainted with the terminology is mandatory when starting to work with OpenSearch.

## What is an index?
Indices are the largest unit of data, OpenSearch organizes data into indices, and an index is a collection of JSON **documents**.

## Sharding an index is useful
Data in an index is partitioned across **shards**, the reasoning being that an index might be too large to fit on a single disk, but shards being smaller can be distributed and allocated across different nodes as needed.

Searches can be run in parallel across different shards speeding up the query processing. When creating an index, you can define how many shards you want. Each shard is an **independent Lucene index** that can be hosted anywhere in the cluster.

## Replication is good
OpenSearch provides a fail-safe mechanism for events like node crashes, by having two types of shards: **primary** and **replica**. By default, OpenSearch creates a replica shard for each primary shard.

Each shard can have several replicas (that can be configured at index creation or changed later on) and to ensure high availability the replicas are NOT PLACED on the same node as the original/primary shards. The primary shard is the main shard that handles the indexing of documents and can also handle the processing of queries.

## Cluster health
OpenSearch cluster health can be either green, yellow, or red.

-  **Green**: all primary shards and their replicas are allocated to nodes

- **Yellow**: all primary shards are allocated to nodes, but some replicas aren’t.

- **Red**: at least one primary shard and its replicas are not allocated to any node

## Rule of thumb

- **Shard size matters** because it impacts both **search latency** and **write performance**, too many small shards will exhaust the memory (JVM Heap) too few large shards prevent OpenSearch from properly distributing requests, a good rule of thumb is to keep shard size between 10–50 GB.

- There is a **limit** on how many **shards** a **node** can **handle**, it’s useful to check how many shards a node can accommodate and search, by inspecting [cluster settings](https://opensearch.org/docs/latest/install-and-configure/configuring-opensearch/cluster-settings/).

> `cluster.max_shards_per_node` (Integer): Limits the total number of primary and replica shards for the cluster. The limit is calculated as follows: `cluster.max_shards_per_node` multiplied by the number of non-frozen data nodes. Shards for closed indexes do not count toward this limit. The default is `1000`

- When adding or searching data within an index, that index it’s in an open state, the _**longer you keep the indices open the more shards you use**_.
- On the other hand, a [closed](https://opensearch.org/docs/latest/api-reference/index-apis/close-index/) index is not accessible for read or write operations and does not consume compute resources, however, it still occupies disk space, one way to reduce the disk space is to [shrink](https://opensearch.org/docs/latest/api-reference/index-apis/shrink-index/) the index (basically moving the data of an existing index into a new index with fewer primary shards).
- Very important red indexes cause red shards, and red shards cause red clusters.
- **Unassigned** shards **cannot** be **deleted**, an unassigned shard is not a corrupted shard, but a **missing replica**.