Monitoring Elasticsearch with Sematext
Tags: monitoring, elasticsearch, metrics
Published: 2019-04-24 11:23:33

---
title: Monitoring Elasticsearch with Sematext
published: true
tags: Monitoring,elasticsearch,metrics,monitoring
canonical_url: https://sematext.com/blog/monitoring-elasticsearch-with-sematext/
---

 ![Elasticsearch monitoring with sematext](https://sematext.com/wp-content/uploads/2019/03/New_Elasticsearch-monitoring-part-4-1024x560.jpg)

As shown in [Elasticsearch Key Metrics](https://sematext.com/blog/top-10-elasticsearch-metrics-to-watch/), the setup, tuning, and operations of Elasticsearch require deep insights into the performance metrics such as index rate, query rate, query latency, merge times, and many more. [Sematext](https://sematext.com/) provides an excellent alternative to other Elasticsearch monitoring tools.

Open-source tools to monitor Elasticsearch are free, but your time is not. Relatively speaking, they’re rather expensive. Thus, [Sematext](https://sematext.com/cloud/)[ Cloud](https://sematext.com/cloud/) aims to save you time, effort, and your hair!

## How Sematext Helps you Save Time

Here are a few **things you will**  **NOT****  have to do when using Sematext for Elasticsearch** monitoring:

- figure out which metrics to collect and which ones to ignore
- give metrics meaningful labels
- hunt for metric descriptions in the docs so that you know what each one actually shows
- build charts to group metrics that you really want on the same charts, not several separate charts
- figure out which aggregation to use for each set of metrics (min? max? avg? something else?)
- build dashboards to combine charts with metrics you typically want to see together
- set up basic alert rules

All of the above is not even a complete story. Do you want to collect Elasticsearch logs? How about structuring them? Sematext does all this automatically for you!

In this post, we will look at how Sematext provides more comprehensive – and easy to set up – monitoring for Elasticsearch and other technologies in your infrastructure. By combining events, logs, and metrics together in one integrated [full stack observability platform](https://sematext.com/cloud) and using the Sematext [open-source monitoring agent](https://sematext.com/blog/now-open-source-sematext-monitoring-agent/) and its integrations, which are also open-source, you can monitor your whole infrastructure and apps, not just your Elasticsearch cluster. You can also get deeper visibility into your entire software stack by collecting, processing, and analyzing your logs.

## **Elasticsearch Monitoring**

### **Collecting Elasticsearch Metrics**

Sematext Elasticsearch integration collects [over 100 different Elasticsearch metrics](https://sematext.com/docs/integration/elasticsearch/) for JVM, index performance, cluster health, query performance and more. Sematext maintains and supports [official Elasticsearch monitoring integration](https://github.com/sematext/sematext-agent-integrations/tree/master/clickhouse). Moreover, the Sematext Elasticsearch integration is customizable and open source.

Bottom line: you don’t need to deal with configuring the agent for metrics collection, which is the first huge time saver!

### **Installing Monitoring Agent**

Setting up the monitoring agent takes less than 5 minutes:

1. Create an Elasticsearch App in the [Integrations / Overview](https://apps.sematext.com/ui/monitoring-create) (or [Sematext Cloud Europe](https://apps.eu.sematext.com/ui/monitoring-create)). This will let you install the agent and control access to your monitoring and logs data. The short [What is an App in Sematext Cloud](https://www.youtube.com/watch?v=TR_qXdR8DVk&index=14&list=PLT_fd32OFYpfLBFZz_HiafnqjdlTth1NS) video has more details.
2. Name your Elasticsearch monitoring App and, if you want to collect Elasticsearch logs as well, create a Logs App along the way.
3. Install the Sematext Agent according to the [setup instructions](https://apps.sematext.com/ui/howto/ClickHouse/overview) displayed in the UI.

![app creation](https://sematext.com/wp-content/uploads/2019/04/create-elasticsearch-app.gif)

App creation and setup instructions in Sematext Cloud

For example, on Ubuntu, add Sematext Linux packages with the following command:

```bash
echo "deb http://pub-repo.sematext.com/ubuntu sematext main" | sudo tee /etc/apt/sources.list.d/sematext.list > /dev/null
wget -O - https://pub-repo.sematext.com/ubuntu/sematext.gpg.key | sudo apt-key add -
sudo apt-get updatesudo apt-get install spm-client
```

Then setup Elasticsearch monitoring by providing Elasticsearch server connection details:

```bash
sudo bash /opt/spm/bin/setup-sematext --monitoring-token --app-type elasticsearch \
--agent-type standalone \
--SPM_MONITOR_ES_NODE_HOSTPORT 'localhost:9200' \
--infra-token
```

In case you have Elasticsearch secured with HTTPS and basic authentication, you can add the following parameters to the command:

```bash
--SPM_MONITOR_ES_NODE_BASICAUTH_USERNAME userName
--SPM_MONITOR_ES_NODE_BASICAUTH_PASSWORD passWord
```

In addition, you need to specify the HTTPS as protocol SPM_MONITOR_ES_NODE_HOSTPORT, as shown in the complete setup command:

```bash
sudo bash /opt/spm/bin/setup-sematext --monitoring-token <your-token-goes-here> \
--app-type elasticsearch \
--agent-type standalone \
--SPM_MONITOR_ES_NODE_HOSTPORT 'https://localhost:9200' \
--infra-token <your-token-goes-here> \
--SPM_MONITOR_ES_NODE_BASICAUTH_USERNAME userName \
--SPM_MONITOR_ES_NODE_BASICAUTH_PASSWORD passWord
```

Go grab a drink, but hurry! Elasticsearch metrics will start appearing in your charts in less than a minute.

### **Elasticsearch Monitoring Dashboard**

When you open the Elasticsearch App you find a predefined set of dashboards that organize more than 100 Elasticsearch metrics and general [server monitoring](https://sematext.com/server-monitoring/) in predefined charts grouped into an intuitively organized set of monitoring dashboards:

- Overview with charts for all key Elasticsearch metrics
- Operating System metrics such as CPU, memory, network, disk usage, etc.
- Java Virtual Machine metrics for Garbage collection, JVM Memory, JVM Threads and JVM open files
- Elasticsearch metrics
  - **Cluster Health** :  The number of Elasticsearch nodes and shard status (active, relocating, initializing, ..)
  - **Shard Stats** : The number of shards, shard status per index
  - **Index Stats** : The number of indexed documents, size on disk, indexing rate, merging rate, merged documents
  - **Search** : Request rate, query and fetch latency, realtime-get latency
  - **Thread Pools** : Number of threads per pool, thread pool size
  - **Circuit Breakers** : Field data stats, request size stats
  - **Connections** : Connected sockets, Node-Node-Transport stats, TCP socket and traffic stats

## ![elasticsearch metrics](https://sematext.com/wp-content/uploads/2019/03/elasticsearch-metrics.gif)

[_Elasticsearch key metrics in Sematext Cloud_](http://www.sematext.com/cloud)

### **Setup Alerts for Elasticsearch Metrics** 

To save you time Sematext automatically creates a set of default alert rules such as alerts for low disk space. You can create additional alerts on any metric. Watch [Alerts in Sematext Cloud](https://www.youtube.com/watch?v=WE9xAUud28o) for more details.

### **Alerting on Elasticsearch Metrics**

There are [3 types of alerts](https://sematext.com/docs/alerts/) in Sematext:

- **Heartbeat alerts** , which notify you when a Elasticsearch DB server is down
- Classic **threshold-based alerts**  that notify you when a metric value crosses a predefined threshold
- Alerts based on statistical **anomaly detection**  that notify you when metric values suddenly change and deviate from the baseline

Let’s see how to actually create some alert rules for Elasticsearch metrics in the animation below. The request query count chart shows a spike. We normally have up to 100 requests, but we see it can jump to over 600 requests. To create an alert rule on a metric we’d go to the pulldown in the top right corner of a chart and choose “Create alert”. The alert rule applies the filters from the current view and you can choose various notification options such as email or configured [notification hooks](https://sematext.com/docs/alerts/#alert-integrations) (PagerDuty, Slack, VictorOps, BigPanda, OpsGenie, Pusher, generic webhooks etc.). Alerts are triggered either by anomaly detection, watching metric changes in a given time window or through the use of classic threshold-based alerts.

![elasticsearch create alert](https://sematext.com/wp-content/uploads/2019/03/elasticsearch-create-alert.gif)

_Alert creation for Elasticsearch request query count metric_

## **Elasticsearch Logs** 

### **Shipping Elasticsearch Logs**

Since having [logs and metrics in one platform](https://sematext.com/metrics-and-logs/) makes troubleshooting simpler and faster let’s ship Elasticsearch logs too. You can use [many log shippers](https://sematext.com/docs/integration/#logging), but we’ll use [Logagent](https://sematext.com/logagent/) because it’s lightweight, easy to set up, and because it can parse and structure logs out of the box. The log parser extracts timestamp, severity, and messages. For query traces, the log parser also extracts the unique query ID to group logs related to query execution.

1. [Create a Logs App](https://apps.sematext.com/ui/integrations) to obtain an App token
2. Install Logagent npm package

```bash
sudo npm i -g @sematext/logagent
```

you don’t have Node.js, you can install it easily. E.g. On Debian/Ubuntu:

```bash
curl -sL https://deb.nodesource.com/setup\_10.x | sudo -E bash -
sudo apt-get install -y nodejs
```

3. Install the Logagent service by specifying the logs token and the path to Elasticsearch log files.  You can use `-g '/var/log/ **/elasticsearch*.log'` to ship only logs from Elasticsearch server. If you run other services, on the same server consider shipping all logs using `-g '/var/log/** /*.log'` The default settings ship all logs from `/var/log/**/*.log` when the -g parameter is not specified. Logagent detects the init system and installs Systemd or Upstart service scripts. On Mac OS X it creates a launchd service. Simply run:

```sh
sudo logagent-setup -i YOUR_LOGS_TOKEN -g '/var/log/**/elasticsearch*.log'
```

for EU region use:

```sh
sudo logagent-setup -i LOGS_TOKEN \
-u logsene-receiver.eu.sematext.com \
-g '/var/log/**/elasticsearch*.log'
```

The setup script generates the configuration file in /etc/sematext/logagent.conf and starts Logagent as system service.

Note, if you run Elasticsearch in containers, [setup Logagent for container logs](https://sematext.com/docs/logagent/installation-docker/).

### **Log Search and Dashboards**

Once you have logs in Sematext you can search through them when troubleshooting, save queries you run frequently or [create your individual logs dashboard](https://sematext.com/product-updates/#/2018/custom-reports-for-monitoring-logs-apps).

![logs search](https://sematext.com/wp-content/uploads/2019/03/clickhouse-logs-search.png)

_Search for Elasticsearch Logs_

**Log Search Syntax**

If you know how to search with Google, you’ll know [how to search your logs](https://sematext.com/docs/logs/search-syntax/) in Sematext Cloud.

- Use AND, OR, NOT operators – e.g. (error OR warn) NOT exception
- Group AND, OR, NOT clauses – e.g. message:(exception OR error OR timeout) AND severity:(error OR warn)
- Don’t like Booleans? Use + and – to include and exclude – e.g. +message:error -message:timeout -host:db1.example.com
- Use field references explicitly – e.g. message:timeout
- Need a phrase search? Use quotation marks – e.g. message:”fatal error”

When digging through logs you might find yourself running the same searches again and again. To solve this annoyance, Sematext lets you save queries so you can re-execute them quickly without having to retype them. Please watch how [using logs for troubleshooting](https://www.youtube.com/watch?v=glwZ8OCV0kc)simplifies your work.

### **Alerting on Elasticsearch Logs**

To create an alert on logs we start by running a query that matches exactly those log events that we want to be alerted about. To create an alert just click to the floppy disk icon.

![alerts on elasticsearch logs](https://sematext.com/wp-content/uploads/2019/03/pasted-image-0-2.png)

Similar to the setup of metric alert rules, we can define threshold-based or anomaly detection alerts based on the number of matching log events the alert query returns.

![alert screen](https://sematext.com/wp-content/uploads/2019/03/Screen-Shot-2019-02-13-at-12.32.04.png)

Please watch [Alerts in Sematext Cloud](https://www.youtube.com/watch?v=WE9xAUud28o) for more details.

### **Elasticsearch Metrics and Log Correlation**

A typical troubleshooting workflow starts from detecting a spike in the metrics, then digging into logs to find the root cause of the problem. Sematext makes this really simple and fast. Your metrics and logs live under the same roof. Logs are centralized, the search is fast, and the powerful [log search syntax](https://sematext.com/docs/logs/search-syntax/) is simple to use.  Correlation of metrics and logs is literally one click away.

![logs and metrics correlation](https://sematext.com/wp-content/uploads/2019/03/pasted-image-0-1.png)

_Elasticsearch logs and metrics in a single view_

## **Monitor Elasticsearch with Sematext**

Comprehensive monitoring for Elasticsearch involves identifying key metrics for Elasticsearch, collecting metrics and logs, and then connecting everything in a meaningful way. In this post, we’ve shown you how to monitor Elasticsearch metrics and logs in one place. We used OOTB and customized dashboards, metrics correlation, log correlation, anomaly detection, and alerts. And with other [open-source integrations](https://sematext.com/blog/now-open-source-sematext-monitoring-agent/), like [Apache Kafka](https://sematext.com/integrations/kafka-monitoring/), you can easily start monitoring Elasticsearch alongside metrics, logs, and distributed request traces from all of the other technologies in your infrastructure. Get deeper visibility into Elasticsearch today with a [free Sematext trial](https://apps.sematext.com/ui/registration).

The post [Monitoring Elasticsearch with Sematext](https://sematext.com/blog/monitoring-elasticsearch-with-sematext/) appeared first on [Sematext](https://sematext.com).