Monitoring ElasticSearch Cluster using Prometheus and Grafana
Tags: linux, webdev, productivity, devops
Published: 2020-08-31 11:34:47


We are going to monitor our production ElasticSearch Cluster using `elasticsearch exporter plugin` and store those metrics on the Prometheus server and later visualize it using Grafana Dashboard. The exporter will scrape the metrics from the cluster and make it available on default port `9114`(which can be changed) and Prometheus will scrape those metrics from the exporter and store it on its TSD.

> ES Cluster >> es_exporter >> Prometheus >> Grafana

  

#### ElasticSearch Exporter:
ElasticSearch Exporter is a ElasticSearch stats exporter for Prometheus which is written in GO LANG. 

***NOTE:*** Exporter fetches metrics from an ElasticSearch cluster on every scrape, therefore having a too short scrape interval can impose load on ES master nodes, particularly if you run with `--es.all` and `--es.indices`.  

Link for [ElasticSearch Exporter](https://github.com/justwatchcom/elasticsearch_exporter) github project.

I'm going to use pre-built binaries here.

```yaml
wget https://github.com/justwatchcom/elasticsearch_exporter/releases/download/v1.1.0/elasticsearch_exporter-1.1.0.linux-amd64.tar.gz
tar xvf elasticsearch_exporter-1.1.0.linux-amd64.tar.gz
cd elasticsearch_exporter-1.1.0.linux-amd64
cp elasticsearch_exporter /usr/local/bin/es_exporter
```
If you need any further information regarding binary then use `--help` option.
```yaml
elasticsearch_exporter --help
```

### SYSTEMD SERVICE
Now create systemd service file for elasticsearch exporter in your ES server. Define your ES cluster username password if set.  
`vim /etc/systemd/system/es_exporter.service`
```yaml
[Unit]
Description=Prometheus ES_exporter
After=local-fs.target network-online.target network.target
Wants=local-fs.target network-online.target network.target

[Service]
User=root
Nice=10
ExecStart = /usr/local/bin/es_exporter --es.uri=http://elastic_user:XXXXXXXXXXX@localhost:9200 --es.all --es.indices --es.timeout 20s
ExecStop= /usr/bin/killall es_exporter

[Install]
WantedBy=default.target
```

```yaml
systemctl start es_exporter.service
systemctl enable es_exporter.serivce
```

### PROMETHEUS
config for `prometheus.yml`

`vim /etc/prometheus/prometheus.yml`
Add below lines.
```yaml
  - job_name: elasticsearch
    scrape_interval: 60s
    scrape_timeout:  30s
    metrics_path: "/metrics"
    static_configs:
    - targets: ['es-01:9114' ]
      labels:
        service: elasticsearch
    relabel_configs:
    - source_labels: [__address__]
      regex: '(.*)\:9108'
      target_label:  'instance'
      replacement:   '$1'
    - source_labels: [__address__]
      regex:         '.*\.(.*)\.lan.*'
      target_label:  'environment'
      replacement:   '$1'
```
```yaml
systemctl restart prometheus
```

### GRAFANA DASHBOARD

Import [https://grafana.com/grafana/dashboards/2322](https://grafana.com/grafana/dashboards/2322) Grafana Dashboard and start monitoring your ElasticSearch Cluster metrics. You can either download in JSON format and upload in your grafana server or copy ID and import directly.


![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/bdnesoh5u6g6xz1ethmj.png)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/u2fgtsv6mdkutzf5vq55.png)