Solr Open Source Monitoring Tools
Tags: solr, prometheus, observability, devops
Published: 2019-05-20 13:24:23

---
title: Solr Open Source Monitoring Tools
published: true
description: Solr open source monitoring tools including Prometheus, Graphite, Grafana and more. Take a look how to monitor your Solr and SolrCloud instances with freely available tools.
tags: #solr #prometheus #observability #devops
series: Monitoring Solr
cover_image: https://thepracticaldev.s3.amazonaws.com/i/d586bm0e09i9y2nu7xx6.png
---


Open source software adoption continues to grow. Tools like [Kafka](https://sematext.com/blog/monitoring-kafka-with-sematext/) and [Solr](http://lucene.apache.org/solr/) are widely used in small startups, ones that are using cloud ready tools from the start, but also in large enterprises, where legacy software is getting faster by incorporating new tools. In this second part of our Solr monitoring series (see the first part discussing [Solr metrics to monitor](https://sematext.com/blog/solr-key-metrics-to-monitor/)), we will explore some of the open source tools available to monitor Solr nodes and clusters. We'll take the opportunity to look into what it takes to install, configure and use each tool in a meaningful way.

#The Importance of Solr Monitoring

Operating, managing and maintaining distributed systems is not easy. As we explored in the first part of our monitoring Solr series there are more than forty metrics that we need to have full visibility into our Solr instances and the full cluster. Without any kind of monitoring tool, it is close to impossible to have a full view over all the needed pieces to be sure that the cluster is healthy or to react properly when things are not going the right way.

When searching for an open source tool to help you track [Solr metrics](https://sematext.com/solr-api-metrics-cheat-sheet/), look at the following qualities:
1. The ability to monitor and manage multiple clusters
2. An easy, at-glance overview of the whole cluster and its state
3. Clear information about the crucial performance metrics
4. Ability to provide historical metrics for post mortem analysis
5. Combines low-level OS metrics, JVM metrics, and Solr specific metrics
6. Ability to set up alerts

Let's now explore some of the available options.

#Prometheus with Solr Exporter

[Prometheus](https://github.com/prometheus) is an open-source monitoring and alerting system that was originally developed at SoundCloud. Right now it is a standalone open source project and it is maintained independently from the company that created it initially. [Prometheus](https://github.com/prometheus) project, in 2016, joined the [Cloud Native Computing Foundation](https://www.cncf.io/) as the second hosted project, right after Kubernetes.

Out of the box Prometheus supports flexible query language on top of the multi-dimensional data model based on [TSDB](http://opentsdb.net/) where the data can be pulled using the HTTP based protocol:

![](https://thepracticaldev.s3.amazonaws.com/i/v05vtsdx90q2d8cl32r4.png)

Source: [https://prometheus.io/docs/introduction/overview/](https://prometheus.io/docs/introduction/overview/)

For Solr to be able to ship metrics to Prometheus we will use a tool called [Exporter](http://opentsdb.net/). It takes the [metrics from Solr](https://sematext.com/solr-api-metrics-cheat-sheet/) and translates them into a format that is understandable by Prometheus itself. The Solr Exporter is not only able to ship metrics to Prometheus, but also responses for requests like [Collections API](https://lucene.apache.org/solr/guide/7_7/collections-api.html) commands, [ping](https://lucene.apache.org/solr/guide/7_7/ping.html) requests and [facets](https://lucene.apache.org/solr/guide/7_7/json-facet-api.html) gathered from search results.

The Prometheus Solr Exporter is shipped with Solr as a contrib module located in the *contrib/prometheus-exporter* directory. To start working with it we need to take the *solr-exporter.xml* file that is located in the *contrib/prometheus-exporter/conf* directory. It is already pre-configured to work with Solr and we will not modify it now. However, if you are interested in additional metrics, ship additional facet results or send fewer data to Prometheus you should look and modify the mentioned file.

Once we have the exporter configured we need to start it. It is very simple. Just go to the *contrib/prometheus-exporter* directory (or the one where you copied it in your production system) and run appropriate command, depending on the architecture of Solr you are running.

For Solr master-slave deployments you would run:

```bash
./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f 
./conf/solr-exporter-config.xml -n 8
```

For SolrCloud you would run:

```bash
./bin/solr-exporter -p 9854 -z localhost:2181/solr -f 
./conf/solr-exporter-config.xml -n 16
```

The above command runs Solr exporter on the *9854* port with 8 threads for Solr master-slave and 16 for SolrCloud. In case of SolrCloud we are also pointing exporter to the Zookeeper ensemble that is accessible on port *2181* on the *localhost*. Of course, you should adjust the commands to match your environment.

After the command was successfully run you should see the following:

```
INFO  - 2019-04-29 16:36:21.476; 
org.apache.solr.prometheus.exporter.SolrExporter; Start server
```

We have Solr master-slave/SolrCloud running and we have our Solr Exporter running, this means we are ready to take the next step and configure our Prometheus instance to fetch data from our Solr Exporter. To do that we need to adjust the *prometheus.yml* file and add the following:

```yaml
scrape_configs:
  - job_name: 'solr'
    static_configs:
      - targets: ['localhost:9854']
```

Of course, in the production system, our Prometheus will run on a different host compared to our Solr and Solr Exporter - we can even run multiple exporters. That means that we will need to adjust the targets property to match our environment.

After all the preparations we can finally look into what Prometheus gives us. We can start with the main Prometheus UI.

![Prometheus UI](https://thepracticaldev.s3.amazonaws.com/i/lplgm7lyukpev5j3xfly.png)

It allows for choosing the metrics that we are interested in, graph it, alert on it and so on. The beautiful thing about it is that the UI support the [full Prometheus Query Language](https://prometheus.io/docs/prometheus/latest/querying/basics/) allowing the use of operators, functions, subqueries and many, many more.

When using the visualization functionality of Prometheus we get the full view of the available metrics by using a simple dropdown menu, so we don't need to be aware of each and every metric that is shipped to Solr.

![Prometheus Metrics](https://thepracticaldev.s3.amazonaws.com/i/7f4miljpgsez0ci9g8a6.png)

![Prometheus Graph](https://thepracticaldev.s3.amazonaws.com/i/kqi2iwemizyc35b90hcy.png)

The nice thing about Prometheus is that we are not limited to the default UI, but we can also use [Grafana](https://grafana.com/) for dashboarding, alerting and team management. Defining the new, Prometheus data source is very, very simple:

![Prometheus Data Sources](https://thepracticaldev.s3.amazonaws.com/i/pup8vugmg2icuzcoq4wo.png)

Once that is done we can start visualizing the data:

![Prometheus Grafana Graph](https://thepracticaldev.s3.amazonaws.com/i/o01kjrctg7cfoennvoqk.png)

However, all of that requires us to build rich dashboards ourselves. Luckily Solr comes with an example [pre-built Grafana dashboard](https://lucene.apache.org/solr/guide/7_7/monitoring-solr-with-prometheus-and-grafana.html#sample-grafana-dashboard) that can be used along with the metrics scrapped to Prometheus. The example dashboard definition is stored in the *contrib/prometheus-exporter/conf/grafana-solr-dashboard.json* file and can be loaded to Grafana giving a basic view over our Solr cluster.

![Example Grafana Dashboard using Prometheus metrics](https://thepracticaldev.s3.amazonaws.com/i/r68um0la1lid77b7tu6h.png)

Of dashboards with metrics is not everything that Grafana is capable of. We are able to set up teams, users, assign roles to them, set up alerts on the metrics and include multiple data sources within a single installation of Grafana. This allows us to have everything in one place - metrics from multiple sources, logs, signals, tracing and whatever we need and can think of.

#Graphite & Graphite with Grafana

[Graphite](https://graphiteapp.org/) is a free open-sourced monitoring software that can monitor and graph numeric time-series data. It can collect, store and display data in a real-time manner allowing for fine-grained metrics monitoring. It is composed of three main parts - [Carbon](https://github.com/graphite-project/carbon), the daemon listening for time-series data, [Whisper](https://github.com/graphite-project/whisper) - database for storing time-series data and the [Graphite web-app](https://github.com/graphite-project/graphite-web) that is used for on-demand metrics rendering.

To start monitoring Solr with Graphite as the platform of choice we assume that you already have Graphite up and running, but if you don't we can start by using the provided Docker container:

```bash
docker run -d
  --name graphite
  --restart=always
  -p 80:80
  -p 2003-2004:2003-2004
  -p 2023-2024:2023-2024
  -p 8125:8125/udp
  -p 8126:8126
  graphiteapp/graphite-statsd
```

To be able to get the data from Solr we will use [Solr metrics registry](https://lucene.apache.org/solr/guide/7_7/metrics-reporting.html#metric-registries) along with the [Graphite reporter](https://lucene.apache.org/solr/guide/7_7/metrics-reporting.html#graphite-reporter). To configure that we need to adjust the *solr.xml* file and add the metrics part to it. For example, to monitor information about the JVM and the Solr node the metrics section would look as follows:

```xml
<metrics>
  <reporter name="graphite" group="node, jvm"
 class="org.apache.solr.metrics.reporters.SolrGraphiteReporter">
     <str name="host">localhost</str>
     <int name="port">2003</int>
     <int name="period">60</int>
  </reporter>
</metrics>
```

So we pointed Solr to the Graphite server that is running on the *localhost* on the port *2003* and we defined the period of data writing to 60, which means that Solr will push the JVM and Solr node metrics once every 60 seconds.

Keep in mind that by default Solr will write by using the plain text protocol. This is less efficient than using the pickled protocol. If you would like to configure Solr and Graphite in production we suggest using setting the pickled property to true in the reporter configuration and using the port for the pickled protocol, which in case of our Docker container would be 2004.

We can now easily navigate to our Graphite server, available at *127.0.0.1* on port *80* with our container and graph our data:

![Graphite graph](https://thepracticaldev.s3.amazonaws.com/i/3d4lni3uz0lyqce7cjlb.png)

All the metrics are sorted out and easily accessible in the left menu allowing for rich dashboarding capabilities.

If you are using [Grafana](https://lucene.apache.org/solr/guide/7_7/metrics-reporting.html#graphite-reporter) it is easy to setup Graphite as yet another data source and uses its graphing and dashboarding capabilities to correlate multiple metrics together, even ones that are coming from different data sources.

![](https://thepracticaldev.s3.amazonaws.com/i/vf5b89i281bondfeedd7.png)

Next, we need to configure Graphite as the data source. It is as easy as providing the proper Graphite URL and setting the version:

![](https://thepracticaldev.s3.amazonaws.com/i/j5d81cint3zjfmyblq5e.png)

And we are ready to create our visualizations and dashboards, which is very easy and powerful. With the autocomplete available for metrics we don't need to recall any of the names and Grafana will just show them for us. An example of a single metric dashboard can looks as follows:

![](https://thepracticaldev.s3.amazonaws.com/i/obft0engio79jrjl2730.png)

#Ganglia

[Ganglia](http://ganglia.info/) is a scalable distributed monitoring system. It is based on a hierarchical design targeted for a large number of clusters and nodes. It is using XML for data representation, [XDR](https://tools.ietf.org/html/rfc4506) for data transport and [RRD](http://www.rrdtool.org/) for data storage and visualization. It has been used to connect clusters across university campuses and is proven to handle clusters with 2000 nodes.

To start monitoring Solr master-slave or SolrCloud clusters with Ganglia we will start with setting up the metrics reporter in the *solr.xml* configuration file. To do that we add the following section to the mentioned file:

```xml
<metrics>
  <reporter name="ganglia" group="node, jvm"
class="org.apache.solr.metrics.reporters.SolrGangliaReporter">
    <str name="host">localhost</str>
    <int name="port">8649</int>
  </reporter>
</metrics>
```

Next thing that we need to do is allow Solr to understand the XDR protocol used for data transport. We need to download the [oncrpc-1.0.7.jar](http://www.rrdtool.org/) jar file and place it either in your Solr classpath or include the path to it in your *solrconfig.xml* file using the lib directive.

One all of the above is done and assuming our Ganglia is running on *localhost* on port *8649* that is everything that we need to do to have everything ready and start shipping Solr nodes and JVM metrics.

By visiting Ganglia and choosing the Solr node we can start looking into the metrics:

![Ganglia Metrics](https://thepracticaldev.s3.amazonaws.com/i/hu0ease5z649rhnzws36.png)

We can jump to the graphs right away, choose which group of metrics interested in and basically see most of the data that we are interested in right away.

![Ganglia graph](https://thepracticaldev.s3.amazonaws.com/i/a41jex5hhd11oy6j25ei.png)

Ganglia provides us with all the visibility for our metrics, but out of the box, it doesn't support one of the crucial features that we are looking for - alerting. There is a project called [ganglia-alert](http://www.rrdtool.org/), which is a user contributed extension to Ganglia.

#Conclusion

As you can see there is a wide variety of tools that help you monitor Solr. What you have to keep in mind is that each requires setting up, configuration and manual dashboard building in order to get meaningful information. All of that may require deep knowledge across the whole ecosystem.

If you are looking for a Solr monitoring tool that you can set up in minutes and have pre-built dashboards with all necessary information, alerting and team management take a look at the third part of the Solr monitoring series to learn more about [production ready Solr monitoring with Sematext](https://sematext.com/blog/solr-monitoring-made-easy-with-sematext/).

If you need full-stack observability for your software stack, check out [Sematext](https://sematext.com/). We’re pushing to [open source our products](https://github.com/sematext) and make an impact.