SonarQube: solving “Unrecoverable indexation failures” and Elasticsearch “Disk watermark exceeded” errors
Tags: jenkins, sonarqube, elasticsearch
Published: 2019-08-29 17:15:11

---
title: SonarQube: solving “Unrecoverable indexation failures” and Elasticsearch “Disk watermark exceeded” errors
published: true
tags: jenkins,sonarqube,elasticsearch
canonical_url: https://rtfm.co.ua/en/sonarqube-solving-unrecoverable-indexation-failures-and-elasticsearch-disk-watermark-exceeded-errors/
---

![](https://rtfm.co.ua/wp-content/uploads/2019/06/sonarqube-logo-e1560940372101.png)We are using SonarQube started from Jenkins jobs.

See more at [SonarQube: running tests from Jenkins Pipeline in Docker](https://rtfm.co.ua/en/sonarqube-running-tests-from-jenkins-pipeline-from-docker/).

So, the SonarQube is started from the Jenkins – all good here, but inside SonarQube we can see errors:

> java.lang.IllegalStateException: Unrecoverable indexation failures: 1 errors among 1 requests  
> at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39)  
> at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:122)  
> …

[![](https://rtfm.co.ua/wp-content/uploads/2019/08/Screenshot_20190829_123022.png "SonarQube: Unrecoverable indexation failures и Elasticsearch "Disk watermark exceeded"")](https://rtfm.co.ua/wp-content/uploads/2019/08/Screenshot_20190829_123022.png)

And host logs for SonarQube:

> Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1  | 2019.08.29 09:30:26 ERROR web[][o.s.s.es.BulkIndexer] index [components], type [auth], id [AWzcfdpYaLFDDA6l6km3], message [ClusterBlockException[blocked by: [FORBIDD  
> EN/12/index read-only / allow delete (api)];]]  
> Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1  | 2019.08.29 09:30:26 ERROR web[][o.s.s.e.RecoveryIndexer] Elasticsearch recovery – too many failures [167/167 documents], waiting for next run  
> Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1  | 2019.08.29 09:30:26 INFO  web[][o.s.s.e.RecoveryIndexer] Elasticsearch recovery – 167 documents processed [167 failures] | time=51ms  
> Aug 29 12:30:48 jenkins-production docker-compose[12591]: sonarqube_1  | 2019.08.29 09:30:48 INFO  es[][o.e.c.r.a.DiskThresholdMonitor] low disk watermark [85%] exceeded on [mjJvGbfOSZyVqbKWKCGwxw][sonarqube][/opt/sonarqube/data/es6/nodes  
> /0] free: 11.6gb[11.8%], replicas will not be assigned to this node

Check the documentation about [Disk-based shard allocation](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html) – need to set `cluster.routing.allocation.disk.watermark.low` or disable those checks at all by setting the `"cluster.routing.allocation.disk.threshold_enabled" : false`.

Although we have 12 GB free from the 100 GB total on the Jenkins host – but ElasticSearch by default count in percents so it’s not enough for him.

Because Elasticsearch is running inside (!) of the SonarQube’s container – let’s find where is the ElasticSearch config is stored there:

```
root@jenkins-production:/opt/jenkins# docker exec -ti jenkins_sonarqube_1 find / -name elasticsearch.yml
find: ‘/proc/tty/driver’: Permission denied
/opt/sonarqube/temp/conf/es/elasticsearch.yml
/opt/sonarqube/elasticsearch/config/elasticsearch.yml
```

Here we are – `/opt/sonarqube/elasticsearch/config/elasticsearch.yml`.

Sonar is started from a Docker Compose file with some files/dirs mapping:

```
version: '3.5'
networks:
  sonar:
    external:
      name: jenkins
services:
  sonarqube:
    user: 1004:1004
    image: sonarqube:7.9.1-community
    ports:
      - "9000:9000"
    networks:
      - sonar
    environment:
      - sonar.jdbc.url=jdbc:postgresql://db:5432/sonar
    volumes:
      - /data/sonarqube/conf:/opt/sonarqube/conf
      - /data/sonarqube/logs:/opt/sonarqube/logs
      - /data/sonarqube/temp:/opt/sonarqube/temp
      - /data/sonarqube/data:/opt/sonarqube/data
      - /data/sonarqube/extensions:/opt/sonarqube/extensions
      - /data/sonarqube/bundled_plugins:/opt/sonarqube/lib/bundled-plugins
    logging:
      driver: "journald"
...
```

Available options for the disk management are:

- `cluster.routing.allocation.disk.threshold_enabled`: будет ли Elasticsearch вообще проверять состояние диска
- `cluster.routing.allocation.disk.watermark.low`: default 85%, Elasticsearch не будет размещать шарды на этой ноде
- `cluster.routing.allocation.disk.watermark.high`: default 90%, Elasticsearch попробует перенести шарды с этой ноды на другие
- `cluster.routing.allocation.disk.watermark.flood_stage`: default 95%, запрещает обновление индексов, шарды которых на этой ноде

Create a new config on the host – `/data/sonarqube/conf/elasticsearch.yml`:

```
cluster.routing.allocation.disk.watermark.flood_stage: 95%
cluster.routing.allocation.disk.watermark.high: 90%
```

Map it to the SonarQube’s container SonarQube:

```
...
    volumes:
      - /data/sonarqube/conf:/opt/sonarqube/conf
      - /data/sonarqube/conf/elasticsearch.yml:/opt/sonarqube/elasticsearch/config/elasticsearch.yml
...
```

To access the Elasticsearch – need to set `sonar.search.httpPort` for the SonarQube, see the [How to Monitor ElasticSearch](https://docs.sonarqube.org/display/SONARqube71/ElasticSearch+Activity).

Enable this connector _9100_:

```
...
    environment:
      - sonar.jdbc.url=jdbc:postgresql://db:5432/sonar
      - sonar.search.httpPort=9100
...
```

Restart the service:

```
root@jenkins-production:/home/admin# systemctl restart sonarqube
```

And a bit logs says this option will be deprecated soon:

> 2019.08.29 10:56:15 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001  
> 2019.08.29 10:56:15 WARN  app[][o.s.a.es.EsSettings] Elasticsearch HTTP connector is enabled on port 9100. MUST NOT BE USED FOR PRODUCTION  
> 2019.08.29 10:55:10 WARN  es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.

Check parameters after the restart to make sure Elasticsearch sees them.

As Elasticsearch listen on the localhost only inside (again!) the SonarQube’s container – use `curl` to check from the SonarQube itself:

```
root@jenkins-production:/home/admin# docker exec -ti jenkins_sonarqube_1 curl localhost:9100/_cluster/settings?include_defaults=true | jq '.[].cluster.routing.allocation.disk'
null
null
{
  "threshold_enabled": "true",
  "watermark": {
  "low": "85%",
  "flood_stage": "95%",
  "high": "90%"
},
  "include_relocations": "true",
  "reroute_interval": "60s"
}
```

Okay – new settings applied, can proceed with builds now.

Used disc space per node can be checked by calling the next API:

```
root@jenkins-production:/home/admin# docker exec -ti jenkins_sonarqube_1 curl 'http://localhost:9100/_cat/allocation?v&pretty'

shards disk.indices disk.used disk.avail disk.total disk.percent host      ip        node

24      613.4kb    86.3gb     11.5gb     97.9gb           88 127.0.0.1 127.0.0.1 sonarqube
```

All available parameters for the Elasticsearch – [https://nickcanzoneri.com/elasticsearch-settings](https://nickcanzoneri.com/elasticsearch-settings).

Done.

### Similar posts
 
- <small>06/18/2019</small> [SonarQube: running tests from Jenkins Pipeline in Docker](https://rtfm.co.ua/en/sonarqube-running-tests-from-jenkins-pipeline-from-docker/)
- <small>06/18/2019</small> [SonarQube: запуск в Docker и вызов из Jenkins Pipeline](https://rtfm.co.ua/sonarqube-zapusk-v-docker-i-vyzov-iz-jenkins-pipeline/)
- <small>06/19/2019</small> [SonarQube: the “SCM provider autodetection failed” error](https://rtfm.co.ua/en/sonarqube-the-scm-provider-autodetection-failed-error/) 
- <small>02/27/2017</small> [Docker: Compose down – network jmadmin\_default has active endpoints](https://rtfm.co.ua/docker-compose-down-network-jmadmin_default-has-active-endpoints/)