Writing integration tests for your spring-data-elasticsearch-based applications
Tags: springdata, elasticsearch, testing, java
Published: 2019-10-30 06:00:34

We all know it by now, testing is an important part of our job as Software Engineers. If that is not currently the case for you or your organization, I couldn't recommend enough, that you start including it in your development workflow, asap.

We should also agree that testing, in particular integration testing, can sometimes be challenging or too complex to put together. Which is probably why, some developers simply skip that essential part.

To make things less complicated and more accessible, libraries like `assert4j`, `mockito`, `spring-test`, `testcontainers` have been created over the years. The easier it is to write tests, the more people will be willing to write them.


In a recent project I worked on, that uses `Elasticsearch` via `Spring Data`, we decided to write integration tests to validate that:
* our data were being correctly indexed in the Elasticsearch Server (ES), 
* our search queries were working and performing as expected

For that purpose, we needed a way to easily load fresh data into a testing <abbr title="Elasticsearch server">ES</abbr>, and then run our search queries on it. We already had a tool to massively inject data from our <abbr title="User Acceptance Tests">UAT</abbr> environment, so it would be nice if we could somehow extract these data back and use them as fixtures to feed our testing <abbr title="Elasticsearch Server">ES</abbr>.

After a few searches online, not finding a good tool for the job, I decided to create and publish my own take on the problem : [spring-esdata-loader](https://github.com/tinesoft/spring-esdata-loader).
 
It is an open source `Java 8+` testing library to help write integration tests for [spring-data elasticsearch](https://spring.io/projects/spring-data-elasticsearch)-based projects, by allowing you to easily load data into <abbr title="Elasticsearch Server">ES</abbr> using simple annotations, leveraging Spring's entity mappings (i.e classes annotated with `@Document`, `@Field`, etc) and via specific `JUnit 4`'s Rules or `JUnit Jupiter`'s Extensions.

The library reads all the metadata it needs from the entity classes (index name, index type, etc) , uses them to create/refresh the related indices and, feeds <abbr title="Elasticsearch Server">ES</abbr> with the data using the `ElasticsearchOperations` already present in your Spring's test application context.

It offers the following out-of-the-box 📦 :

## Features

* **Simple API** and no configuration required
* Support  for  **JUnit 4** via `@LoadEsDataRule`, `@DeleteEsDataRule`
* Support  for  **JUnit  Jupiter** via 
  * `@LoadEsDataConfig` / `@LoadEsDataExtension` 
  * `@DeleteEsDataConfig` / `@DeleteEsDataExtension`
* Built-in support for **gzipped data**
* **Multiple data formats**(dump, manual)
* **Fast** and **Memory-efficient** loading of data
* Written  in  **Java  8**
* Based on **Spring (Data, Test)**

## Installation & Usage

The library is split into 2 independent sub-modules, both are available on [JCenter](https://bintray.com/bintray/jcenter?filterByPkgName=spring-esdata-loader) and [Maven Central](https://search.maven.org/search?q=spring-esdata-loader) for download:

* `spring-esdata-loader-junit4` for testing with **JUnit 4**
* `spring-esdata-loader-junit-jupiter` for testing with **JUnit Jupiter**

So, to get started:

1.Add the appropriate dependency to your `gradle` or `maven` project

|    | **JUnit 4**   |      **JUnit Jupiter**      |
|:----|----------|-------------|
| **Gradle**  | dependencies {<br />&nbsp;&nbsp;&nbsp;&nbsp;testImplementation 'com.github.spring-esdata-loader:spring-esdata-loader-junit4:VERSION'<br />} |  dependencies {<br />&nbsp;&nbsp;&nbsp;&nbsp;testImplementation 'com.github.spring-esdata-loader:spring-esdata-loader-junit-jupiter:VERSION'<br />} |
| **Maven**   | &lt;dependency&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&lt;groupId&gt;com.github.spring-esdata-loader&lt;/groupId&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&lt;artifactId>spring-esdata-loader-junit4&lt;/artifactId&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&lt;version>VERSION&lt;/version&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&lt;scope>test&lt;/scope&gt;<br />&lt;/dependency&gt; |  &lt;dependency&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&lt;groupId&gt;com.github.spring-esdata-loader&lt;/groupId&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&lt;artifactId>spring-esdata-loader-junit-jupiter&lt;/artifactId&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&lt;version>VERSION&lt;/version&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&lt;scope>test&lt;/scope&gt;<br />&lt;/dependency&gt; |

2.Write your test class. 

You can have a look at :

* [JUnit4 Sample](https://github.com/tinesoft/spring-esdata-loader/junit4) - for an example using **JUnit 4** API
* [JUnit Jupiter Sample](https://github.com/tinesoft/spring-esdata-loader/junit-jupiter) - if you prefer **JUnit Jupiter** API

## Supported Data Formats

`spring-esdata-loader` currently supports 2 formats to load data into Elasticsearch : **DUMP** and **MANUAL**.

### Dump data format

Here is an example :
```json
{"_index":"author","_type":"Author","_id":"1","_score":1,"_source":{"id":"1","firstName":"firstName1","lastName":"lastName1"}}
{"_index":"author","_type":"Author","_id":"2","_score":1,"_source":{"id":"2","firstName":"firstName2","lastName":"lastName2"}}
{"_index":"author","_type":"Author","_id":"3","_score":1,"_source":{"id":"3","firstName":"firstName3","lastName":"lastName3"}}
{"_index":"author","_type":"Author","_id":"4","_score":1,"_source":{"id":"4","firstName":"firstName4","lastName":"lastName4"}}
{"_index":"author","_type":"Author","_id":"5","_score":1,"_source":{"id":"5","firstName":"firstName5","lastName":"lastName5"}}
{"_index":"author","_type":"Author","_id":"6","_score":1,"_source":{"id":"6","firstName":"firstName6","lastName":"lastName6"}}
{"_index":"author","_type":"Author","_id":"7","_score":1,"_source":{"id":"7","firstName":"firstName7","lastName":"lastName7"}}
{"_index":"author","_type":"Author","_id":"8","_score":1,"_source":{"id":"8","firstName":"firstName8","lastName":"lastName8"}}
{"_index":"author","_type":"Author","_id":"9","_score":1,"_source":{"id":"9","firstName":"firstName9","lastName":"lastName9"}}
{"_index":"author","_type":"Author","_id":"10","_score":1,"_source":{"id":"10","firstName":"firstName10","lastName":"lastName10"}}

```
You can use a tool like [elasticdump](https://npmjs.com/package/elasticdump) (*requires* [NodeJS](https://nodejs.org/)) to extract existing data
from your Elasticsearch server, and them dump them into a JSON file.

```
$ npx elasticdump --input=http://localhost:9200/my_index --output=my_index_data.json
```

> The above command will run `elasticdump` to extract data from an index named `my_index` on a ES server located at http://localhost:9200 and then save the result into a file named `my_index_data.json`

> If you change the `--output` part above into `--output=$ | gzip my_data.json.gz` the data will be automatically gzipped

### Manual data format

In this format, you specify your target data directly (no metadata like `_index`, `_source`, ...), as an Array of JSON objects.

This is more suitable when you create test data from scratch (as opposed to dumping existing ones from a ES server) because it is easier to tweak later on to accommodate future modifications in tests.

Here is an example :
```json
[
    {"id":"1","firstName":"firstName1","lastName":"lastName1"},
    {"id":"2","firstName":"firstName2","lastName":"lastName2"},
    {"id":"3","firstName":"firstName3","lastName":"lastName3"},
    {"id":"4","firstName":"firstName4","lastName":"lastName4"},
    {"id":"5","firstName":"firstName5","lastName":"lastName5"},
    {"id":"6","firstName":"firstName6","lastName":"lastName6"},
    {"id":"7","firstName":"firstName7","lastName":"lastName7"},
    {"id":"8","firstName":"firstName8","lastName":"lastName8"},
    {"id":"9","firstName":"firstName9","lastName":"lastName9"},
    {"id":"10","firstName":"firstName10","lastName":"lastName10"}
]
```


More complete examples can be found on the project's github under [demo/](https://github.com/tinesoft/spring-esdata-loader/demo).


Happy Hacking 📠!