Elasticsearch Sample Data Generator
Tags: elasticsearch, bulk, data, dump
Published: 2021-03-24 01:44:59

Recently, I am trying to learn Elasticsearch once again. I used "once again" because I wanted to learn it since late 2016 and in between the time frame, I tried learning it several times and as always I have failed myself to learn it. And just like every other time, I am motivated this time as well :wink:

## Motive
To learn elasticsearch, you need lots of data to make queries as you want. I searched a few places to get a valid dump. But I couldn't find any dump that I can go with. Also what I found online, I am not familiar with the types of data. So, I thought to make a generator of my own. I have already used [Artisan Console](https://laravel.com/docs/master/artisan) and [fzaninotto/Faker](https://github.com/fzaninotto/Faker), that's why I thought to make a generator that anyone can use with their terminal and generate the dump the way they wish.

## The repository
This is the repository that you can use to generate the dump.

{% github ssi-anik/elasticsearch-sample-data-generator %}

## Installation [without docker]
- Clone the repository. 
- If your machine has PHP version `>=7.3` and `<8.0` and composer installed, then just run `composer install` being in the root of the repository. It'll install the project dependencies.

That's all.

## Installation [with docker]
- Clone the repository.
- Uncomment the line `Copy . /app` in the `Dockerfile`.
- Uncomment the line `RUN composer install` in the `Dockerfile`.
- Copy the `docker-compose.yml.example` to `docker-compose.yml`.
- Comment the line `.:/app` in your docker-compose.yml's `services.php.volumes`.
- Uncomment the line `./dumps:/app/dumps` in your docker-compose.yml's `services.php.volumes`.
- If you don't need elasticsearch and kibana services, then just delete them.
- Run `docker-compose up -d --build` to run your containers.
- To exec into the PHP service, run `docker-compose exec php bash`.

That's all for the docker-based installation. If you're good at docker, you can tweak these things as well by going through the `Dockerfile` and the `docker-compose.yml`.

## Usage
The repository contains one executable `elasticsearch-dump` in the root of it. We'll have to use this to run commands and generate dumps.

`./elasticsearch-dump generate` is the base command. Let's have a look at the available arguments and options.

```bash
./elasticsearch-dump generate --help

Description:
  Generate dump for elasticsearch bulk API upload

Usage:
  generate [options] [--] <fields>

Arguments:
  fields               Enter the fields definition (required)

Options:
  --file[=FILE]        Enter the file name [default: "dumps/dump.json"]
  --entries[=ENTRIES]  Enter the number of entries [default: "1"]
  --action[=ACTION]    Enter the action name [index or create] [default: "index"]
  --index[=INDEX]      Enter the index name [default: "my-index"]
  --id[=ID]            Enter the sequence start value [default: "1"]
  --append             Append to existing file
  --force              Does not ask for confirmation
  --uuid               UUID based ID generation
```

## Options
Before we check the required argument, let's explore the options first. There are few options that expect values and a few are boolean flags. And all the options are optional. You'll override the common values passing these options.

- `--file` - Default is `dumps/dump.json`. You can pass the file name where you want to save the dump. You can pass a relative or absolute path. If the path starts with `/` then it'll use it as an absolute path. Otherwise, it'll always dump in the `dumps` directory and considers the file name only.
- `--entries` - Default is `1`. The number of entries you want to generate.
- `--action` - Default is `index`. The type of the action. Either it can be `index` or `create`.
- `--index` - Default is `my-index`. The name of the index where you'll put these values.
- `--id` - Default is `1`. The start position of the sequence. It can only generate a numeric sequence.
- `--append` - A boolean flag. If exists then it'll append to the existing file. If the file doesn't exist, then it'll create the file and put contents on it.
- `--force` - A boolean flag. By default, the command will ask you for confirmation. By providing this flag, you can bypass the confirmation.
- `--uuid` - A boolean flag. If passed, the `--id` will not be considered and will generate the UUID-based IDs.

## Arguments
The command generates data utilizing the [PHP's Faker library](https://github.com/fzaninotto/Faker). We have to pass the fields that we want to generate with the fake data.

Suppose we want to generate `name` and `address` fields. When you pass the fields, you can use the pipe `|` to separate each field. So, the command looks like the following.

Example:
```bash
./elasticsearch-dump generate --entries 10 "name|address"
```

Here, both the **`name`** and **`address`** fields are resolved to the Faker's `name` and `address` properties. If we have to have a different key for the objects, we can use a colon `:` to separate them. So, if we want to have `firstName` in our name fields, and `streetAddress` in our address field, then we can simply use the following.

Example:
```bash
./elasticsearch-dump generate --entries 10 \
  "name:firstName|address:streetAddress"

# Generates
# {"name":"Roosevelt","address":"45647 Judy Isle"}
```

Here, the `name` key will be in the object, containing the `firstName` as well as the `streetAddress` value in the `address` key. Now, `firstName` and the `streetAddress` are resolved to the faker's property.

If the faker wants you to pass a method, you can also do it by passing as a method.

Example:
```bash
./elasticsearch-dump generate --entries 10 \
  "name:firstName|id:numerify('ID-####')|amount:numberBetween(1000, 9000)"

# Generates
# {"name":"Lourdes","id":"ID-4912","amount":1004}
```

## Object nesting
When passing your fields to the command's argument, you can pass nest objects using the dot notation.

Example:
```bash
./elasticsearch-dump generate --entries 10 \
"student.name:firstName|student.age:numberBetween(20, 27)|id:numerify('ID-####')"

# Generates
# {"student":{"name":"Chandler","age":20},"id":"ID-4386"}
```

Check the JSON. The `student` object contains the `name` and `age` within it. The ID field is outside the `student` object.

## Extending the faker functionality
If the faker doesn't provide the type of data you want and you want to extend it, you can also do so by providing an array of values in the project's [`config/source.php`](https://github.com/ssi-anik/elasticsearch-sample-data-generator/blob/master/config/source.php) file. The file already contains `designation` as an example. You can call the custom provider using the `custom('key')` format.

Example:
```bash
./elasticsearch-dump generate "name|designation:custom('designation')"

# Generates
# {"name":"Annabelle Balistreri","designation":"HR Managers"}
```

So, for our case `custom('designation')`, where `designation` is the key in the `config/source.php` file.

---

Hope this helps you to generate lots of data. 

Happy coding. :heart: