2.9 - ElasticSearch Autocomplete Indexing
Published: 2022-04-02 01:54:10

[Release 2.9](https://github.com/Seneca-CDOT/telescope/releases/tag/2.9.0) was a partial success on my part, cause in the end I did manage to get [`Author` autocomplete indexing](https://github.com/Seneca-CDOT/telescope/pull/3301) into the back-end.

Admittedly, I've put this blog off a little too long. Partly, because my other courses needed my explicit attention lest I want to fail them all, but also, because I'm rather afraid of having to explain just exactly how I got it working.

A long story short, lots of time playing around in the Kibana dev tools, lots of time pouring into the oddly structured ElasticSearch documentation, and lots of time waiting for docker images to be rebuilt and posts to be re-indexed. All in all, many many trial and errors. 

But why does it work? Well, that's a whole other story. I can not guarantee complete accuracy, and can only explain it to how I understand it. 

## Explaining the code
To begin with, here's the code, and in case this ever gets lost, the [permalink to the code](https://github.com/Seneca-CDOT/telescope/blob/2bb611eae1344bbaecadad9bd84be165ea2b663a/src/backend/utils/indexer.js#L11-L89) with a huge blob of comment consisting of many links to various documentation.
```js
const setupPostsIndex = async () => {
  try {
    const response = await client.indices.exists({ index });
    // If the index doesn't exist, 404 statusCode is returned
    if (response.statusCode === 404) {
      await client.indices.create({
        index,
        body: {
          settings: {
            analysis: {
              analyzer: {
                autocomplete_analyzer: {
                  tokenizer: 'autocomplete',
                  filter: ['lowercase', 'remove_duplicates'],
                },
                autocomplete_search_analyzer: {
                  tokenizer: 'lowercase',
                },
              },
              tokenizer: {
                autocomplete: {
                  type: 'edge_ngram',
                  min_gram: 1,
                  max_gram: 20,
                  token_chars: ['letter', 'digit'],
                },
              },
            },
          },
          mappings: {
            properties: {
              author: {
                type: 'text',
                fields: {
                  autocomplete: {
                    type: 'text',
                    analyzer: 'autocomplete_analyzer',
                    search_analyzer: 'autocomplete_search_analyzer',
                  },
                },
                analyzer: 'standard',
              },
            },
          },
        },
      });
    }
  } catch (error) {
    logger.error({ error }, `Error setting up ${index} index`);
  }
};
```

For starters, ElasticSearch will complain if we try to create an index if it already exists. To save ourselves from the error spam in case ElasticSearch data was stored and the index already exists, an extra if-statement was added.

It was decided to use the [Edge N-gram tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-edgengram-tokenizer.html). To make use of that we'd need to set up a [custom analyzer](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-custom-analyzer.html) which would be set to use our custom [tokenizer](https://www.google.com/search?client=firefox-b-d&q=elasticsearch+tokenizer) of the `Edge N-gram` type and with our chosen settings. All of this is to be done in the [analysis](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis.html) process inside of index [settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-modules-settings).

After that, we'd need to specifically [map](https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html) which [properties](https://www.elastic.co/guide/en/elasticsearch/reference/current/properties.html) (in this case, `author`) are to use the custom analyzer. For Telescope, I used a trick to designate a custom [field](https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html) inside the `author` property to use the custom analyzers instead. This way, our original search query on the index wouldn't be affected. 

The decision to create an index instead of updating the already existing index is because
1. The `index.create()` function we use to index posts will create the index (if it doesn't already exist), and add a document to this index. If there weren't any previous settings and mappings, it'd only apply the standard options. 
2. We can [only update an index setting on a closed index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html#update-settings-analysis), this would involve closing, updating, and re-opening the index. 
3. Furthermore, the [Update Settings API](https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-update-settings.html) hasn't been implemented yet.

Now with this new indexing, while searching, instead of querying on the `author` property for an exact match, we can now search on the `author.autocomplete` field for more detailed, possible matches. 
```js
    query: {
      match: {
        'author.autocomplete': {
          query: author,
          operator: 'and',
        },
      },
    },
    highlight: {
      fields: {
        'author.autocomplete': {},
      },
    },
```
[highlight](https://www.elastic.co/guide/en/elasticsearch/reference/current/highlighting.html) is an ES Search option to show where the query matches are in the search results of our designated field. 

Since each author could have more than one post, we had a problem with multiples of the same author showing up in the queries. [Josue](https://github.com/manekenpix) (alumni) introduced me to the [reduce()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/Reduce) function as a possibility to deal with it. 

Now a query on the new `authors/autocomplete/author=r l` will look something like this. 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rqzg9t6wb496c46xwha8.png)

---

## Conclusion

This may have looked easy, but one thing I learned the hard way was that we had to specify `settings` and `mappings` in the `body` in JavaScript mode, while this wasn't needed in console mode. Hopefully this knowledge will save time for anyone else implementing autocomplete stuff in the future.

Amusingly, before I got a chance to break Telescope, I also encountered Telescope seemingly breaking my WSL, during the many times of re-indexing and re-building docker images. 

During my local runs I like to run
```terminal
docker-compose --env-file ./config/env.development up --build (whatever images I want to build)
```
and
```
docker-compose --env-file ./config/env.development down
```
to build and stop images. 
Sometimes, my WSL trips up and returns a message such as this.
```
Traceback (most recent call last):
  File "docker-compose", line 3, in <module>
  File "compose/cli/main.py", line 81, in main
  File "compose/cli/main.py", line 200, in perform_command  
  File "compose/cli/command.py", line 40, in project_from_options
  File "compose/config/environment.py", line 67, in from_env_file
  File "compose/config/environment.py", line 57, in _initialize
FileNotFoundError: [Errno 2] No such file or directory      
[22287] Failed to execute script docker-compose
```
When this happens, my solution is to close all the existing terminals, open up WSl, and run the commands again. Doing it this way always works, so my only conclusion is that somehow my WSL got confused and messed up in between running all those commands.

The next part is to try get this indexing into Parser, which surprisingly giving me some problems. Then, implementing autocomplete in the front-end. Finally, it'd be to add some tests so we can catch errors when we try to update to ES 8.0+.