Elasticsearch Tokenizers – Word Oriented Tokenizers
Tags: elasticsearch, tokenizers, word
Published: 2021-05-22 10:09:56

https://grokonez.com/elasticsearch/elasticsearch-tokenizers-word-oriented-tokenizers

Elasticsearch Tokenizers – Word Oriented Tokenizers

A tokenizer breaks a stream of characters up into individual tokens (characters, words...), then outputs a stream of tokens. We can also use tokenizer to record the order or position of each term (for phrase and word proximity queries), or the start and end character offsets of the original word which the term represents (for highlighting search snippets).

In this tutorial, we're gonna look at how to use some Word Oriented Tokenizers which tokenize full text into individual words.

<!--more-->
<h2>1. Standard Tokenizer</h2>
<code>standard</code> tokenizer provides grammar based tokenization:
<pre><code class="language-json">
POST _analyze
{
  "tokenizer": "standard",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
</code></pre>
Token:
<pre><code class="language-json">
{
  "tokens": [
    {
      "token": "The",
      "start_offset": 0,
      "end_offset": 3,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "2",
      "start_offset": 4,
      "end_offset": 5,
      "type": "<NUM>",
      "position": 1
    },
    {
      "token": "QUICK",
      "start_offset": 6,
      "end_offset": 11,
      "type": "<ALPHANUM>",
      "position": 2
    },
    {
      "token": "Brown",
      "start_offset": 12,
      "end_offset": 17,
      "type": "<ALPHANUM>",
      "position": 3
    },
    {
      "token": "Foxes",
      "start_offset": 18,
      "end_offset": 23,
      "type": "<ALPHANUM>",
      "position": 4
    },
    {
      "token": "jumped",
      "start_offset": 24,
      "end_offset": 30,
      "type": "<ALPHANUM>",
      "position": 5
    },
    {
      "token": "over",
      "start_offset": 31,
      "end_offset": 35,
      "type": "<ALPHANUM>",
      "position": 6
    },
    {
      "token": "the",
      "start_offset": 36,
      "end_offset": 39,
      "type": "<ALPHANUM>",
      "position": 7
    },
    {
      "token": "lazy",
      "start_offset": 40,
      "end_offset": 44,
      "type": "<ALPHANUM>",
      "position": 8
    },
    {
      "token": "dog's",
      "start_offset": 45,
      "end_offset": 50,
      "type": "<ALPHANUM>",
      "position": 9
    },
    {
      "token": "bone",
      "start_offset": 51,
      "end_offset": 55,
      "type": "<ALPHANUM>",
      "position": 10
    }
  ]
}
</code></pre>
To keep things simple, we can write term from tokens in this way:
<pre><code class="language-json">
[ The, 2, QUICK, Brown, Foxes, jumped, over, the, lazy, dog's, bone ]
</code></pre>
<h3>Max Token Length</h3>
We can configure maximum token length (<code>max_token_length</code> - Defaults to 255).
If a token exceeds this length, it is split at <code>max_token_length</code> intervals.
For example, we set <code>max_token_length</code> to 4, it makes <strong>QUICK</strong> separate to <strong>QUIC</strong> and <strong>K</strong>.

More at:

https://grokonez.com/elasticsearch/elasticsearch-tokenizers-word-oriented-tokenizers

Elasticsearch Tokenizers – Word Oriented Tokenizers