Handling synonyms in Neo4j Full Text Search
Tags: neo4j, lucene, search, synonym
Published: 2019-11-11 11:44:19

So you have followed the [Deep Dive into Neo4j's Full Text Search tutorial](https://graphaware.com/neo4j/2019/01/11/neo4j-full-text-search-deep-dive.html), learned even how to [create custom analyzers](https://graphaware.com/neo4j/2019/09/06/custom-fulltext-analyzer.html) and finally watched the [Full Text Search tips and tricks](https://graphaware.com/resources/#video-full-text-search-nodes-19) talk at the [Neo4j](https://graphaware.com/neo4j-consultancy/) Nodes19 online conference?

Still, searching for `boat` does not yield results containing `yacht` or `ship`, and you're wondering how to make your search engine a bit more relevant for your users?  

Don't go any further, you'll learn how to do it, now!

## Synonyms

A synonym is a word or phrase that means exactly or nearly the same as another word or phrase.

## Why synonyms ? 

It's all about recall! In other words, to help your users find the content they're interested in without them having to know specific terms.

A user searching for `coffee` should probably be seeing results containing `latte macchiato`, `espresso` or even `ristretto`.

## Lists of synonyms

You can find 3rd party word lists for synonyms, such as WordNet or ConceptNet5, howeveer, appropriate word lists are domain/application/use-case dependent, and the best fit is generally a self-curated synonyms word list.

## How to use them ?

The first thing to do, is to create a word list with the following format :

```text
coffee,latte macchiato,espresso,ristretto
boat,yacht,sailing vessel,ship
fts,full text search, fulltext search
```

The next step is to create a custom analyzer using the synonym filter. Since we're using an analyzer the first question that might come to mind is :

> Do I have to reindex all the documents when my synonyms list change ?

The answer is yes, using a query time synonym filter is _very bad(TM)_, for the following reasons :

* The QueryParser tokenizes before giving the text to the analyzer, so if a user searches for `sailing vessel`, the analyzer will be given the words `sailing` and `vessel` separately, and will not know they match a synonym

* Multi-Word synonyms will also not work in phrase queries

* The IDF of rare synonyms will be boosted

More information can be found in the [Solr documentation](https://cwiki.apache.org/confluence/display/solr/AnalyzersTokenizersTokenFilters#solr.SynonymFilterFactory).

Let's create our custom analyzer for synonyms then : 


```java
@Service.Implementation(AnalyzerProvider.class)
public class SynonymAnalyzer extends AnalyzerProvider {

    public static final String ANALYZER_NAME = "synonym-custom";

    public SynonymAnalyzer() {
        super(ANALYZER_NAME);
    }

    @Override
    public Analyzer createAnalyzer() {
        try {
            String synFile = "synonyms.txt";
            Analyzer analyzer = CustomAnalyzer.builder()
                    .withTokenizer(StandardTokenizerFactory.class)
                    .addTokenFilter(StandardFilterFactory.class)
                    .addTokenFilter(SynonymFilterFactory.class, "synonyms", synFile)
                    .addTokenFilter(LowerCaseFilterFactory.class)
                    .build();

            return analyzer;
        } catch (Exception e) {
            throw new RuntimeException("Unable to create analyzer", e);
        }
    }

    @Override
    public String description() {
        return "The default, standard analyzer with a synonyms file. This is an example analyzer for educational purposes.";
    }
}
```

A *very* important note is that the `LowerCaseFilter` comes after the `SynonymFilter`, in some use cases it causes synonyms to not be recognized, for example with the following list : 

```
GB,gibabyte
```

If the lowercase filter is applied before synonyms, then the tokens will not match.


Create a `synonyms.txt` file with your synonyms list in the `conf/` directory of your Neo4j instance :

`conf/synonyms.txt`
```text
coffee,latte macchiato,espresso,ristretto
boat,yacht,sailing vessel,ship
fts,full text search, fulltext search
```

Build your analyzer jar and put it in the `plugins` directory of Neo4j and restart the database if needed.

### Create the Index

```cypher
CALL db.index.fulltext.createNodeIndex(
  'syndemo', 
  ['Article'], 
  ['text'], 
  {analyzer:'synonym-custom'}
)
```

Create an Article node with some text :


```cypher
CREATE (n:Article {text: "This is an article about Full Text Search and Neo4j, let's go !"})
```

### Query the index : 

```cypher
CALL db.index.fulltext.queryNodes('syndemo', 'fts')
```

```
╒══════════════════════════════════════════════════════════════════════╤══════════════════╕
│"node"                                                                │"score"           │
╞══════════════════════════════════════════════════════════════════════╪══════════════════╡
│{"text":"This is an article about Full Text Search and Neo4j, let's go│1.2616268396377563│
│ !"}                                                                  │                  │
└──────────────────────────────────────────────────────────────────────┴──────────────────┘
```

Similarly, a search for `fulltext` will return the result as well. But let's get fancy, heuu `fuzzy !` :

```cypher
CALL db.index.fulltext.queryNodes('syndemo', 'fullt*')

No results, no records
```

### Prefix and synonyms ? 

There is one limitation : prefix,fuzzy,.. queries do not use the analyzer, they produce term or multiterm queries instead.

But there is a trick you can use, add an `NgramFilter` to your analyzer and use a phrase query, so fts and its synonyms will have their ngrams tokenized and stored/retrieved in the index :


```java
Analyzer analyzer = CustomAnalyzer.builder()
                    //...
                    .addTokenFilter(NGramFilterFactory.class, "minGramSize", "3", "maxGramSize", "5")
                    .build();

            return analyzer;
```


The `NgramTokenFilter` will tokenize the inputs into n-grams of the given sizes, here min 3 and max 5. So for the following input : 

```
fulltext search
```

The index will contain the n-grams `ful, full, fullt, ull, ullt, ullte, lte, ltex, ltext`.

You can also use the `EdgeNgramFilter` will will produce n-grams only from the beginnig of the token, for the same example as above the n-grams will be `ful, full, fullt`.


Re-deploy your plugin, restart the database, drop and recreate the index and now : 

```cypher
CALL db.index.fulltext.queryNodes('syndemo', '"fullt*"')

╒══════════════════════════════════════════════════════════════════════╤═══════════════════╕
│"node"                                                                │"score"            │
╞══════════════════════════════════════════════════════════════════════╪═══════════════════╡
│{"text":"This is an article about Full Text Search and Neo4j, let's go│0.04872262850403786│
│ !"}                                                                  │                   │
└──────────────────────────────────────────────────────────────────────┴───────────────────┘
```


To finalize, let's try some other phrase queries : 

```cypher
CALL db.index.fulltext.queryNodes('syndemo', '"article fullte*"~2')

╒══════════════════════════════════════════════════════════════════════╤══════════════════╕
│"node"                                                                │"score"           │
╞══════════════════════════════════════════════════════════════════════╪══════════════════╡
│{"text":"This is an article about Full Text Search and Neo4j, let's go│2.3429081439971924│
│ !"}                                                                  │                  │
└──────────────────────────────────────────────────────────────────────┴──────────────────┘
```


## Conclusion


Synonyms are a valuable asset when building search engines, offering a better recall and thus a better user experience. 

You can find all the code from this blog post on this example [Github repository](https://github.com/graphaware/neo4j-full-text-search-extra)