Adding N-gram LM for ASR Models
State: closed | Pull request: yes

This PR adds the support for training and evaluating N-gram LM for ASR models. 

The N-gram language model is trained with KenLM library (https://github.com/kpu/kenlm) which can be used with the beam search decoders on top of the ASR models. It supports both character level and BPE level encodings and models which is detected automatically from the type of the model. 
The KenLM library and the beam search decoders need to get installed to use this feature. Please refer to 'scripts/ngram_lm/install_beamsearch_decoders.sh' on how to install them.

How to train an N-gram LM:

```
python train_kenlm.py --nemo_model_file <path to the .nemo file of the model> \
                              --train_file <path to the training text or json manifest file \
                              --kenlm_bin_path <path to the bin folder of KenLM library> \
                              --kenlm_model_file <path to store the binary KenLM model> \
                              --ngram_length <order of N-gram model>
```

After training is done, the binary LM model is stored at the path specified by '--kenlm_model_file'. Then you may use 'scripts/ngram_lm/eval_beamsearch_ngram.py' to evaluate it on an ASR model. This script would evaluate an N-gram language in fusion with beam search decoders on top of a trained ASR model. NeMo's beam search decoders are capable of using the KenLM's N-gram models to find the best candidates. 

You may evaluate a model with N-gram LM and beam search decoding as the following:

```
 python eval_beamsearch_ngram.py --nemo_model_file <path to the .nemo file of the model> \
                                         --input_manifest <path to the evaluation Json manifest file \
                                         --kenlm_model_file <path to the binary KenLM model> \
                                         --beam_width <list of the beam widths> \
                                         --beam_alpha <list of the beam alphas> \
                                         --beam_width <list of the beam betas> \
                                         --preds_output_folder <optional folder to store the predictions> \
                                         --decoding_mode beam_search_ngram
```

The beam search decoder would calculated the scores as:
```
final_score = acoustic_score + beam_alpha * lm_score + beam_beta * seq_length

beam_alpha: specifies the The amount of importance to place on the N-gram language model.
beam_beta: a penalty term to consider the sequence length in the scores. Negative values for beta will give penalty to longer sequences and make the decoder to prefer shorter predictions, while positive values would result in longer candidates.

```

The results would be reported in Word Error Rate (WER) and Character Error Rate (CER). The results if the best candidate is selected from the candidates is also reported as the best WER/CER. It can show how good the predicted candidates are.

Hyperparameter Grid Search:
Beam search decoding with N-gram LM has three main hyperparameters: beam_width, beam_alpha, and beam_beta. The accuracy of the model is dependent to the values of these parameters, specially beam_alpha and beam_beta. You may specify a single or list of values for each of these parameters to perform grid search. It would perform the beam search decoding on all the combinations of the these three hyperparameters. For instance, the following set of parameters would results in 2*1*2=4 beam search decodings:

```
python eval_beamsearch_ngram.py ... \
                     --beam_width 64 128 \
                     --beam_alpha 1.0 \
                     --beam_beta 1.0 0.5 \
```
The user can select the decoding strategy from the following list:
--decoding_mode: ["greedy", "beamsearch", "beamsearch_ngram"]

The "beamsearch" mode would ignore the LM and calculates the performs regular beam search decoding.

This pull request **introduces 2 alerts** when merging a5a058a6d13568253e1a2e49d996cdc6cda8c522 into 21a17b267fac68d4cdd20f3969a580a0a40dbdb4 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d8d2c21d7f0c2b2b08d057ea93e8c9524bb8826e)

**new alerts:**

* 1 for Unused local variable
* 1 for Variable defined multiple times

This pull request **introduces 4 alerts** when merging f65dcb8fc43a37883b2343b497f3b18df9d9f543 into 21a17b267fac68d4cdd20f3969a580a0a40dbdb4 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5835aa9bd664c563372cbc59bb7d9fb020b375b7)

**new alerts:**

* 2 for Unused local variable
* 1 for Unused import
* 1 for Variable defined multiple times

This pull request **introduces 4 alerts** when merging 2430c462a3149d4f23282ee4656669d72583466b into 21a17b267fac68d4cdd20f3969a580a0a40dbdb4 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-75fdc635019f8a33e0eb0f85f8233f3dd0a07f64)

**new alerts:**

* 3 for Unused local variable
* 1 for Variable defined multiple times

This pull request **introduces 1 alert** when merging 5b4ec59ec2d5ba1d08cee6480a6ed1fe7b67cda4 into 21a17b267fac68d4cdd20f3969a580a0a40dbdb4 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b971f3c77917d12253297b3fdb3113f788478f88)

**new alerts:**

* 1 for Unused local variable

This pull request **introduces 1 alert** when merging 52939fe8ce2bd485df8ac801f452d96b4d43a4e0 into ba027d0e88a588ed4bd7e4142366db127c396b2d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-38e635a5f00f4add2b108de3d5ba48046f96a9a9)

**new alerts:**

* 1 for Unused local variable

This pull request **introduces 1 alert** when merging abb0cbed21a24e9684f154cb59386224f138f8b1 into 54e6f6ee688f09810d3e54661275fd5c8718db00 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-cf3527523c3b634b3e0cdec6d178dad54e2d265f)

**new alerts:**

* 1 for Testing equality to None