Error loading KenLM for beam decoding with ngram
State: closed | Pull request: no
Labels: name: bug


[data.zip](https://github.com/NVIDIA/NeMo/files/8322010/data.zip)
**Describe the bug**

I am trying to decode hindi speech file with recently released STT Hi Conformer-CTC Medium.
Once loaded AM and LM I run script scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py as described in tutorial.
After passing greedy decoging phase, script starts the beam search decoding but fails to load KenLM binary.

**Steps/Code to reproduce bug**

1. Download the latest NeMo docker container with using command:   
docker pull nvcr.io/nvidia/nemo:22.01
2. Run docker container using command:  
docker run --runtime=nvidia -it --rm -v --shm-size=16g --ulimit memlock=-1 --ulimit stack=67108864 \
           --mount=type=bind,source=/mnt/asr/korenevsky,target=/mnt/asr/korenevsky nvcr.io/nvidia/nemo:22.01
3. Download stt_hi_conformer_ctc_medium.nemo and 5-gram-kenlm.bin from https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_hi_conformer_ctc_medium and put them into the folder /mnt/asr/korenevsky/test_nemo_hindi/ which is mouned to docker container.
4. Put wav-file to recognize and corresponding json manifest into /mnt/asr/korenevsky/test_nemo_hindi/data (see attached zip-file)
5. Run decoding script using following command
`python /workspace/nemo/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py \
                                     --nemo_model_file /mnt/asr/korenevsky/test_nemo_hindi/stt_hi_conformer_ctc_medium.nemo \
                                     --input_manifest /mnt/asr/korenevsky/test_nemo_hindi/data/manifest.json \
                                     --kenlm_model_file /mnt/asr/korenevsky/test_nemo_hindi/5-gram-kenlm.bin \
                                     --acoustic_batch_size 32 \
                                     --beam_width 128 \
                                     --beam_alpha 1.5 \
                                     --beam_beta 2.0 \
                                     --preds_output_folder /mnt/asr/korenevsky/test_nemo_hindi/data/predictions \
                                     --decoding_mode beamsearch_ngram
`

This causes the following messages:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/modules/beam_search_decoder.py", line 64, in __init__
    from ctc_decoders import Scorer, ctc_beam_search_decoder_batch
ModuleNotFoundError: No module named 'ctc_decoders'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/nemo/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py", line 357, in <module>
    main()
  File "/workspace/nemo/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py", line 341, in main
    beam_search_eval(
  File "/workspace/nemo/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py", line 71, in beam_search_eval
    beam_search_lm = nemo_asr.modules.BeamSearchDecoderWithLM(
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/modules/beam_search_decoder.py", line 66, in __init__
    raise ModuleNotFoundError(
ModuleNotFoundError: BeamSearchDecoderWithLM requires the installation of ctc_decoders from scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoder
s.sh

6. Install Beamsearch decoder using following commands:
export BOOST_ROOT=/mnt/asr/korenevsky/boost   (otherwise CMake error occures)
cd /workspace/nemo/scripts/asr_language_modeling/ngram_lm
bash ./install_beamsearch_decoders.sh

7. Then run step 5 again. The following messages appear:
[NeMo I 2022-03-22 06:40:39 eval_beamsearch_ngram:325] ==============================Starting the beam search decoding===============================
[NeMo I 2022-03-22 06:40:39 eval_beamsearch_ngram:326] Grid search size: 1
[NeMo I 2022-03-22 06:40:39 eval_beamsearch_ngram:327] It may take some time...
[NeMo I 2022-03-22 06:40:39 eval_beamsearch_ngram:328] ==============================================================================================
terminate called after throwing an instance of 'lm::FormatLoadException'
  what():  kenlm/lm/vocab.cc:52 in void lm::ngram::{anonymous}::ReadWords(int, lm::EnumerateVocab*, lm::WordIndex, uint64_t) threw FormatLoadException because `expected_count != index'.
The binary file has the wrong number of words at the end.  This could be caused by a truncated binary file.
./test_beam_1file.sh: line 10:  1454 Aborted                 (core dumped) python /workspace/nemo/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py --nemo_model_file /mnt/asr/korenevsky/test_nemo_hindi/stt_hi_conformer_ctc_medium.nemo --input_manifest /mnt/asr/korenevsky/test_nemo_hindi/data/manifest.json --kenlm_model_file /mnt/asr/korenevsky/test_nemo_hindi/5-gram-kenlm.bin --acoustic_batch_size 32 --beam_width 128 --beam_alpha 1.5 --beam_beta 2.0 --preds_output_folder /mnt/asr/korenevsky/test_nemo_hindi/data/predictions --decoding_mode beamsearch_ngram

But KenLM 5-gram-kenlm.bin can be loaded successfully with KenLM python interface, so it doesn't seem to be truncated.

**Expected behavior**

Successful beam search decoding

**Environment overview (please complete the following information)**
docker pull nvcr.io/nvidia/nemo:22.01
docker run --runtime=nvidia -it --rm -v --shm-size=16g --ulimit memlock=-1 --ulimit stack=67108864 \
           --mount=type=bind,source=/mnt/asr/korenevsky,target=/mnt/asr/korenevsky nvcr.io/nvidia/nemo:22.01

**Environment details**
setting environment variable BOOST_ROOT to point Boost build

**Additional context**
GPU A100 was used