Is there a way to modify Offline_ASR tutorial so that it builds KenLM binary file?
State: closed | Pull request: no

Hello, thank you again for the amazing project. I am trying to modify the code in https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/asr/Offline_ASR.ipynb#scrollTo=I9eIxAyKHREB.

Am I correct, based on (https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/zh/latest/asr/tutorial.html#using-kenlm) that this does not actually build the KenLM binary? 

When I try to use the train_kenlm.py script, it says I don't have /content/NeMo/decoders/kenlm/build/bin/lmplz, but reading through the KenLM README and the various setup.sh and setup.py scripts in NeMo and OpenSeq2Seq, I cannot seem to get it to build this /bin folder. I am sure this is my user error :-)

Thanks!

These docs are quite outdated, could you point me to where they were referenced so we can update them ?

These are the latest docs - https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/

This section (https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/asr_language_modeling.html)  has the steps to build KenLM and train a KenLM model, plus perform beam search using it. 

Thanks! This answer has a 404 link, so I searched for the link I shared: https://github.com/NVIDIA/NeMo/issues/577

In the link above, I see that it mentions install_beamsearch_decoders.sh to install the decoders. But does this build the KenLM binary folder (kenlm/build/bin)? When I run install_beamsearch_decoders.sh, then run train_kenlm.py, it gives me the error decoders/kenlm/build/bin/lmplz is not found, so I believe I am missing a step where I build the binary?

Thanks!

EDITED with fixed github issue link

I fixed my user errors, but wanted to share what worked for me if others try to compile KenLM on Google Colab:

1) I updated cmake to latest version: 3.20, using method B here: https://askubuntu.com/questions/355565/how-do-i-install-the-latest-version-of-cmake-from-the-command-line (takes about 10 minutes, and I hard-coded the version and build numbers, as it failed with variables for some reason)
2) I ran KenLM build instructions after running install_beamsearch_decoders.sh, using here: https://github.com/kpu/kenlm/blob/master/BUILDING

Thank you so much, Somshubra, for pointing me in the right direction and responding so quickly! If I could make a constructive comment, I might add into the user guide you linked to that one needs to explictly compile KenLM after running install_beamsearch_decoders.sh. I had been unsure if that was being done by the script itself (since it does download KenLM and creates a build folder).

That's a very good point, we'll update the docs to reflect that KenLM must still be compiled after performing the setup steps. @vsl9 @VahidooX  fyi

The script 'install_beamsearch_decoders.sh' has been already updated to install the binary kenlm. I think @vsl9 can verify this.
Here is the commit in the openseq2seq:
https://github.com/NVIDIA/OpenSeq2Seq/commit/5cb391bba14a09142b82c68c5de17d52ab98b51f

Did you get any error message when ran the script?

If the script could not install the binary kenlm, then it can be a bug or some of the dependencies of the kenlm was not installed. You may find more info on its dependencies here:
https://kheafield.com/code/kenlm/dependencies/

I am going to add more hints and instructions for tackling such issues in the doc. 

> The script 'install_beamsearch_decoders.sh' has been already updated to install the binary kenlm. I think @vsl9 can verify this.
> Here is the commit in the openseq2seq:
> [NVIDIA/OpenSeq2Seq@5cb391b](https://github.com/NVIDIA/OpenSeq2Seq/commit/5cb391bba14a09142b82c68c5de17d52ab98b51f)
> 
> Did you get any error message when ran the script?
> 
> If the script could not install the binary kenlm, then it can be a bug or some of the dependencies of the kenlm was not installed. You may find more info on its dependencies here:
> https://kheafield.com/code/kenlm/dependencies/
> 
> I am going to add more hints and instructions for tackling such issues in the doc.

Yes, at first the script could not install the binary kenlm because I was missing Boost package, so I had to install dependencies as you mentioned :
`sudo apt-get install build-essential libboost-all-dev cmake zlib1g-dev libbz2-dev liblzma-dev` 
After that, I rerun `install_beamsearch_decoders.sh` script and everything were installed correctly.