issues with eval_beamsearch_ngram.py
State: closed | Pull request: no
Labels: name: bug

This is really two issues but the first one is really simple and pertains to the same script.

I'm using a fastconformer hybrid RNNT-CTC model and have trained an LM with Kenlm so i can decode with eval_beamsearch_ngram.py. However, I'm seeing two issues so far:

1. When specifying the probs_cache_file flag to store the results, the code errors out if i point to a file in a directory that does not exist yet. Can you add a quick check to ensure the directory is made if it doesn't already exist (this would be around line 337 in https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py#L337C13-L337C13)?

2. When i select a single value for each of the various beam hyperparameters (beam_width, beam_alpha, beam_beta) everything works fine. However, when i try to do a grid search of beam_width, beam_alpha, and beam_beta then i get an error. It seems that the model_decoding_strategy is getting reset (https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py#L143) but then isn't updated correctly, particularly because I'm using the hybrid RNNT-CTC model which requires special handling.

For example, when I run this:
```
python ~/git/NeMo/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py \
  nemo_model_file=./results/stt_ft_bsz128_lr0.0020_wu1000/checkpoints/stt_ft_bsz128_lr0.0020_wu1000.nemo \
  input_manifest=./data/manifest.debug.json \
  kenlm_model_file=./data/lm/lm.bin \
  beam_alpha=[-0.1,0.1] \
  beam_beta=[10] \
  beam_width=[10] \
  preds_output_folder=./foo_lm \
  probs_cache_file=./foo_lm/probs.pkl \
  decoding_mode=beamsearch_ngram \
  decoding_strategy=flashlight \
  beam_batch_size=128 \
  use_amp=True \
  acoustic_batch_size=4
```
I get the following error:
```
Traceback (most recent call last):
  File "/panfs/g52-panfs/home/jo23828/git/NeMo/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py", line 457, in <module>
    main()
  File "/panfs/g52-panfs/home/jo23828/git/NeMo/nemo/core/config/hydra_runner.py", line 126, in wrapper
    _run_hydra(
  File "/panfs/g52-panfs/exp/venv/jo23828/.conda/envs/nemo_r1.21.0/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/panfs/g52-panfs/exp/venv/jo23828/.conda/envs/nemo_r1.21.0/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/panfs/g52-panfs/exp/venv/jo23828/.conda/envs/nemo_r1.21.0/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/panfs/g52-panfs/exp/venv/jo23828/.conda/envs/nemo_r1.21.0/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/panfs/g52-panfs/exp/venv/jo23828/.conda/envs/nemo_r1.21.0/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/panfs/g52-panfs/exp/venv/jo23828/.conda/envs/nemo_r1.21.0/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/panfs/g52-panfs/exp/venv/jo23828/.conda/envs/nemo_r1.21.0/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/panfs/g52-panfs/exp/venv/jo23828/.conda/envs/nemo_r1.21.0/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/panfs/g52-panfs/home/jo23828/git/NeMo/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py", line 417, in main
    candidate_wer, candidate_cer = beam_search_eval(
  File "/panfs/g52-panfs/home/jo23828/git/NeMo/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py", line 143, in beam_search_eval
    model.change_decoding_strategy(None)
  File "/panfs/g52-panfs/home/jo23828/git/NeMo/nemo/collections/asr/models/hybrid_rnnt_ctc_bpe_models.py", line 399, in change_decoding_strategy
    self.decoding = RNNTBPEDecoding(
  File "/panfs/g52-panfs/home/jo23828/git/NeMo/nemo/collections/asr/metrics/rnnt_wer_bpe.py", line 208, in __init__
    super(RNNTBPEDecoding, self).__init__(
  File "/panfs/g52-panfs/home/jo23828/git/NeMo/nemo/collections/asr/metrics/rnnt_wer.py", line 234, in __init__
    raise ValueError(f"Decoding strategy must be one of {possible_strategies}")
ValueError: Decoding strategy must be one of ['greedy', 'greedy_batch', 'beam', 'tsd', 'alsd', 'maes']
```