Different behavior in ASR prediction if handle 1 element or many in manifest
State: closed | Pull request: no
Labels: name: bug

I was using methods from this [tutorial for ASR](https://colab.research.google.com/github/NVIDIA/NeMo/blob/r1.0.0b3/tutorials/asr/01_ASR_with_NeMo.ipynb) and you provide two possible ways for inference. 
1. model.transcribe()
2. [Pure PyTorch inference loop](https://colab.research.google.com/github/NVIDIA/NeMo/blob/r1.0.0b3/tutorials/asr/01_ASR_with_NeMo.ipynb#scrollTo=7mP4r1Gx_Ilt&line=5&uniqifier=1)
I have trained russian language quartznet15x5 and when I use method "1."  my output is [""] predicting nothing. When I use method "2." and handle my test dataset (500 audio .wav files) it gives pretty good predictions and for the same audio file I get correct prediction insted of [""]. However, when I pass manifest with only 1 audio file insted of ~500 prediction gets worse, not getting the same correct prediction as with many audio files in manifest. For the method '2.' it is not correct because prediction of a single audio file does not affect the prediction of another. Can you explain the problems with both "1" and "2" methods? If it is needed I can provide the same example with english model not russian. 

What exactly do you mean by "Method 1" and "Method 2". Could you please paste code snippets? 

"Method 1. "
```
print(first_asr_model.transcribe(paths2audio_files=['pass path to .wav file']))
```
"Method 2."
```
# Bigger batch-size = bigger throughput
params['model']['validation_ds']['batch_size'] = 16

# Setup the test data loader and make sure the model is on GPU
first_asr_model.setup_test_data(test_data_config=params['model']['validation_ds'])
first_asr_model.cuda()

# We will be computing Word Error Rate (WER) metric between our hypothesis and predictions.
# WER is computed as numerator/denominator.
# We'll gather all the test batches' numerators and denominators.
wer_nums = []
wer_denoms = []

# Loop over all test batches.
# Iterating over the model's `test_dataloader` will give us:
# (audio_signal, audio_signal_length, transcript_tokens, transcript_length)
# See the AudioToCharDataset for more details.
for test_batch in first_asr_model.test_dataloader():
        test_batch = [x.cuda() for x in test_batch]
        targets = test_batch[2]
        targets_lengths = test_batch[3]        
        log_probs, encoded_len, greedy_predictions = first_asr_model(
            input_signal=test_batch[0], input_signal_length=test_batch[1]
        )
        # Notice the model has a helper object to compute WER
        first_asr_model._wer.update(greedy_predictions, targets, targets_lengths)
        _, wer_num, wer_denom = first_asr_model._wer.compute()
        wer_nums.append(wer_num.detach().cpu().numpy())
        wer_denoms.append(wer_denom.detach().cpu().numpy())

# We need to sum all numerators and denominators first. Then divide.
print(f"WER = {sum(wer_nums)/sum(wer_denoms)}")
```
Did not changed and took it from tutorial.

@okuchaiev  any comments why is it so?

Could this be because of Batch Normalization?

In method two, I dont see any line where you call `model.eval()`. Assuming the notebook is being run multiple times, the model can predict random items if its still in train mode when being called manually - batch effects will be in play since batchnorm will try updating running statistics.

To go one step further wrap the first and second method with :
```python

asr_model.freeze()
with torch.no_grad():
   ...
```

This is generally not necessary though, and simply calling model.eval() along with torch.no_grad() should do the trick.

@topusOctopus  Theres also a PR which should (hopefully) simplify transcription for all NeMo ASR models in the future - can you take my branch and give your eval a try? https://github.com/NVIDIA/NeMo/pull/1786

@titu1994 I will try your suggestion today