Transcribe() WER results differ from validation results for Fastconformer Hybrid RNNT-CTC
State: closed | Pull request: no
Labels: name: bug

name: stale

name: ASR

**Describe the bug**
Hello and, first of all, thank you for your work! I have the following problem:

While training a Fastconformer Hybrid RNNT-CTC model, I perform multi-validation on several datasets at the end of each epoch. The reported results on these datasets at validation, greatly differ from the results obtained when using **speech_to_text_eval.py** (or inherently the **transcribe_speech.py** script). They differ by a margin of +-2% WER on almost all datasets, some having smaller differences than others. (for default RNNT decoding) When I trained the normal fastconformer_transducer model, I haven't experienced such big gaps in WER between training and offline inference.

I have tried providing the model as it is to the _model_path_ argument or firstly changed the decoder to rnnt and then load it, but I obtained same results in both cases. I am also aware of the _dither_ argument in the _preprocessor_ from other issues, and tried changing it to **0**, but didn't obtain any change in the results. 

I have trained the model using bfloat16, but at inference i can't use bfloat16 as I am using a **CPU only** machine, might this be a reason for such a great difference? Should I use a different script for Hybrid models? 

Differences for dataset 1: (10.93% train vs 11.68% inf)
![ssc-eval1_merge](https://github.com/NVIDIA/NeMo/assets/53811655/21a53f1c-1d6d-4578-b7fc-9fd5dc5fc778)

Differences for dataset 2: (12.84% train vs 14.08% inf)
![ssc-eval2_merge](https://github.com/NVIDIA/NeMo/assets/53811655/a297c56d-b9de-40c8-96dd-abbc6307ca1e)

Differences for dataset 3: (3.149% train vs 2.94% inf)
![rsc_eval_merge](https://github.com/NVIDIA/NeMo/assets/53811655/ac0a84fc-de77-47e4-b92e-5fdc452ac6c9)

**Steps/Code to reproduce bug**

Config file used for training:


https://drive.google.com/file/d/1S4eLgDMSsRoLYYxqey-kPOTOHjZfwCOA/view?usp=drive_link


Script used:
```
python3 speech_to_text_eval.py model_path=<path_to_hybrid_model> dataset_manifest=<path_to_manifest_file> output_filename=<path_to_out_json>
```

The **transcribe_speech.py** and **speech_to_text_eval.py** are unchanged. The **config for the fastconformer hybrid** model is almost the same as the normal one.

Unfortunately, I can't provide the datasets I trained/evaluated on, but I also tried to reproduce the experiments on Common Voice Spanish 7.0 dataset (cv-corpus-7.0-2021-07-21), but with a much smaller difference between the results..

Results on dev-set: (6.96% train vs 6.86% inf)
![cv_dev_merge](https://github.com/NVIDIA/NeMo/assets/53811655/ffedb14f-bb04-4f43-bbec-4df1cc662d37)

**Expected behavior**

Same or similar results ( difference of 0.1-0.2% WER ) between validation and offline inference (when using transcribe_speech.py) for consistent training.

**Environment details**

- pytorch-lightning             1.9.4
- torch                         2.1.0a0+fe05266
- torch-tensorrt                1.4.0.dev0
- torchaudio                    2.1.0
- torchmetrics                  0.11.4
- torchvision                   0.15.0a0
- python         3.8.10

**Environment overview (please complete the following information)**

 - Tested offline inference using the main branch code of NeMo repo at 18.09.2023 .

**Additional context**

Trained on A100 gpu with bf16 precision, inference on CPU only. I am also using online noise and speed augmentations while training. I initialiazed the Hybrid model from a transducer pretrained model, but I obtained the same behavior (big differences in WER) while also initializing from a hybrid model.