speech_to_text_with_vad doesn't work correctly with stt_*_conformer_ctc_large models
State: closed | Pull request: no
Labels: name: bug

name: stale

Hi👋 I've been trying to use [this script](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/asr_vad/speech_to_text_with_vad.py) for ASR without VAD, but I keep getting strange results. I've tried this script with 2 models: `stt_en_conformer_ctc_large` and `stt_ru_conformer_ctc_large`.
____________________________________________________________________________________________________________________________________________
For the first model `stt_en_conformer_ctc_large` I used this command:
```
python speech_to_text_with_vad.py \
    manifest_filepath=input_manifest.json \
    asr_model=stt_en_conformer_ctc_large \
    use_rttm=False \
    vad_config=vad_inference_postprocessing.yaml
```
`input_manifest.json`:
```
{"audio_filepath": "an4_diarize_test.wav", "offset": 0, "duration": 5.2, "label": "infer", "text": "eleven twenty seven fifty seven october twenty four nineteen seventy", "num_speakers": 2, "rttm_filepath": "an4_diarize_test.rttm", "uem_filepath": null}
```
This is what I get as a result:
```
{"audio_filepath": "an4_diarize_test.wav", "offset": 0, "duration": 5.2, "label": "infer", "text": "eleven twenty seven fifty seven october twenty four nineteen seventy", "num_speakers": 2, "rttm_filepath": "an4_diarize_test.rttm", "uem_filepath": null, "pred_text": "it's a cretyve new complex na bit very strany you onur", "feature_file": "outputs/features/c45bc24490f3e2395bdd4649bc16f165.pt"}
```
I use the audio `an4_diarize_test.wav` from [this tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Diarization_Inference.ipynb). 
___________________________________________________________________________________________________________________________________________
As for the second model `stt_ru_conformer_ctc_large`, I used this command:
```
python speech_to_text_with_vad.py \
    manifest_filepath=input_manifest.json \
    asr_model=stt_ru_conformer_ctc_large \
    use_rttm=False \
    vad_config=vad_inference_postprocessing.yaml
```
`input_manifest.json`:
```
{"audio_filepath": "/nas/Data/Audio/Processed/RU/Golos/test/crowd/files/c45bc24490f3e2395bdd4649bc16f165.wav", "offset": 0, "duration": 4.84, "label": "infer", "num_speakers": 1, "text": "закажите мне комплексный обед в ресторане в номер"}
```
I use the audio file from the dataset Golos https://github.com/sberdevices/golos , tried it with multiple files from the dataset and got the same result as below:
```
{"audio_filepath": "/nas/Data/Audio/Processed/RU/Golos/test/crowd/files/c45bc24490f3e2395bdd4649bc16f165.wav", "offset": 0, "duration": 4.84, "label": "infer", "num_speakers": 1, "text": "закажите мне комплексный обед в ресторане в номер", "pred_text": " ⁇ ", "feature_file": "/nas/PipelineService/From1Server/speaker_identification/outputs/features/c45bc24490f3e2395bdd4649bc16f165.pt"}
```

I keep getting these results regardless of using VAD or not, I checked it for multiple VAD models (`vad_multilingual_marblenet` and `vad_marblenet`). Also, I tried the same command as above with `stt_ru_conformer_transducer_large` model and it worked just fine:
```
{"audio_filepath": "/nas/Data/Audio/Processed/RU/Golos/test/crowd/files/c45bc24490f3e2395bdd4649bc16f165.wav", "offset": 0, "duration": 4.84, "label": "infer", "num_speakers": 1, "text": "закажите мне комплексный обед в ресторане в номер", "pred_text": "закажите мне комплексный обед в ресторане в номер", "feature_file": "/nas/PipelineService/From1Server/speaker_identification/outputs/features/c45bc24490f3e2395bdd4649bc16f165.pt"}
```
So, it seems to be just conformer_ctc type of models problem.

I use this config file `vad_inference_postprocessing.yaml`:
```
name: &name "vad_inference_postprocessing"

dataset: null # Path of json file of evaluation data. Audio files should have unique names
num_workers: 4
sample_rate: 16000

# functionality
gen_seg_table: True # whether to converting frame level prediction to speech/no-speech segment in start and end times format
write_to_manifest: True # whether to writing above segments to a single manifest json file.

prepare_manifest:
    auto_split: True # whether to automatically split manifest entry by split_duration to avoid potential CUDA out of memory issue.
    split_duration: 400 # try smaller number if you still have CUDA memory issue
    
    
vad:
    model_path: null #.nemo local model path or pretrained model name or none
    parameters: # Parameters were tuned on 0~20db SNR noisy and clean multilingual ASR data.  
      normalize_audio: False
      window_length_in_sec: 0.63  # window length in sec for VAD context input 
      shift_length_in_sec: 0.08 # shift length in sec for generate frame level VAD prediction, Here we use 0.08 for faster inferene
      smoothing: False # false or type of smoothing method (eg: median, mean)
      overlap: 0.875 # overlap ratio for overlapped mean/median smoothing filter. If smoothing=False, ignore this value.
      postprocessing: 
        onset: 0.5 # onset threshold for detecting the beginning and end of a speech 
        offset: 0.3 # offset threshold for detecting the end of a speech. 
        pad_onset: 0.2 # adding durations before each speech segment 
        pad_offset: 0.2 # adding durations after each speech segment 
        min_duration_on: 0.5 # threshold for small non_speech deletion
        min_duration_off: 0.5 # threshold for short speech segment deletion
        filter_speech_first: True 
          
prepared_manifest_vad_input: null # if not specify, it will automatically generated be "manifest_vad_input.json"
frame_out_dir: "vad_frame"
smoothing_out_dir: null # if not specify, it will automatically generated be frame_out_dir + "/overlap_smoothing_output" + "_" + smoothing_method + "_" + str(overlap)
table_out_dir: null # if not specify, it will automatically be frame_out_dir + "/table_output_tmp_" + key and value in postprocessing params
out_manifest_filepath: null # if not specify it will automatically be "vad_out.json"

# json manifest line example
# {"audio_filepath": "/path/to/audio_file", "offset": 0, "duration": null, "label": "infer", "text": "-"}
```

I use nemo ver. 1.15.0.

What could be the issue here? Maybe I'm missing something important in config files? 
Thanks!