ASR with Speaker Diarization not working for 'stt_en_conformer_ctc_xlarge' on GPU
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

ASR with Speaker Diarization not working for 'stt_en_conformer_ctc_xlarge' on colab GPU (Tesla T4)

**Steps/Code to reproduce bug**

Run the aforementioned colab notebook listed in tutorials on a GPU till the following section 'Run ASR and get word timestamps' included. It should raise an Assertion Error on the run_ASR() method.

** Stack Trace**
```
[NeMo I 2022-10-20 03:36:13 speaker_utils:93] Number of files to diarize: 1
[NeMo I 2022-10-20 03:36:13 cloud:66] Downloading from: https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_conformer_ctc_xlarge/versions/1.10.0/files/stt_en_conformer_ctc_xlarge.nemo to /root/.cache/torch/NeMo/NeMo_1.12.0/stt_en_conformer_ctc_xlarge/eff9a44c668d74bfee7b59f7def80bf4/stt_en_conformer_ctc_xlarge.nemo
[NeMo I 2022-10-20 03:37:09 common:910] Instantiating model from pre-trained checkpoint
[NeMo I 2022-10-20 03:37:20 mixins:172] Tokenizer SentencePieceTokenizer initialized with 128 tokens
[NeMo W 2022-10-20 03:37:20 modelPT:143] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config : 
    manifest_filepath:
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket1/tarred_audio_manifest.json
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket2/tarred_audio_manifest.json
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket3/tarred_audio_manifest.json
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket4/tarred_audio_manifest.json
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket5/tarred_audio_manifest.json
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket6/tarred_audio_manifest.json
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket7/tarred_audio_manifest.json
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket8/tarred_audio_manifest.json
    sample_rate: 16000
    batch_size: 1
    shuffle: true
    num_workers: 4
    pin_memory: true
    use_start_end_token: false
    trim_silence: false
    max_duration: 20
    min_duration: 0
    is_tarred: true
    tarred_audio_filepaths:
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket1/audio__OP_0..8191_CL_.tar
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket2/audio__OP_0..8191_CL_.tar
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket3/audio__OP_0..8191_CL_.tar
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket4/audio__OP_0..8191_CL_.tar
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket5/audio__OP_0..8191_CL_.tar
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket6/audio__OP_0..8191_CL_.tar
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket7/audio__OP_0..8191_CL_.tar
    - - /data/NeMo_ASR_SET/English/v3.0/train_bucketed/bucket8/audio__OP_0..8191_CL_.tar
    shuffle_n: 2048
    bucketing_strategy: synced_randomized
    bucketing_batch_size:
    - 64
    - 64
    - 32
    - 32
    - 16
    - 16
    - 16
    - 16
    
[NeMo W 2022-10-20 03:37:20 modelPT:150] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s). 
    Validation config : 
    manifest_filepath:
    - /data/librispeech_withsp2/manifests/librivox-dev-other.json
    - /data/librispeech_withsp2/manifests/librivox-dev-clean.json
    - /data/librispeech_withsp2/manifests/librivox-test-other.json
    - /data/librispeech_withsp2/manifests/librivox-test-clean.json
    sample_rate: 16000
    batch_size: 4
    shuffle: false
    num_workers: 4
    pin_memory: true
    use_start_end_token: false
    is_tarred: false
    tarred_audio_filepaths: na
    min_duration: 0
    
[NeMo W 2022-10-20 03:37:20 modelPT:156] Please call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s).
    Test config : 
    manifest_filepath:
    - /data/librispeech_withsp2/manifests/librivox-test-other.json
    - /data/librispeech_withsp2/manifests/librivox-dev-clean.json
    - /data/librispeech_withsp2/manifests/librivox-dev-other.json
    - /data/librispeech_withsp2/manifests/librivox-test-clean.json
    sample_rate: 16000
    batch_size: 4
    shuffle: false
    num_workers: 4
    pin_memory: true
    use_start_end_token: false
    is_tarred: false
    tarred_audio_filepaths: na
    
[NeMo I 2022-10-20 03:37:20 features:225] PADDING: 0
[NeMo I 2022-10-20 03:37:25 audio_preprocessing:491] Numba CUDA SpecAugment kernel is being used
[NeMo I 2022-10-20 03:37:33 save_restore_connector:243] Model EncDecCTCModelBPE was successfully restored from /root/.cache/torch/NeMo/NeMo_1.12.0/stt_en_conformer_ctc_xlarge/eff9a44c668d74bfee7b59f7def80bf4/stt_en_conformer_ctc_xlarge.nemo.
[NeMo W 2022-10-20 03:37:33 decoder_timestamps_utils:67] `ctc_decode` was set to True. Note that this is ignored.
[NeMo I 2022-10-20 03:37:33 features:225] PADDING: 0
[NeMo I 2022-10-20 03:37:33 features:225] PADDING: 0
[NeMo I 2022-10-20 03:37:33 decoder_timestamps_utils:657] Running ASR model stt_en_conformer_ctc_xlarge
[NeMo I 2022-10-20 03:37:33 decoder_timestamps_utils:661] [1/1] FrameBatchASR: /content/data/an4_diarize_test.wav
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
[<ipython-input-9-15c5a63eedb2>](https://localhost:8080/#) in <module>
      2 asr_ts_decoder = ASR_TIMESTAMPS(**cfg.diarizer)
      3 asr_model = asr_ts_decoder.set_asr_model()
----> 4 word_hyp, word_ts_hyp = asr_ts_decoder.run_ASR(asr_model)
      5 
      6 print("Decoded word output dictionary: \n", word_hyp['an4_diarize_test'])

2 frames
[/usr/local/lib/python3.7/dist-packages/nemo/collections/asr/parts/utils/decoder_timestamps_utils.py](https://localhost:8080/#) in get_ts_from_decoded_prediction(self, decoded_prediction, hypothesis, char_ts)
    137                 stitched_word = ''.join(decoded_char_list[stt_ch_idx : end_ch_idx + 1]).replace(space, '')
    138                 word_seq.append(stitched_word)
--> 139         assert len(word_ts) == len(hypothesis.split()), "Hypothesis does not match word time stamp."
    140         return word_ts
    141 

AssertionError: Hypothesis does not match word time stamp.
```


**Expected behavior**

Works fine for  'stt_en_conformer_ctc_large' on GPU so I expected to get two lists one containing a list of words and other being the word timestamps.

**Environment overview (please complete the following information)**

 - Environment location: Colab
 - Method of NeMo install: pip installation from github
 
**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 18.04.6 LTS
- PyTorch version: 1.12.1+cu113
- Python version: 3.7.15

**Additional context**

Link to Colab Notebook [ASR_with_SpeakerDiariazation.ipynb](https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/speaker_tasks/ASR_with_SpeakerDiarization.ipynb#scrollTo=chL48Ttmgl0r)