nemo_asr grammar seems to be incorrect
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

NeMo ASR grammar seems to be incorrect on any version except v1.4.0.

Besides the odd behavior with the grammar we can also notice that Punctuation takes much longer (100x~) in v1.4.0 compared to other versions that worked for my setup which leaves me with the only choice of using v1.4.0 for transcriptions and v1.6.0 for punctuation, eg:
```
NeMo Toolkit v1.4.0
Punctuation took: 13.06 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very like the old portrait."]
```
```
NeMo Toolkit v1.6.0
Punctuation took: 0.08 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very likt the old portrait."]
```


**Steps/Code to reproduce bug**

`pip install Cython`
```
pip install nemo_toolkit[all]==1.6.1
pip install nemo_toolkit[all]==1.6.0
pip install nemo_toolkit[all]==1.5.1
pip install nemo_toolkit[all]==1.4.0
```
I simply ran one of these every time I wanted to switch a version

**Expected behavior**

Let's inspect the debug log that I repeatedly tested multiple times on different versions of NeMo on the same wave file:
https://dldata-public.s3.us-east-2.amazonaws.com/2086-149220-0033.wav
If you will notice, only version 1.4.0 will get the word `like` right while all the others will sort of "make up" the word `likt` that doesn't really exists in any known dictionary, besides names, which I get the complexity of the manner, but am curios what "changed" or more like what got messed up so we can now see this behavior:

**NeMo Toolkit v1.4.0**

```
NeMo Toolkit v1.4.0
Init took: 27.58 seconds
Transcribing: 100%|█████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.14it/s]
Transcriptions took: 0.93 seconds
["well i don't wish to see it any more observed phoebe turning away her eyes it is certainly very like the old portrait"]
Punctuation took: 13.77 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very like the old portrait."]
Everything took: 42.28 seconds

NeMo Toolkit v1.4.0
Init took: 29.06 seconds
Transcribing: 100%|█████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.15s/it]
Transcriptions took: 1.22 seconds
["well i don't wish to see it any more observed phoebe turning away her eyes it is certainly very like the old portrait"]
Punctuation took: 12.90 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very like the old portrait."]
Everything took: 43.18 seconds

NeMo Toolkit v1.4.0
Init took: 28.26 seconds
Transcribing: 100%|█████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.29it/s]
Transcriptions took: 0.83 seconds
["well i don't wish to see it any more observed phoebe turning away her eyes it is certainly very like the old portrait"]
Punctuation took: 12.20 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very like the old portrait."]
Everything took: 41.30 seconds

NeMo Toolkit v1.4.0
Init took: 27.88 seconds
Transcribing: 100%|█████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.16it/s]
Transcriptions took: 0.92 seconds
["well i don't wish to see it any more observed phoebe turning away her eyes it is certainly very like the old portrait"]
Punctuation took: 13.06 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very like the old portrait."]
Everything took: 41.86 seconds
```

**NeMo Toolkit v1.5.1**

```
NeMo Toolkit v1.5.1
Init took: 24.72 seconds
Transcribing: 100%|█████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.00s/it]
Transcriptions took: 1.09 seconds
["well i don't wish to see it any more observed phoebe turning away her eyes it is certainly very likt the old portrait"]
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 11.76batch/s]
Punctuation took: 0.10 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very likt the old portrait."]
Everything took: 25.91 seconds

NeMo Toolkit v1.5.1
Init took: 24.43 seconds
Transcribing: 100%|█████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.08it/s]
Transcriptions took: 1.01 seconds
["well i don't wish to see it any more observed phoebe turning away her eyes it is certainly very likt the old portrait"]
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 11.36batch/s]
Punctuation took: 0.10 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very likt the old portrait."]
Everything took: 25.54 seconds

NeMo Toolkit v1.5.1
Init took: 24.63 seconds
Transcribing: 100%|█████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.06it/s]
Transcriptions took: 1.03 seconds
["well i don't wish to see it any more observed phoebe turning away her eyes it is certainly very likt the old portrait"]
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 12.99batch/s]
Punctuation took: 0.09 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very likt the old portrait."]
Everything took: 25.76 seconds

NeMo Toolkit v1.5.1
Init took: 23.22 seconds
Transcribing: 100%|█████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.10it/s]
Transcriptions took: 0.99 seconds
["well i don't wish to see it any more observed phoebe turning away her eyes it is certainly very likt the old portrait"]
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 14.29batch/s]
Punctuation took: 0.08 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very likt the old portrait."]
Everything took: 24.30 seconds
```

**NeMo Toolkit v1.6.0**

```
NeMo Toolkit v1.6.0
Init took: 22.43 seconds
Transcribing: 100%|█████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.27it/s]
Transcriptions took: 0.87 seconds
["well i don't wish to see it any more observed phoebe turning away her eyes it is certainly very likt the old portrait"]
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 15.38batch/s]
Punctuation took: 0.07 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very likt the old portrait."]
Everything took: 23.37 seconds

NeMo Toolkit v1.6.0
Init took: 22.57 seconds
Transcribing: 100%|█████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.25it/s]
Transcriptions took: 0.88 seconds
["well i don't wish to see it any more observed phoebe turning away her eyes it is certainly very likt the old portrait"]
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 15.38batch/s]
Punctuation took: 0.07 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very likt the old portrait."]
Everything took: 23.53 seconds

NeMo Toolkit v1.6.0
Init took: 22.14 seconds
Transcribing: 100%|█████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.22it/s]
Transcriptions took: 0.91 seconds
["well i don't wish to see it any more observed phoebe turning away her eyes it is certainly very likt the old portrait"]
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 15.63batch/s]
Punctuation took: 0.07 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very likt the old portrait."]
Everything took: 23.12 seconds

NeMo Toolkit v1.6.0
Init took: 22.27 seconds
Transcribing: 100%|█████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.26it/s]
Transcriptions took: 0.88 seconds
["well i don't wish to see it any more observed phoebe turning away her eyes it is certainly very likt the old portrait"]
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 15.15batch/s]
Punctuation took: 0.08 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very likt the old portrait."]
Everything took: 23.22 seconds
```

**NeMo Toolkit v1.6.1**

```
NeMo Toolkit v1.6.1
Init took: 37.51 seconds
Transcribing: 100%|██████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.30s/it]
Transcriptions took: 1.36 seconds
["well i don't wish to see it any more observed phoebe turning away her eyes it is certainly very likt the old portrait"]
100%|█████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.54s/batch]
Punctuation took: 1.58 seconds
["Well, I don't wish to see it any more, observed Phoebe, turning away her eyes. It is certainly very likt the old portrait."]
Everything took: 40.45 seconds
```

**Environment overview**

 - Environment location: Local
 - Method of NeMo install: pip install. as can be seen above.

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Latest
- PyTorch version: as required by each nemo package that I installed on a fresh machine using the commands above ^^
- Python version: 3.7.11 (also tested and got the same behavior on: 3.8.12 and 3.9.7)
- Python versions that didn't run it were: 2.7.18, 3.5.6, 3.6.13, 3.10.0

**Additional context**

I installed apex for cpu using:
```
pip install -v --disable-pip-version-check --no-cache-dir git+https://github.com/NVIDIA/apex
```
(TORCH_CUDA_ARCH_LIST="compute capability")

The following NeMo versions were not able to run at all using the same spec:
```
v1.3.0 and below (tested until v1.0.0)
v1.5.0
```

**Example Code:** (Fixed + added full snippet below)

```
import time
import nemo.collections.asr as nemo_asr
import nemo.collections.nlp as nemo_nlp

if __name__ == "__main__":
    print("NeMo Toolkit vX.Y.Z")
    start = time.time()
    
    asr_model = nemo_asr.models.EncDecCTCModel.from_pretrained("stt_en_quartznet15x5")
    punctuation = nemo_nlp.models.PunctuationCapitalizationModel.from_pretrained(model_name='punctuation_en_distilbert')
    print("Init took: " + '%.2f' % (time.time() - start) + " seconds")
    start = time.time()
    
    transcriptions = asr_model.transcribe(['2086-149220-0033.wav'])
    print("Transcription took: " + '%.2f' % (time.time() - start) + " seconds")
    start = time.time()
    
    res = punctuation.add_punctuation_capitalization(queries=transcriptions)
    print("Punctuation took: " + '%.2f' % (time.time() - start) + " seconds")
    
    print(transcriptions)
    print(res)
```