renwei2024/nemo-github-issues huggingface dataset:
num_samples is zero, but 1172 files were filtered
State: closed | Pull request: no

**Describe the bug**

I am getting this error when I try to instantiate ASR model (following [this](https://github.com/NVIDIA/NeMo/blob/v1.0.0/tutorials/asr/01_ASR_with_NeMo.ipynb) tutorial):

`ValueError: num_samples should be a positive integer value, but got num_samples=0`

But it's quite strange because I am getting these logs during the run:

```
[NeMo I 2022-01-21 11:37:29 audio_to_text_dataset:38] Model level config does not container `sample_rate`, please explicitly provide `sample_rate` to the dataloaders.

[NeMo I 2022-01-21 11:37:29 audio_to_text_dataset:38] Model level config does not container `labels`, please explicitly provide `labels` to the dataloaders.

[NeMo I 2022-01-21 11:37:29 collections:173] Dataset loaded with 0 files totalling 0.00 hours

[NeMo I 2022-01-21 11:37:29 collections:174] 1172 files were filtered totalling 347.43 hours
```

Which shows correct number of files (`1172`) in training dataset.

**Steps/Code to reproduce bug**

```python
trainer = pl.Trainer(gpus=1, max_epochs=50)

train_manifest = './<folder_name>/manifests/train_manifest.json'
test_manifest = './<folder_name>/manifests/test_manifest.json'

params['model']['train_ds']['manifest_filepath'] = train_manifest
params['model']['validation_ds']['manifest_filepath'] = test_manifest

first_asr_model = nemo_asr.models.EncDecCTCModel(cfg=DictConfig(params['model']), trainer=trainer)
```

I have tried to put full path for `train_manifest` and `test_manifest` (`/home/ubuntu/<folder_name>/manifests`), but it didn't help at all.

**Expected behavior**

Running without errors.

**Environment overview (please complete the following information)**

 - Environment location: `Jupyter Notebook` in `AWS EC2` instance node
 - Method of NeMo install: `pip install nemo_toolkit[all]`

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 20.04.3 LTS
- PyTorch version: 1.10.1
- Python version: 3.8.12

**Additional context**
It doesn't seem that there are no files or the path to the `train` and `test` manifests are incorrect. Otherwise I couldn't open manifests via `pandas` (did this for checking purposes). The structure of the manifest looks like this:

`
{"audio_filepath": "/home/ubuntu/<folder_name>/audio/<audio_file_name>.wav", "duration": 1725.84, "text": "..."}
{"audio_filepath": "/home/ubuntu/<folder_name>/audio/<audio_file_name>.wav", "duration": 1725.84, "text": "..."}
`

where `<folder_name>` is the same as in the path for `train_manifest.json` and `test_manifest.json`.
NLPDDPPlugin find_unused_parameters is configurable
State: closed | Pull request: yes

This PR makes `find_unused_parameters` user configurable in `NLPDDPPlugin` by setting a default dictionary with the current value (`False`) and updating that dictionary with user provided kwargs. This ensures that the value is always `False`, even if it changes from its current default (also `False`) in PyTorch's `DistributedDataParallel`.

An alternative option would be to remove it from the initialization call to `DistributedDataParallel` since it is already `False` by default in PyTorch. 

The reason for this request is that `find_unused_parameters` must be `True` for MegaMolBART.

I have read the contribution guidelines. All tests that are currently passing in `main` pass with this commit. (Note that two tests fail in NeMo `main`.) The code has been formatted with black and the commit is signed. This change was discussed with @ericharper several months ago.

Signed-off-by: Michelle Gill <mgill@nvidia.com>
Process correctly label ids dataset parameter + standardize type of label ids model attribute + minor changes (error messages, typing)
State: closed | Pull request: yes
Labels: name: bug

1. If  `BertPunctuationCapitalizationDataset.__init__` arguments `punct_label_ids` and `capit_label_ids` are  `DictConfig`s, then transform them to dictionaries.
2. Make sure that `PunctuationCapitalizationModel` attributes `punct_label_ids` and `capit_label_ids` are always dictionaries and not `DictConfig`s. Previously, type of these attributes varied depending on where the attributes are set.
3. Add more typing and improve error messages.
Test only if the model was trained on single GPU for accurate results.
State: closed | Pull request: yes

# Changelog
- Revert previous code that constructs a trainer on one GPU to evaluate results. The code is misleading because the torch environment is still distributed, and all processes run the code
- From now on, test only if the model was trained on a single GPU with no ddp.

Signed-off-by: smajumdar <titu1994@gmail.com>
nemo2riva BUG
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Error when converting Nemo model to Riva

**Steps/Code to reproduce bug**

Shell command: 

`nemo2riva --out /mnt/devdata/zzh/tao/results/Citrinet/stt_zh_citrinet_512.riva /mnt/devdata/zzh/tao/results/Citrinet/stt_zh_citrinet_512.nemo
`

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: ubuntu18.04
- Python version: 3.8
- nemo2riva: Downloaded from riva_quickstart_v1.8.0-beta/nemo2riva-1.8.0b0-py3-none-any.whl
- stt_zh_citrinet_512.nemo: curl -LO https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_zh_citrinet_512/versions/1.0.0rc1/files/stt_zh_citrinet_512.nemo
**Additional context**

Error message: 

```
(riva) $ nemo2riva --out ./tao/results/Citrinet/stt_zh_citrinet_512.riva ./tao/results/Citrinet/stt_zh_citrinet_512.nemo
INFO: Logging level set to 20
INFO: Restoring NeMo model from '/mnt/devdata/zzh/tao/results/Citrinet/stt_zh_citrinet_512.nemo'
ERROR: Failed to restore model from NeMo file : /mnt/devdata/zzh/tao/results/Citrinet/stt_zh_citrinet_512.nemo. Please make sure you have the latest NeMo package installed with [all] dependencies.
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/riva/bin/nemo2riva", line 8, in <module>
    sys.exit(nemo2riva())
  File "/home/user/miniconda3/envs/riva/lib/python3.8/site-packages/nemo2riva/cli/nemo2riva.py", line 50, in nemo2riva
    Nemo2Riva(args)
  File "/home/user/miniconda3/envs/riva/lib/python3.8/site-packages/nemo2riva/convert.py", line 41, in Nemo2Riva
    raise e
  File "/home/user/miniconda3/envs/riva/lib/python3.8/site-packages/nemo2riva/convert.py", line 34, in Nemo2Riva
    model = ModelPT.restore_from(restore_path=nemo_in)
AttributeError: type object 'ModelPT' has no attribute 'restore_from'

```
User EncDecRNNTBPEModel with Beam Search
State: closed | Pull request: no

I am using a pre-trained  stt_en_conformer_transducer and it is doing well. However, since my dataset has some special terms I want to use KenLM with Beam Search.

In one tutorial, I saw that we can change the decoding strategy to Beam using: 

```
import copy
decoding_config = copy.deepcopy(asr_model.cfg.decoding) 
# Update the config for the decoding strategy
decoding_config.strategy = "beam"  # Options are `greedy`, `greedy_batch`, `beam`, `tsd` and `alsd`
decoding_config.beam.beam_size = 8  # Increase beam size for better scores, but it will take much longer for transcription !
asr_model.change_decoding_strategy(decoding_config)
asr_model.transcribe(['/home/wesam/wavs/X1.wav'], )

```

It works well, but I did not know how to feed it with my own Language Model?
Fine-tune stt conformer transducer EncDecRNNTBPEModel model
State: closed | Pull request: no

I'm trying to fine-tine stt_en_conformer_transducer_large model on a custom dataset of 300H. I have a problem with setting up the training configuration. 

In [this ](https://github.com/NVIDIA/NeMo/blob/8c912f515a77efb0e16b1433030da23e488960d5/tutorials/asr/ASR_with_Transducers.ipynb) example you reported how to train a model from scratch, but I understood from the model card (https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_conformer_transducer_small) that we can fine-tune it.

I tried to load the model, extract the config file, modify it and re-assign it to the mode, but did not work.

Here is what I did:

# Loading the model
```
import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.EncDecRNNTBPEModel.from_pretrained(model_name="stt_en_conformer_transducer_large", map_location='cpu')
```

# Loading the model configurations:
```
import copy
from omegaconf import DictConfig

modelConfig = DictConfig(asr_model.cfg)
modelConfig

```

Set the training data:

```
modelConfig['train_ds']['manifest_filepath'] = TRAIN_MANIFEST
modelConfig['validation_ds']['manifest_filepath'] = TEST_MANIFEST
```

Assigned it to the model:
`asr_model.setup_training_data(modelConfig['train_ds'])`

Here is the error I get:


```
[NeMo I 2022-01-18 18:44:40 collections:173] Dataset loaded with 1503 files totalling 2.81 hours
[NeMo I 2022-01-18 18:44:40 collections:174] 0 files were filtered totalling 0.00 hours
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_3295870/3465286351.py in <module>
----> 1 asr_model.setup_training_data(modelConfig['train_ds'])

~/anaconda3/envs/nemo/lib/python3.7/site-packages/nemo/collections/asr/models/rnnt_models.py in setup_training_data(self, train_data_config)
    499             # If it's an int, we assume that the user has set it to something sane, i.e. <= # training batches,
    500             # and don't change it. Otherwise, adjust batches accordingly if it's a float (including 1.0).
--> 501             if isinstance(self._trainer.limit_train_batches, float):
    502                 self._trainer.limit_train_batches = int(
    503                     self._trainer.limit_train_batches

AttributeError: 'NoneType' object has no attribute 'limit_train_batches'
```


Thanks!
UnicodeEncodeError when deploying Nemo Text Normalization on Triton Inference Server
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I have a problem when deloy Text Nemo Text Normarlzation on Triton Inference Server. The Normalizer can not load the default "whitelist.tsv" file. Whenever I run ```tritonserver --model-repository models/```, the bug will appear.

```python
from nemo_text_processing.text_normalization.normalize import Normalizer
normalizer = Normalizer(input_case='cased', lang='en')
```
Results:
```bash
Internal: UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 150: ordinal not in range(128)

At:
  /usr/lib/python3.8/encodings/ascii.py(26): decode
  /usr/local/lib/python3.8/dist-packages/nemo_text_processing/text_normalization/en/utils.py(41): load_labels
  /usr/local/lib/python3.8/dist-packages/nemo_text_processing/text_normalization/en/taggers/whitelist.py(49): _get_whitelist_graph
  /usr/local/lib/python3.8/dist-packages/nemo_text_processing/text_normalization/en/taggers/whitelist.py(64): __init__
  /usr/local/lib/python3.8/dist-packages/nemo_text_processing/text_normalization/en/taggers/tokenize_and_classify.py(107): __init__
  /usr/local/lib/python3.8/dist-packages/nemo_text_processing/text_normalization/normalize.py(76): __init__

```
**Environment overview (please complete the following information)**

 - Environment: Triton Inference Server 21.10
 - Method of NeMo install: pip install nemo_toolkit['all']

**Additional context**

**There is no problem when I test this code without Triton Inference Server!**.
Text normalization takes too much time for a string which contains a lot of dates
State: closed | Pull request: yes
Labels: name: bug

The PR reduces number of permutations processed in [this](https://github.com/NVIDIA/NeMo/blob/a8154fd825c34de0a59309536e11ba7d5d1f94e1/nemo_text_processing/text_normalization/normalize.py#L149) loop.

1. Count number of permutations that each token gives.
2. Split sequence of tokens into smaller parts so that number of permutations from one part would not exceed certain number.
3. Verbalize every part.
4. Unite verbalizations of parts.

Closes #3450
Add randomized bucketing
State: closed | Pull request: yes

+ This PR Adds different bucketing strategies to improve the performance of ASR bucketing training accuracy-wise. A new parameter added to the data loaders (train_ds.bucketing_strategy) to specify one of the following modes:
1-fixed_order: the same order of buckets are used for all epochs
2-sycned_randomized: each epoch would have a different order of buckets
3-fully_randomized: similar to synced_randomized but each GPU has its own random order. So GPUs would not be synced.

+Also added the support of layer norm for Conformer layers instead of batch norm.
ASR_with_NeMo Pre-trained Model Inference | Pickling Error 
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
On attempting inference of a recording, I received the following error:
PicklingError: Can't pickle <class 'nemo.collections.common.parts.preprocessing.collections.AudioTextEntity'>: attribute lookup AudioTextEntity on nemo.collections.common.parts.preprocessing.collections failed

**Steps/Code to reproduce bug**

```
import nemo
import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En", strict=False)

files = ['./soundsample.wav']
for fname, transcription in zip(files, asr_model.transcribe(paths2audio_files=files)):
  print(f"Audio in {fname} was recognized as: {transcription}")
```

**Environment overview**
 - Conda Virtual Environment
 - Method of NeMo install: from source

**Environment details**
- Mac OS Catalina 10.15.7
- PyTorch version: 1.10.1
- Python version: 3.8.5
Prompt tuning loss mask fix
State: closed | Pull request: yes

Added loss mask configurability to include or exclude the text portion of the input in the prompt tuning loss calculation. 

```
[soft` prompt][Input Text][Answer][EOD Padding]
[000000000000011111111111111111111000000000000]
```
vs 

```
[soft prompt][Input Text][Answer][EOD Padding]
[00000000000000000000000011111111000000000000]
```

I also added prompt tuning CI tests and prompt tuning dataset unit tests that get called during CI testing. 

Calculating the loss on the answer only helped. The 2nd score is from calculating the loss on the full input text and the 3rd is from calculating the loss on the answer only. These scores are on Winogrande formulated as fill in the blank instead of sentence completion. (The GPT eval scores in the spread sheet are for Winogrande as sentence completion. We couldn't find a Winogrande sentence completion training set split to use for prompt tuning, so I added Winogrande fill in the blank to the model eval code on gitlab.)

Allen AI Winogrande
Baseline 1.3B Megatron GPT: 0.489345
PromptTuned 1.3B Megatron GPT Full Text Loss: 0.486977
PromptTuned 1.3B Megatron GPT Answer Only Loss: 0.528019
AttributeError: type object 'EncDecCTCModel' has no attribute 'from_pretrained'
State: closed | Pull request: no
Labels: name: bug

Upon call to retrieve a pre-trained model, the following line fails:
asr_model = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")

Additionally, list_available_models() also fails with the same above error.

I have installed NeMo via:
python -m pip install git+https://github.com/NVIDIA/NeMo.git

OS: Mac OS Catalina 10.15.7
Pytorch: 1.10.0
Python: 3.8.5
Extending input_example() to take max batch and dimension arguments
State: closed | Pull request: yes

This is to implement https://jirasw.nvidia.com/browse/JARS-1682.
It is important to provide user ability to generate input example of particular dimensions - as certain networks would reconfigure themselves according to input dimensions - thus export needs to be performed with real life size example.
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
Working around Pytorch exporter issue with expand()
State: closed | Pull request: yes

WAR changes to prevent issues described in https://nvbugswb.nvidia.com/NvBugs5/SWBug.aspx?bugid=3429305
"ONNX model exported with PyTorch container > 21.06 fails when data of different size is provided to dynamic axes"

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
Speaker Diarization Inference error with pickle 
State: closed | Pull request: no
Labels: name: bug

name: stale

Dear colleagues, after install Nemo r1.5.0 I have error in method oracle_model.diarize() in 
[Speaker_Diarization_Inference] https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Diarization_Inference.ipynb)

```
from nemo.collections.asr.models import ClusteringDiarizer
oracle_model = ClusteringDiarizer(cfg=config)
oracle_model.diarize()

PicklingError: Can't pickle <class 'nemo.collections.common.parts.preprocessing.collections.SpeechLabelEntity'>: attribute lookup SpeechLabelEntity on nemo.collections.common.parts.preprocessing.collections failed
```

Environment:
 - Anaconda 3
 - Python3.8
 - NeMo r1.5.0
AttributeError: type object 'SpectrogramGenerator' has no attribute 'from_pretrained'
State: closed | Pull request: no

I'm working with PyCharm, using Python 3.7. 
In the following code snippet:
 `from nemo.collections.tts.models.base import SpectrogramGenerator, Vocoder`
`def load_spectrogram_model():`
    `pretrained_model = "tts_en_fastpitch"`
    `model = SpectrogramGenerator.from_pretrained(pretrained_model, override_config_path=None)`
    `return model`

`spec_gen = load_spectrogram_model().eval()`
I get the following error on the above line:
 `AttributeError: type object 'SpectrogramGenerator' has no attribute 'from_pretrained'`

Moreover, `print(SpectrogramGenerator.list_available_models())` returns an empty list. However, none of this is an issue in Colab. 
Is it something environment related or am I missing something? 

Any help would be appreciated.
ASR transcribe_speech with Language Model
State: closed | Pull request: no

Hi, 
I have 2 questions about ASR Language Modeling:
1- Is there any implemented code in Nemo, for generating the json file of the predictions of an ASR model with a N-gram LM? Something like the [transcribe_speech_parallel.py](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/transcribe_speech_parallel.py) code. I know that, I can easily implement this code via [Offline_ASR.ipynb Tutorial](https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/asr/Offline_ASR.ipynb#scrollTo=nV1CAy0Dit-g). But I want to know if such a code is already available in the framework?
2- Would you please kindly describe what is the `Oracle WER/CER` mean when using the [eval_beamsearch_ngram.py](https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py) code? 
Best
Download pre-trained Fastspeech2 Hifigan E2E fail
State: closed | Pull request: no

hello, thanks for your great job.

I tried to download pre-trained model of Fastspeech2 Hifigan E2E.
but, browser download doesn't working and wget too.

I attach the wget log below

wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_e2e_fastspeech2hifigan/versions/1.0.0/zip -O tts_en_e2e_fastspeech2hifigan_1.0.0.zip
--2022-01-07 17:17:54--  https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_e2e_fastspeech2hifigan/versions/1.0.0/zip
Resolving api.ngc.nvidia.com (api.ngc.nvidia.com)... 52.38.144.30, 35.165.139.54
Connecting to api.ngc.nvidia.com (api.ngc.nvidia.com)|52.38.144.30|:443... connected.
HTTP request sent, awaiting response... 404 
2022-01-07 17:18:00 ERROR 404: (no description).

Thank you.
Getting error module 'posixpath' has no attribute 'json'
State: closed | Pull request: no

I was trying the single liner diarization as mentioned in this comment 
https://github.com/NVIDIA/NeMo/issues/2722#issuecomment-905306973

But I'm getting this error: AttributeError: module 'posixpath' has no attribute 'json' in this line:
https://github.com/NVIDIA/NeMo/blob/b4a28b4e29a4eb03ac2e0003dfe08fb7ddedea63/nemo/collections/asr/models/clustering_diarizer.py#L395

I was wondering if it should be join instead of json? 
I'm using python 3.7.12 and nemo-toolkit 1.6.0rc0
Weird transcriptions from a fine-tuned model
State: closed | Pull request: no

Hi, 

I fine-tuned a pre-trained French ASR model "stt_fr_conformer_ctc_large" with my own data. And I replaced the tokenizer with my own by `model.change_vocabulary(new_tokenizer_dir=my_own_tokenizer, new_tokenizer_type="bpe")`.

The training was smooth and I got 17% valid WER which is rather good for our dataset, but when I did `transcription=asr_model.transcribe(["example.wav"])` for inference, I got weird transcriptions as below : 
![image](https://user-images.githubusercontent.com/41187612/148400356-a24dfafb-369e-4c53-aa40-3a9ac349cf13.png)
It seems that something changed the alphabet, and here is another example:
![image](https://user-images.githubusercontent.com/41187612/148400928-3c1e21b6-68a7-4781-a226-e47d59afca8c.png)
I looked back into the training log and I found the same thing : 
![image](https://user-images.githubusercontent.com/41187612/148401547-937142cb-f1de-4ab4-8a90-83fa430d37d2.png)

How can I get good transcriptions please ? And is my 17% WER correct ?
Thanks in advance.
[Question] about Speech Data Explorer (SDE) Tool
State: closed | Pull request: no

Hi,
The Speech Data Explorer (SDE) tool is really a great tool for error analysis. I have some questions about it:
1- What is `Vocabulary size` exactly mean in SDE? Is it unique words separated by space in the manifest? (because when I have tested SDE with QuartzNet & Citrinet (with bpe tokenizer) models its vocabulary size have been changed!!)  
2- What does `Score` means in SDE? 
3- In the last plot of SDE (_e.g._, the last plot in `statistics` section of  http://18.223.252.228:8050/) what does `sometimes recognized` exactly means? It means these words recognition accuracy is greater than zero and less than 100%?
4- How one can save or export tables in SDE for example as a csv file (excel file)? Also if one can save them after applying a filter it would be great. 
5- After running the SDE for a manifest according to the predicted results of an ASR model, is there any systematic approach for analyzing and detecting the common errors of the ASR model and address them? In other words, how one can improve the accuracy of an ASR model by using the SDE tool? Is there any paper/tutorial in this area? 
Best
vad infer refactor
State: closed | Pull request: yes

1. Organize and move parameters used in `vad_infer.py` to _yaml_ config file` conf/vad_inference_postprocessing.yaml`.
2. Add gen_overlap_seq, gen_seg_table and write_to_manifest choice to `vad_infer.py` so the script can cover the whole pipeline from 
**frame -> overlapped smoothing filter -> write to rttm-like table -> write to manfiest json file (for possible next step)** instead of using multiple scripts as in previous.
3. Remove duplicate code in  `vad_overlap_posterior.py`  and  `write_long_audio_manifest.py`  in scripts/voice_activity_detection.
4. Reflect updates in `tutorials/asr/Online_Offline_Microphone_VAD_Demo.ipynb ` and `tutorials/asr/Voice_Activity_Detection.ipynb`
5. bugfix for diarization yaml file about input data example.
6. Unify and rename _window_length_in_sec_ and _shift_length_in_sec_ across files to reduce confusion.
Refactor ASR Examples Directory
State: closed | Pull request: yes

# Changelog
- Refactor ASR directory to modularize against - type of task (ASR/Speech Pretraining/Speech Classification) and type of ASR training (CTC/RNNT).
- Keep all inference scripts in base dir for easy access to infer/eval ASR models without tracking down which model to evaluate.
- Move VAD/Speech Classification to its own subdomain
- Move Speech Pretraining to its own subdomain (anticipating more advanced SSL capabilities)
- Separate CTC / RNNT training scripts to inform which type of model is being trained.

Signed-off-by: smajumdar <titu1994@gmail.com>
[NeMoMegatron] Pipeline parallelism for GPT
State: closed | Pull request: yes

PR to add pipeline parallelism to GPT using fwd/bwd functions from Apex.

FP32, FP16, and BF16 are all working now.

When using pipeline parallel it is recommended to use BF16 + Megatron amp O2.
```
model.megatron_amp_O2=True
trainer.precision='bf16'
```

### TODOs
- under review

### Known issues
- complete method will be supported in a subsequent PR
- prompt tuning temporarily disabled, use NeMo 1.6 if needed
- when using tensor parallel only, we're still using sync grad all-reduce which reduces perf. Will be fixed in NeMo 1.8.
update speaker collate functions
State: closed | Pull request: yes

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

This PR 
* removes time-length and shift-length concept for training speaker label models. 
> This assumes all the samples present in training manifest must adhere to max of 8 secs. (3 sec is ideal) 

* removes sliced_seq_collate_fn : Since we introduced segmentation for creating manifest files hence this collate func is no longer required now, and existing seq_collate_fn can be used for getting speaker embeddings for diarization as well.
Learning-Curve
State: closed | Pull request: no
Labels: name: feature

HI Team,
I need to draw learning curve for the trained Biomegatron model. I have trained  the model as per the tutorial https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/nlp/Text_Classification_Sentiment_Analysis.ipynb#scrollTo=DQhsamclRtxJ

Now when I am trying to get the learning-curve which is the plot for each training loss and validation loss as per the number of epochs as sklearn library providing attribute as from sklearn.model_selection import learning_curve

![image](https://user-images.githubusercontent.com/45452951/147752362-c74ad670-a798-4125-9dc1-60056fb06fc7.png)
Fix minor problems in punctuation and capitalization model
State: closed | Pull request: yes
Labels: name: bug

1. Fix creating tarred dataset for cases when tokenizer name contains `/` character, e.g. `'facebook/mbart-large-50'`.
2. Fix subclass check [here](https://github.com/NVIDIA/NeMo/blob/f83b2c5524a787be21ffea170850c4b5486eac2b/nemo/collections/nlp/data/token_classification/punctuation_capitalization_tarred_dataset.py#L383). It did not work if `error_class_or_function` is not a class
3. Fix return typing in a [method](https://github.com/NVIDIA/NeMo/blob/f83b2c5524a787be21ffea170850c4b5486eac2b/nemo/collections/nlp/data/token_classification/punctuation_capitalization_dataset.py#L436) `TokenizeCreateMasksClipWorker.__call__()`.
4. Make comment [here](https://github.com/NVIDIA/NeMo/blob/f83b2c5524a787be21ffea170850c4b5486eac2b/nemo/collections/nlp/data/token_classification/punctuation_capitalization_dataset.py#L67) more comprehensible. Before fix it was identical to a name of a model config section `common_dataset_parameters`, whereas it was not related to the section in any way.
5. In a spirit of fix 1 add checking of characters in `tar_file_prefix` parameter in functions [`create_tarred_dataset()`](https://github.com/NVIDIA/NeMo/blob/f83b2c5524a787be21ffea170850c4b5486eac2b/nemo/collections/nlp/data/token_classification/punctuation_capitalization_tarred_dataset.py#L633) and [`get_args()`](https://github.com/NVIDIA/NeMo/blob/f83b2c5524a787be21ffea170850c4b5486eac2b/examples/nlp/token_classification/data/create_punctuation_capitalization_tarred_dataset.py#L64).
[fix] for resume training on SLURM multi-node multi-gpu
State: closed | Pull request: yes

On SLURM resuming training in a multi-node multi-gpu settings fails, as when `LOCAL_RANK` is undefined `is_globa_rank_zero()` returns true on all processes that run on node 0. In this case `exp_manager.py` https://github.com/NVIDIA/NeMo/blob/f83b2c5524a787be21ffea170850c4b5486eac2b/nemo/utils/exp_manager.py#L446, creates multiple `run_*` folders, and eventually leads to failure (missing files because other processes have moved them already). 

Checking also for `SLURM_PROCID` solves this issue, as the environment variable contains the global rank id.
[Question] save ASR model config
State: closed | Pull request: no

Hi,
1- Would you please help me how I can save the config (`cfg`) of an ASR model as a yaml file? (_i.e.,_ load a trained asr `.nemo` model file and save its cfg as a yaml file)
2- Another question is that, when I follow your [`ASR_CTC_Language_Finetuning`](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/ASR_CTC_Language_Finetuning.ipynb) tutorial, then the `hparams.yaml` file which is saved in the experiments folder, seems related to the previous checkpoint of the model (I mean it is not updated according to my cfg modifications). So, I want to save its modified `hparams.yaml` file according to my new changes. How can I do that? 
Best
from_pretrained() does not update configuration in exp_manager
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When fine tuning from a NeMo model (ex. stt_en_cn1024), the exp_manager's cfg is not updated properly.
I can see that in my run the model uses one config, but WandB reports another.

This issue did not occur in v1.4.0 and happened after I upgraded to v1.5.0.
Maybe it has to do with order of operations? See below.

**Steps/Code to reproduce bug**

```py
import pytorch_lightning as pl
from nemo.collections.asr.models import EncDecCTCModelBPE
from nemo.core.config import hydra_runner
from nemo.utils.exp_manager import exp_manager

@hydra_runner(config_path="conf/citrinet/", config_name="config")
def main(cfg):
    trainer = pl.Trainer(**cfg.trainer)
    log_dir = exp_manager(trainer, cfg.get("exp_manager", None))
    asr_model = EncDecCTCModelBPE.from_pretrained(model_name=cfg.init_from_pretrained_model)
    asr_model.encoder.unfreeze()
    asr_model.change_vocabulary(
      new_tokenizer_dir=cfg.model.tokenizer.dir,
      new_tokenizer_type=cfg.model.tokenizer.type
    )
    asr_model.setup_optimization(cfg.model.optim)
    asr_model.setup_training_data(cfg.model.train_ds)
    asr_model.setup_multiple_validation_data(cfg.model.validation_ds)
    asr_model.spec_augmentation = asr_model.from_config_dict(cfg.model.spec_augment)
    asr_model.set_trainer(trainer)
    trainer.fit(asr_model)
```

**Expected behavior**

WandB cfg should display the proper config ([Pastebin of the WandB config](https://pastebin.com/Bc29P9FL))

**Environment overview (please complete the following information)**

 - Environment location: Docker (nvcr.io/nvidia/pytorch:21.10-py3) on AWS EC2 using `docker run -it bash <image>`
 - Method of NeMo install: `pip install nemo_toolkit[asr]==1.5.1`

**Additional context**

GPU model: V100
Nvidia driver: 460
shape mismatch while loading my pretrained model
State: closed | Pull request: no

Hi, 

I trained a corformer CTC BPE model with 8k wav data, but when I tried to load the model for decoding with 
`asr_model = nemo_asr.models.ASRModel.restore_from(nemo_model_file)` I got this shape mismatch error 
![image](https://user-images.githubusercontent.com/41187612/147249585-2c748058-b9b8-4cfc-bee6-0ef59574e5f4.png)
I don't want to train another model with 16k data, is there a way that I load this pretrained model please ?
ASR Tutorial | ASR_with_Nemo | from_pretrained not defined.
State: closed | Pull request: no
Labels: name: bug

I am trying the tutorial ASR_with_NeMo.ipynb. It's giving me following error. I checked the code for EncDecCTCModel, it doesn't contain the function "from_pretrained". How to load the pre-trained models:
I even tried to use restore_from after downloading the respective *.nemo file. But getting same issue there as well (_'EncDecCTCModel' has no attribute 'restore_from'_)

**# This line will download pre-trained QuartzNet15x5 model from NVIDIA's NGC cloud and instantiate it for you
quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")**

Error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_349758/1493780281.py in <module>
      1 # This line will download pre-trained QuartzNet15x5 model from NVIDIA's NGC cloud and instantiate it for you
----> 2 quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")

**AttributeError: type object 'EncDecCTCModel' has no attribute 'from_pretrained'**
Byte-level Multilingual NMT
State: closed | Pull request: yes

Add support for Byte-level multilingual NMT + Byte-level preproc fixes.

Other change to note:
* `train_tokenizers::preproc_mt_data` Instead of throwing an error if the tokenizer is not trainable, train_tokenizer now returns None. This is done to keep the pipeline simple and same for tokenizers like byte-level which do not require any explicit training.
Update Mixer-TTS, FastPitch and TTSDataset
State: closed | Pull request: yes

- [x] Update nemo.collections.tts.torch.data.TTSDataset and nemo.collections.tts.torch.data.MixerTTSDataset
    - [x] Rename "duration_prior" to "align_prior_matrix"
    - [x] Add `normalized_text` field 
    - [x] Fix minor bugs
- [x] Update FastPitch 
    - [x] Comment unnecessary code and suggest to remove it (+ need to discuss with @blisc)
    - [x] Refactor config 
    - [x] Add correct setup for text normalization and text tokenization to support new config
    - [x] Fix minor bugs
- [x] Update Mixer-TTS 
    - [x] Refactor config 
    - [x] Add correct setup for text normalization and text tokenization to support new config
    - [x] Fix minor bugs
    - [x] Add test to Jenkinsfile for Mixer-TTS
    - [x] Update documentation for Mixer-TTS and Mixer-TTS-X
    - [x] Add checkpoints from NGC Catalog to code
 - [x] Update HiFi-GAN
    - [x] Add checkpoints from NGC Catalog to code (+ need to discuss with @blisc)
- [x] Add & update notebooks
    - [x] Add `MixerTTS_Training.ipynb` notebook
    - [x] Update `Inference_ModelSelect.ipynb` notebook
    - [x] Fix bugs in `FastPitch_Finetuning.ipynb` notebook
  

Signed-off-by: Oktai Tatanov oktai.tatanov@gmail.com
Support Changing Number of Tensor Parallel Partitions for Megatron
State: closed | Pull request: yes

Added `megatron_change_num_partitions.py` that supports converting a Megatron model with an arbitrary number of Tensor Parallel (TP) partitions N to any other number of TP partitions M.

In order to do so, the script intermediately converts all models to TP=1.
All the models/tensors are created/loaded on CPU.

Usage:
```
python examples/nlp/language_modeling/megatron_change_num_partitions.py \
   --model_file=PATH_TO_SRC_FILE \
   --target_file=PATH_TO_TGT_FILE \
   --tensor_model_parallel_size=2 \
   --target_tensor_model_parallel_size=1
```
FastSpeech2_HiFiGAN_E2E training doesn't work
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

The class for training FastSpeech2_HiFiGAN end-to-end model uses variables that aren't present in the core NeMo classes, so the python errors out before training begins.

**Steps/Code to reproduce bug**

Try training such a model using NeMo 1.5.1

**Expected behavior**

A model can be trained
Updates on ASR with diarization util files
State: closed | Pull request: yes

This PR includes the following changes:
- Added CTM file based (word-by-word) diarization evaluation for calculating accurate WDERs.
- Created decoder_timestamps_utils.py that contains all the word time-stamp generation scripts.
- pyctcdecode support in decoder_timestamps_utils.py.
- LM based diarization transcript realigning in diarization_utils.py.
- offline_diarization_with_asr.py example and its yaml file.
- Updates on diarization README.md.
- Fixed unique ID indexing bug in scripts/speaker_tasks/pathsfiles_to_manifest.py.
NeMo Megatron Epoch training is confusing
State: closed | Pull request: no
Labels: name: feature

Currently, the NeMo Megatron has the intrinsic epoch concept in the dataset which is confusing as pytorch lightning has its own epoch concept. The `sample_mappling` logic prepares data samples for multiple epochs (the number of epoch is determined by the max_steps and number of documents in datafile) and creates the `*.npy` mapping files. The dataloader will take this dataset and sample in batches to go over the data for multiple times according to `*.npy` files. The training just need one pytorch lightning epoch. 

Ideally, we should use pytorch lightning epoch only and the NeMo Megatron dataset should only concern with one epoch of data.
WER results are different 
State: closed | Pull request: no

### Discussed in https://github.com/NVIDIA/NeMo/discussions/3339

<div type='discussions-op-text'>

<sup>Originally posted by **andrusenkoau** December 15, 2021</sup>
Hi everyone!
I am trying to decode all librispeech dev and test sets for my Conformer-CTC-BPE model by "eval_beamsearch_ngram.py"  script in greedy way. But I am getting WER results that are different from training logs. For example, for the last 100th epoch I have got  WER = **10.43%** for dev-clean according to "lightning_logs.txt". But after "eval_beamsearch_ngram.py" I have got WER = **12.01%**. Maybe i am doing something wrong?

This is my decoding command:

```
model_dir="nemo_experiments/Conformer-CTC-BPE/2021-12-14_12-47-27"
model_file="${model_dir}/checkpoints/Conformer-CTC-BPE.nemo"

for test_set in dev-clean dev-other test-clean test-other; do
    decode_dir=${model_dir}/decoding_${test_set}
    decode_log=${model_dir}/decoding_${test_set}.log
    decode_manifest=data/manifest_${test_set}.json
    python ${NGRAM_PATH}/eval_beamsearch_ngram.py --nemo_model_file ${model_file} \
                                                 --input_manifest ${decode_manifest} \
                                                 --acoustic_batch_size 10 \
                                                 --preds_output_folder ${decode_dir} \
                                                 --decoding_mode greedy | tee ${decode_log}
done
```</div>
How to restore a ckpt conformer model for inference/finetuning?
State: closed | Pull request: no
Labels: name: bug

For conformer model 

we can restore a nemo model by  
model = nemo_asr.models.EncDecCTCModelBPE.restore_from(name+'.nemo') 
and use it for inference.
But the models during training were saved in ckpt format.

And I tried model.load_state_dict(torch.load(model.ckpt))  but there are different keys.

How can I load a ckpt which was saved during training for inference? or how to convert a ckpt model to the target nemo model?

Thanks
Self-supervised tutorial & update
State: closed | Pull request: yes

- Basic tutorial for self-supervised pre-training and subsequent supervised fine-tuning
- Config for conformer SSL (may be changed later)
- Update to convert_to_tarred script to enable not having "text" field in manifest, and allow using "offset" for manifest without labels
- Add optional non-stride layers to reconstruction decoder
- Add option to load part of model from second nemo checkpoint from config (for example if we want to load encoder from one pre-trained model, and decoder from another)

# Changelog
- New way to initialize parts of models into current model for ssl

```markdown
  init_from_nemo_model: Str path to a .nemo model in order to load state_dict from single nemo file;
  if loading from multiple files, pass in a dict where the values have the following fields:
      - path: Str path to .nemo model
      - include: Optional list of strings, at least one of which needs to be contained in parameter name
      to be loaded from this .nemo file. Default: everything is included.
      - exclude: Optional list of strings, which can be used to exclude any parameter containing one of
      these strings from being loaded from this .nemo file. Default: nothing is excluded.
      hydra usage example:

```


```yaml
...

init_from_nemo_model:
    model0:
        path:<path/to/model1>
        include:["encoder"]
    model1:
        path:<path/to/model2>
        include:["decoder"]
        exclude:["embed"]

```
German normalization fails due to apparent installation problems.
State: closed | Pull request: no
Labels: name: bug

name: TN/ITN

**Describe the bug**

German normalization fails due to apparent installation problems.

**Steps/Code to reproduce bug**

If installed with:
`pip install nemo-toolkit[all]` the nemo_text_processing.text_normalization.de.taggers.tokenize_and_classify is not in the project in order to normalize german phrases although used in the nemo_text_processing/text_normalization/normalize.py file.

If installed with:
`python -m pip install git+https://github.com/NVIDIA/NeMo.git@main` the file nemo_text_processing/inverse_text_normalization/vi/taggers/money.py is not found.

If installed with:
`git clone -q https://github.com/NVIDIA/NeMo.git`
when normalizing, getting the following issue: [https://github.com/NVIDIA/NeMo/issues/3341]

**Expected behavior**

Successful German normalization

**Environment overview**

 - Environment location: Bare-metal: Ubuntu 20.04.3 LTS

**Environment details**

- OS version: Ubuntu 20.04.3 LTS
- PyTorch version:  Torch 1.10.0
- Python version: Python 3.8.12
nemo text normalizer fails to normalize in german - _pywrapfst.FstOpError: Operation failed
State: closed | Pull request: no
Labels: name: bug

name: TN/ITN

**Describe the bug**

Nemo normalizer fails to normalize phrases in german. Operation fails with a _pywrapfst.FstOpError caused by line 464 in the extensions/_pynini.pyx file.

**Steps/Code to reproduce bug**

`from nemo_text_processing.text_normalization.normalize import Normalizer
normalized_text = Normalizer(input_case='cased', lang='de').normalize('0.12', verbose=True)
`

**Expected behavior**

`normalized_text = 'null komma eins zwei'`

**Environment overview**

 - Environment location: Bare-metal: Ubuntu 20.04.3 LTS
 
 - Method of NeMo install: 
`git clone -q https://github.com/NVIDIA/NeMo.git`
`pip install -e NeMo`

**Environment details**

- OS version: Ubuntu 20.04.3 LTS
- PyTorch version:  Torch 1.10.0
- Python version: Python 3.8.12
NMT Shared Embeddings Weights
State: closed | Pull request: yes

This PR closes https://github.com/NVIDIA/NeMo/issues/3220 by adding support to share encoder and decoder embeddings and pre-softmax projection layers when the same vocabulary is used in the encoder and decoder.

**NOTE:** Only the shape of the embedding matrix is validated during runtime (i.e., vocabulary size, vocabulary dimension). It is the user's responsibility to make sure the same vocabulary is being used in the encoder and decoder tokenizers.

# Usage:

Set the following flag in `yaml` config to `true` (default is `false`):

```
model:
  shared_embeddings: true
```
Serving trained ASR models via TorchServe
State: closed | Pull request: no
Labels: name: feature

Hi,
Thanks for your great framework. 
Do you have any plan for adding serving ASR models via TorchServe (_e.g._, REST API or gRPC API)? I think it would be great (but I think at first, one must implement an appropriate handler for it, because to the best of my knowledge TorchServe at current state doesn't have any handler for speech models such as QuartzNet ASR model).
Best
Warning about slow loading of non-wav formats
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**

Recently I've tried to understand why is my gpu usage spiking from 0 to 100% and why does training seem to halt every num_workers batches. What I found is that my dataset consisted of .opus and .mp3 files, and that they load 2-30 times slower than those same files when converted to .wav.

**Describe the solution you'd like**

A warning message that informs the user that non-wav files might become a bottleneck for training.

It seems wrong to show this message in segment.py cause it would be displayed for each of those files and pollute the logs.
Maybe the ManifestProcessor can keep a list of found formats and in the end of processing compare it to soundfile.available_formats()?
Add uniform ASR evaluation script for all models
State: closed | Pull request: yes

# Changelog
- Add speech_to_text_eval.py which utilizes `transcribe_speech.py` to first perform transcription (for all asr models) and then uses its output to compute scores against some ground truth (via manifest).
- Remove speech_to_text_infer.py since it was hand tuned for just 1 type of model and was not an inference script but an evaluation script that required ground truth.
- Improve docstrings for transcribe_speech.py and transcribe_speech_parallel.py
AttributeError: type object 'EncDecCTCModel' has no attribute 'from_pretrained
State: closed | Pull request: no

I'm running into the same issue .
Python :3.8.12
Running inside docker image :pytorch:21.11-py3

Code: :
 ```
asr_model = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name='QuartzNet15x5Base-En', strict=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: type object 'EncDecCTCModel' has no attribute 'from_pretrained'

```
Installation steps :

- apt-get update && apt-get install -y libsndfile1 ffmpeg
- pip install Cython
- python -m pip install git+https://github.com/NVIDIA/NeMo.git@r1.5.0

_Originally posted by @architlfc in https://github.com/NVIDIA/NeMo/issues/2978#issuecomment-993514325_
AttributeError: Can't pickle local object 'AudioToBPEDataset.__init__.<locals>.TokenizerWrapper'
State: closed | Pull request: no

Hi, I can not run transcribe audio method using STT En Conformer-CTC Small model on windows 10.
I am using the following code: 
```
import nemo.collections.asr as nemo_asr

asr_model = nemo_asr.models.EncDecCTCModelBPE.restore_from("models/stt_en_conformer_ctc_small.nemo")

files = ['1.wav']

print(asr_model.transcribe(files, batch_size=1))
```
I'm using branch r1.5.0
As a result, I get the following error: 
**AttributeError: Can't pickle local object 'AudioToBPEDataset\.\_\_init\_\_.\<locals\>\.TokenizerWrapper'**

On the colab everything worked without problems. What could be the problem and how to solve it? Thank you.