renwei2024/nemo-github-issues huggingface dataset:
upgrade PTL trainer flags
State: closed | Pull request: yes

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

# What does this PR do ?

Upgrade pytorch_lightning trainer flags that are going to be deprecated in future

**Collection**: Core and All collections yaml files 

# Changelog 
- Update pytorch_lightning deprecated variables, namely:

> -  accelerator to the trainer now takes the type of accelerator device than distributed type
> -  distributed type is handled by strategy flag
> -  previously: accelerator='ddp' now accelerator='gpu' strategy='ddp'
> -  trainer flag checkpoint_callback is renamed to enable_checkpointing
> - ModelCheckpoint argument evert_n_val_epochs renamed to every_n_epochs
> - Update `trainer.gpus` to `trainer.devices` ( With this change setting devices=0 doesnt auto select gpu instead need to make accelerator='cpu' and also note accelerator=cpu now will only accept >0 as values for devices) 
> - `examples/asr/transcribe_speech.py` script modified to remove cuda option. Script automatically searches for GPU and if its not present then falls back to CPU  
> - `checkpointing_callback` -> `enable_checkpointing` 
> - `flush_logs_every_n_steps` removed from trainer flags
> - `sync_batchnorm` and `num_nodes` removed from NLPDPPPlugins 

# Usage
- 

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [] Did you write any new necessary tests?
- [] Did you add or update any necessary documentation?
- [] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
-
Upgrade ptl trainer flags
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Merge r1.6.1 main
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Merge bugfixes from r1.6.1 into main.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] update waveglow export
State: closed | Pull request: yes

# What does this PR do ?

Closes #3509 on the NeMo side. Requires a fix in Riva as well.

**Collection**: TTS

# Changelog 
- There was a name collusion with audio that resulted in a onnx dynamic axes error
- input_example was changed to be compliant with the new spec of (return dict,) as opposed to the older return dict
  
**PR Type**:
- [x] Bugfix
[TTS] Update UnivNet, HiFi-GAN and WaveGlow, small fixes in Mixer-TTS, FastPitch and Exportable
State: closed | Pull request: yes

**Collection**: [tts]

# Changelog 
- refactoring of HiFi-GAN code and config, replacing deprecated datasets with `VocoderDataset`
- refactoring of UnivNet code and config, replacing deprecated datasets with `VocoderDataset`, add exportability (it works correctly only for TorchScript)
- refactoring of WaveGlow code and config, replacing deprecated datasets with `VocoderDataset`
- minor refactoring of FastPitch code 
- minor refactoring of Mixer-TTS code 
- fixing small bugs in Exportable
  
**PR Type**:
- [x] New Feature
- [x] Bugfix
Fixed EH and error reporting in restore_from
State: closed | Pull request: yes

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

# What does this PR do ?

Fixes 'lost cascading error info' due to constructor not being under try:.
As a result, my previously added code to save initial error was not executed. 
Also streamlined control flow in that block.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add rank check and barrier
State: closed | Pull request: yes

# What does this PR do ?

Adds rank check and barrier when compiling helpers in megatron dataset

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add guards
State: closed | Pull request: yes

Signed-off-by: ericharper <complex451@gmail.com>

# What does this PR do?
Adds apex import guards.
Please have one of the following:

* Fixes # (issue)
* New feature: (short feature description)

## Before your PR is "Ready for review"
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or updated any necessary documentation?


If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
fix embedding name for verifying speakers
State: closed | Pull request: yes

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

# What does this PR do?

Please have one of the following:

* Fixes verify helper function for speaker verification 

## Before your PR is "Ready for review"
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or updated any necessary documentation?


If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Update pr template and re-add Changelog builder
State: closed | Pull request: yes

**Collection**: [docs]

# Changelog 
- Improves the PR template to include meta info + help reviewers in detailed review

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
ITN bug fixes, ip address, card num support, whitelist clean up
State: closed | Pull request: yes

Signed-off-by: ekmb <ebakhturina@nvidia.com>

# What does this PR do?

Please have one of the following:

* Fixes # (issue) ITN whitelist issues
* New feature: (short feature description): support for IP address, card num, country code

## Before your PR is "Ready for review"
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [X] Did you write any new necessary tests?
- [x] Did you add or updated any necessary documentation?


If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
GPT Test sync handler
State: closed | Pull request: yes

# What does this PR do?

Please have one of the following:

* Fixes # (issue)
* New feature: (short feature description)

## Before your PR is "Ready for review"
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or updated any necessary documentation?


If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Upgrade to 22.01 container
State: closed | Pull request: yes

# What does this PR do?
Upgrade to 22.01 container

Please have one of the following:

* Fixes # (issue) n/a
* New feature: (short feature description)
Upgrade to 22.01 container

## Before your PR is "Ready for review"
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or updated any necessary documentation?


If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
[problem of connecting lexicon or vocab to SDE tool] OOV doesn't show in wordstable
State: closed | Pull request: no
Labels: name: bug

@vsl9 
Hi, 
Thank you very much for the great Speech Data Explorer (SDE) tool. I have faced a problem when connecting a simple lexicon (or vocab as the --vocab argument) to the SDE. My vocab file has just one word in each line and I have followed the SDE code and it can load it correctly. But the OOV words doesn't show in the `wordstable`. I think the problem is in [`update_wordstable`](https://github.com/NVIDIA/NeMo/blob/main/tools/speech_data_explorer/data_explorer.py#L495) method. Please help me to address this issue. 
I also send you a simple (my tmp) lexicon.  
Best
[lexicon.csv](https://github.com/NVIDIA/NeMo/files/7970171/lexicon.csv)
[DRAFT] [TTS] FastPitch training tutorial
State: closed | Pull request: yes

# Addition of FastPitch training tutorial

Please have one of the following:

* New feature: (short feature description) FastPitch training tutorial.
This PR adds FastPitch tutorial to the same notebook as MixerTTS as both generate spectrogram. This tutorial goes through the data download, preprocess, FastPitch import and training pipeline.
Tutorial was added with careful consideration to make minimal changes to MixerTTS tutorial.

## Before your PR is "Ready for review"
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or updated any necessary documentation?


If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Problems with WaveGlow
State: closed | Pull request: no
Labels: name: bug

Hi all,

I am trying to train a TTS model with NeMo implementations of **Tacotron2** and **WaveGlow** architectures. The dataset is my own and includes 15 hours of speech in Slovenian language (sample rate 22050, 16-bit, mono). Label set has 83 symbols, all within 0-255 ASCII. 

The dataset has been tested and should not be problematic – I used it before for training a TTS model with Tacotron2/WaveGlow architectures from the **DeepLeaningExamples** (DLE). The model developed with DLE produced rather clear speech. 

With the NeMo implementation of the Tacotron2 and WaveGlow I am however not able to develop a model that would produce speech of at least a moderate quality. According to the encoder/decoder alignment it seems that Tacotron2 learns ok, while WaveGlow never learns as it should. The NeMo version I use is 1.5.0.

Here is what I have tried so far:

1.	I trained with 16 and 32 precisions
2.	I trained with batch sizes of 10, 16 and 20.
3.	I trained 1000 and 1500 epochs
4.	I trained on different HW settings: I used one node with 4 x A100 GPU and even 8 nodes with 4 x A100 GPU

No matter what I try, results are disappointing. Here are also some figures:

This is what the model produces after 1000 epochs (fig above) and what it should produce (fig below) (bs=20):
<img width="452" alt="WaveGlow predicted" src="https://user-images.githubusercontent.com/16853936/151710024-4e28a9a6-148c-459a-947f-af2d70988f46.png">
<img width="452" alt="WaveGlow target" src="https://user-images.githubusercontent.com/16853936/151710040-1190ab0c-e2ac-430c-a58c-5e3b1a840458.png">

I know that val loss is not very informative for WaveGlow, but nevertheless, here is what is shows:
<img width="452" alt="loss" src="https://user-images.githubusercontent.com/16853936/151710461-f4948328-6da3-4ebd-b9a3-c63e55289772.png">

And last but not least: I trained HifiGan instead of WaveGlow and results seem ok. So I am wondering what am I doing wrong with WaveGlow. Any help/hint will be appreciated.
How about supporting Unigram tokenizer in NeMO?
State: closed | Pull request: no
Labels: name: feature

Currently NeMo has BPE but I don't see Unigram.

[Unigram is better than BPE.](https://arxiv.org/pdf/2004.03720.pdf)

[SpeechBrain supports Unigram.](https://colab.research.google.com/drive/12yE3myHSH-eUxzNM0-FLtEOhzdQoLYWe?usp=sharing)
[TTS] Fix bugs in HiFi-GAN (scheduler, optimizers) and add input_example() in Mixer-TTS/Mixer-TTS-X
State: closed | Pull request: yes

# What does this PR do?

* Fixes: 
  * https://github.com/NVIDIA/NeMo/issues/3557 
  * https://github.com/NVIDIA/NeMo/issues/3549
  * Use `self.optimizers` in HiFi-GAN
  * Fix typo in FastPitch's `input_example()`

Signed-off-by: Oktai Tatanov <oktai.tatanov@gmail.com>
Fix restoring from checkpoint for case when `model.common_dataset_parameters.label_vocab_dir` is provided
State: closed | Pull request: yes
Labels: name: bug

# What does this PR do?

Please have one of the following:

* Fixes #3561

## Before your PR is "Ready for review"
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or updated any necessary documentation?


If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
`PunctuationCapitalizationModel` restoring from checkpoint is not working if a config field `model.common_dataset_parameters.label_vocab_dir` is set
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

If model is being restored from checkpoint path from a config field `model.common_dataset_parameters.label_vocab_dir` should not be prepended to config fields `model.class_labels.punct_labels_file` and `model.class_labels.punct_labels_file`. This because in a checkpoint these labels files are in the root of a checkpoint archive.

**Steps/Code to reproduce bug**

1. Create a config provides label ids using  `model.class_labels.punct_labels_file` and `model.class_labels.punct_labels_file`.  `model.common_dataset_parameters.label_vocab_dir` should not be empty.
2. Create a `.nemo` checkpoint for a model created via the config.
3. Try to restore from the `.nemo` checkpoint.


**Expected behavior**

A restoring without errors

**Environment overview (please complete the following information)**

 - Environment location: Bare-metal
 - Method of NeMo install: `pip install nemo_toolkit[all]`

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 20.04.1 LTS
- PyTorch version: 1.10.2
- Python version: 3.8.12
No module named 'apex' in colab notebook 00_NeMo_Primer.ipynb
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When running `00_NeMo_Primer.ipynb` notebook during `import nemo.collections.nlp as nemo_nlp` raises an error:
`ModuleNotFoundError: No module named 'apex'`

**Steps/Code to reproduce bug**

Opened in colab tutorial notebook [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/00_NeMo_Primer.ipynb) and "Run all cells". Then after installation section at importing stage error raises.


**Expected behavior**

End-to-end notebook executions without errors.

**Environment overview (please complete the following information)**

 - Environment location: colab
 - Method of NeMo install: as in notebook
 ```
BRANCH = 'main'
!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]
```

**Environment details**
- OS version: linux
- PyTorch version: 1.10.0+cu111
- Python version: 3.7.12

**Additional context**
Error occurs in both kernel-types: CPU and GPU.
![apex1](https://user-images.githubusercontent.com/36787333/151659365-9b5a7593-a343-4a8f-8f41-83ff04dfcfc4.jpg)
![apex2](https://user-images.githubusercontent.com/36787333/151659369-7af2fed5-83e7-4ff6-8917-d1960f473453.jpg)
Error when exporting mixer_tts model to onnx model
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I got this error when exporting mixer_tts model to onnx model:
AttributeError: 'MixerTTSModel' object has no attribute 'input_example'

**Steps/Code to reproduce bug**

```
from nemo.collections.tts.models import MixerTTSModel
spec_generator = MixerTTSModel.from_pretrained("tts_en_lj_mixerttsx")

spec_generator.export('mixter_tts.onnx')
```

**Expected behavior**

onnx model generated succefully

**Environment overview (please complete the following information)**

pip3 install nemo_toolkit['all']

NeMo version is 1.6.0
Update deprecated pytorch_lightning flags in NeMo
State: closed | Pull request: yes

# What does this PR do?

Please have one of the following:

- Update pytorch_lightning deprecated variables, namely:

> - accelerator to the trainer now takes the type of accelerator device than distributed type 
> - distributed type is handled by strategy flag
> - previously: accelerator='ddp' now accelerator='gpu' strategy='ddp' 
> - trainer flag checkpoint_callback is renamed to enable_checkpointing
> - ModelCheckpoint argument evert_n_val_epochs renamed to every_n_epochs 

Reference: https://pytorch-lightning.readthedocs.io/en/1.5.9/common/trainer.html#trainer-flags 

## Before your PR is "Ready for review"
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?


If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?
@titu1994
Issues viewing NMTensors and 
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I am able to load a pretrained version of Quartznet, and when I explore the encodings, I can't figure out a way to convert them into something I can analyze, like a list, np.ndarray or pandas Series. This is for a voice analysis project and I am looking to access the encodings directly. thanks so much



**Steps/Code to reproduce bug**
https://colab.research.google.com/github/tugstugi/dl-colab-notebooks/blob/master/notebooks/NVidiaQuartzNetMic.ipynb
Go here, Run All using the Co-Lab. Then after putting in your voice for the 10s sample recording, 

1. Add breakpoint just after line 63 
<img width="533" alt="image" src="https://user-images.githubusercontent.com/13772726/151621762-82e07180-21b7-46be-a562-bbfb300bce14.png">
to
<img width="542" alt="image" src="https://user-images.githubusercontent.com/13772726/151621800-8d66a5dd-b26a-481c-b90a-8a6133d7989f.png">
`import pdb; pdb.set_trace()`

2. Add new cell at bottom with `wav_to_text(create_manifest('test.wav'))`
3. once PDB debugger opens, type `encoded` and see this result:
<img width="550" alt="image" src="https://user-images.githubusercontent.com/13772726/151621897-065bef93-6ea7-4820-b785-273603db8f72.png">

I just want to convert that `nemo.core.neural_types.NmTensor` into a list, pandas series, etc. Something with numbers that I can look at.

```
encoded, encoded_len = jasper_encoder(audio_signal=processed_signal,
                                          length=processed_signal_len)
```
Add support for Gradient Clipping (clamp) in RNNT Numba loss
State: closed | Pull request: yes

# Changelog
- Add support for `clamp` kwarg in rnnt loss numba. 
- Update configs to support this argument.

# Note
- This is different from using Pytorch Lightning `gradient_clip_val>0` as that will clip all gradients of the entire network by the limits of the clamp, this will only clip the gradient of the joint tensor itself and not the rest of the network.
[BUG or INTENDED?] optim.sched vs optim and sched on same level
State: closed | Pull request: no
Labels: name: bug

Reading through https://github.com/NVIDIA/NeMo/blob/main/examples/tts/conf/hifigan/hifigan.yaml, I noticed that hifigan specifies the `optim` and `sched` on the same level, on quick check in other config files the `sched` is a sub parameter of `optim`. Any special reason?

Checking out the docs https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/core/core.html?highlight=scheduler#optimization, https://github.com/NVIDIA/NeMo/blob/main/docs/source/core/core.rst#optimization, I notice that it is quite misleading. It is not evident which version is the correct one.
Megatron encoder-decoder refactor
State: closed | Pull request: yes

This PR refactor Megatron T5 to a modular base encoder-decoder class hierarchy, and a T5 class which customizes the base class. The goal of the refactor is to support multiple encoder-decoder models with minimal code redundancy. 

# Refactor High Level Description

Hierarchy tree is expressed below via list alignment (i.e., right alignment for child class).

* `MegatronModule` sub-classes:
	* `MegatronTransformerEncoderModule`/`MegatronTransformerDecoderModule` - various architectures for manipulation of hidden --> hidden
	* `MegatronTransformerEncoderDecoderModule` - contains above classes and generates required joint encoder/decoder mask given independent masks, and reparametrization (e.g., vae)
	* `TokensEncoderDecoderModule` - modality-specific (e.g., tokens-tokens) model which map between pre-processed data (e.g., token ids) and encoder/decoder continuous input (i.e., hidden); handles embedders (i.e., embedding + positional embedding); handles projection from hidden to data distribution (e.g., project from hidden dimensions to the number of tokens)
* `NLPModel` sub-classes:
	* `MegatronBaseModel` - common initialization of Megatron
		* `MegatronLMEncoderDecoderModel` - modality-specific (e.g., text-text) model which pre-process raw data (e.g., tokenizer pre-process text to token ids); generic modality-specific training/testing/inference/loss methods
			* `MegatronT5Model` - a specific model (i.e., T5) which prepares datasets (i.e., with corresponding data augmentation), and model-specific features (e.g., sentinels in tokenizer)

# Notable Changes

1. Encoder/decoder mask is 2D with `1` for active element, and `0` for masked element (i.e., as opposed to 3D Megatron  mask with `False` for active and `True` for masked). This allows easy manipulation of masks. Conversion to 3D Megatron masks is done on-the-fly.
2. Joint cross-attention mask is not precomputed, but is rather done on-the-fly.
_youtokentome.so: undefined symbol: _ZNSt7__cxx...
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I am unable to use nemo models due to some issue with the tokenizer: 
```
    from nemo.collections.asr.models      import EncDecCTCModel
  File "/home/linux/miniconda3/envs/env/lib/python3.9/site-packages/nemo/collections/asr/__init__.py", line 15, in <module>
    from nemo.collections.asr import data, losses, models, modules
  File "/home/linux/miniconda3/envs/env/lib/python3.9/site-packages/nemo/collections/asr/models/__init__.py", line 18, in <module>
    from nemo.collections.asr.models.classification_models import EncDecClassificationModel
  File "/home/linux/miniconda3/envs/env/lib/python3.9/site-packages/nemo/collections/asr/models/classification_models.py", line 33, in <module>
    from nemo.collections.common.metrics import TopKClassificationAccuracy
  File "/home/linux/miniconda3/envs/env/lib/python3.9/site-packages/nemo/collections/common/__init__.py", line 16, in <module>
    from nemo.collections.common import data, losses, parts, tokenizers
  File "/home/linux/miniconda3/envs/env/lib/python3.9/site-packages/nemo/collections/common/tokenizers/__init__.py", line 24, in <module>
    from nemo.collections.common.tokenizers.youtokentome_tokenizer import YouTokenToMeTokenizer
  File "/home/linux/miniconda3/envs/env/lib/python3.9/site-packages/nemo/collections/common/tokenizers/youtokentome_tokenizer.py", line 17, in <module>
    import youtokentome as yttm
  File "/home/linux/miniconda3/envs/env/lib/python3.9/site-packages/youtokentome/__init__.py", line 1, in <module>
    from .youtokentome import BPE
  File "/home/linux/miniconda3/envs/env/lib/python3.9/site-packages/youtokentome/youtokentome.py", line 4, in <module>
    import _youtokentome_cython
ImportError: /home/linux/miniconda3/envs/env/lib/python3.9/site-packages/_youtokentome_cython.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE7reserveEv

```

**Steps/Code to reproduce bug**

```
from nemo.collections.asr.models import EncDecCTCModel
```

**Environment overview (please complete the following information)**

 - Environment location: local
 - Method of NeMo install: `python -m pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[all]`

**Environment details**

- OS version: `Pop OS 21.10` (similar issue on Ubuntu 21.04)
- PyTorch version: `1.10.1`
- Python version: `3.9`
Add Change Log builder to NeMo
State: closed | Pull request: yes

# Changelog
- Build a raw changelog automatically for every new NeMo release based on a comparison from PRs between newest version and previous version. 
- Raw changelog is then tidied to display on final page. 

# Usage

Call the builder and visit the GitHub Action that builds the changelog. It's "Print Changelog" section will have the raw changelog, which needs to be tidied up. 

```sh
git tag vX.YZ
git push origin vX.Y.Z
```

## Redo steps
If something should go wrong tag can be deleted using 

```sh
git tag vX.YZ -d
git push origin vX.Y.Z -d
```
refactor dialogue state tracking for modelling/dataset interoperability
State: closed | Pull request: yes

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

Design based on attached diagram
[UML proposed dialogue1.pdf](https://github.com/NVIDIA/NeMo/files/7946212/UML.proposed.dialogue1.pdf)

Currently supports SGD-QA (Bert-based) and Huggingface GPT2 models, and planning to support other models.
Fixed section typo
State: closed | Pull request: yes

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Fixed Bugs 3509889 and 3509839:

https://nvbugswb.nvidia.com/NVBugs5/redir.aspx?url=/3509889 [ML-NeMo][Tesla A30] Token_Classification-BioMegatron.ipynb contains a typo 

https://nvbugswb.nvidia.com/NVBugs5/redir.aspx?url=/3509839 [ML-NeMo][Tesla A30] Relation_Extraction-BioMegatron.ipynb contains a typo
Enforce utf-8 on all file r/w
State: closed | Pull request: yes

# Bugfix
- Enforce UTF-8 on all file reading and writing. Fixes a bug with some locales not enforcing utf 8.
- If you are still facing Unicode errors, you have to pass `LC_ALL=en_US.UTF-8` before launching your python command. 

Fixes https://github.com/NVIDIA/NeMo/issues/3452
Fixed duplicate cell bug
State: closed | Pull request: yes

Fixes Duplicate cells in entity linking notebook bug

Bug 3509934 (https://nvbugswb.nvidia.com/NVBugs5/redir.aspx?url=/3509934 [ML-NeMo][Quadro GV100] Entity_Linking_Medical.ipynb tutorial notebook contains duplicate cells)

Signed-off-by: Virginia Adams <vadams@nvidia.com>
Add HuggingFace Datasets to NeMo ASR Dataset script
State: closed | Pull request: yes

# Changelog
- Add script to convert HuggingFace ASR datasets to preprocessed NeMo compatible datasets.
- These datasets can then be converted to tarred datasets as always.

# NOTE:
- The converted datasets DO NOT PERFORM TEXT PREPROCESING. That should be handled by the user at manifest level.
- The offline mode dataset transformation is fast but requires 3 copies of the dataset in its entirety at any given time - so it needs a lot of disk space.
nemo2riva Tacotron2 model
State: closed | Pull request: no
Labels: name: bug

When trying to convert a Tacotron2 TTS model trained with nemo I receive the following error:
```
W0125 21:12:08.102883 140527509010240 schema.py:118] Validation schema not found for nemo.collections.tts.models.Tacotron2Model.                
That means Riva does not yet support a pipeline for this network and likely will not work with it.                                              
I0125 21:12:08.860513 140527509010240 artifacts.py:54] Found model at ./model_weights.ckpt                                                      
I0125 21:12:08.861304 140527509010240 artifacts.py:120] Retrieved artifacts: dict_keys(['model_config.yaml'])                                   
E0125 21:12:08.861998 140527509010240 cookbook.py:58] Your NeMo model class (nemo.collections.tts.models.tacotron2.Tacotron2Model) is not Exportable.                                                                                                                                           
```
regardless that on https://docs.nvidia.com/deeplearning/riva/user-guide/docs/service-tts.html#model-deployment, mention both FastPitch+HifiGAN and Tacotron2+Waveglow.
E2E TTS fixes
State: closed | Pull request: yes

Several fixes to get the FastSpeech2-HiFiGAN training to work (#3360). The normal training command should work now.

- Config reference to n_mels corrected
- Fix output type issue when `splices` is None for FS2-HiFiGAN
- Removed `spec` from calls to forward function in FS2-HiFiGAN
- Changed type of FS2-HiFiGAN output to AudioSignal
- Added keywords to functions that require kwargs due to typing
- `precessor` -> `preprocessor` typo fixes
- FastSpeech2 duration loss type fixed, model output type changed to reflect reality

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>
ITN: Assets not installed when pip install from source
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When installing nemo[all] via pip from source, some of the data files used to build the french ITN grammar are missing from site-packages (e.g. those in folders electronic, measurements, money, numbers, time). These folders are the only ones that are not packages i.e. missing `__init__.py` files.

**Steps/Code to reproduce bug**

`
python -m pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[all]
ls /venv/nemo/lib/python3.9/site-packages/nemo_text_processing/inverse_text_normalization/fr/data/
`
> `fractions.tsv  __init__.py  months.tsv  ordinals  __pycache__  roman  whitelist.tsv`

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: venv on local machine
 - Method of NeMo install:
    `apt-get update && apt-get install -y libsndfile1 ffmpeg`
    `pip install Cython`
    `python -m pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[all]`

**Environment details**


    OS version - Ubuntu 18.04.5 LTS
    PyTorch version - 1.10.1
    Python version - 3.8.11
integrate with Lightning ecosystem CI
State: closed | Pull request: no
Labels: name: feature

name: stale

Hello and so happy to see you use Pytorch-Lightning! :tada: 
Just wondering if you already heard about quite the new **Pytorch Lightning (PL) ecosystem CI** where we would like to invite you to... You can check out our blog post about it: [Stay Ahead of Breaking Changes with the New Lightning Ecosystem CI](https://devblog.pytorchlightning.ai/stay-ahead-of-breaking-changes-with-the-new-lightning-ecosystem-ci-b7e1cf78a6c7) :zap: 
As you use PL framework for your cool project, we would like to enhance your experience and offer you safe updates to our future releases. At this moment, you run tests with a particular PL version, but it may accidentally happen that the next version will be incompatible with your project... :confused: We do not intend to change anything on our project side, but still here we have a solution - ecosystem CI with testing both - your and our latest development head we can find it very early and prevent releasing eventually bad version... :+1: 

**What is needed to do?**
- have some tests, including PL integration
- add config to ecosystem CI - https://github.com/PyTorchLightning/ecosystem-ci

**What will you get?**
- scheduled nightly testing configured for development/stable versions
- slack notification if something went wrong to investigate
- testing also on multi-GPU machine as our gift to you :rabbit:
Dependence of the WER on batch size in val set? 
State: closed | Pull request: no

**Describe the bug**

Trained the QuartzNet15x5 from scratch on the dataset An4 for 200 epochs. When I try to calculate the WER using the method given in the first tutorial, I noticed a strange dependence of the batch size on the size of the resulting error value. The larger the batch size, the smaller the error, what is the reason for this behaviour? 
Results looks like that: 
4 batch wer0.51
8 batch wer0.46
16 batch wer0.44
32 batch wer0.439
64 batch wer0.434
128 batch wer0.430
256 batch wer0.434
512 batch wer0.404
1024 batch wer0.399
2048 batch wer0.397
4096 batch wer0.398

**Steps/Code to reproduce bug**

    # Bigger batch-size = bigger throughput
    params['model']['validation_ds']['batch_size'] = 4096

    # Setup the test data loader and make sure the model is on GPU
    first_asr_model.setup_test_data(test_data_config=params['model']['validation_ds'])
    first_asr_model.cuda()

    # We will be computing Word Error Rate (WER) metric between our hypothesis and predictions.
    # WER is computed as numerator/denominator.
    # We'll gather all the test batches' numerators and denominators.
    wer_nums = []
    wer_denoms = []

    # Loop over all test batches.
    # Iterating over the model's `test_dataloader` will give us:
    # (audio_signal, audio_signal_length, transcript_tokens, transcript_length)
    # See the AudioToCharDataset for more details.
    for test_batch in first_asr_model.test_dataloader():

            test_batch = [x.cuda() for x in test_batch]
            targets = test_batch[2]
            targets_lengths = test_batch[3]        



            log_probs, encoded_len, greedy_predictions = first_asr_model(
                input_signal=test_batch[0], input_signal_length=test_batch[1]
            )
            # Notice the model has a helper object to compute WER
            first_asr_model._wer.update(greedy_predictions, targets, targets_lengths)
            _, wer_num, wer_denom = first_asr_model._wer.compute()

            first_asr_model._wer.reset()
            wer_nums.append(wer_num.detach().cpu().numpy())
            wer_denoms.append(wer_denom.detach().cpu().numpy())

            # Release tensors from GPU memory
            del test_batch, log_probs, targets, targets_lengths, encoded_len, greedy_predictions

    # We need to sum all numerators and denominators first. Then divide.
    print(f"WER = {sum(wer_nums)/sum(wer_denoms)}")


**Expected behavior**

I thought that nothing should change and the size of the validation batch only determines the speed at which the error is calculated 

**Environment overview (please complete the following information)**

 - Environment location: [Collab]
Unable to load WaveGlow checkpoints or models: TypeError: Expected data to be int, Sequence or Mapping, but got NoneType
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I can train WaveGlow models, but when I try to load checkpoints or `.nemo` files back, I am not able to.

**Steps/Code to reproduce bug**

Train, in a Google Colab GPU environment with or without Apex, a Waveglow 16khz model for one epoch. When attempting to load it back, either the checkpoint or the `.nemo` archive, I get the following error consistently:

```
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/supporters.py", line 589, in _nested_calc_num_data
    raise TypeError(f"Expected data to be int, Sequence or Mapping, but got {type(data).__name__}")
TypeError: Expected data to be int, Sequence or Mapping, but got NoneType
```

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

1. Train a new WaveGlow model for one epoch using the defaults, following the example in https://github.com/NVIDIA/NeMo/blob/main/examples/tts/waveglow.py. I have barely altered this, only adding a snippet to restore a model from a file.
2. Use `model = WaveGlowModel.restore_from(path_to_my_model)`. It will fail with the above error. Loading the checkpoint also fails with the same error.


**Expected behavior**

I expect to be able to load either checkpoints or `.nemo` files.
**Environment overview (please complete the following information)**

 - Environment location: colab
 - Method of NeMo install: [pip install or from source]. pip
 
**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 18.04 .5 LTS
- PyTorch version: 1.10.0+cu111
- Python version: Python 3.7.12

**Additional context**
I'm not sure if the fact that it's 16khz matters, but it's the only meaningful deviation  have from the example. Some online research suggests that this is because there's some mixed CPU/GPU training going on, but I wasn't able to find much more information than that, or how to fix it.

I will also note, that I expect a `.nemo` file for each checkpoint, and this does not seem to be the behavior - there seems to be just one `.nemo` file and many large `.ckpt` files, but this may be unrelated.
Always get RuntimeError: CUDA out of memory. Tried to allocate XX.XX MiB
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
I am trying to train a TTS model. I followed the steps of the documentation but still cannot train the model.
I always get RuntimeError: Cuda out of memory. So the program crash.

**Steps/Code to reproduce bug**

The way to reproduce the error is to execute this command from the console in a linux:
python3 tacotron2.py sample_rate=22050 train_dataset=files.json validation_datasets=files.json trainer.max_epochs=1 trainer.accelerator=null trainer.check_val_every_n_epoch=1

The file tacotron2.py is the next:
https://github.com/NVIDIA/NeMo/blob/main/examples/tts/tacotron2.py

The config file for tacotron2.py is https://github.com/NVIDIA/NeMo/blob/main/examples/tts/conf/tacotron2.yaml

I edited tacotron2.yaml in my computer to reduce the batch_size and num_workers
The values are:
batch_size=1
num_workers=1

So, I reduced the batch_size to the minium to try to avoid the cuda error with No success.

The file "files.json":
I started with 100.000 lines. Each line like this: (I mean the file is written as explained in the documentation)
{"audio_filepath": "4da6b70e-0108-4f75-80ae-3d71f1dd2c2b.wav", "text": "y aquí en dos palotadas hemos encontrado robustas columnas donde apoyar la grandiosa fábrica de su alcurnia", "duration": 6.84}

I tried to train the model reducing the number of lines. I Started with 100000. Then I reduced to the file "files.json" to only 100. Unfortunately, it crashed as well..

**Expected behavior**

I would like to find a way to train a TTS model managing properly the GPU memory.

**Environment overview**

Computer: MSI Apache Pro GE70 - 2PE
Operating system: Ubuntu 20.04.3 LTS
GPU: NVIDIA GTX 860 (2GB)
nemo-toolkit                  1.5.1               
pytorch-lightning             1.5.7               
torch                         1.10.1              
torch-stft                    0.1.4               
torchaudio                    0.10.1+cu102        
torchmetrics                  0.6.2               
torchvision                   0.11.2+cu102   


Could you help me to solve it?

Thank you!
[TTS] New structure for TTS datasets in scripts/dataset_processing, VocoderDataset, update TTSDataset
State: closed | Pull request: yes

- [x] Create unify downloading and processing TTS datasets in `scripts/dataset_proccesing/tts`
- [x] Add `VocoderDataset`
- [x] Fix problem with collision in name (e.g. LibriTTS suffers) for supplementary data in `TTSDataset`
- [x] Other minor updates in `collection.tts.torch` 


Signed-off-by: Oktai Tatanov <oktai.tatanov@gmail.com>