renwei2024/nemo-github-issues huggingface dataset:
Fix T5 Encoder Mask while decoding
State: closed | Pull request: yes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

# What does this PR do ?

Fixes a bug with T5 decoding present with encoder mask computation.

NOTE: This does not affect fine-tuning that calls `.decode()` directly. It only affects `.complete()`.

**Collection**: NLP

# Changelog 
- One line removal of encoder_mask = encoder_mask < 0.5

# Usage
N/A

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Set find_unused_parameters to False in GPT example script
State: closed | Pull request: yes

Signed-off-by: ericharper <complex451@gmail.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
parse_dataset_as_name returns an empty string if a manifest is named "manifest.json"
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When instantiating a new model, in `nemo.utils.model_utils.parse_dataset_as_name`, if the name of the manifest passed is `manifest.json`, then eventually the manifest name will be empty and an index error will occur during this check:

```py
if '_' != name[-1]:
    name = name + '_'
```

**Steps/Code to reproduce bug**

Run any ASR example/tutorial with a cfg value `model.(train/val)_ds.manifest_filepath = "/path/to/manifest.json"`

```py
from omegaconf import OmegaConf
import pytorch_lightning as pl
from nemo.collections.asr.models import EncDecCTCModelBPE

cfg = OmegaConf.load("/path/to/cfg.yaml")
trainer = pl.Trainer(**cfg.trainer)
asr_model = EncDecCTCModelBPE(cfg=cfg.model, trainer=trainer)
```

**Expected behavior**

I should be able to create a model with train/val_ds whose manifest file name is `manifest.json`

**Environment overview (please complete the following information)**

 - Environment location: Docker
 - Method of NeMo install: `pip install nemo_tookit[all]==1.7.0`
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used
 Docker image: `nvcr.io/nvidia/pytorch:22.02-py3`

I can of course change the name of my manifest file, but I don't think this is valid assumption that we will not be naming our files `manifest.json` and not sure why this parsing is necessary.
Simple resolution is to just make sure the `name` string never becomes empty and early return if it will.
If it does, maybe use the parent directory name as the dataset name?
If this is necessary, can make the change on my end, but at least provide a different error informing the user to change their manifest name.
Add text_generation_util to support TopK, TopP sampling + Tabular Data Generation.
State: closed | Pull request: yes

# What does this PR do?

Added the text_generation_util to support the MegatronLM way of text generation. It supports top_k, top_p sampling. 

Added the Tabular Data tokenizer which recognizes the tabular data structure. 
Added the unit test for column type encode and decode.
Added tabular data generation which considers the tabular data structure.

To run text generation. here is one example.
```python
python examples/nlp/language_modeling/megatron_gpt_eval.py \
            model_file=/path/gpt_model.nemo
```
the inference parameters are set in file
```
megatron_gpt_inference.yaml
```
Update: 

Included GPT inference server. To use it  
```python
python examples/nlp/language_modeling/megatron_gpt_eval.py \
            model_file=/path/gpt_model.nemo \
            server=True \
            port=5555
```
Here is the example code to run inference as a client
```python
import json
import requests

batch_size = 8
port_num = 5555
headers = {"Content-Type": "application/json"}


def request_data(data):
    resp = requests.put('http://localhost:{}/generate'.format(port_num),
                        data=json.dumps(data),
                        headers=headers)
    sentences = resp.json()['sentences']
    return sentences


data = {
    "sentences": [""] * batch_size,
    "tokens_to_generate": 300,
    "temperature": 1.0,
    "add_BOS": True,
    "top_k": 0,
    "top_p": 0.9,
    "greedy": False,
    "all_probs": False,
    "repetition_penalty": 1.2,
    "min_tokens_to_generate": 2,
}

sentences = request_data(data)

for i in range(batch_size):
    s = sentences[i]
    print(s)
```
fix: multilingual transcribe does not require lang id param
State: closed | Pull request: yes

# What does this PR do ?

Fixes model.transcribe() for multilingual models.

**Collection**: ASR

# Changelog 
- multilingual models require language id for manifests during training. The transcribe method used to generate a transient
manifest that had 'nothing' as placeholder text. Replacing this placeholder text with ''. If this empty string is detected , the manifest language id requirement gets lifted in this PR.

# Usage
* No changes to interface

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Cant load model from path with 1.6
State: closed | Pull request: no
Labels: name: bug

I have a 1.6 model trying to load with the following line:

`nemo_asr.models.EncDecCTCModelBPE.restore_from`

And I get this:

```
[NeMo E 2022-03-11 17:35:11 common:504] Traceback (most recent call last):
      File "/root/.cache/pypoetry/virtualenvs/--IulmO6hw-py3.7/lib/python3.7/site-packages/nemo/core/classes/common.py", line 477, in from_config_dict
        instance = imported_cls(cfg=config, trainer=trainer)
      File "/root/.cache/pypoetry/virtualenvs/--IulmO6hw-py3.7/lib/python3.7/site-packages/nemo/collections/asr/models/ctc_bpe_models.py", line 194, in __init__
        self._setup_tokenizer(cfg.tokenizer)
      File "/root/.cache/pypoetry/virtualenvs/--IulmO6hw-py3.7/lib/python3.7/site-packages/nemo/collections/asr/parts/mixins/mixins.py", line 84, in _setup_tokenizer
        self.tokenizer = tokenizers.SentencePieceTokenizer(model_path=model_path)
      File "/root/.cache/pypoetry/virtualenvs/--IulmO6hw-py3.7/lib/python3.7/site-packages/nemo/utils/exceptions.py", line 31, in __init__
        raise LightningNotInstalledException(self)
    nemo.utils.exceptions.LightningNotInstalledException:  You are trying to use <nemo.collections.common.tokenizers.SentencePieceTokenizer object at 0x7f582a0ad950> without installing all of pytorch_lightning, hydra, and omegaconf. Please install those packages before trying to access <nemo.collections.common.tokenizers.SentencePieceTokenizer object at 0x7f582a0ad950>.
    
```

All there of those packages are installed and importable
gradient noise vs variational weight noise
State: closed | Pull request: no

Hi, I notice that the variational noise for RNNT model is implemented as gradient noise addition (line 878 of `nemo/collections/asr/models/rnnt_models.py`). The variational noise cited by the Conformer-Transducer paper is actually adding noise to weights. So, is this a mistake or because injecting gradient noise works better than adding weight noise?
Add new pretrained Spanish ASR models
State: closed | Pull request: yes

Signed-off-by: Elena Rastorgueva <erastorgueva@nvidia.com>

# What does this PR do ?
Add new pretrained Spanish ASR models (Citrinet 1024, Conformer CTC Large, Conformer Transducer Large, ContextNet 1024) to `list_available_models` and to documentation

**Collection**: [Note which collection this PR will affect]
ASR

# Changelog 
* Add new pretrained Spanish ASR models (Citrinet 1024, Conformer CTC Large, Conformer Transducer Large, ContextNet 1024) to the method `list_available_models` in `EncDecCTCModelBPE` and `EncDecRNNTBPEModel`.
* Added the 4 models also to the list of Spanish ASR models in the documentation. 

# Usage


# Before your PR is "Ready for review"
**Pre checks**:
- [y] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [n/a] Did you write any new necessary tests?
- [y] Did you add or update any necessary documentation?
- [n] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [y] New Feature
- [ ] Bugfix
- [y] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Question about LM adapted T5 dataset
State: closed | Pull request: no

Hi all,

Thank you for your effort for these work. I have a little question about the split point of the input source. As mentioned in T5 paper, they choose a random point to split it into prefix and target portions. However, you set a specific length to split text sequence into two parts as shown in the following code

https://github.com/NVIDIA/NeMo/blob/49183b7c59c3b4243671d5eb4d3f202a4e4739ce/nemo/collections/nlp/data/language_modeling/megatron/lm_adapted_t5_dataset.py#L49-L52

I am curious about this part. Could you please explain why this set is different from T5 paper? Thank you in advance.
Offline VAD+ASR tutorial
State: closed | Pull request: yes

# What does this PR do ?

This PR adds functions and a tutorial about how to perform offline VAD before ASR.
Pipeline: audio -> [VAD] -> speech segments -> [ASR] -> segmented ASR output -> stitched ASR output

**Collection**:  ASR

# Changelog 
- Transcribe partial audio (for example speech segments) with manifest.
- Stitch segmented ASR output.
- Construct the manifest that aligns stitched ASR outputs and ground truth text. 
- And a walkthrough tutorial for the whole process. 

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
get_statistics_with_mask (mean vs sum)
State: closed | Pull request: no

Thank you for providing a very good framework and solution for speaker recognition.

To understand more about it, I read through source code and paper. I want to clarify one of the calculation in `get_statistics_with_mask`. To calculate statistics, the variable name and description is `mean` while the implementation is `sum`  when calculating input and mask.

May I confirm whether it is `mean` or `sum` operation? 

https://github.com/NVIDIA/NeMo/blob/v1.7.0/nemo/collections/asr/parts/submodules/tdnn_attention.py#L87
Fix MegatronPretrainingRandomSampler by taking `data_parallel_size` into account
State: closed | Pull request: yes

# What does this PR do ?

Use `data_parallel_size` appropriately.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- 

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

ccing @ericharper @MaximumEntropy 

# Additional Information
* Related to #3665
NaN error while training Conformer-Transducer with automatic mixed precesion
State: closed | Pull request: no
Labels: name: bug

Hi, thanks for the great toolkit!

I'm training the conformer-transducer-medium on librispeech-960hr from scratch, following the config in `examples/asr/conf/conformer/conformer_transducer_bpe.yaml`

I use a sentencepiece unigram tokenizer with a vocab size of 1024 generated using the script in NeMo. The batch size is set at 32 on each GPU for a global batch size of 256 (8 A100 GPUs are used). I use automatic mixed precision with amp_level O1. I tried a few different settings for the learning rate and weight decay to adjust for the batch size of 256, which is smaller than the default 2k batch size, and here are the results:

1. Learning rate = 2, weight decay = 1e-6. The training went well and got 2.8/7.6 WER on the dev-clean/dev-other sets at epoch 144. But suddenly got NaN error at epoch 145. I tried resuming the training from checkpoint but got NaN error again at the end of epoch 145.

2. Learning rate = 5, weight decay = 1e-3 (the default setting). The training ran for 25 epoch and got NaN error at epoch 26.

3. Learning rate = 2, weight decay = 1e-4. The training ran for 58 epochs and got NaN error at epoch 59.

Given that the NaN error occurred so late in the training, I doubt that the error was caused by some training example. So, any suggestions to find the root causes would be really appreciated!

Thank you!
Merge r1.7.1 to main
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Patch GPT Sampler
State: closed | Pull request: yes

# What does this PR do ?

GPT sampler has a typo.

**Collection**: NLP

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
GPT multinode eval
State: closed | Pull request: yes

# What does this PR do ?

This PR adds multi-node eval for GPT. Also, it adds the ability to do GPT eval from PTL checkpoint. This is useful when models are very large.

**Collection**: NLP

# Usage
```python
python /code/examples/nlp/language_modeling/megatron_gpt_eval.py \
	--devices ${SLURM_NTASKS_PER_NODE} \
	--num_nodes ${SLURM_JOB_NUM_NODES} \
	--checkpoint_dir "/checkpoint_dir" \
	--checkpoint_name ${CHECKPOINT_NAME} \
	--hparams_file /hparams_dir/${HPARAMS_FILE} \
	--prompt "In critical moments, people sometimes see exactly"  \
	--tensor_model_parallel_size ${TENSOR_MODEL_PARALLEL_SIZE} \
	--pipeline_model_parallel_size ${PIPELINE_MODEL_PARALLEL_SIZE} \
	--tokens_to_generate 32 \
	--stop_after_sentence True \
	--precision ${PRECISION}
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Error selecting speaker while inferencing
State: closed | Pull request: no
Labels: name: bug

Hi all.

I've trained FastPitch using both LJSpeech and my dataset, as it is suggested in the FastPitch_finetuning.ipynb colab. I've balanced the datasets by time instead of by number of samples. It turns out that the voice generated is from the woman of LJSpeech, not from my dataset.

While trying to execute the inference, with the function `str_to_audio()`:

```
def str_to_audio(inp, pace=1.0, durs=None, pitch=None):
    with torch.no_grad():
        tokens = fastpitch.parse(inp)
        spec, _, durs_pred, _, pitch_pred, *_ = fastpitch(text=tokens, durs=durs, pitch=pitch, speaker=None, pace=pace)
        audio = hifigan.convert_spectrogram_to_audio(spec=spec).to('cpu').numpy()
    return spec, audio, durs_pred, pitch_pred
```

I want to change the parameter speaker to 0 or 1, but I get the following error:

```
[NeMo E 2022-03-09 19:12:15 vocabs:324] Torch distributed needs to be initialized before you initialized <nemo.collections.asr.data.vocabs.Phonemes object at 0x7f876a7febd0>. This class is prone to data access race conditions. Now downloading corpora from global rank 0. If other ranks pass this before rank 0, errors might result.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-69-2fd102bd26e0>](https://localhost:8080/#) in <module>()
      5   phraseNumber = i+1
      6   with torch.no_grad():
----> 7       spec_norm, audio_norm, durs_norm_pred, pitch_norm_pred = str_to_audio(p)
      8 
      9       # Let's try to make the speech more solemn

9 frames
[/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py](https://localhost:8080/#) in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   2042         # remove once script supports set_grad_enabled
   2043         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 2044     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   2045 
   2046 

TypeError: embedding(): argument 'indices' (position 2) must be Tensor, not int
```

Thank you in advance.
patch librosa deprecation and fix
State: closed | Pull request: yes

Signed-off-by: fayejf <fayejf07@gmail.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.
Fix the deprecation of librosa get_duration in vad_utils.py and fix the error happened in http://10.110.40.131:8080/blue/organizations/jenkins/NeMo/detail/PR-3771/8/pipeline

**Collection**: [Note which collection this PR will affect]
ASR
# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
http://10.110.40.131:8080/blue/organizations/jenkins/NeMo/detail/PR-3771/8/pipeline
bert_module: fix inputs of export model
State: closed | Pull request: yes

Exported ONNX model had to be passed with "attention_mask" and
"token_type_ids" inputs swapped to get correct output.

Changing the input order fixes this issue.
Also return a dict instead of list for correctly passing inputs.

# What does this PR do ?

Fix swapped model inputs for exported ONNX/TRT model.

**Collection**: NLP

# Changelog 
- Fix order of inputs returned via input_types()

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests? - Updated related tests.
- [x] Did you add or update any necessary documentation? - Not required
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc) - No
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] Bugfix

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?
@MaximumEntropy, @ericharper, @ekmb, @yzhang123

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Problem loading wavfiles using AudioSegment.
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I am getting a bug loading wavfiles using AudioSegment.

File /mnt/disks/uberduck-experiments-v0/NeMo/nemo/collections/asr/parts/preprocessing/segment.py:180, in AudioSegment.from_file(cls, audio_file, target_sr, int_values, offset, duration, trim, orig_sr)
    178     if duration > 0:
    179         seconds = duration * 1000
--> 180         samples = samples[: int(seconds)]
    181     samples = np.array(samples.get_array_of_samples())
    182 except CouldntDecodeError as err:

File /opt/conda/envs/ub_march/lib/python3.8/site-packages/pydub/audio_segment.py:331, in AudioSegment.__getitem__(self, millisecond)
    326     if missing_frames > self.frame_count(ms=2):
    327         raise TooManyMissingFrames(
    328             "You should never be filling in "
    329             "   more than 2 ms with silence here, "
    330             "missing frames: %s" % missing_frames)
--> 331     silence = audioop.mul(data[:self.frame_width],
    332                           self.sample_width, 0)
    333     data += (silence * missing_frames)
    335 return self._spawn(data)

error: Size should be 1, 2, 3 or 4

Here, size refers to "self.sample_width" I believe

**Steps/Code to reproduce bug**

```
from nemo.collections.asr.parts.preprocessing import AudioSegment
file_path=  path_to_wav
AudioSegment.from_file(
file_path,
target_sr=22050,
int_values=True,
offset=0,
duration=1000,
trim=0,
orig_sr=22050,
)
```

**Expected behavior**

I would expect this to load the file.  scipy.io.wavfile.read is able to load file_path no problem.

**Environment overview (please complete the following information)**

GCP
NeMo from source (download and ./reinstall.sh)

**Environment details**

- Debian GNU/Linux 10
- PyTorch '1.10.2+cu102'
- Python 3.8

**Additional context**

T4 GPU
These files load on a Collab notebook running NeMo 1.0.2 and Pydub 25.1. However, when I install NeMo 1.0.1 and Pydub on this GCP machine, I get the same error.

Any help appeciated.  Thanks!
Add readme files describing model execution flow for ASR tasks
State: closed | Pull request: yes

Signed-off-by: smajumdar <titu1994@gmail.com>

# What does this PR do ?

Adds high level architecture / model execution flow diagrams to better enable users to understand how to write model scratch training / fine-tuning steps / inference steps

**Collection**: [ASR]

# Changelog 
- Add flow diagrams for CTC/RNNT model training
- Add flow diagram for ASR inference
- Add flow diagram for Speech classification
- Add flow diagram for Speech SSL

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
date prefered over serial
State: closed | Pull request: yes

Signed-off-by: ekmb <ebakhturina@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Raise error if FP16 training is tried with O2 recipe.
State: closed | Pull request: yes

# What does this PR do ?

Raise an error if fp16 training is tried with O2 recipe. This is not yet supported.

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fixed soft prompt eval loading bug
State: closed | Pull request: yes

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Soft prompt weights were accidently being over written when loading a prompt tuned model for evaluation. This PR fixes this bug. 

**Collection**: BigNLP?

# Changelog 
Fixed evaluation bug and updated prompt tuning config with default hyperparams found to lead to better performance. 

# Usage
Same as before

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Missing dependencies in TTS examples
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
Pyini (and probably some other things) are missing when running in a Colob environment.

**Steps/Code to reproduce bug**
Run FastPitch_Finetuning.ipynb on a Colab instance.


**Expected behavior**
Unless I'm missing something, I should be able to hit run all after I uncomment the first few lines to fetch dependencies, right? 
**Environment overview (please complete the following information)**
Colab
**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version - colab
- PyTorch version 1.10.0+cu111
- Python version  - Python 3.7.12

**Additional context**
I keep hitting this regardless of which notebook I pick.
Multi-GPU / Multi-Node Worker Crash / Deadlock When Returning {"loss": None, "lr": None}
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When running multi-GPU / multi-node training of NLP Entity Linking model against full UMLS dataset, a return of None for loss & lr from training_step results in a crash of a worker and deadlock (at 100% GPU / CPU utilization). It appears this is incompatible with multi-GPU, but entity linking notebooks reference multi-gpu training.

**Steps/Code to reproduce bug**

Training fails at around 75% of first epoch, which is ~18 hours in on 4x A100s, or more than double that on single GPU setups.

Error message is "pytorch_lightning.utilities.exceptions.MisconfigurationException: In automatic_optimization, when `training_step` returns a dict, the 'loss' key needs to be present"

Relevant code in training_step: https://github.com/NVIDIA/NeMo/blob/69385daa999907a15f86e461b670a866424f5bb9/nemo/collections/nlp/models/entity_linking/entity_linking_model.py#L83-L103

Note PyTorch Lightning docs on [training_step](https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html#training-step) return value:
```
None - Training will skip to the next batch. This is only for automatic optimization.
**This is not supported for multi-GPU, TPU, IPU, or DeepSpeed.**
```

Thus this will work on single GPU, but not multi-GPU according to the docs.

**Expected behavior**

Training completes without crashing.

**Environment overview (please complete the following information)**

 - Environment location: Docker 
 - Method of NeMo install: Nemo NGC v.1.6.1
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used
 Pull: docker pull nvcr.io/nvidia/nemo:1.6.1
Run: docker run --gpus all -it -v /opt/nvidia/nemo:/workspace/nemo/local --shm-size=16g --ulimit memlock=-1 --ulimit stack=67108864 -v /dev/infiniband --uts=host --net=host --ipc=host --cap-add IPC_LOCK --device=/dev/snd --device /dev/infiniband/uverbs0 --device /dev/infiniband/rdma_cm --device /dev/infiniband/umad0 nvcr.io/nvidia/nemo:1.6.1 /bin/bash

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Tested on A100 & Titan RTX - same results
Single vs. Multi-node does not appear to make a difference either.
Fix spelling error in "num_workers" parameter to actually set number of dataset workers specified in yaml configs
State: closed | Pull request: yes

Signed-off-by: Michael Menefee <mikem@mnetwork.org>

# What does this PR do ?

Corrects config file parsing to use number of workers specified instead of default "2", which generates warnings and slows training.

**Collection**: NLP

# Changelog 
- Fix spelling error in "num_workers" parameter to actually set number of dataset workers specified in yaml configs

# Usage
* Validating fix - 

```
cd examples/nlp/entity_linking 
vim ./conf/umls_medical_entity_linking_config.yaml
## Change num_workers ~= nproc
python data/umls_dataset_processing.py
python self_alignment_pretraining.py project_dir=. 
## should no longer see num_workers warnings
```

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [N/A] Did you write any new necessary tests?
- [N/A] Did you add or update any necessary documentation?
- [N/A] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Support for Camembert Huggingface bert-like models
State: closed | Pull request: yes

Signed-off-by: Iztok Lebar Bajec <ilb@fri.uni-lj.si>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: nlp

# Changelog 
- Support for loading Huggingface Camembert models, e.g. https://huggingface.co/camembert-base

# Usage
* You can potentially add a usage example below
In `/examples/nlp/token_classification/conf/punctuation_capitalization_config.yaml` set
```python
pretrained_model_name: camembert-base
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Fix: Tacotron2 decoder inference thread-safety bug
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do?

This PR makes Tacotron2 decoder module inference thread-safe and avoids unintentional side-effects that can occur due to concurrent invocation of the function.

**Collection**: TTS

# Changelog 
- Encapsulated the mutable internal decoder states into a dataclass called `DecoderStates`
- Initialize new local `DecoderStates` object at every invocation of Tacotron2 Decoder module inference instead of shared variable states.
- Decode method writes to said local Decoder states instead of overwriting shared variable states.

# Usage
Usage hasn't changed but this PR prevents RuntimeError or prediction quality issues from happening due to crosstalk between threads when making multi-threaded invocations of the Tacotron2 Inference method in eval mode.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests? No
- [x] Did you add or update any necessary documentation? No
- [x] Does the PR affect components that are optional to install? No
- [x] Reviewer: Does the PR have correct import guards for all optional libraries? N/A
  
**PR Type**:
- [x] Bugfix

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* This PR fixes #3797
size mismatch for fastpitch encoder
State: closed | Pull request: no

I am training my model on (Hi-Fi TTS) dataset and when I am calling model. fit, it is giving me the following error.
`Traceback (most recent call last):
  File "fastpitch_finetune.py", line 33, in main
    model.maybe_init_from_pretrained_checkpoint(cfg=cfg)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py", line 50, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/core/classes/modelPT.py", line 998, in maybe_init_from_pretrained_checkpoint
    self.load_state_dict(restored_model.state_dict(), strict=False)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1370, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for FastPitchModel:
	size mismatch for fastpitch.encoder.word_emb.weight: copying a param with shape torch.Size([115, 384]) from checkpoint, the shape in current model is torch.Size([114, 384]).`

I am using Nemo branch 1.7.1 for this
Diarization with ASR JSON output now also includes sentences.
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

ASR

# Changelog 

Diarization with ASR JSON output now also includes sentences.

# Usage

You can use the `examples/speaker_tasks/diarization/offline_diarization.py` script and check the output JSON file (`.../pred_rttms/uniq_name.json`).

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md).
- [x] Did you write any new necessary tests? **No.**
- [x] Did you add or update any necessary documentation? **No.**
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)? **No.**
- [x] Reviewer: Does the PR have correct import guards for all optional libraries? **N/A.**

**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

## Who can review?

@titu1994, @redoctopus, @jbalam-nv, or @okuchaiev

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #3708 (https://github.com/NVIDIA/NeMo/issues/3708)

When implementing I used an approach that changes the original codebase as little as possible. In other words, I tried to be non-invasive. After reviewing the existing code for sentences (`string_out`) preparation in `make_json_output()` of `/nemo/collections/asr/parts/utils/diarization_utils.py` I feel like the whole process could be done much more efficiently and elegantly. Currently, the sentence preparation is appending words to a string where the beginning of the string are the time stamps. When the ending time stamp needs to be corrected the code then uses string splitting to extract and replace/correct the end time value. This is quite clunky and inefficient. It would be much more elegant if time stamps and the sentences were stored and built within a dictionary and then printed out at the end.
se pool step calculation 
State: closed | Pull request: no

https://github.com/NVIDIA/NeMo/blob/256236f684cd08abd27008d6a9c41d77993be138/nemo/collections/asr/parts/submodules/jasper.py#L174

https://github.com/NVIDIA/NeMo/blob/256236f684cd08abd27008d6a9c41d77993be138/nemo/collections/asr/parts/submodules/jasper.py#L506

It seems that no matter what timesteps and context_window are, same formula is applied. 

By the way, suggesting to use `.to` instead of `.type`. [Reference](https://discuss.pytorch.org/t/why-does-pytorch-needs-the-three-functions-to-type-and-type-as/20164/2)
Conformer WARs for TRT8.2
State: closed | Pull request: yes

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

# What does this PR do ?

With those changes, exported Conformer with autocast can be converted to TRT with TRT 8.2 (e.g. current nemo2riva).
Also, rolled back previous change of expand->repeat as expand yields better performance. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)