renwei2024/nemo-github-issues huggingface dataset:
Update megatron interface to dialogue 
State: closed | Pull request: yes

# What does this PR do ?

Update megatron interface to dialogue 

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Speaker Diarization with Marblenet and ClusterDiarizer issue
State: closed | Pull request: no

**Describe the bug**
I am not getting correct time stamps for speech segment and many speech chunks are removed. I am using pretrained Marblenet and speakerdiarization_speakernet models. It removes lots of speech data in between the chunks as below:

['0.0 0.23 speaker_1', '1.55 1.99 speaker_2', '3.18 4.25 speaker_2', '5.77 11.35 speaker_1',

**Steps/Code to reproduce bug**

from nemo.collections.asr.models import ClusteringDiarizer
sd_model = ClusteringDiarizer(cfg=cfg)
sd_model.diarize()

It generates RTTM file with speaker and timestamp which itself is wrong. Tried with multiple CFG parameter changes but no change in the output. 

**Expected behavior**

The chunks should be continous one after another so that no loss of actual speech data.

**Environment overview (please complete the following information)**

 - Environment location: Collab
 - Method of NeMo install: [pip install or from source]. PIP install from git as per the tutorials and git
 - If method of install is [Docker], NO

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version Windows 10
- PyTorch version 1.11
- Python version 3.7

**Additional context**

Add any other context about the problem here.
Example: GPU model
Integrating support for GPT/T5/BART for Question Answering
State: closed | Pull request: yes

Signed-off-by: Ameya Mahabaleshwarkar <ameyasm1154@gmail.com>

# What does this PR do ?

Integrating support for GPT/T5/BART for Question Answering

**Collection**: nlp

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add NSys Profiling Interface to NeMo Core
State: closed | Pull request: yes

# What does this PR do ?

The interface to generate nsys profile results. 

**Collection**: [Core]

# Changelog 
- Enable profile
- Specify range to profile by batch step
- Specify rank to profile
- Specify profiling details

# Usage
Enable knobs to profile

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Weighted bucketing
State: closed | Pull request: yes

# What does this PR do ?
- Adds weighted bucketing to speaker classification.

**Collection**: [Note which collection this PR will affect]
- ASR
- Copied bucketing logic as used in https://github.com/NVIDIA/NeMo/pull/4474 to `audio_to_label_dataset` for Speaker classification

# Before your PR is "Ready for review"
**Pre checks**:
- [Y ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ N] Did you write any new necessary tests?
- [ N] Did you add or update any necessary documentation?
- [ N] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [Y ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Sequence parallel rebase with bugfixes
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Megatron: NameError: name 'ensure_divisibility' is not defined
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

Trying to run `megatron_gpt_pretraining.py`. During the init, the NameError occurs, Traceback below:

```
Traceback (most recent call last):
  File "/NeMo/examples/nlp/language_modeling/megatron_gpt_pretraining.py", line 92, in <module>
    main()
  File "/NeMo/nemo/core/config/hydra_runner.py", line 104, in wrapper
    _run_hydra(
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 377, in _run_hydra
    run_and_report(
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
    raise ex
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 378, in <lambda>
    lambda: hydra.run(
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 111, in run
    _ = ret.return_value
  File "/opt/conda/lib/python3.8/site-packages/hydra/core/utils.py", line 233, in return_value
    raise self._return_value
  File "/opt/conda/lib/python3.8/site-packages/hydra/core/utils.py", line 160, in run_job
    ret.return_value = task_function(task_cfg)
  File "/NeMo/examples/nlp/language_modeling/megatron_gpt_pretraining.py", line 86, in main
    model = MegatronGPTModel(cfg.model, trainer)
  File "/NeMo/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py", line 79, in __init__
    super().__init__(cfg, trainer=trainer, no_lm_init=True)
  File "/NeMo/nemo/collections/nlp/models/language_modeling/megatron_base_model.py", line 73, in __init__
    initialize_model_parallel_for_nemo(
  File "/NeMo/nemo/collections/nlp/modules/common/megatron/megatron_init.py", line 68, in initialize_model_parallel_for_nemo
    ) = fake_initialize_model_parallel(
  File "/NeMo/nemo/collections/nlp/modules/common/megatron/megatron_init.py", line 181, in fake_initialize_model_parallel
    ensure_divisibility(world_size, tensor_model_parallel_size * pipeline_model_parallel_size)
NameError: name 'ensure_divisibility' is not defined
```

**Expected behavior**

Manually importing `ensure_divisibility` from apex.transformer.utils works fine. Running the code should properly import it as well.

**Environment overview (please complete the following information)**

singularity image, based on nvcr.io/nvidia/pytorch:22.06-py3 docker image, NeMo installed with
```
git clone https://github.com/NVIDIA/NeMo
cd NeMo
python -m pip install -r requirements/requirements.txt \
            -r requirements/requirements_common.txt \
            -r requirements/requirements_lightning.txt \
            -r requirements/requirements_nlp.txt \
            -r requirements/requirements_test.txt
./reinstall.sh
```
**Update**
Update: no issue when using 22.05 image AND NeMo release v1.10.0. I did not yet check which of the two changes is responsible for the issue.
Additional sentencepiece args - Byte fallback, split digits, split_on_whitespace
State: closed | Pull request: yes

# What does this PR do ?

Adds some sentencepiece args useful for NLP.

**Collection**: NLP

# Changelog 
- Adds byte_level fallback, split_digits and split_on_whitespace args for spm.

# Usage

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[tutorials][asr] Fixed WER initialization in ASR_with_Nemo notebook
State: closed | Pull request: yes

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[NLP] punctuation_capitalization resume training fails
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When training PC from scratch, using the example configs, everything works fine, however if I try to resume an interrupted training from the last checkpoint, I get the following log entry
```
[NeMo W 2022-07-07 21:18:38 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:378: UserWarning: `ModelCheckpoint(monitor='val_loss')` could not find the monitored key in the returned metrics: ['epoch', 'step']. HINT: Did you call `log('val_loss', value)` in the `LightningModule`?
      warning_cache.warn(m)
```
and all new checkpoints get a `val_loss=0.0000`

**Environment details**

- pytorch:22.04-py3
- nemo:1.8.2 installed with ./reinstall.sh, without numba install
[TTS] [bugfix] German tutorial directory correction
State: closed | Pull request: yes

Signed-off-by: Akshit Arora <akshit.arora@colorado.edu>

# What does this PR do ?

Corrected the directory created to download validation_ds, required for HiFi-GAN fine-tuning.

**Collection**: TTS

# Changelog 
From 
```
!cd NeMoGermanTTS && mkdir -p model/train_ds && cd model/validation_ds && wget https://raw.githubusercontent.com/nvidia/NeMo/$BRANCH/examples/tts/conf/hifigan/model/validation_ds/val_ds_finetune.yaml
```
to:
```
!cd NeMoGermanTTS && mkdir -p model/validation_ds && cd model/validation_ds && wget https://raw.githubusercontent.com/nvidia/NeMo/$BRANCH/examples/tts/conf/hifigan/model/validation_ds/val_ds_finetune.yaml
```

**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
r1.10.0 untie embeddings weights
State: closed | Pull request: yes

# What does this PR do ?

This PR adds the config option to prevent weights tying between encoder embeddings, decoder embeddings, and decoder tokens head.
This PR does not support PP, only TP.
## Added configs

```
model:
  share_word_embeddings: True # If True share encoder/decoder embeddings
  share_decoder_tokens_head_embeddings: True # If True share decoder embeddings and decoder projection to logits
```

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] bug fix - sample rate was being ignored in vocoder dataset
State: closed | Pull request: yes

Signed-off-by: Paarth Neekhara <paarth.n@gmail.com>

# What does this PR do ?

Fixes a bug where sample rate was being ignored in VocoderDataset when loading just the audio and audio lengths and not the precomputed mel.

**Collection**: tts

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
dataset = VocoderDataset(manifest_filepath=manifest_path, sample_rate=16000)
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix duplex inference with grammars
State: closed | Pull request: yes

Signed-off-by: ekmb <ebakhturina@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add ITN pt
State: closed | Pull request: yes

# What does this PR do ?

Adds ITN rules for Portuguese.
In addiction, tests for these rules were added.

**Collection**: [Text Processing]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
problem of load ASR lstm trained model
State: closed | Pull request: no
Labels: name: bug

name: stale

hello. I have train a "LSTM-Transducer-BPE" model some days ago. and to save the model, I have used checkpoint. for this train, I have used the default config.
after that, when i want to load this model by `EncDecRNNTBPEModel` method,

i got an error and i could not use this model:
```
RuntimeError: Error(s) in loading state_dict for EncDecRNNTBPEModel:
	Missing key(s) in state_dict: "decoder.prediction.dec_rnn.lstm.weight_hr_l0", "decoder.prediction.dec_rnn.lstm.weight_hr_l1". 
	size mismatch for decoder.prediction.dec_rnn.lstm.weight_ih_l0: copying a param with shape torch.Size([2560, 640]) from checkpoint, the shape in current model is torch.Size([8192, 640]).
	size mismatch for decoder.prediction.dec_rnn.lstm.weight_hh_l0: copying a param with shape torch.Size([2560, 640]) from checkpoint, the shape in current model is torch.Size([8192, 640]).
	size mismatch for decoder.prediction.dec_rnn.lstm.bias_ih_l0: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([8192]).
	size mismatch for decoder.prediction.dec_rnn.lstm.bias_hh_l0: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([8192]).
	size mismatch for decoder.prediction.dec_rnn.lstm.weight_ih_l1: copying a param with shape torch.Size([2560, 640]) from checkpoint, the shape in current model is torch.Size([8192, 640]).
	size mismatch for decoder.prediction.dec_rnn.lstm.weight_hh_l1: copying a param with shape torch.Size([2560, 640]) from checkpoint, the shape in current model is torch.Size([8192, 640]).
	size mismatch for decoder.prediction.dec_rnn.lstm.bias_ih_l1: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([8192]).
	size mismatch for decoder.prediction.dec_rnn.lstm.bias_hh_l1: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([8192]).
```
please help me.
thank you.
OSError: libtorch_hip.so: cannot open shared object file in Speaker_Diarization_Inference
State: closed | Pull request: no
Labels: name: bug


Followed the steps here : https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Diarization_Inference.ipynb
to run in jupyter notebook and just ran the whole notebook as it is. But got the error `OSError: libtorch_hip.so: cannot open shared object file: No such file or directory` when importing `nemo.collections.asr.models`


**Environment details**
![image](https://user-images.githubusercontent.com/66141447/177719440-0e6a7cbe-4266-4f69-ae8c-01ebb527b50d.png)

- OS version Ubuntu 20.04
- PyTorch version 1.11.0
- Python version 3.9.12
Add silence handling for speaker diarization pipeline
State: closed | Pull request: yes

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

# What does this PR do ?

Provides fix for handling silence wav files for diarization

**Collection**: [ASR - Speaker Diarization]

# Changelog 
- Added `validate_vad_manifest` function to compare the uniq_ids in vad manifest file to manifest file provided by user for diarization
- Fixed an issue to check for a VAD `.frame` file, even if the duration of that audio is <=0. 

# Usage

```bash
python offline_diarization.py
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
[ASR][Breaking Change] Update signature of Hypothesis alignments
State: closed | Pull request: yes

# What does this PR do ?

Updates signature of Hypothis.alignments such that it now contains a tuple (logprob, label), such that CTC, RNNT greedy and RNNT beam search all yield the logprob tensor of shape (V+1,) where V is the vocab size (and last index is CTC/RNNT Blank). This can be used to compute beam search results.

For CTC Models, hypothesis.alignments now returns a tuple of tensors (logp, labels) rather than just logp to match the RNNT signature.

**Collection**: [ASR]

# Changelog 
- [BREAKING CHANGE] Force `preserve_alignments` flag to preserve a tuple of values (logp, label) for both CTC and RNNT.
- This enables external beam search rescoring to work with CTC and RNNT models.
- Also enables the possibility of RNNT based char/word time stamp calculation
- In totality, enables CTC/RNNT models to compute word confidence.
- Add respective tests

# Usage

```python

model = EncDecRNNTModelBPE.from_pretrained(...)
model.eval()

decoding_cfg. model.cfg.decoding
decoding_cfg.preserve_alignments = True
model.change_decoding_strategy(decoding_cfg)

# For RNNT
hyps, _ = model.transcribe(..., return_hypothesis=True)
# For CTC
hyps = model.transcribe(..., return_hypothesis=True)

# select hypothesis of 1st sample
hyp = hyps[0]

# extract alignments
for t in range(len(hyp.alignments)):
  for u in range(len(hyp.alignments[t])):
    logprob, label = hyp[t][u]
    ....

```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [x] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.
[TTS] Update cmudict
State: closed | Pull request: yes

phoneme IY1 -> IH1 in NVIDIA
Added phonemes for CUSTOMIZABLE

Update the cmudict version and its reference.

# What does this PR do ?

Update phoneme typo and a new phoneme word

**Collection**: [Text to speech]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
update fastpitch to add export controls
State: closed | Pull request: yes

# What does this PR do ?

Adds options for fastpitch export. One for ragged_batch and one for volume. Undos the hard assumption that `volume` must be input and output as done in #4167

**Collection**: TTS

# Changelog 
- Adds ragged_batch and volume options for fastpitch export

**PR Type**:
- [x] New Feature
[Add] Support for Different LRs with Param Groups
State: closed | Pull request: yes

This PR aims to add support for different learning rates for different modules in a model. For example, when finetuning an ASR model on small datasets, we might want the encoder to have smaller learning rate than the decoder. With this PR, we can do this by adding a `optim_param_groups` section under `model` in the config file:
```yaml
model:
    optim_param_groups:
        "encoder": 
            "lr": 1e-4
            "momentum": 0.8
    optim:
        "name": sgd
        "lr": 1e-3   # default lr for non-specified parameters
```
In this example, all parameters in the encoder will have learning rate `1e-4` while all other parameters (e.g., decoder) will have learning rate `1e-3`. 

For logging learning rates, the current models only log the first group of `param_groups`, which in this case is the default `lr`.  To log all learning rates, we need to modify the `training_step()` function in the model.
wav2vec CTC val_wer is always 1
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Hi, I am having issues with training wav2vec models. It seems like the model starts training, there are no warnings or errors, everything runs. However, after every epoch val_wer is equal to 1.0. For [wav2vec_pretrain.yaml](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/ssl/wav2vec/wav2vec_pretrain.yaml) I am not sure where to start as there are no tutorials or examples. So I have been using [wav2vecCTC.yaml](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/wav2vec_ctc/wav2vecCTC.yaml) and the example script for training CTC models.

**Steps/Code to reproduce bug**

Below is the exact used command, used datasets work perfectly with other NeMo ASR models. It is the same command that I use for all CTC models.

```
python /workspace/nemo/examples/asr/asr_ctc/speech_to_text_ctc_bpe.py \\
  --config-path=/workspace/nemo/examples/asr/conf/wav2vec_ctc \\
  --config-name=wav2vecCTC \\
  exp_manager.name=wav2vec} \\
  +exp_manager.exp_dir=/experiments \\
  exp_manager.resume_if_exists=true \\
  exp_manager.resume_ignore_no_checkpoint=true \\
  +exp_manager.checkpoint_callback_params.save_best_model=true \\
  model.train_ds.manifest_filepath=/data/train.nemo \\
  model.train_ds.batch_size=8 \\
  model.train_ds.num_workers=4 \\
  model.train_ds.max_duration=17 \\
  +model.train_ds.min_duration=1 \\
  +model.train_ds.parser=base \\
  model.validation_ds.manifest_filepath=/data/validation.nemo \\
  model.validation_ds.batch_size=8 \\
  model.validation_ds.num_workers=4 \\
  +model.validation_ds.parser=base \\
  model.tokenizer.dir=/data/out/tokenizer_spe_unigram_v128 \\
  model.tokenizer.type=bpe \\
  trainer.precision=bf16 \\
  trainer.devices=-1
```

**Expected behavior**

I expect val_wer to drop with training and not remain at its maximum value (1.0).

**Environment overview (please complete the following information)**

 - Official latest NeMo container where the NeMo repository is updated to the current tip of the master branch.

**Environment details**

 - Official latest NeMo container where the NeMo repository is updated to the current tip of the master branch.

**Additional context**

Running on 4 RTX A6000 GPUs.
Update finetune label models
State: closed | Pull request: yes

# What does this PR do ?

Upgraded speaker recognition fine-tune script

**Collection**: [ASR - Speaker Recognition]

# Changelog 
- Removed set_up_finetune_model definition in label_models
- Updated speaker_reco_finetune.py script, to resemble PTL training method
- Now, fine-tune can be directly used from config. Added titanet-finetune.yaml config
- Updated notebook tutorials for the fine-tuning section

# Usage
* You can potentially add a usage example below

```bash
python speaker_reco_finetune.py 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation
Conditionally skip input loading at pipeline-parallel training
State: closed | Pull request: yes

# What does this PR do ?

At pipeline-parallel training, set the pipeline stage that the current GPU belongs to skip loading inputs
Intermediate stage: do not load any inputs
Fist pipeline stage: load only 'tokens' and 'position_ids'
Last pipeline stage: load only 'labels' and 'loss_mask'

The gain of this PR increases when used with interleaved pipelining because the baseline code loads inputs at each virtual pipe stage.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
This is enabled when pipeline-model size is larger than 1

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix runtime check
State: closed | Pull request: yes

# What does this PR do ?
Post-export runtime check refinements - allow checking multiple input sizes.
Restored max_dim argument functionality in export.py
Also, added a cast to ensure ASR nets are always exported with output in FP32 - even with autocast.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
NMESC speaker counting algorithm update
State: closed | Pull request: yes

# What does this PR do ?

`nmesc_clustering.py`

This PR adds an option named "maj_vote_spk_count" to NMESC clustering module.
Originally, speaker number was obtained from one p-value.
In the new implementation, maj_vote_spk_count = True, the system takes majority vote on
all p-values that are included.
This feature almost halves the speaker counting error for 1.5sec window setting.
(CH109, 1.5/0.75 win/shift, speaker counting error becomes 12.8% -> 7.3%)
Default will be False for now. Yaml file will be later updated with other updates to provide the best performance.
`diarizer.clustering.parameters.maj_vote_spk_count=False`

This PR also adds error handling for extreamly short segments (1~10 segments)
when creating p_value_list, the minimum length is now forcibly assigned so that we can avoid having an empty list.

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Changes to make Megatron NMT exportable
State: closed | Pull request: yes

# What does this PR do ?

Adding requirements to make Megatron NMT exportable

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Added input/output types for encoder/decoder/tokens_head
- Added input_example

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [] New Feature
- [X] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Cannot run Fastpitch on CPU
State: closed | Pull request: no
Labels: name: bug

Hi, I ran the code at tutorials/tts/FastPitch_MixerTTS_Training.ipynb and received the error as following when run python fastpitch.py
![image](https://user-images.githubusercontent.com/73210249/177138773-df77f68c-c910-445e-8ba9-f0bbf08358a1.png)

I change config file to run on CPU 
![image](https://user-images.githubusercontent.com/73210249/177235371-3ecf1e17-2963-4e20-b95e-060ad6d60dfb.png)

- OS version: Ubuntu 20.04
- PyTorch version: 

![image](https://user-images.githubusercontent.com/73210249/177235550-414ce314-c00e-4acc-9f1c-3169fbfe5eda.png)

- Python version: 3.8

Is there anything need to be changed to be able to train on CPU?
Wav2vec deviations from fairseq implementation
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I recently loaded the weights of a pre-trained "[XLS-R](https://github.com/facebookresearch/fairseq/tree/main/examples/wav2vec/xlsr)" wav2vec SSL model into the NeMo implementation and found a couple subtle deviations from the fairseq implementation that led to a different computational graph with the same state dict.

E.g., in the fairseq feature extractor, layer norm is applied [before](https://github.com/facebookresearch/fairseq/blob/5307a0e078d7460003a86f4e2246d459d4706a1d/fairseq/models/wav2vec/wav2vec2.py#L604) projecting the embeddings to the encoder dim whereas in the NeMo implementation the layer norm is applied [after](https://github.com/NVIDIA/NeMo/blob/8a172df69250da231742893387e9a1dd01b16e3d/nemo/collections/asr/modules/wav2vec_modules.py#L196).

After a couple changes I was able to run the XLS-R fairseq model in NeMo and get the same results from the feature extractor and encoder graphs.

**Are these deviations intentional?** It seems like most of the code is just copy-pasted except for these changes.

If these are intentional deviations, I will simply keep my patch local and close the issue.

If these deviations are accidental or if it would be useful to have an option to rearrange the graph to match the fairseq implementation, I will submit my changes.
GPT Prompt Learning Improvements
State: closed | Pull request: yes

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Makes several improvements to the gpt prompt learning code including:
- faster training time using less memory 
- switch `global_batch_size` to refer to actual global batch size and not local mini batch size

**Collection**: BigNLP

# Changelog 
- Update GPT prompt learning model class

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Megatron BART BOS / EOS bug fix
State: closed | Pull request: yes

Fixes Megatron BART dataset missing <EOS> for decoder output.
`BARTDataset` output description:
1. Encoder input: tokens (no `<BOS>` or `<EOS>` added)
2. Decoder input: [`<BOS>`] + tokens
3. Decoder output: tokens + [`<EOS>`]

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Gradient Accumulation fusion to Linear layer weight gradient computation
State: closed | Pull request: yes

# What does this PR do ?

Fuse gradient accumulation ops to the weight gradient computes of linear layers.
This feature should be used with (1) serialized gradient allreduce at this update and (2) O2.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
Add below knob to the commanline.

```python
training.model.gradient_accumulation_fusion=True
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [] Did you write any new necessary tests?
- [] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Traing to fine tune pre-trained model but getting "...shape torch.Size([69, 512])..."
State: closed | Pull request: no
Labels: name: bug

I'm traing to fine tune pre-trained model, and I'm getting this error:

`RuntimeError: Error(s) in loading state_dict for Tacotron2Model:
	size mismatch for text_embedding.weight: copying a param with shape torch.Size([69, 512]) from checkpoint, the shape in current model is torch.Size([83, 512]).`

I know that is been caused by updating the charters(labels) dictionary, is there any way to compute new dict size?

Thnx in advance.
Fix for GeGLU activations with not using fusion
State: closed | Pull request: yes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

# What does this PR do ?

Fixes GeGLU when not using fusion.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Updates if/else chain when deciding how to use activation functions in transformer.py

# Usage
N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)