renwei2024/nemo-github-issues huggingface dataset:
[Tutorials] Fix matplotlib version and directory name in Multispeaker_Simulator
State: closed | Pull request: yes

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

# What does this PR do ?

Fix a couple minor issues in `Multispeaker_Simulator` notebook

**Collection**: Tutorials

# Changelog 

- added double quotes around the version specification to make sure the expected `matplotlib` is installed
- `ls` the correct directory with created files

# Usage
* n/a

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* n/a
Added P&C lexical audio model
State: closed | Pull request: yes

# What does this PR do ?

Add audio and lexical model to NeMo and necessary support.

**Collection**:  nlp
# Changelog 
- Added `PunctuationCapitalizationLexicalAudioModel` which is inherited from `PunctuationCapitalizationModel`.
- Extended `BertPunctuationCapitalizationDataset` and `BertPunctuationCapitalizationTarredDataset` to support audio as well.
- Added default configs stored in `examples/nlp/token_classification/conf/punctuation_capitalization_lexical_audio_config.yaml`.
- Added script to run train and evaluate at `examples/nlp/token_classification/punctuation_capitalization_lexical_audio_train_evaluate.py`.

# Usage
```
python punctuation_capitalization_lexical_audio_train_evaluate.py \
        model.train_ds.ds_item=<PATH/TO/TRAIN/DATA> \
        model.train_ds.text_file=<NAME_OF_TRAIN_INPUT_TEXT_FILE> \
        model.train_ds.labels_file=<NAME_OF_TRAIN_LABELS_FILE> \
        model.validation_ds.ds_item=<PATH/TO/DEV/DATA> \
        model.validation_ds.text_file=<NAME_OF_DEV_INPUT_TEXT_FILE> \
        model.validation_ds.labels_file=<NAME_OF_DEV_LABELS_FILE
```

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [X] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [X] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Audio and lexical P&C model
State: closed | Pull request: yes

# What does this PR do ?

Add audio and lexical model to NeMo and necessary support.

**Collection**:  nlp
# Changelog 
- Added `PunctuationCapitalizationLexicalAudioModel` which is inherited from `PunctuationCapitalizationModel`.
- Extended `BertPunctuationCapitalizationDataset` and `BertPunctuationCapitalizationTarredDataset` to support audio as well.
- Added default configs stored in `examples/nlp/token_classification/conf/punctuation_capitalization_lexical_audio_config.yaml`.
- Added script to run train and evaluate at `examples/nlp/token_classification/punctuation_capitalization_lexical_audio_train_evaluate.py`.

# Usage
```
python punctuation_capitalization_lexical_audio_train_evaluate.py \
        model.train_ds.ds_item=<PATH/TO/TRAIN/DATA> \
        model.train_ds.text_file=<NAME_OF_TRAIN_INPUT_TEXT_FILE> \
        model.train_ds.labels_file=<NAME_OF_TRAIN_LABELS_FILE> \
        model.validation_ds.ds_item=<PATH/TO/DEV/DATA> \
        model.validation_ds.text_file=<NAME_OF_DEV_INPUT_TEXT_FILE> \
        model.validation_ds.labels_file=<NAME_OF_DEV_LABELS_FILE
```

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [X] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [X] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add support for Apex distributed Adam optimizer with Megatron-LM models
State: closed | Pull request: yes

# What does this PR do ?

Generalize dist Adam support for GPT-3 to T5 and other Megatron-LM models.

**Collection**: NLP

# Changelog 
- When params are BF16, distributed Adam will store 16-bit param remainders instead of FP32 main params
- Decouples distributed Adam support from Megatron O2-level optizations
- Add support for Apex distributed Adam optimizer with Megatron-LM models

# Usage

Set optimizer to `distributed_fused_adam` in config file:

https://github.com/NVIDIA/NeMo/blob/265f7b17876aef7937cf0fedf7a76885bdc63288/examples/nlp/language_modeling/conf/megatron_t5_config.yaml#L137

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

Pinging @ericharper and @yaoyu-33.

# Additional Information
* Option for distributed Adam to store 16-bit remainders with BF16 params was added in https://github.com/NVIDIA/apex/pull/1463
* Closes https://github.com/NVIDIA/NeMo/pull/4796
Add ability to give seperate datasets for test, train and validation
State: closed | Pull request: yes

# What does this PR do ?
Gives the user the ability to specify separate train,test and validation datasets as a dictionary in data_prefix for gpt model

**Collection**: nlp/language_modelling

# Changelog 
- Supports dictionary format in data_prefix for megatron gpt config file.
- Uses the existing data loader to load each individual dataset appropriately by changing split string

# Usage
* data_prefix: {train:/path/to/train, test:/path/to/test, validation:/path/to/validation}

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Add ability to give seperate train test and validation splits
State: closed | Pull request: yes

# What does this PR do ?
Gives the user the ability to specify separate train,test and validation datasets as a dictionary in data_prefix for gpt model

**Collection**: nlp/language_modelling

# Changelog 
- Supports dictionary format in data_prefix for megatron gpt config file.
- Uses the existing data loader to load each individual dataset appropriately by changing split string

# Usage
* data_prefix: {train:/path/to/train, test:/path/to/test, validation:/path/to/validation}

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Avoid storing extra copy of params in distributed Adam optimizer
State: closed | Pull request: yes

# What does this PR do ?

https://github.com/NVIDIA/apex/pull/1463 adds an option to the distributed Adam optimizer that reduces memory usage for BF16 models. Instead of storing FP32 main params in the optimizer, we can just store 16-bit param remainders and reconstruct the FP32 main params from the BF16 model params.

**Collection**: NLP

# Changelog 
- Store 16-bit param remainders in dist Adam instead of full FP32 main params
- Update Dockerfile to build Apex with dist Adam support

# Usage
Set optimizer to `distributed_fused_adam` in config file:
https://github.com/NVIDIA/NeMo/blob/c0bfa6f07f766a3abd1804f5b666474887e0a1e4/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml#L150

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

Pinging @ericharper and @yaoyu-33.

# Additional Information
* Depends on https://github.com/NVIDIA/apex/pull/1463
Colab py37 compatibility megatron
State: closed | Pull request: yes

# What does this PR do ?

Update Megatron LM Encoder Decoder to maintain compatibility with python 3.7 to enable notebooks to run on Colab

**Collection**: nlp

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Set the number of workers to 0 for validation and test sets in all enc-dec models
State: closed | Pull request: yes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

# What does this PR do ?

Sets the number of data workers to 0 for validation and test sets for all encoder-decoder models to sidestep a strange CUDA error.

NOTE: This is not a permanent solution and just a temporary fix.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Implement Phonemizer/eSpeak based TTS tokenizer
State: closed | Pull request: yes

Signed-off-by: Ryan <rlangman@nvidia.com>

# What does this PR do ?

Introduce a experimental implementation of a generic IPA tokenizer using phonemizer/eSpeak for text normalization & G2P. 

You can still optionally use NeMo text normalizer as well by providing it in the config or putting the text through it prior to calling this G2P class, similar to existing tokenizers.

**Collection**: [TTS]

# Changelog 
- Added PhonemizerTokenizer class implementation.
- Added phonemizer as a TTS dependency (though eSpeak backend cannot be installed with pip or conda).
- Added an empty pronounce dictionary which we could extend with additional language-agnostic vocabulary, or could replace with language specific pronunciations of certain words.
- Added unit tests
- Install eSpeak on docker & Jenkins

# Usage

```python
import os
from nemo.collections.tts.torch.tts_tokenizers import PhonemizerTokenizer

nemo_root_dir = "..."

ipa_dict_path = os.path.join(nemo_root_dir , "scripts", "tts_dataset_files", "ipa_dict-default.txt")

tokenizer = PhonemizerTokenizer(
    language="es",
    phoneme_dict=ipa_dict_path,
    pad_with_space=True
)

text = "la ciudad es bellísima y tan alegre"
tokens = tokenizer(text)
phonemes = tokenizer.text_to_phonemes(text)

# phonemes -> " lˈa θjuðˈad ˈes beʎˈisima i tˈan alˈeɣɾe "
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

# Additional Information

Here are examples of the output for a few Spanish sentences from openSLR. Please review and let me know how consistent or inconsistent the character-set/rules/semantics are with the other IPA systems currently being developed.

```
Input:  alex, ¿podrás buscar algún restaurante para hoy en la noche?
Output: alˈeks, ¿poðɾˈas buskˈaɾ alɣˈun ɾɾˌestaʊɾˈante pˈaɾa ˈoɪ ˈen lˈa nˈotʃe?

Input:  es muy importante el apoyo nasal en la pronunciación
Output: ˈes mˈujj ˌimpoɾtˈante ˈel apˈojjo nasˈal ˈen lˈa pɾˌonunθjaθjˈon

Input:  la ciudad es bellísima y tan alegre
Output: lˈa θjuðˈad ˈes beʎˈisima i tˈan alˈeɣɾe
```
Fix qa notebook typos and branch
State: closed | Pull request: yes

Signed-off-by: ericharper <complex451@gmail.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Merge r1.11.0 main
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix logger reference
State: closed | Pull request: yes
Labels: name: bug

# What does this PR do ?

Closes #4772 

**Collection**: NLP

# Changelog 
- Fix logger reference in `text_memmap_dataset.py`

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Code switching
State: closed | Pull request: yes

# What does this PR do ?

* Adds scripts for synthetic code-switched data generation for ASR
* Adds the En-Es code-switched ASR models (stt_enes_conformer_transducer_large_codesw and stt_enes_conformer_ctc_large_codesw)

**Collection**:  ASR

# Changelog 
- Lines 537 - 543 in nemo/collections/asr/models/ctc_bpe_models.py
- Lines 200 - 206 in nemo/collections/asr/models/rnnt_bpe_models.py
- File code_switching_audio_data_creation.py in scripts/speech_recognition/code_switching/
- File code_switching_manifest_creation.py in scripts/speech_recognition/code_switching/
- File README.md in scripts/speech_recognition/code_switching/

# Usage
1. Create the (intermediate) manifest file using `code_switching_manifest_creation.py`.  It's usage is as follows:

    `python code_switching_manifest_creation.py --manifest_language1 <absolute path of Language 1's manifest file> --manifest_language2 <absolute path of Language 2's manifest file> --manifest_save_path <absolute path to save the created manifest> --id_language1 <language code for language 1 (e.g. en)> --id_language2 <language code for language 2 (e.g. es)> --max_sample_duration_sec <maximum duration of generated sample in seconds> --min_sample_duration_sec <maximum duration of generated sample in seconds> --dataset_size_required_hrs <size of generated synthetic dataset required in hrs>`

2. Create the syntheic audio data and the corresponding manifest file using `code_switching_audio_data_creation.py` It's usage is as follows: 

    `python code_switching_audio_data_creation.py --manifest_path <absolute path to intermediate CS manifest generated in step 1> --audio_save_folder_path <absolute path to directory where you want to save the snythesized audios> --manifest_save_path <absolute path to save the created manifest> --audio_normalized_amplitude <sclaed normalized amplitude desired> --sample_beginning_pause_msec <pause to be added to the beginning of the generated sample in milli seconds> --sample_joining_pause_msec <pause to be added between segments while joining, in milli seconds> --sample_end_pause_msec <pause to be added to the end of the generated sample in milli seconds> `


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
amend rnnt word timestamps
State: closed | Pull request: yes

# What does this PR do ?

This PR amends the start time for the last token in RNNT decoding because the code aggregates (inclusively) since the previous_token_index all tokens in the current word. This has tripped me up in my own implementations of similar code external to NeMo in the past. My apologies for not catching it in #4779. Without these changes, some transcription invocations fail.

**Collection**: ASR

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] Bugfix

@titu1994 I tested this code against an ASR corpus of 163,708 words, and it's looking stable with no more errors thrown. My tests were in WER rather than time stamps. There's a jitter of about 0.1%pt WER on clean English and 0.5%pts on noisy English, which is similar to what I've seen before on repeated invocations of custom models, though perhaps a bit larger for noisy audio. I don't know where to enumerate more test cases in this codebase, but this logic is a bit difficult to exhaustively test. I've seen 200-300 millisecond drifts in word timestamps across different models, so I don't know how to formalize a test for these outputs more specifically.
[TTS] Fix text normalizer bugs in TTS data loader
State: closed | Pull request: yes

Signed-off-by: Ryan <rlangman@nvidia.com>

# What does this PR do ?

Fixes two bugs recently introduced into the TTS data loader initialization.

1. When pynini is not installed, then Normalizer = None. This causes `isinstance(self.text_normalizer, Normalizer)` to throw an exception.
2. The statement `("text_normalized" not in item) or ("normalized_text" not in item)` should be an `and `rather than an `or`. But rather than make that specific change, I rewrote the condition checks to be less confusing/error prone.

**Collection**:  [TTS]

# Changelog 
- Avoid TypeError when user does not use a text normalizer and pynini is not installed.
- Avoid applying text normalization when "text_normalized" is provided as a field

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
[TTS] deprecated old scripts for ljspeech.
State: closed | Pull request: yes

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix handling of the final word for rnnt word timestamps
State: closed | Pull request: yes

# What does this PR do ?

Fixes another observed edge case for the final word in RNNT word timestamps

**Collection**: ASR

# Changelog 
- the last word in rnnt decoding for a segment may contain multiple tokens, but the data model for how these were aggregated 

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

# Additional Information
* Related to # (issue)

Example object from real use case
```python
from torch import tensor
offsets = [{'char': [tensor(456, dtype=torch.int32), tensor(1024, dtype=torch.int32)],
  'start_offset': 0,
  'end_offset': 1},
 {'char': [tensor(28, dtype=torch.int32),
   tensor(291, dtype=torch.int32),
   tensor(1024, dtype=torch.int32)],
  'start_offset': 2,
  'end_offset': 3}]
built_token = [456, 28, 291]
```

If you attempt to index offsets via `offsets[-len(built_token)]]` you get an error because `len(offsets)==2` and `len(built_token)==3`
add conditional logic to rnnt_wer to handle when arrays have no elements
State: closed | Pull request: yes

# What does this PR do ?

Adds conditional logic to rnnt_wer to check if arrays exist and have elements before accessing them. This error occasionally blocks transcription.


**Collection**:  ASR

# Changelog 

Bug fix for transcription with rnnt sentencepiece tokens


# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)

~- [ ] Did you write any new necessary tests?~ Unable to share specific test file that found this error
- [X] Did you add or update any necessary documentation?

~- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)~

  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation
Broken link to notebook in documentation
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
Hi,

I have just found a broken link to a notebook in this page: https://docs.nvidia.com/deeplearning/nemo/best-practices/index.html

In the "Table 1. Tutorials", the "Question answering with SQuAD" notebook link is broken.

Instead of:

https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/nlp/Question_Answering_Squad.ipynb

it should be:

https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/nlp/Question_Answering.ipynb

Hope it helps!
Problems initializing StepLR
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I am using a pre-trained Conformer Transducer and want to initialize an optimizer and an LR scheduler using its method `setup_optimization`, which returns the optimizer and the LR scheduler.

The problem I am having is that the ASR model's method `setup_optimization` method requires `max_steps` to be part of the scheduler config. Otherwise, it doesn't initialize the scheduler. But given that  `StepLR` does not accept `max_steps` as an argument, it raises an error.

**Steps/Code to reproduce bug**

```python
from nemo.collections.asr.models.rnnt_bpe_models import EncDecRNNTBPEModel

config = {
    "name": "adamw",
    "lr": 0.1,
    "betas": [0.9, 0.98],
    "weight_decay": 0.001,
    "sched": {"name": "StepLR", "step_size": 1, "last_epoch": 1},
}

model = EncDecRNNTBPEModel.from_pretrained("stt_de_conformer_transducer_large")
model.setup_optimization(config)
```

Running this script outputs the following log:

```
[NeMo W 2022-08-19 10:30:46 lr_scheduler:816] Neither `max_steps` nor `iters_per_batch` were provided to `optim.sched`, cannot compute effective `max_steps` !
    Scheduler will not be instantiated !
```

When adding `max_steps` with an integer value (e.g. `"max_steps". 1`) to `config["sched"]`, the following exception occurs:

```
Exception has occurred: TypeError
__init__() got an unexpected keyword argument 'max_steps'
  File "/home/carlos/repos/accsr/lr_issue.py", line 12, in <module>
    model.setup_optimization(config)
```

**Expected behavior**

StepLR does not require a `max_steps` argument and should be initialized without it.

**Environment overview (please complete the following information)**

 - Environment location: Local machine, conda environment
 - Method of NeMo install: pip install (version 1.10.0)
 
**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 20.04
- PyTorch version: 1.12.0
- Python version: 3.8.0
Online Multispeaker Audio Simulation
State: closed | Pull request: yes

# What does this PR do ?

Creates online data simulation in NeMo dataloader for training diarization models

**Collection**: [Note which collection this PR will affect]

ASR

# Changelog 
- Add SyntheticDataloader class
- Add AudioToSpeechMSDDSyntheticTrainDataset class
- Modified msdd_models.py for online simulated data generation

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Use of undeclared `logger`.
State: closed | Pull request: no

`logger` is used here but not defined

https://github.com/NVIDIA/NeMo/blob/6abfbbfda654f44313068b950edb0f70b01449b1/nemo/collections/nlp/data/language_modeling/text_memmap_dataset.py#L157

I believe this should be switched to `logging`, which is used throughout the rest of the module.
T5 prompt learning fixes
State: closed | Pull request: yes

# What does this PR do ?

Fixes the following things for T5 prompt-learning:

1. T5 models with RPE.
2. Models trained with asymmetric enc-dec configurations where hidden size is present in cfg.encoder.hidden_size.
3. Aligns config variables names with the GPT prompt learning config.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Add single male speaker German FastPitch and HiFiGAN NGC checkpoints
State: closed | Pull request: yes

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix logging in gradient clipping with PTL 1.7.2
State: closed | Pull request: yes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

# What does this PR do ?

Fixes the use of `self.log()` inside `configure_gradient_clipping` in `megatron_base_model.py`.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update CMUdict with more recent 0.7b entries
State: closed | Pull request: yes

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

# What does this PR do ?

Merges the existing NeMo CMUdict with a version of 0.7b with added entries.

**Collection**: TTS

# Changelog 
- Added entries from https://github.com/Alexir/CMUdict/blob/master/cmudict-0.7b
- I've omitted entries with British spellings
Cannot train colab with Mixer-TTS-X
State: closed | Pull request: no
Labels: name: bug

Hi, I'm getting an error trying to execute the colab example: https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_MixerTTS_Training.ipynb with Mixer-TTS-X.

Execution cell:

```
assert isinstance(spec_gen, SpectrogramGenerator)

if isinstance(spec_gen, FastPitchModel):
    tokens = spec_gen.parse(str_input="Hey, this produces speech!")
else:
    tokens = spec_gen.parse(text="Hey, this produces speech!")

spectrogram = spec_gen.generate_spectrogram(tokens=tokens)

# Now we can visualize the generated spectrogram
# If we want to generate speech, we have to use a vocoder in conjunction to a spectrogram generator.
# Refer to the Inference_ModelSelect notebook on how to convert spectrograms to speech.
imshow(spectrogram.cpu().detach().numpy()[0,...], origin="lower")
plt.show()
```
The same error occurs if I try with MixerTTSModel:

```
assert isinstance(spec_gen, SpectrogramGenerator)

if isinstance(spec_gen, MixerTTSModel):
    tokens = spec_gen.parse("Hey, this produces speech!")
else:
    tokens = spec_gen.parse(text="Hey, this produces speech!")

spectrogram = spec_gen.generate_spectrogram(tokens=tokens)

# Now we can visualize the generated spectrogram
# If we want to generate speech, we have to use a vocoder in conjunction to a spectrogram generator.
# Refer to the Inference_ModelSelect notebook on how to convert spectrograms to speech.
imshow(spectrogram.cpu().detach().numpy()[0,...], origin="lower")
plt.show()
```
Just in case, the MixerTTSModel loaded is Mixer-TTS-X:

```
# In the same way, we can load the pre-trained Mixer-TTS model as follows
pretrained_model = "tts_en_lj_mixerttsx"
spec_gen = MixerTTSModel.from_pretrained(pretrained_model)
spec_gen.eval();
```

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-6-e1795f3aeeee>](https://localhost:8080/#) in <module>
      6     tokens = spec_gen.parse(text="Hey, this produces speech!")
      7 
----> 8 spectrogram = spec_gen.generate_spectrogram(tokens=tokens)
      9 
     10 # Now we can visualize the generated spectrogram

1 frames
[/usr/local/lib/python3.7/dist-packages/nemo/collections/tts/models/mixer_tts.py](https://localhost:8080/#) in generate_spectrogram(self, tokens, tokens_len, lm_tokens, raw_texts, norm_text_for_lm_model, lm_model)
    619         if self.cond_on_lm_embeddings and lm_tokens is None:
    620             if raw_texts is None:
--> 621                 raise ValueError("raw_texts must be specified if lm_tokens is None")
    622 
    623             lm_model_tokenizer = self._get_lm_model_tokenizer(lm_model)

ValueError: raw_texts must be specified if lm_tokens is None
```

Thanks,
Add EMA support to NeMo
State: closed | Pull request: yes

# What does this PR do ?

Adds Exponential Moving Average (EMA) support. We've seen promising results in reducing training times to reach convergence.

Things to do:

- [x] Capture speed degradation using the EMA callback.

There is practically none when the EMA weights are already on CUDA. Below are convergence results using LibriSpeech (SP) and the Conformer RNNT config for ASR:

**Precision 32**

WER vs Global Step

![Screenshot 2022-09-14 at 15 33 17](https://user-images.githubusercontent.com/6707363/190184437-2d5f043a-7e32-4a46-9f02-cea4ead8556c.png)

WER vs Process Time

![Screenshot 2022-09-14 at 15 34 40](https://user-images.githubusercontent.com/6707363/190184580-c71e9a90-7e0b-4fd4-8915-bf817180f2a3.png)

**Precision 16**

WER vs Global Step

![Screenshot 2022-09-14 at 15 33 43](https://user-images.githubusercontent.com/6707363/190184495-a2d1c692-0e14-4969-9763-6a5d443ca490.png)

WER vs Process Time

![Screenshot 2022-09-14 at 15 34 17](https://user-images.githubusercontent.com/6707363/190184541-911e7042-342a-4dfc-97b6-fe349aff14e9.png)

- [x] Explore keeping EMA weights on the CPU (transfer to GPU just for the multi-tensor apply, then bring it back to CPU)
- [x] Test using Quartznet/Conformer to ensure we see expected improvements using the EMA callback
- [ ] Add multi-node/gpu correctness test (EMA weights should be the same across processes) to jenkinsfile

**Collection**:
- Common

# Changelog 
- Adds Exponential Moving Average (EMA) support.

# Usage

The callback can be added to pure lightning, however it requires Apex to be installed (for the multi-tensor apply).

```python
...
lenet5 = MNISTLeNet5(MNISTLeNet5Config())
lenet5.setup_training_data()
lenet5.setup_optimization()

trainer = Trainer(
    accelerator='gpu',
    devices=1,
    callbacks=EMA(ema=0.999)
)
trainer.fit(model=lenet5)
```

Currently I've exposed this through the exp manager (questionable if this is the right approach or not) to use through the hydra configs.
```bash
python examples/asr/asr_ctc/speech_to_text_ctc.py \
    model.train_ds.manifest_filepath=... \
    model.validation_ds.manifest_filepath=... \
    trainer.devices=1 \ 
    +exp_manager.enable_ema=True
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

cc @titu1994
[TTS] Add multi-speaker German FastPitch and HiFiGAN NGC checkpoints
State: closed | Pull request: yes

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)