renwei2024/nemo-github-issues huggingface dataset:
When training HiFiGAN, after some epochs, `d_loss`, `g_loss`, etc. become `NaN`
State: closed | Pull request: no
Labels: name: stale

Hello,
I'm training a 44100 HifiGAN model on my own dataset. 
After some epochs, `d_loss`, `g_loss`, etc. become `NaN`,

![20230407110450](https://user-images.githubusercontent.com/16176151/230533410-2e69084a-686c-4747-a159-a7c179faa9d4.jpg)
![20230407110514](https://user-images.githubusercontent.com/16176151/230533449-7c93218c-f65f-4163-a8bd-e3489a41adfb.jpg)
![20230407110942](https://user-images.githubusercontent.com/16176151/230533478-68da3b74-ead8-48f8-a6ed-e5e558ae01bd.jpg)

and `val_loss` increases, 
![20230407110538](https://user-images.githubusercontent.com/16176151/230533562-3a20e813-2173-406a-b624-4ecdf2be9076.jpg)

audio quality drops dramatically.

What seems to be the problem?
Thank you!
Add [Draft][TTS] FastPitch multi-speaker pre-train and adapter fine-tune
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

name: NLP

# What does this PR do ?

Adds FastPitch speaker adaptation with adapters.

**Collection**: TTS

# Changelog 
- Adds multi-speaker FastPitch pre-training with 1) Looked-up speaker embedding 2) Speaker verification model speaker embedding 3) Global Style Tokens 4) Conditional Layer Normalization 
    - `nemo/collections/tts/data/tts_dataset.py`
    - `nemo/collections/tts/torch/tts_data_types.py`
    - `nemo/collections/tts/models/fastpitch.py`
    - `nemo/collections/tts/modules/fastpitch.py`
    - `nemo/collections/tts/modules/transformer.py`
    - `nemo/collections/tts/modules/submodules.py`
- Add adapter modules for FastPitch fine-tuning
    - `nemo/collections/tts/models/fastpitch.py`
    - `nemo/collections/tts/modules/fastpitch.py`
    - `nemo/collections/tts/modules/transformer.py`
    - `nemo/collections/tts/modules/aligner.py`
    - `nemo/collections/tts/parts/mixins/__init__.py`
    - `nemo/collections/tts/parts/mixins/fastpitch_adapter_mixins.py`
- Add config and fine-tuning python script
    - `examples/tts/conf/fastpitch_speaker_adaptation.yaml`
    - `examples/tts/fastpitch_finetune_adapters.py`
- Add tutorial
    - `tutorials/tts/FastPitch_SpeakerAdaptation.ipynb`
- Add fast text normalization script
    - `scripts/dataset_processing/tts/add_normalized_text.py`
- Fix aligner `nan` loss bug
    - `nemo/collections/tts/losses/aligner_loss.py`
 
# Usage
* Follow `tutorials/tts/FastPitch_SpeakerAdaptation.ipynb`
    * Pre-train multi-speaker FastPitch
    * Fine-tune HiFiGAN with mel-spectrograms from pre-trained FastPitch
    * Fine-tune FastPitch with adapters
    * Fine-tune HiFiFAN with mel-spectrograms from fine-tuned FastPitch

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add support for Megatron GPT Untied Embd TP PP Change
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Adds support for untied embeddings for Megatron GPT for TP PP change

**Collection**: [NLP]

# Changelog 
- Generalize indexing for tied vs untied gpt embeddings

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
R1.17.0
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: stale

name: ASR

name: NLP

name: Speaker Tasks

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
user selected max_seq_len should be less than model's max_seq_len
State: closed | Pull request: yes
Labels: name: cherry-pick

name: stale

name: NLP

# What does this PR do ?

user selected max_seq_len should be less than model's max_seq_len
**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Ensure fine-tuning/prompt learning work for T5
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Adds support for T5 Finetuning/Prompt tuning.

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Installation fails for python 3.10
State: closed | Pull request: no
Labels: name: bug

name: stale

name: MacOS

Executing `python3.10 -m pip install mecab-python3` produces the error

```
Collecting mecab-python3
  Using cached mecab-python3-1.0.6.tar.gz (77 kB)
  Preparing metadata (setup.py) ... done
Installing collected packages: mecab-python3
  DEPRECATION: mecab-python3 is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for mecab-python3 ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for mecab-python3 did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
        warnings.warn(
      WARNING: The wheel package is not available.
      WARNING: The wheel package is not available.
      WARNING: The wheel package is not available.
      WARNING: The wheel package is not available.
      running install
      /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.9-universal2-cpython-310
      creating build/lib.macosx-10.9-universal2-cpython-310/MeCab
      copying src/MeCab/__init__.py -> build/lib.macosx-10.9-universal2-cpython-310/MeCab
      copying src/MeCab/cli.py -> build/lib.macosx-10.9-universal2-cpython-310/MeCab
      running build_ext
      error: [Errno 2] No such file or directory: 'mecab-config'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> mecab-python3

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
```

MacOS Ventural 13.3, M2 processor.
fixed torch warning on using a list of numpy arrays
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: ASR

# Changelog 
- Fixed torch warning about creating a tensor from a list of numpy arrays and sped up execution by putting ```audio``` in a list when calling ```np.array()``` instead of when calling ```torch.tensor()```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Onnx export of cache aware conformer fails
State: closed | Pull request: no
Labels: name: bug

name: stale

Latest fixes of  onnx export of cache aware conformer still fail.
PR [ #6311](https://github.com/NVIDIA/NeMo/pull/6311) (This is not yet in nemo 1.17.0, only https://github.com/NVIDIA/NeMo/pull/5837 is with more problems)


The model get's exported, but inferencing results in an exception.
This can be seen when the onnx model inference results get's verified with the pytorch model as a last step of the export. This fails with:

[ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_1530' Status Message: D:\a_work\1\s\onnxruntime\core\providers\cpu\tensor\reshape_helper.h:26 onnxruntime::ReshapeHelper::ReshapeHelper i < input_shape.NumDimensions() was false. The dimension with value zero exceeds the dimension size of the input tensor.
(I had to change CUDAExecutionProvider to CPUExecutionProvider)

The model was trained with nemo, but I can not share such a model.

Also I would want to remark that the dimension 0 and 1 of tensors: cache_last_channel and cache_last_time get transposed at the input and transposed again at the output, while this transposing is not needed and adds overhead.
Add opengraph support for NeMo docs
State: closed | Pull request: yes

# What does this PR do ?

Adds opengraph support for NeMo documentation page

**Collection**: [Docs]

# Changelog 
- Add opengraph support for NeMo docs

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
updated lr for FC configs
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Refines the suggested training parameters for the Fast Conformer models.

**Collection**: ASR

# Changelog 
- In fast-conformer_ctc_bpe.yaml, change min_lr to 1e-4 (from 1e-6)
- in fast-conformer_transducer_bpe.yaml, change lr to 5e-3 and min_lr to 5e-4 (from 1e-6)

# Usage
* Train as usual but you will be able to use checkpoint averaging to improve results

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update gpt_training.rst
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add installation and ASR inference instructions for Mac
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Adds installation and inference instructions for Apple Mac.

**Collection**: [ASR (docs)]

# Changelog 
- installation instructions
- ASR inference instructions

# Usage

Developing NeMo on Mac devices.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #6289
Adapter tuning accepts expanded language model dir
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

set model extracted dir such that the save restore connector can load expanded base models. This is required to load large base models so that we can avoid the untaring step during training.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Make possible to control tqdm progress bar in ASR models
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Now it's possible to control display of progress bar when calling `transcribe` method of ASR models.

**Collection**: ASR

# Changelog 
- `transcribe` method of ASR models now have `verbose` param, which allows to control display of tqdm progress bar

# Usage
```python
import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.EncDecRNNTBPEModel.from_pretrained("nvidia/stt_en_conformer_transducer_xlarge")

# Will display progress bar by default
asr_model.transcribe(['2086-149220-0033.wav'])

# Won't display progress bar when verbose is set to False
asr_model.transcribe(['2086-149220-0033.wav'], verbose=False)
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
None
Update wfst_text_normalization.rst
State: closed | Pull request: yes
Labels: name: NLP

Add Hungarian (incoming in NeMo-text-processing)

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [X] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Bugfix for BF16 grad reductions with distopt (#6340)
State: closed | Pull request: yes
Labels: name: core

name: NLP

Cheery-pick from `main` to `r1.17.0_pt_23.04`


* Debug distopt support for BF16 grad reductions
* Dump and load FP32 main params
* Style tweaks



---------

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add conformer configs for hat model
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add conformer char and bpe configs for hat model (https://arxiv.org/abs/2003.07705)

**Collection**: [ASR]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
[Experimental] Update black to 23.x
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: Speaker Tasks

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update script for ngram rnnt and hat beam search decoding
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Update previous script for ngram RNNT/HAT beam search decoding by analogy with CTC ngram decoding.

**Collection**: [ASR]

# Changelog 
- Add option to return `return_encoder_embiddings` for `rnnt_mode.transcribe()` method (analogy of logits for CTC)
- Update `eval_beamsearch_ngram_transducer.py` by analogy with `eval_beamsearch_ngram.py`
- Add RNNT/HAT ngram decoding description to docs

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
ASR_with_Subword_Tokenization.ipynb's guide on process_asr_text_tokenizer script is broken
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

When following guide to Build a Customize Tokenizer, process_asr_text_tokenizer.py does not create intended tokenizer folder with required content. Unable to pre-trained on any models that use BPE vocab since no one can pass anything to the new_tokenizer_dir argument now. 

**Steps/Code to reproduce bug**
Try to run below command as taught by the tutorial, see if it works for you.
`!python /content/drive/MyDrive/Colab\sNotebooks/AI6127\sNLP/GroupProject/Data/process_asr_text_tokenizer.py \
  --manifest="/content/drive/MyDrive/Colab Notebooks/AI6127 NLP/GroupProject/Data/manifest_train.json" \
  --data_root="/content/drive/MyDrive/Colab Notebooks/AI6127 NLP/GroupProject/Data" \
  --vocab_size=32 \
  --tokenizer="spe" \
  --no_lower_case \
  --log`

**Expected behavior**

![image](https://user-images.githubusercontent.com/1837762/230006481-47a2e2e2-f0bf-4d95-8eae-2267bad5f565.png)

**Environment overview (please complete the following information)**

Colab Pro +

**Environment details**

NA
**Additional context**

NA
[NeMo Megatron] Cleanup configs to infer the models TP PP config automatically
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

**Collection**: [NLP]

# Changelog 
- Update configs to set tp pp and pp rank all to -1
- Check and update -1 to the model config results

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Word boosting for Flashlight decoder
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Adds word boosting to the Flashlight decoder in Nemo to mirror this capability in Riva

For more info see here: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-python-advanced-wordboosting.html

And here: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-customizing.html#word-boosting

**Collection**: ASR

# Changelog 
- Added word boosting to collections/asr/modules/flashlight_decoder.py
- Added new config line item to flashlight_cfg in ctc_beam_decoding.py

(I noticed that flashlight_decoder.py had Windows EOLs, so I changed them to Unix, but unfortunately this makes git show as if I had changed the entire file which isn't the case, sorry about this)

# Usage
* In order to boost certain words, you need to create a flat file which is tab separated. The format of this file is:

```
<word>\t<boost score>
```

For example, here is a valid format boost file on disk, not that it is a single tab separated file:
```
nvidia	40
geforce	50
riva	80
turing	30
badword	-100
```
Positive scores boost words higher in the LM decoding step so they show up more frequently, whereas negative scores squelch words so they show up less frequently. The recommended range for the boost score is 20 to 100.

The boost file handles both in-vocabulary words and OOV words just fine, so feel free to mix and match.

You can then add it to your flashlight config object in Nemo like this:

```python
ctc_decoding.beam.flashlight_cfg.boost_path='/path/to/my_boost_file.boost'
```

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [X] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
taskname and early stopping for adapters
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Cherrypick mlperf prs
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update readme
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update readme
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Tensor-parallel communication overlap with userbuffer backend
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Add (1) interfaces to TE and initialized (2) process group setting to support tensor-parallel communication overlap with userbuffer backend.

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
Set `ub_tp_comm_overlap` to `True`

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[pre-commit.ci] pre-commit suggestions
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: Speaker Tasks

name: common

<!--pre-commit.ci start-->
updates:
- [github.com/pre-commit/pre-commit-hooks: v4.3.0 → v4.4.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.3.0...v4.4.0)
- [github.com/psf/black: 19.10b0 → 23.3.0](https://github.com/psf/black/compare/19.10b0...23.3.0)
<!--pre-commit.ci end-->
using mmap dataset for ptuning
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

replaces standard pytorch dataset with a mem mapped json dataset.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Correct image logo in NeMo Website
State: closed | Pull request: yes

# What does this PR do ?

Use better image for the website default card.

**Collection**: [Docs]


# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Correct tag for image url in NeMo Website
State: closed | Pull request: yes

# What does this PR do ?

Fix image tag for open graph

**Collection**: [Docs]

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Update support for twitter cards in NeMo Website
State: closed | Pull request: yes

# What does this PR do ?

Adds support for twitter cards in addition to open graph

**Collection**: [Docs]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Add OpenGraph metadata for better links to NeMo Website
State: closed | Pull request: yes

# What does this PR do ?

Adds [OpenGraph](https://ogp.me/) support to NeMo website for richer URL linking.

**Collection**: [Docs]

# Usage

```markdown

---
title: Template 2022
author: [First Name, Second Name]
author_gh_user: [Github User 1, Github User 2]
read_time: Time to read
publish_date: Date of publishing

# Optional: OpenGraph metadata
og_title: Title of the blog post for Rich URL previews
og_image (or image): Image for Rich URL previews (absolute URL)
og_image_type: Image type (e.g. image/png). Defaults to image/png.
page_path: Relative path to the image from the website root (e.g. /assets/images/)
description: Description of the post for Rich URL previews

# DO NOT CHANGE _BELOW
template: blog.html
---
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation
extract inference weights
State: closed | Pull request: yes

# What does this PR do ?
Adds a script to extract p_tuned inference weights. 

More details: The ptuning training saves all params including the prompt encoder weights. These are typically not required after training is completed. The script allows you to save the required representations for inference in a separate file and you can delete the .nemo if wanted.


**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
docs on the use of heterogeneous test / val manifests
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Adds docs for the use of multiple test / validation manifests as enabled in #6306

**Collection**: ASR

# Changelog 
- Added docs to `docs/source/asr/configs.rst`

# Usage
Docs only

```python
# Add a code snippet demonstrating how to use this 
n/a
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)