renwei2024/nemo-github-issues huggingface dataset:
Merge r1.18.0 bugfixes and doc updates to main
State: closed | Pull request: yes
Labels: name: TTS

name: NLP

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Merge r1.18.0 to main
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: Speaker Tasks

name: CI

name: common

# What does this PR do ?

Make sure all doc updates and bugfixes that were made to the release branch back into main.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
WER for the loanword when validating code-switching ASR model
State: closed | Pull request: no

Hello, I have question for the Error rate for the code-switching ASR model.

There are many loanwords in korean which from english.
For example, "Coffee" is "커피" in korean. These two words have same pronunciation and meaning.

The sentence "커피 마실래?" is "would you want coffee?" in english.

Let's say there is some audio input, and there are two situation in output.

1. "커피 마실래?"
2. "coffee 마실래?"

Actually, both two sentence are correct and natural for korean people. 

Then, how to calculate error rate?
if the label is '커피' then 'coffee' is wrong?
or, '커피' and 'coffee' are both correct?
How the gradients are updated when using grad_accumulation
State: closed | Pull request: no

Hello, 

The paper of CitriNet said they used 32 V100 GPUs, with a batch size of 32 per GPU.
then, the gradients of each GPU are accumulated and updated together?

When I trained CitriNet-384 model with single gpu, with batch size of 32, then grad_accumulation=32 will be same condition with the paper?

Also, Im training Multilingual ASR model which includes English (LibriSpeech 960h, wpe 256) and Korean (KsponSpeech 1000h, bpe 1641) with CitriNet-384. I made the hyperparameters same with paper (NovoGrad optimizer with learning rate (LR) of 0.05, β1 = 0.8, β2 = 0.25 and weight decay of 0.001). However, validation_loss seem to not decrease until epoch 18. 

is there any tips for this? Im testing with lr=0.01 now. 

Thank you.
fix conversion and eval
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

fixes conversion of sft ckp to nemo and allows eval of sft .nemo using peft_eval scripts.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Move black parameters to pyproject.toml
State: closed | Pull request: yes

# What does this PR do ?

Move `black` formatter parameters to `pyproject.toml`

Advantages:
- parameters are stored in one place
- one can use `black .`/`black <file>` command to reformat code
- more robust: if someone tries to reformat code with the newest `black` version, the error will be thrown (instead of reformatting the whole codebase)

**Collection**: [Note which collection this PR will affect]

# Changelog 
- add `black` params to `pyproject.toml`
- remove params from `setup.py` and `.pre-commit-config.yaml`

# Usage
* You can potentially add a usage example below

```
black .
black edited_file.py
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
lang field in Multilingual ASR models
State: closed | Pull request: no

Hello, 

I have a question about Multilingual ASR models.

In a Multilingual ASR model, when using an Aggregate tokenizer, the "lang" field needs to be added in the manifest. From what I understand from tech blogs and tutorials, the "lang" field is primarily used to assign offset indices to tokens for each language during token aggregation.

However, I'm curious if the "lang" field in Multilingual models is utilized for purposes other than token indexing. For example, I would like to know if it is designed to provide language information in the form of a one-hot vector or embedding vector, which can assist in the training process of the model.

Thank you
Fix for interctc test random failure
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

I finally figured out the reason for random failure of this test. It turns out that sometimes our input signal is such that for some models it results in all-zeros in the output of some initial encoder layers. This is then propagated all the way through the model and so the assert that final layer output is different from middle layer fails. After changing the code to pass processed signal directly, this should not happen anymore.

**Collection**: ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Remove dup
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Remove from jenkins
State: closed | Pull request: yes
Labels: name: cherry-pick

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Remove from jenkins
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add megatron_core to requirements
State: closed | Pull request: yes
Labels: name: cherry-pick

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add megatron_core to requirements
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Disable interctc tests
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
sft model can use this script for eval
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

sft model can use `megatron_gpt_peft_eval.py`

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
small bugfix for asr_evaluator
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix for old models in change_attention_model
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fixes some issues:
-change_attention_model should work for old models which don't have some of the relevant parameters in config
-also it shouldn't run when nothing to change was submitted

Original PR: https://github.com/NVIDIA/NeMo/pull/6608

**Collection**: ASR

  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation
fix module names to reflect the module names in the GPTModel from meg…
State: closed | Pull request: yes
Labels: name: core

name: stale

name: NLP

…atron core

# What does this PR do ?

Corrects the names of modules that are expected to be in GPTModel object coming from megatron-core

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Patch decoding for PC models
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

# What does this PR do ?

Remove extraneous spaces between word and punctuation

**Collection**: [ASR]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Patch decoding for PC models
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Remove extraneous spaces between word and punctuation

**Collection**: [ASR]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Decoding patch for PC models
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Removes spaces before `.` `,` or `?` punctuations generated by PC models

**Collection**: ASR

# Changelog 
- wer.py regex, line 545 
- rnnt_wer.py regex, line 503

# Usage
n/a

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Fix TTS audio preprocessing bugs
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Fix a few bugs in TTS preprocess_audio.py and MelSpectrogramFeaturizer.

**Collection**: [TTS]

# Changelog 
- Add additional config to mel featurizer to make it more consistent with past TTS configs
- Add check and unit test for normalize_volume when audio input is empty array.
- Modify preprocess script to avoid error when resample_rate=0.
- Modify preprocess script to round duration to 2 decimals
- Move normalize_volume unit tests to correct test class, following an earlier refactoring.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
AttributeError: module 'megatron.core.parallel_state' has no attribute 'get_amax_reduction_group'
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When running megatron_gpt_eval.py with an FP8 model, The FP8 path is run and not able to find `get_amax_reduction_group()` in parallel_state.

The error is triggered at this line:
https://github.com/NVIDIA/NeMo/blob/21048627b3923c9268842990aafdef141bd14bd1/nemo/collections/nlp/modules/common/megatron/transformer.py#LL1432C70-L1432C70

**Steps/Code to reproduce bug**

Get a model trained with TE FP8

Run:
python megatron_gpt_eval.py gpt_model_file=models/5b_fp8_tp1.nemo

**Expected behavior**

The script is expected to finish and generate outputs.

**Environment overview (please complete the following information)**

 - Environment location: 
 Container: nvcr.io/nvidia/pytorch:23.03-py3
 - Method of NeMo install:
 Call `./reinstall.sh` on main branch commit c3deeac8d58f48e9283d13fb7c32fd259605705f
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used
 docker run --gpus '"device=0"' -it --ipc=host --ulimit memlock=-1 -v /home/scratch.yenshiw_sw/NeMo:/workspace/local-nemo --ulimit stack=67108864 nvcr.io/nvidia/pytorch:23.03-py3

**Environment details**

**Additional context**

I cannot find name `get_amax_reduction_group` in megatron source code (parallel_state):
https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/parallel_state.py
Exchange indices with torch.distributed
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Allows to exchange GPTDataset indices with torch.distributed (instead of filesystem) by broadcasting indices from rank 0.

**Collection**: NLP

# Changelog 
- Allows to exchange GPTDataset indices with torch.distributed

# Usage
Turn on with a `model.data.exchange_indices_distributed=1` flag

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update core commit hash in readme
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Update core commit hash in readme

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update core commit hash in readme
State: closed | Pull request: yes

# What does this PR do ?

Update core commit hash in readme

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add HAT image to docs
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

# What does this PR do ?

Add forgotten image to hat docs description

**Collection**: [ASR]


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation
ASR-TTS Models: Support hybrid RNNT-CTC, improve docs.
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

# What does this PR do ?

Improve ASR-TTS models code and docs. Address issues raised in #6594:
- (new feature) support hybrid RNNT-CTC models
- improve documentation:
  - add required docs for sampling techniques
  - add a note about appending values to config for training
  - add initial enhancer stubs in docs (detailed documentation will be added later by @racoiaws)
- do not warn on import for optional library
- (bugfix) fix instantiation from ASR model config: `use_start_end_token` parameter, broken after #6426, #6467 

**Collection**: [ASR, TTS]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [x] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add HAT image to docs
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add forgotten image to hat docs description

**Collection**: [ASR]


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation
Exchange indices with torch.distributed
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Allows to exchange GPTDataset indices with torch.distributed (instead of filesystem) by broadcasting indices from rank 0.

**Collection**: NLP

# Changelog 
- Allows to exchange GPTDataset indices with torch.distributed

# Usage
Turn on with a `model.data.exchange_indices_distributed=1` flag

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
EnglishPhonemesTokenizer skips <oov> when encoding
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

Calling `EnglishPhonemesTokenizer.encode_from_g2p()` with text that a list of phonemes that contains `<oov>` as one of the items simply skips the `<oov>` item even if `<oov>` is a part of the `EnglishPhonemesTokenizer.tokens`.

**Steps/Code to reproduce bug**

```
from nemo.collections.tts.torch.tts_tokenizers import EnglishPhonemesTokenizer
from nemo.collections.tts.torch.g2ps import EnglishG2p

g2p = EnglishG2p(ignore_ambiguous_words=False)
tokenizer = EnglishPhonemesTokenizer(g2p, punct=True, stresses=True)
tokens = tokenizer.encode_from_g2p(['OY1', '<oov>'])
assert len(tokens) == 2
```

**Expected behavior**

Tokenizer to correctly tokenize `<oov>` token
[Question] How to perform audio controls(pace and pitch) with ACEVC model？
State: closed | Pull request: no
Labels: name: stale

Hi. Thank you very much for your implementation on ACEVC. I'm interested in this Zero-shot Voice Conversion project. I have finished training the model and successfully conveted the voice by following the document on the website https://paarthneekhara.github.io/ace/code.html. However, I didn't know how to control pace or pitch as the website  https://paarthneekhara.github.io/ace/ mentioned. Are there any hyperparameters or instructions for pace and pitch controls on audio. I'm sincerely thankful with your help!
Patch memory used for NeMo Megatron models
State: closed | Pull request: yes
Labels: name: core

name: NLP

# What does this PR do ?

Reduce the peak memory usage during model building and restoration.

**Collection**: [NLP]

# Changelog 
- Force state dict onto CPU during restoration
- Add param_dtype in multiple places to correctly build tensor with trainer.precision dtype


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Confidence ensembles implementation
State: closed | Pull request: yes
Labels: name: core

name: ASR

# What does this PR do ?

Adds confidence ensembles implementation.

I will follow up with more features (e.g., tuning of the LR/confidence parameters) as well as documentation and probably a tutorial notebook. But the basic functionality is fully implemented.

**Collection**: ASR

# Changelog 
- Add new script to `scripts/confidence-ensembles/build_ensemble.py`. It can be run to build an ensemble by providing models/data. It will train confidence-based model selection block and output .nemo checkpoint containing all models + pkl file for the model selection.
- Add new argument to `transcribe_speech.py` `return_transcriptions` which can be used to select if transcriptions or config is returned. Default to current behavior of returning config.
- Add a new temperature parameter to the decoding strategies and ctc/rntt decoder/joint blocks. What's currently called `temperature` in the confidence config is not really a temperature, but instead an `alpha` parameter as defined here https://arxiv.org/abs/2212.08703. Temperature should be applied before softmax, so it cannot be directly implemented inside confidence computation.
- Add a new `ConfidenceEnsembleModel` class that supports running `transcribe` method for the ensemble. It does not support training or any other kind of inference (`forward` method is not implemented). This is by design as we only combine the models through model selection block and it operates on the full transcriptions, not intermediate results.
- Add `scripts/confidence_ensembles/test_confidence_ensembles.py` which contains a few integration tests of building->running ensemble of es-it models. The plan is to include it in the nightly tests, since it depends on ~200mb of data and takes 5-10 minutes to run.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Cherry pick commits in #6528 to main
State: closed | Pull request: yes
Labels: name: bug

name: cherry-pick

name: NLP

name: CI

# What does this PR do ?

Manually cherry-pick changes from https://github.com/NVIDIA/NeMo/pull/6528. Automatic merging probably failed due to merge conflicts with https://github.com/NVIDIA/NeMo/pull/6510.

**Collection**: NLP

# Changelog 
- Cherry-pick changes from https://github.com/NVIDIA/NeMo/pull/6528
- Move duplicated logic for distributed optimizer into recently-added [`MegatronGPTModel.fwd_bwd_step` function](https://github.com/NVIDIA/NeMo/blob/24c7b4bdcb794536a52a38bbd1737422f095681f/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py#L333).


# Usage

Set `virtual_pipeline_model_parallel_size > 1`:
https://github.com/NVIDIA/NeMo/blob/5468077f5127be1a4c88065de2544f4268b9a6e4/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml#L51

This requires `pipeline_model_parallel_size > 2`  and `virtual_pipeline_model_parallel_size*pipeline_model_parallel_size` divides `num_layers`.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [x] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Cherry-pick from https://github.com/NVIDIA/NeMo/pull/6528
* Fixes merge conflicts with https://github.com/NVIDIA/NeMo/pull/6510
t5 lora tuning
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

LoRA support for T5 models.

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Cherry pick commits in #6601 to main
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Manually Cherry pick commits in https://github.com/NVIDIA/NeMo/pull/6601/commits merged to r1.18.0 due to failure of creating automatic cherry pick and merge

**Collection**: ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add GPT eval mode fix for interleaved to main (#6449)
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

add fix from #6449 into main

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review
?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)