renwei2024/nemo-github-issues huggingface dataset:
Fix TTS FastPitch tutorial
State: closed | Pull request: yes
Labels: name: TTS

name: cherry-pick

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix dimensionality in get_dist function
State: closed | Pull request: yes
Labels: name: TTS

name: cherry-pick

# What does this PR do ?

Adds back a `squeeze(1)` to the Aligner `get_dist()` function.

This should fix a few cells erroring out in the Aligner inference notebook.

See #7490 for more info.

**Collection**:TTS

**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
How do I save all checkpoints in exp_manager
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I'm trying to accomplish a seemingly trivial task of saving all checkpoints during training - many hours later after trying many different configurations I still can't do and I see multiple problems

**Steps/Code to reproduce bug**

Here is the relevant part of the config:
```
trainer:
  enable_checkpointing: False # checkpointing provided by exp_manager
  limit_val_batches: 0
  limit_test_batches: 0
exp_manager:
  checkpoint_callback_params:
    every_n_train_steps: 5
    every_n_epochs: null
    monitor: null # want all checkpoints, 
    save_top_k: -1 # save all checkpoints
    save_last: True
```

The main doc in PTL is https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.ModelCheckpoint.html#modelcheckpoint

There are multiple problems here, but the key is to get rid of the message as I'm not running eval - only training and be able to save all checkpoints. The message that I get is:

```
Epoch 0, global step 10: '' was not in top -1
```
The PL doc says:
> monitor (Optional[str]) – quantity to monitor. By default it is None which saves a checkpoint only for the last epoch.

 if I set it to null it breaks:
 
```
nemo/nemo/utils/exp_manager.py", line 889, in configure_checkpointing
    if "val" in params.monitor:
TypeError: argument of type 'NoneType' is not iterable
```

and looking at PL's source code:

```
    def _save_topk_checkpoint(self, trainer: "pl.Trainer", monitor_candidates: Dict[str, Tensor]) -> None:
        if self.save_top_k == 0:
            return

        # validate metric
        if self.monitor is not None:
            if self.monitor not in monitor_candidates:
```
it shouldn't go into validation metric if it's set to None, but NeMo doesn't allow me to do that.


Then the PL doc says:
> save_top_k
> (int) – if save_top_k == k, the best k models according to the quantity monitored will be saved. if save_top_k == 0, no models are saved. if save_top_k == -1, all models are saved. Please note that the monitors are checked every every_n_epochs epochs. if save_top_k >= 2 and the callback is called multiple times inside an epoch, the name of the saved file will be appended with a version count starting with v1.

but I still get only the last checkpoint saved, 

and this message:

```
Epoch 0, global step 10: 'blah' was not in top -1
```
looks wrong because it shouldn't even go there, as it's set to -1 == meaning do not check.

**Expected behavior**

I need to be able:

1. save all checkpoints
2. while not running eval

Thank you!
dllogger - log on rank 0 only
State: closed | Pull request: yes

# What does this PR do ?

As discussed in https://github.com/NVIDIA/NeMo/issues/7495 @titu1994 helped me to make Dllogger to log on rank0 as otherwise the metrics log happens 512 times per step on 512 gpus.

Fixes: https://github.com/NVIDIA/NeMo/issues/7495

**Collection**: [Note which collection this PR will affect]

# Changelog 
- make Dllogger to log on rank0 as otherwise the metrics log happens 512 times per step on 512 gpus.


# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to https://github.com/NVIDIA/NeMo/issues/7495
bugfix: specify trainer.strategy=auto when devices=1
State: closed | Pull request: yes
Labels: name: TTS

name: cherry-pick

# What does this PR do ?

the script called default `conf/vits.yaml` config: https://github.com/NVIDIA/NeMo/blob/main/examples/tts/conf/vits.yaml#L192, which applied `strategy=ddp` by default. When `devices=1` in this tutorial, we should specify `auto` to override `ddp`.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
SFT model parallel fix for dist ckpt
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

SFT model parallel fix for dist ckpt

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
remove auto generated examples
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

mark the dp inserted example and remove it for test.

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
bugfix: specify trainer.strategy=auto when devices=1
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

the script called default `conf/vits.yaml` config: https://github.com/NVIDIA/NeMo/blob/main/examples/tts/conf/vits.yaml#L192, which applied `strategy=ddp` by default. When `devices=1` in this tutorial, we should specify `auto` to override `ddp`.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
updated peft metric script
State: closed | Pull request: yes

# What does this PR do ?

evaluates peft outputs.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fastconform SSL pretrained recipe failed potentially due to PTL 2.0
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

the `examples/asr/speech_pretraining/speech_pre_training.py` is not working properly on the lates main branch, but working in 1.20. At first glance, the error could be related to the upgrade of Pytorch lightning 2.0.

**Error Brief**
```bash
lightning_fabric.utilities.exceptions.MisconfigurationException: `ModelCheckpoint(monitor='val_loss')` could not find the monitored key in the returned metrics: ['learning_rate', 'global_step', 'train_contrastive', 'train_backward_timing in s', 'train_step_timing in s', 'epoch', 'step']. HINT: Did you call `log('val_loss', value)` in the `LightningModule`?
```

**Error Details**

[error_logs.zip](https://github.com/NVIDIA/NeMo/files/12719689/error_logs.zip)
Fix dimensionality in get_dist function
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Adds back a `squeeze(1)` to the Aligner `get_dist()` function.

This should fix a few cells erroring out in the Aligner inference notebook.

See #7490 for more info.

**Collection**:TTS

**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Fix PTL2.0 related ASR bugs in r1.21.0: Val metrics logging, None dataloader issue
State: closed | Pull request: yes
Labels: name: core

name: ASR

# What does this PR do ?

This PR added fixes for PTL2.0 related ASR bugs in r1.21.0: Val metrics logging, None dataloader issue

**Collection**: 
ASR, Core

# Changelog 
- val_dataloader() and test_dataloader() functions in nemo/core/classes/modelPT.py updated to handle cases where dataloader is None
- correct logging of validation metrics in nemo/collections/asr/models/rnnt_models.py and nemo/collections/asr/models/hybrid_rnnt_ctc_models.py


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

# Additional Information
* Related to [Bug4296343](https://nvbugswb.nvidia.com/NvBugs5/SWBug.aspx?bugid=4296343&cmtNo=), [Bug4296057](https://nvbugswb.nvidia.com/NvBugs5/SWBug.aspx?bugid=4296057&cmtNo=), [Bug4246804](https://nvbugswb.nvidia.com/NvBugs5/SWBug.aspx?bugid=4246804&cmtNo=)
* Fixes issues raised in #7450, #7507
Fix exp manager check for sleep
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Fixes a bug in checking the number of ranks before sleeping.

**Collection**: [Core]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Fix exp manager check for sleep
State: closed | Pull request: yes

# What does this PR do ?

Fixes a bug in checking the number of ranks before sleeping.

**Collection**: [Core]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Hotfix: change the order of config overrides
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Remove incorrect extra argument of load_from_checkpoint_dir()
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

The function load_from_checkpoint_dir() in `examples/nlp/language_modeling/megatron_gpt_continue_training.py` has four arguments:
```python
def load_from_checkpoint_dir(cls, cfg, trainer, modify_confg_fn):
```
But the calling side is:
```python
model = load_from_checkpoint_dir(MegatronGPTModel, cfg, trainer, gpt_cfg, modify_confg_fn=_modify_config)
```
The argument `gpt_cfg` is extraneous.

# Changelog 
- Remove extraneous argument of load_from_checkpoint_dir()

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

@MaximumEntropy @ericharper @ekmb @yzhang123 @VahidooX @vladgets @titu1994 @XuesongYang 

# Additional Information
* Related to #7480
add sleep
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Fixes https://github.com/NVIDIA/NeMo/issues/7460. Exp manager now has a sleep argument to sleep the non rank 0 ranks.
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

# Additional Information
* Related to # (issue)
#7460
add sleep
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Fixes https://github.com/NVIDIA/NeMo/issues/7460. Exp manager now has a sleep argument to sleep the non rank 0 ranks.
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

# Additional Information
* Related to # (issue)
#7460
add init group during exp manager
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Fixes https://github.com/NVIDIA/NeMo/issues/7460. Exp manager now initializes distributed groups to force a barrier.

# Changelog 
- Add specific line by line info of high level changes in this PR.


  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

# Additional Information
* Related to # (issue)
#7460
fix a typo
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix a typo.

**Collection**: NLP

# Changelog 
Fix a typo in the comment part in `nemo/collections/nlp/data/language_modeling/megatron/helpers.cpp`.

# Usage
Do not affect any usage

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
How do I make nemo log only on rank 0?
State: closed | Pull request: no
Labels: name: bug

I can't figure out how to make the logging not repeat on 512 ranks. Is there a setting to tell nemo to log only on rank 0? at the very least for the iteration logs?

I currently have:
```
  exp_manager.create_dllogger_logger: True
```
but I'd happy with any other way.

What I ideally would like is the original Megatron-LM-like style logging where it logs iteration metrics from rank 0.

Thank you!
Fix TTS FastPitch tutorial
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] bugfix: trainer.accelerator=auto from None.
State: closed | Pull request: yes
Labels: name: TTS

name: cherry-pick

# What does this PR do ?

fixed default value for `trainer.accelerator` after upgading PTL 2.0.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] bugfix: trainer.accelerator=auto from None.
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

fixed default value for `trainer.accelerator` after upgading PTL 2.0.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS][SpeechLLM] Merge ASR SFT changes to SpeechLLM T5
State: closed | Pull request: yes
Labels: name: TTS

name: NLP

name: CI

# What does this PR do ?

ASR SFT related code changes.

**Collection**: [Note which collection this PR will affect]
NLM

# Changelog 
At some places the flow needs to go into an if-else statement based on the task at hand - ASR or TTS.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix errors caused by different dimensionality in get_dist function
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

name: NLP

name: Speaker Tasks

name: CI

# What does this PR do ?

The TTS Aligner's `get_dist()` now has an extra dimension (from PR #6806); this PR fixes errors stemming from this in the Aligner notebook.

@rlangman -- was the change from returning a (B, C, T)-dimensional tensor to a  (B, 1, C, T)-dimensional one intentional? Do we want to keep that or change it back by adding a squeeze(1)?

**Collection**: TTS
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Fix tts fastpitch tutorial
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: CI

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Safeguard nemo_text_processing installation on ARM
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

name: NLP

# What does this PR do ?

Removes nemo_text_processing as a requirement on ARM.

Some scripts will fail on ARM due to missing nemo_text_processing

**Collection**: TTS

# Changelog 
- removes nemo_text_processing as a requirement on arm machines

# Usage
N/A

**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
List index out of range when transcribing with RNNT conformer large when batch size > 1
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Hello, although #7192 fixed most of our errors, we are still running into some specific cases where the model still fails with the following stack trace.
![image](https://github.com/NVIDIA/NeMo/assets/10922796/db248328-3490-43e6-9f3a-fa667f955f57)

I also noticed that this error only happens when the batch size is higher than 1, while the model transcribes fine with batch_size 1, although of course being slower with a lot of audio files.
The weird part is that this only seems to happen with a combination of certain audio files, as in transcribing 1 of the 2 files 20 times doesnt give an error.

**Steps/Code to reproduce bug**

Luckily I was able to reproduce this error in the English model.

I used a new docker ( nvcr.io/nvidia/pytorch:23.04-py3 ).

apt-get update && apt-get install -y libsndfile1 ffmpeg
git clone https://github.com/NVIDIA/NeMo
cd NeMo
./reinstall.sh

The 2 audio files can be downloaded here: https://easyupload.io/m/980zua

Next I ran the following script:

import nemo
import nemo.collections.asr as nemo_asr
import copy
from omegaconf import DictConfig, OmegaConf, open_dict
from nemo.collections.asr.parts.utils.asr_confidence_utils import ConfidenceConfig

conformer = nemo_asr.models.EncDecRNNTBPEModel.from_pretrained(model_name="stt_en_conformer_transducer_large")

decoding_config = copy.deepcopy(conformer.cfg.decoding)
confidence_cfg = ConfidenceConfig(preserve_word_confidence=True, preserve_token_confidence=False)

with open_dict(decoding_config):
    decoding_config.preserve_alignments = True,
    decoding_config.confidence_cfg = confidence_cfg

conformer.change_decoding_strategy(decoding_config)

audio_segment_name_list = [ '/workspace/FailingCuts/12.wav', '/workspace/FailingCuts/13.wav']
hypotheses = conformer.transcribe(paths2audio_files=audio_segment_name_list, return_hypotheses=True, batch_size=2)

**Expected behavior**

Transcription succeeding without errors when using a batch size higher than 1

**Additional context**

Example: GPU model

Ran on a 4090 GPU.

@GNroy, would you be able to look at this again?
Cache FP8 weight and transpose only at the first micro-batch in each validation and test routine
State: closed | Pull request: yes
Labels: name: cherry-pick

name: stale

name: NLP

# What does this PR do ?

Cache FP8 weight and transpose only at the first micro-batch in each validation and test routine, which is the first micro-batch using the updated weights.
The following micro-batches within a validation or test routine, use pre-cached FP8 weight and transpose. 

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix bug when loading dist ckpt in peft
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix sft chat dataset truncation
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix error for gpt sft chat dataset.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Read audio as int32 to avoid flac read errors
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

# What does this PR do ?

As described here https://github.com/bastibe/python-soundfile/issues/274 when reading flac files there is bug in libsnd in which it sometimes warps the audio, and as a result sometimes causes an 'Internal psf_fseek() failed.' error when reading.

The only known solution is to read audio as int32 instead of float32. AudioSegment class already has the logic to convert the int32 to float32 [-1, 1] representation during initialization, we just need a flag in segment_from_file() to utilize it.

**Collection**: [TTS]

# Changelog
- Add flag to segment_from_file() to specify dtype to read audio as (the AudioSegment will automatically convert it to float32 afterwards)
- Read audio in VocoderDataset as int32.
- Remove retrial logic which should no longer be necessary.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Add an option to disable manual GC during validation
State: closed | Pull request: yes
Labels: name: cherry-pick

name: stale

name: NLP

# What does this PR do ?

Disable manual GC during the validation routine. This is valid only when `gc_interval` is set larger than 0.
To disable, set `NEMO_MANUAL_GC_IN_VALIDATION=0`

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Finetuning hifigan
State: closed | Pull request: no
Labels: name: bug

name: TTS

name: stale

**Hydra conf path not found**

I have been trying to run fastpitch_finetune.ipynb and from inside the nemo docker container. While the code ran bug free till inference of fastpitch model, when I am trying to run the steps given to run hifigan_finetune.py it throws an error related to config path

Primary config directory not found.
Check that the config directory '/TTS/conf/hifigan' exists and readable

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Or 
path not present in hydra_search_config.


I have been running it on a vm instance with a t4 gpu.
Please reach out if you know why this is happening and how to resolve this.
I have checked hydra directory for this error and tried a few solutions from there the problem is persistent.
Use the eval_beamsearch_ngram.py and add the LM made the CER worse.
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
Hi 

I found that using eval_beamsearch_ngram.py to evaluate the language model (LM) could reduce the CER by how much, but I actually found that adding the LM made it worse.

I used pretrain am model=" Conformer-CTC-L-unified_char_ja-JP_6.0.nemo"
and pretrain LM model ="riva_zh_asr_set_2.1_4gram.bin"
my testing data is used commo

I use decoding_mode=greedy and get 
`[NeMo I 2023-09-21 15:44:32 eval_beamsearch_ngram:372] Greedy WER/CER = 43.38%/10.48%
`
but when i use 
decoding_mode=beamsearch_ngram 
decoding_strategy="flashlight"
 
`[NeMo I 2023-09-21 15:46:57 eval_beamsearch_ngram:237] WER/CER with beam search decoding and N-gram model = 99.80%/24.11%
[NeMo I 2023-09-21 15:46:57 eval_beamsearch_ngram:248] Oracle WER/CER in candidates with perfect LM= 43.36%/10.72%
[NeMo I 2023-09-21 15:46:57 eval_beamsearch_ngram:253] =================================================================================
[NeMo I 2023-09-21 15:46:57 eval_beamsearch_ngram:439] Best WER Candidate = 99.80% :: Beam size = 64, Beam alpha = 1.0, Beam beta = 0.0
[NeMo I 2023-09-21 15:46:57 eval_beamsearch_ngram:444] Best CER Candidate = 24.11% :: Beam size = 64, Beam alpha = 1.0, Beam beta = 0.0
[NeMo I 2023-09-21 15:46:57 eval_beamsearch_ngram:448] =================================================================================`

CER got worse.

**Describe the solution you'd like**
Here is the code I am using

and I found that after each sentence in preds_output_folder, an unrelated English word is appended.

外交与全球性议题E	-40.78262926785392
外交与全球性议题L	-41.35890473572654
外交与全球性议题I	-41.67608906475944
外交与全球性议题A	-41.78252627102775
外交与全球性议题D	-41.92528642861289
外交与全球性议题J	-42.08723165242118
外交与全球性议题K	-42.095088259858315


**Steps/Code to reproduce bug**

```
python eval_beamsearch_ngram.py nemo_model_file=/model/zh-TW/stt_zh_citrinet_1024_gamma_0_25.nemo \
       input_manifest=/manifests/zh-TW/commonvoice_test_manifest_processed.json \
       kenlm_model_file=/model/zh-TW/lm/riva_zh_asr_set_2.1_4gram.bin \
       preds_output_folder=./ \
       probs_cache_file=null \
       beam_width=[20] \
       beam_alpha=[1.0] \
       beam_beta=[1] \
       decoding_mode=beamsearch_ngram \
       decoding_strategy="flashlight"
```

**Expected behavior**

How to reduce CER while increasing the LM?
Cache FP8 weight and transpose only at the first micro-batch in each validation and test routine
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Cache FP8 weight and transpose only at the first micro-batch in each validation and test routine, which is the first micro-batch using the updated weights.
The following micro-batches within a validation or test routine, use pre-cached FP8 weight and transpose. 

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Distributed optimizer support for experimental FP8 tensors
State: closed | Pull request: yes
Labels: name: core

name: stale

name: NLP

# What does this PR do ?

This PR integrates with experimental `Float8Tensor`s from the [Transformer Engine float8tensor_experiments branch](https://github.com/sudhakarsingh27/transformerengine/tree/float8tensor_experiments). This allows the model to only store FP8 weight matrices. The distributed optimizer will store an FP32 master copy of the weights and will perform param all-gathers in FP8.

**Collection**: NLP

# Changelog 
- Add logic to initialize GPT with FP8 weight matrices
- Add distributed optimizer support for FP8 weight matrices, including FP8 param all-gathers

# Usage
Enable FP8 support:
https://github.com/NVIDIA/NeMo/blob/19a3b7015fe353199af97903df1814e3a470b503/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml#L169

Use Megatron-core model:
https://github.com/NVIDIA/NeMo/blob/19a3b7015fe353199af97903df1814e3a470b503/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml#L49

Set the optimizer to `distributed_fused_adam`:

https://github.com/NVIDIA/NeMo/blob/f8be40b75ee1f8437b56fcc9602dc2aaddfb0643/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml#L228

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

Pinging @sudhakarsingh27.

# Additional Information
* Depends on https://github.com/sudhakarsingh27/transformerengine/tree/float8tensor_experiments, but there are import guards to handle the case that it isn't available.
* Depends on https://github.com/NVIDIA/apex/pull/1723
* Depends on https://gitlab-master.nvidia.com/ADLR/megatron-lm/-/tree/run_fp8_poc_with_nemo/
* Behavior is unchanged for non-FP8 models.
* If any FP8 params are detected, overlapping is disabled for non-FP8 grad reductions and param all-gathers.