renwei2024/nemo-github-issues huggingface dataset:
Upgrade to pytorch lightning 2.0
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: CI

name: common

# What does this PR do ?

This PR upgrades NeMo to use lightning 2.0. Majority of the changes are done based on the migration guide provided by lightning: [https://lightning.ai/docs/pytorch/stable/upgrade/from_1_9.html](url)

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Add tutorials for FastPitch TTS speaker adaptation with adapters
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Add two tutorials for FastPitch TTS speaker adaptation with adapters. 
One is FastPitch multi-speaker pretraining, the other is FastPitch adapter finetuning.

**Collection**: [TTS]

# Changelog 
- `tutorials/tts/FastPitch_MultiSpeaker_Pretraining.ipynb`
- `tutorials/tts/FastPitch_Adapter_Finetuning.ipynb.`

# Usage

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #6416 #6417
Upgrade pt 23.03
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Impulse perturbation normalization typo
State: closed | Pull request: no
Labels: name: bug

As this is my first issue, I MUST thank anyone contributing to this great repository, so I do :).

Without any context, [this](https://github.com/NVIDIA/NeMo/blob/50f16ba4f445e312b38b75a93881b9db2a10791c/nemo/collections/asr/parts/preprocessing/perturb.py#L377) normalization line seems wrong (first `min` should be `np.mean`).
`impulse_norm = (impulse.samples - min(impulse.samples)) / (max(impulse.samples) - min(impulse.samples))`

This totally changes the RIR signal and adds an overlapping echo to the output. Seems like nobody is using this augmentation :) or they didn't manually check the outputs.

An example RIR file from [BUT reverb dataset](https://speech.fit.vutbr.cz/software/but-speech-fit-reverb-database) is attached for anyone like myself who seeks a RIR example to check the augmentation.
[rir_1sec.zip](https://github.com/NVIDIA/NeMo/files/11230838/rir_1sec.zip)
check if grad is none before calling all_reduce
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

To enable PeFT training via adapter mixins, we should ensure all_reduce is not called on params' grads when the param.requires_grad is False. This does not happen for normal (full fine-tuning) but will happen for PeFT.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix CPU overheads of GPT synthetic dataset
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Move mask and position id generation in GPT synthetic dataset to __init__. They are same for each data loading, so doing it for every data loading is redundant, it also can create big CPU overheads while seq_len is long. Hence, move them to __init__.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Added/updated new Conformer configs
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Added some new configurations and updated some of the older configuration files for Conformer models.

# Changelog 
- Added configs for cache-aware fastconformer and hybrid ASR.
- Added more info on batch size and precision suggestions in the configs.

**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation
Add Frame-VAD
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

name: NLP

name: CI

name: common

# What does this PR do ?

Add Frame-VAD, including related datasets and models for multi-label audio classification. Also, ASR+VAD pipeline is updated to use Frame-VAD, and support for dropping frames instead of masking is added. 

**Collection**: [ASR]

# Changelog 
- Frame-VAD: 
    - Model: `nemo.collections.asr.models.classification_models.EncDecFrameClassificationModel`
    - Dataset: 
        - `nemo.collections.asr.datasets.audio_to_label.AudioToMultiLabelDataset`
        - `nemo.collections.asr.datasets.audio_to_label.TarredAudioToMultiLabelDataset`
        - `nemo.collections.asr.datasets.audio_to_label_dataset.get_audio_multi_label_dataset`
        - `nemo.collections.asr.datasets.audio_to_label_dataset.get_tarred_audio_multi_label_dataset`
    - Utils: `nemo.collections.asr.parts.utils.vad_utils.py` 
    - Example: 
        - `examples/asr/speech_classifications/speech_to_frame_label.py`
        - `examples/asr/speech_classifications/frame_vad_infer.py`
        - `examples/asr/conf/marblenet/marblenet_3x2x64_20ms.yaml`
- ASR+VAD:
    - Example:
        - `examples/asr_vad/speech_to_text_with_frame_vad.py`
    - Datasets:
        - `nemo/collections/asr/data/feature_to_text.py`
        - `nemo/collections/asr/data/feature_to_label.py`


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?  [Todo in next PR]
- [ ] Did you add or update any necessary documentation?  [Todo in next PR]
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
add rampup batch size support for Megatron GPT
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

This PR adds ramp up batch size support for GPT models. It allows to increase the batch size up to global batch size linearly during training in a given number of samples.

**Collection**: NLP

# Changelog 
- Add rampup_batch_size value to the training config.
- Input rampup_batch_size to the apex microbatch_calculator.
- Add current global_batch_size value to the progress bar when using rampup_batch_size.
- Update the apex microbatch_calculator after each training step.

# Usage
Can be set via config or CLI.

**Config:**
```python
rampup_batch_size: [8, 4, 10000] # Should be a list of 3 values: [<start_batch_size>, <batch_size_increment>, <rampup_samples>]
```
**CLI:**
```python
model.rampup_batch_size=[8,4,10000]
```
# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fixing the ability to use temp sampling with concat datasets
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

fixing the ability to use temp sampling with concat datasets

**Collection**: ASR

# Changelog 
In `audio_to_text_dataset.py` check for sampling probabilities only if the sampling technique is random

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Cache and reuse inputs when no reset
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Cache inputs (attention mask, loss mask, and position ids) and reuse.
This mitigates CPU-side overhead.

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
exceptions with empty test / val ds config sections
State: closed | Pull request: yes

# What does this PR do ?

Resolves exceptions when trying to train with empty test/ val ds config sections

**Collection**: ASR

# Changelog 
In `nemo/utils/model_utils.py` check if ds_values has values in it before trying to assign names to entries within. These names would have become visible in the test / validation sections of experiment manager (e.g. wandb) - but if these sections are emptty, there is no need to try.

# Usage
* You can potentially add a usage example below

```python
# should no longer fail:
python asr_transducer/speech_to_text_rnnt_bpe.py --config-path=../conf/contextnet_rnnt/ --config-name=config_rnnt_bpe.yaml model.tokenizer.dir=data/tokenizer/tokenizer_bpe_v1024/ model.tokenizer.type="bpe" model.train_ds.manifest_filepath=data/LibriSpeech/train_clean_100.json model.train_ds.batch_size=32 model.joint.fuse_loss_wer=true model.joint.fused_batch_size=8 model.decoding.greedy.max_symbols=5 ~model.validation_ds +accelerator="gpu" +strategy="ddp" +devices=4 trainer.max_epochs=1 exp_manager.checkpoint_callback_params.monitor="train_loss" "
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to bug 4067510
Bug 4067510
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

fixes exceptions when running with empty test /val ds sections

**Collection**: ASR

# Changelog 
in `/nemo/utils/model_utils.py`, check if len(ds_values) > 0 before trying to assign dataset names

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Refactor PP conversion + add support for TP only conversion
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Updates the TP PP conversion script to support TP only conversion (back ported from NeMo 1.16)
Refactor PP conversion mechanism.

**Collection**: [NLP]

# Changelog 
- Refactor components to functions, split apart TP and PP conversion components
- Refactor PP conversion to model specific classes for PP conversion support
- Backport TP only conversion support (irrespective of whether PP conversion is supported or not)

# Usage

```python

python megatron_change_num_partitions.py \
    --model_file="ckpt/megatron_gpt_model.nemo" \
    --target_file="ckpt/megatron_gpt_model_tp2_pp1.nemo" \
    --tensor_model_parallel_size=1 \
    --target_tensor_model_parallel_size=2 \
    --pipeline_model_parallel_size=1 \
    --target_pipeline_model_parallel_size=1 \
    --tp_conversion_only

```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
[TTS] FastPitch speaker encoder
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Adds FastPitch speaker encoder with reference audio and reference speaker embedding

**Collection**: TTS

# Changelog 
- Adds multi-speaker FastPitch pre-training with 1) Speaker verification model speaker embedding (reference speaker embedding) and 2) Global Style Tokens (reference audio)
    - `nemo/collections/tts/data/tts_dataset.py`
    - `nemo/collections/tts/torch/tts_data_types.py`
    - `nemo/collections/tts/models/fastpitch.py`
    - `nemo/collections/tts/modules/fastpitch.py`
    - `nemo/collections/tts/modules/submodules.py`
- Modified config and fine-tuning python script
    - `examples/tts/conf/fastpitch_speaker_adaptation.yaml`
    - `examples/tts/fastpitch_finetune_adapters.py`
 
# Usage

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] FastPitch adapter fine-tune and conditional layer normalization
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Adds FastPitch pre-training with CLNs and fine-tuning with adapters.

**Collection**: TTS

# Changelog 
- Adds multi-speaker FastPitch pre-training with Conditional Layer Normalization 
    - `nemo/collections/tts/modules/fastpitch.py`
    - `nemo/collections/tts/modules/transformer.py`
    - `nemo/collections/tts/modules/submodules.py`
- Add adapter modules for FastPitch fine-tuning
    - `nemo/collections/tts/models/fastpitch.py`
    - `nemo/collections/tts/modules/fastpitch.py`
    - `nemo/collections/tts/modules/transformer.py`
    - `nemo/collections/tts/modules/aligner.py`
    - `nemo/collections/tts/parts/mixins/__init__.py`
    - `nemo/collections/tts/parts/mixins/fastpitch_adapter_mixins.py`
- Add config and fine-tuning python script
    - `examples/tts/conf/fastpitch_speaker_adaptation.yaml`
    - `examples/tts/fastpitch_finetune_adapters.py`
- Fix aligner `nan` loss bug
    - `nemo/collections/tts/losses/aligner_loss.py`
 
# Usage

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix FastConformer config: correct bucketing strategy
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fix typo in bucketing strategy in FastConformer configs.

**Collection**: [ASR]

# Changelog 
- Add specific line by line info of high level changes in this PR.


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] whitelist broken path fix.
State: closed | Pull request: yes

# What does this PR do ?

fixed broken file path.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue): https://github.com/NVIDIA/NeMo/issues/6408
update container info in readme
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Is MoE implemented correctly?
State: closed | Pull request: no
Labels: name: bug

MoE FFNs are supposed to be sharded between GPUs. It seems like the current MoE implementation puts all FFNs on each GPU. See e.g. `https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/modules/common/megatron/mlp.py#L321`. Am I missing something here?
Config `share_embeddings_and_output_weights` does not work.
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

The code [here](https://github.com/NVIDIA/NeMo/blob/ecc116c29a14534abfe393f6eee4339b24ca6e57/nemo/collections/nlp/modules/common/megatron/module.py#L262) init the super class [MegatronModule](https://github.com/NVIDIA/NeMo/blob/ecc116c29a14534abfe393f6eee4339b24ca6e57/nemo/collections/nlp/modules/common/megatron/module.py#L46)  with default parameter `share_token_embeddings=True`, so the attribute `share_embeddings_and_output_weights` of the input `module` parameter will never be used in `Float16Module`. Is that a bug or designed?
Update hyperparameter recommendation based on experiments
State: closed | Pull request: yes

# What does this PR do ?

Update hyperparameter recommendation based on experiments

**Collection**: tools

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add documentation for preemption support
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Adds documentation for preemption support.
**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Logprob for stt_conformer_transducer_large
State: closed | Pull request: no

The Quartz model uses [logprobs](https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/asr/Offline_ASR.ipynb#scrollTo=ZhWmR7lbvwSm) , which are very helpful in assessing how well speech is spoken. They make it possible to construct a spectrogram, but it is inferior to the model stt_conformer_transducer_large. Is it possible to get log samples or something similar for this model? Maybe I don't know something and there are other ways to get probabilities for tokens in this model?Thanks in advance for the answer
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV) while trying to import from nemo.collections.asr.models import EncDecCTCModelBPE
State: closed | Pull request: no
Labels: name: bug

**I get an "Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)" when I trying to import nemo.collections.asr**

I would like to use NeMo model, but have an error Process finished with exit code 139 (interrupted by signal 11: SIGSEGV) at the asr importing step.

from nemo.collections.asr.models import EncDecCTCModelBPE

file_name = 'audio_test.wav'

model = EncDecCTCModelBPE.from_pretrained(model_name="stt_en_conformer_ctc_large_ls")
transcribed = model.transcribe(paths2audio_files=file_name, batch_size=1, logprobs=False)

Can successfully import nemo, collections, but asr causes this error.

How can I fix this problem?
remove gc collect bottleneck code
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

Remove bottleneck code in `perform_clustering` function

**Collection**: [ASR]

# Changelog 
- remove `gc.collect()` code since it makes too slow.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix prompt template unescaping
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Unescape the prompt template to allow providing escape sequences in the prompt template from the CLI.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Inconsistent documentation?
State: closed | Pull request: no
Labels: name: bug

name: stale

The documentation at [https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/parallelisms.html] states that

`NeMo Megatron supports 4 types of parallelisms (can be mixed together arbitraritly)`

The documentation at [https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/megatron.html#model-parallelism] states that

`Currently, NeMo only supports tensor model parallelism.`

This seems inconsistent. What kind of parallelisms are supported for training and eval? Thanks!
GPT Training process hangs during sanity checking dataloader with TP=2, PP=2 and bias=false
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

With TP=2 ,PP=2 and model.bias=false, GPT Training process hangs during dataloader sanity checking

**Steps/Code to reproduce bug**
Clone r1.15.0 realese branch and then run the offical GPT pretrain example with my trivial modification
```shell
git clone https://github.com/NVIDIA/NeMo.git
cd Nemo
git branch r1.15.0
bash reinstall.sh
cd examples/nlp/language_modeling
bash train.sh
``` 
Below is the content of train.sh:
```shell
MASTER_ADDR=${MASTER_ADDR:-"127.0.0.1"}
MASTER_PORT=29503

NUM_NODES=1
NODE_RANK=0
GPUS_PER_NODE=8

DATA_PREFIX="[1.0,/data/pretrain/openwebtext2/hfbpe_gpt_training_data_text_document]"

torchrun --nnodes ${NUM_NODES} --nproc_per_node ${GPUS_PER_NODE} \
    --master_addr ${MASTER_ADDR} --master_port ${MASTER_PORT} --node_rank ${NODE_RANK} \
    ./megatron_gpt_pretraining.py  \
    --config-name=megatron_gpt_config.yaml \
    trainer.devices=${GPUS_PER_NODE} \
    trainer.num_nodes=${NUM_NODES} \
    model.data.data_prefix=${DATA_PREFIX} \
    model.tensor_model_parallel_size=2 \
    model.pipeline_model_parallel_size=2 \
    +model.bias=false \
    model.bias_activation_fusion=false \
    model.bias_dropout_add_fusion=false \
``` 
As we can see, the modification here is just: 
- disable all the bias item of linear layer in GPT model
- diabale bias related CUDA kernal fusion, as we do not need bias yet

The program hangs during dataloader sanity checking at very early stage of training.

It is OK if I turn on the bias and the related kernal fusion, or if I set PP=1

Logically, there should be no relevance between bias and pipeline. So I think it is an unexpected behaviour.

**Expected behavior**

It is expected that NeMo is able to train very large model without any bias term in pipeline parallel mode, such as LLama-65B.

**Environment overview (please complete the following information)**

 - Environment location: Docker
 - Method of NeMo install: `bash reinstall.sh`
 - If method of install is [Docker], provide `docker pull nvcr.io/nvidia/nemo:23.01 ` & `docker run --shm-size 64g --network=host -v ${HOME}:${HOME} -v ${DATAROOT}:/data -it nvcr.io/nvidia/nemo:23.01 bash` commands used

```[tasklist]
```
Sharded manifests for tarred datasets
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Helps accelerate manifest loading via optional manifest sharding. This should speed up the loading of large manifests and reduce memory footprint - each worker loads just the shards that it needs.

**Collection**: ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
Use the same notation for manifests as for tarred audio files.  There should be a 1:1 correspondence between a sharded audio tar file and a manifest file.   The defer_setup flag must be set, otherwise torch.distributed will not be initialized during manifest processing and we will not be able to determine the dataloader length across all loaded manifest shards.
```
model.train_ds.tarred_audio_filepaths='audio__OP_0..511_CL_.tar'
model.train_ds.manifest_filepath='manifest__OP_0..511_CL_.json'
+model.train_ds.shard_manifests=true
+model.train_ds.defer_setup=true
```
# when generating the tarred manifest, do:
```python

python convert_to_tarred_audio_dataset.py \
    --manifest_path=<path to the manifest file> \
    --target_dir=<path to output directory> \
    --num_shards=<number of tarfiles that will contain the audio> \
    --max_duration=<float representing maximum duration of audio samples> \
    --min_duration=<float representing minimum duration of audio samples> \
    --shuffle --shuffle_seed=1 \
    --sort_in_shards \
    --shard_manifests  \
    --workers=-1


# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add back ptuning pp2 test
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Move Parallelism usage from Apex -> Megatron Core
State: closed | Pull request: yes
Labels: name: core

name: NLP

name: CI

# What does this PR do ?

This PR moves model parallelism in NeMo to use Megatron-core instead of Apex.

* All models use dataloader_iter in train and eval step (except eval for T5 p-tuning)
* All pre-training models that use a constant sequence length sample based on microbatches
* All downstream tasks - p-tuning, adapter etc - that depend on dynamic seq len use global batch based sampling

We still use Apex for microbatch calculator, some enums/utils [both to be soon shifted to core],  LayerNorm/other things with kernel


**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
remove gc bottleneck code
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Remove bottleneck code in `perform_clustering` function

**Collection**: [ASR]

# Changelog 
- remove `gc.collect()` code since it makes too slow.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Framework for PEFT via mixins 
State: closed | Pull request: yes
Labels: name: core

name: NLP

# What does this PR do ?

PTuning and Adapters/IA3 have been implemented in diverging styles. Adapters and IA3 uses the elegant adapter_mixins framework. This PR implements P-Tuning via the same adapter_mixins framework.

The high level goal is to unify all PEFT methods to use adapter_mixins. This is allow us to the GPTSFTDataset to be used for all future PEFT methods.

Currently this PR does not remove the GPTPromptLearingModel functionality, but once adaper-mixin based p-tuning has been tested we can deprecate GPTPromptLearingModel GPTPromptLearningDataset classes.

PTuning does not need to flexibility to insert prompt tokens anywhere in the input. This feature has been removed for simplicity.

Also added:
- mix-n-match PEFT scheme like `adapter_and_ptuning` can be easily extended to do `ia3_and_ptuning` or `lora_and_ptuning`
- grad clipping functionality for PEFT training
- LoRA adapters added for GPT


**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)