renwei2024/nemo-github-issues huggingface dataset:
Refactor LLM pretraining examples
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Simplify LLM pretraining example scripts by moving common logic into a new `TrainerBuilder` and `exp_manager`.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- add a `TrainerBuilder` type that hides some common logic for setting up a `Trainer`
- ~~move logic to handle `resume_from_checkpoint` arg to `exp_manager`~~ Current disabling `resume_from_checkpoint` logic and leaving a TODO comment since its current behavior is not a desirable user experience. Will make a separate PR to improve this.
- utilize above refactors to reduce length of Megatron LLM pretraining example scripts

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation
- [x] Refactor

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Incorrect padding for get_seq_len of features.py
State: closed | Pull request: no
Labels: name: stale

https://github.com/NVIDIA/NeMo/blob/4d78737f612964ff9cce2120bb28ec94f874ab07/nemo/collections/asr/parts/preprocessing/features.py#L392C1-L392C32

I don't know what was intended here, but to me it's clear there is redundency or error, because `self.n_fft // 2 * 2` is the same as just `self.n_fft`. Perhaps it was intended to be the power of two (aka `2 ** 2`)?
```
# Assuming that center is True is stft_pad_amount = 0
pad_amount = self.stft_pad_amount * 2 if self.stft_pad_amount is not None else self.n_fft // 2 * 2
```
What was the intended purpose of the line? Perhaps we want n_fft to be even?
Changed vocoder dataset
State: closed | Pull request: yes
Labels: name: TTS

name: stale

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Support audio offsets in TTS data loaders
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

This PR contains all of the changes needed to support training TTS models with large, multi-utterance files.

Related features are all experimental, and the change is not backwards compatible as I had to change feature files from _.pt_ to _.npy_ so I could memory map them to avoid reading the entire feature file into memory.

**Collection**: [TTS]

# Changelog 
- Add support for "offset" and "duration" in manifest (similar to ASR) to allow reading a subset of audio in the data loader.
- Modify featurizer classes so that they save features as .npy files, and reads subsets of the files using numpy memory mapping when "offset" is present in the manifest.
- Add logic for computing pitch in small chunks for large audio files (to avoid out of memory issues).
- Add on-the-fly volume normalization to featurizers and data loader, as we can not do it as a preprocessing step for multi-utterance files. Volume effects feature computation, as energy is proportional to volume, and volume can change the voiced/unvoiced classification during pitch computation.
- Unit tests to validate that the saving/loading logic works as expected.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
NFA bugfix: remove any empty segments
State: closed | Pull request: yes

# What does this PR do ?

Bugfix to prevent errors being thrown if the substring specified as an `additional_segment_grouping_separator` is placed at the beginning or end of reference text, or if 2+ of these substrings are placed next to each other in the reference text.

**Collection**: tools/nemo_forced_aligner

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
No usage changes

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ N/A ] Did you write any new necessary tests?
- [ N/A ] Did you add or update any necessary documentation?
- [ N/A ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Remove nested TTS configs
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

In a separate PR (https://github.com/NVIDIA/NeMo/pull/6852) it was brought up that TTS is using nested hydra config, which is against NeMo 1 config file per model policy.

This PR removes nested config from our experimental FastPitch and HiFiGan in favor of a flat structure with duplicated code between 22kHz and 44kHz versions.

Our older non-experimental configs are left unchanged for backwards compatibility.

**Collection**: [TTS]

# Changelog 
- Remove nested configuration from FastPitch and HiFiGan configs using new data loader
- Add separate 22kHz and 44kHz examples for each model.
- Moved hifigan_data.yaml config to separate "hifigan_dataset" folder (not sure if that is a good name) now that it does not need to reuse the generator config files. 

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Fix incorrect embedding grads with distopt BF16 grad reductions
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: cherry-pick

name: ASR

name: NLP

name: Speaker Tasks

name: CI

name: common

# What does this PR do ?

In the case where we are training GPT with distopt, BF16 grads, and pipeline parallelism, we are not properly synchronizing the embedding grads between the first and last pipeline stage. The embeddings were being accumulated into BF16 buffers and the pipeline-parallel reduction was on FP32 buffers (so just all-reducing zeros). Note that we synchronize these buffers before the data-parallel reduction, so that is still being done properly in FP32.

This PR fixes the issue by eagerly accumulating the embedding grads into FP32 buffers.

**Collection**: NLP

# Changelog 
- Distopt accumulates grads for FP32 params into FP32 buffers.

# Usage

In the config file, set the optimizer with `name: distributed_fused_adam` and `grad_sync_dtype: bf16`.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Depends/includes https://github.com/NVIDIA/NeMo/pull/6886
freeze base mode on init during peft
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

adds a self.freeze() during peft model init to prevent memory spike during startup.

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
FSDP + (TP)
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Support FSDP + (Tensor parallelism) for Transformer models.

What's missing in the PR
1. Sharded checkpoint store and load (not critical at the moment)
2. Decouple the module granularity of full-activation recomputation and FSDP sharding to remove AG in recomputation
3. CPU initialization with TE enabled

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
multiple fields can form a context
State: closed | Pull request: yes
Labels: name: NLP

name: common

# What does this PR do ?

* Add support for multiple contexts and truncation_field
* Add support for truncation in the middle of prompt_template
* Arguments
  * Remove context_key
  * Remove separate_prompt_and_response_with_newline
  * Add truncate_method
* Change the truncation logic to the following
  1. Split prompt_template by placeholder
  2. Tokenize each part to ids
  3. Truncate each part
  4. Concatenate all part 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix default context size
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: cherry-pick

name: ASR

name: NLP

name: CI

name: common

# What does this PR do ?

There is a bug when performing RNNT/CTC decoding with hybrid models. 

By default `att_context_size` should be [-1,-1] which is what set as default, but if None is passed through cfg, it replaces the current value, which shouldn't be the case. 

**Collection**: ASR

# Changelog 
- change value only if  `cfg.att_context_size` is not None

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Fix default context size
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: CI

name: common

# What does this PR do ?

There is a bug when performing RNNT/CTC decoding with hybrid models. 

By default `att_context_size` should be [-1,-1] which is what set as default, but if None is passed through cfg, it replaces the current value, which shouldn't be the case. 

**Collection**: ASR

# Changelog 
- change value only if  `cfg.att_context_size` is not None

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
[DRAFT] DistributedOptimizer integration for distr ckpt
State: closed | Pull request: yes
Labels: name: core

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
How can I run NeMo for BERT-LARGE finetuning on SQuAD dataset for multinode multigpu configuration using Megatron-LM checkpoint?
State: closed | Pull request: no
Labels: name: stale

I have searched for documentation on how can I use NeMo for downstream tasks, I found this :
https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/v1.0.0/nlp/megatron_finetuning.html
However, I haven't found any steps to get started with that do I need to make any code changes so that I can run the code on multinode-multiple config?

Thanks
Rename `megatron_amp_o2` to `megatron_amp_O2` in source code to be consistent
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Rename `megatron_amp_o2` to `megatron_amp_O2` in source code to be consistent

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fixed main and merging this to r1.20
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

# What does this PR do ?

Fix vad_utils.py for r1.12.0 
Redoing https://github.com/NVIDIA/NeMo/pull/7113 for r1.20.0


**Collection**: ASR

# Changelog 
The lines that are treating "None" in vad_utils.py 

# Usage
Run Speaker_Diarization_Inference.ipynb

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in NeMo ASR

# Additional Information
* Related to # (issue)
Update notebook branch
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update notebook branch
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
SpellMapper / Missing generate_configs.sh
State: closed | Pull request: no
Labels: name: bug

I am currently looking into your code base and am wondering where to find the above-mentioned script that you are referring to [here](https://github.com/NVIDIA/NeMo/blob/2fa7abfb7311f1a4115dfa47afb80339697ec88d/examples/nlp/spellchecking_asr_customization/run_training.sh#L34C29-L34C29). I have already searched in your repo but can't seem to find it. 

Any help would be greatly appreciated.
Grouped query attention support
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Support grouped query attention for language modelling, with torch attention and flash attention 2.

**Collection**: NLP

# Changelog 
Support grouped query attention for language modelling, with torch attention and flash attention 2.
- Added parameter parse to enable GQA for gpt like models (change in nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py, nemo/collections/nlp/models/language_modeling/megatron/gpt_model.py)
- Added support for grouped query attention for language modeling (change in nemo/collections/nlp/modules/common/megatron/language_model.py, nemo/collections/nlp/modules/common/megatron/transformer.py, nemo/collections/nlp/modules/common/megatron/attention.py)
- Not implemented for flash attention triton yet, as it is marked as experimental
- Not implemented for xpos yet, as it is marked as experimental


# Usage
Configure num_kv_attention_heads in model config yaml file, e.g. add in examples/nlp/language_modeling/conf/megatron_gpt_config.yaml, in model configuration.

num_kv_attention_heads is optional configuraiton, if not configured or configured same as num_attention_heads, the existing multi head attention will be used instead. 

Requires num_attention_heads as same or multiple of num_kv_attention_heads (same -> multi head attention), num_kv_attention_heads as same or multiple of tensor_model_parallel_size. 

Same convention as flash attention, e.g. if Q has 6 heads and K, V have 2 heads, head 0, 1, 2 of Q will attention to head 0 of K, V, and head 3, 4, 5 of Q will attention to head 1 of K, V.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
Add updated fc ctc and rnnt xxl models
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: cherry-pick

name: ASR

name: NLP

name: Speaker Tasks

name: CI

name: common

# What does this PR do ?

Add updated FC-XXL RNNT and CTC models

**Collection**: ASR

# Changelog 
- Add pre trained models to code
- Add to docs

# Usage

```python
# FC Transducer BPE models
from nemo.collections.asr.models import EncDecRNNTBPEModel 
fc_xxl_rnnt = EncDecRNNTBPEModel.from_pretrained('stt_en_fastconformer_transducer_xxlarge')

# FC CTC BPE models
from nemo.collections.asr.models import EncDecCTCModelBPE
fc_xxl_ctc = EncDecCTCModelBPE.from_pretrained('stt_en_fastconformer_ctc_xxlarge') 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation
Add updated fc ctc and rnnt xxl models
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: Speaker Tasks

name: CI

name: common

# What does this PR do ?

Add updated FC-XXL RNNT and CTC models

**Collection**: ASR

# Changelog 
- Add pre trained models to code
- Add to docs

# Usage

```python
# FC Transducer BPE models
from nemo.collections.asr.models import EncDecRNNTBPEModel 
fc_xxl_rnnt = EncDecRNNTBPEModel.from_pretrained('stt_en_fastconformer_transducer_xxlarge')

# FC CTC BPE models
from nemo.collections.asr.models import EncDecCTCModelBPE
fc_xxl_ctc = EncDecCTCModelBPE.from_pretrained('stt_en_fastconformer_ctc_xxlarge') 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation
Fixed main and merging this to r1.20
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fix vad_utils.py for r1.12.0 
Redoing https://github.com/NVIDIA/NeMo/pull/7113 for r1.20.0


**Collection**: ASR

# Changelog 
The lines that are treating "None" in vad_utils.py 

# Usage
Run Speaker_Diarization_Inference.ipynb

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in NeMo ASR

# Additional Information
* Related to # (issue)
Fix import guard checks
State: closed | Pull request: yes

# What does this PR do ?
Fix import guards for flash attention and DALI

**Collection**: [ASR, NLP]

# Changelog 
- Add AttributeError support for import where classpath resolution fails.
- Add import guards to pynvml

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

Closes #7122
Refactoring to make MegatronGPTModel dataset building more flexible
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Refactoring to make MegatronGPTModel dataset building more flexible.

In particular this allows subclasses to use a different `build_datasets_strategy` without having to copy / paste the full `build_train_valid_test_datasets()` method.

**Collection**: `nlp`

# Changelog 
- Add a new attribute `build_datasets_strategy` to the constructor of `MegatronGPTModel` to make it easier to override the dataset building logic

# Usage

```python
from my_module import my_build_datasets_strategy

class MyClass(MegatronGPTModel):

    def __init__(self):
        super().__init__(build_datasets_strategy=my_build_datasets_strategy)
```

# Discussion

There are at least two potential controversial design choices, happy to discuss them here (marking PR as draft in the meantime):
1. Naming: the most "natural" name would be `build_train_valid_test_datasets_strategy`, but I felt it was starting to get annoyingly long, which is why I picked a shorter variant
2. Implementation: alternatively, we could (a) avoid changing the signature of `__init__()`, letting subclasses override `self._build_datasets_strategy` as they see fit (I didn't go this route because I tend to prefer customization options to be clearly exposed through the API), or (b) make it part of the config, using Hydra's instantiate feature to obtain the strategy (although I actually like this option better, I wanted to discuss it here before making changes to the config). [edit: see [comment below](https://github.com/NVIDIA/NeMo/pull/7123#issuecomment-1665501843) for a follow-up discussion]

I'll look into adding a test once the implementation is approved.

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation
Installation instructions should better indicate mandatory steps to make tests pass (or reinstall.sh needs an update)
State: closed | Pull request: no

**Is your feature request related to a problem? Please describe.**

I wanted to setup a dev conda environment for NeMo, so I followed steps at https://github.com/NVIDIA/NeMo/tree/main#from-source

Afterwards `pytest --cpu` was failing (before it could even run any test) with two errors:
* `module 'nvidia' has no attribute 'dali'`
* `No module named 'pynvml'`

**Describe the solution you'd like**

After manually installing both libraries with
* pip install --extra-index-url https://developer.download.nvidia.com/compute/redist --upgrade nvidia-dali-cuda120
* pip install pynvml

the tests were able to pass (`1420 passed, 304 skipped, 261 warnings`).

Ideally these libraries would be installed automatically by the `reinstall.sh` script.
RoPE length extrapolation with interpolation (#7005)
State: closed | Pull request: yes
Labels: name: stale

name: NLP

* Push changes



* Fixes



* add continue training script



* [WIP] nonlinear interp



* Fix



* override encoder_seq_len



* Remove nonlinear



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* sft with pi (#7006)

* sft with pi



* update values only if not None"



---------



* Address comments



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add info



* Empty



---------

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
In MegatronGPTModels, Creation of AutoCastTransformerLayer with TransformerEngine ignores chosen activation 
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I was training a MegatronGPTModel using `megatron_gpt_pretraining.py` and a machine with H100 chips. When I set `transformer_engine=true` I later noticed that my requested `activation=fast-swiglu` was ignored. 

I traced the issue back to the [transformer.py](./nemo/collections/nlp/modules/common/megatron/transformer.py) file, in the definition of the class `AutocastTransformerLayer`. In the initializer there is no `activation` parameter being accepted or passed into the layer. 

**Steps/Code to reproduce bug**

python megatron_gpt_pretraining.py model.activation=swiglu

**Expected behavior**

The model to use the SwiGLU activation function, which can be seen by the `layernorm_mlp.fc1_weight` parameter (or `mlp.dense_h_to_4h` in the non transformer_engine code) being of dimension `2 * config.ffn_hidden_size` .

**Environment overview (please complete the following information)**

 - Environment location: [Docker]
 - Method of NeMo install: [from source]. Please specify exact commands you used to install.

```bash
git clone https://github.com/NVIDIA/NeMo.git
DOCKER_BUILDKIT=1 docker build -f Dockerfile -t nemo:latest .
```
Pass `rotary_pos_emb` and `relative_position_bias` to TE
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Correctly pass `rotary_pos_emb` and `relative_position_bias` to TE.

**Collection**: [Note which collection this PR will affect]

[NLP]

# Changelog 
- nemo/collections/nlp/modules/common/megatron/transformer.py

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fixed small bug with NoisePerturbationWithNormalization
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fixes a small indexing bug inside the NoisePerturbationWithNormalization class in perturb.py

**Collection**: ASR

# Changelog 
- nemo/collections/asr/parts/preprocessing/perturb.py: added safety check to ensure you don't index into scalar value

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix links in Segmentation tutorial
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Set a default value when getting `rampup_batch_size`
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Set a default value when getting `rampup_batch_size`. Otherwise, configs without setting the value would result in errors.

**Collection**: [Note which collection this PR will affect]

[NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
online diarizer
State: closed | Pull request: no
Labels: name: stale

Hi!

I was trying an online diarizer with reference to this PR(https://github.com/NVIDIA/NeMo/pull/5609). I am using the Nemo Frame VAD and modified it to directly output vad_timestamps (e.g., tensor([[0.0000, 1.5200, 1.5300]])). Then, I call diarize_step function every two seconds, as I'm not quite sure what one "step" (1 frame) represents. However, I only get one speaker output each time, despite calling diarize_step multiple times. Could you please help me identify what might be going wrong? I will provide some relevant code snippets and logs below. Thank you in advance.

```python
MODEL_CONFIG = "/workspace/diar_infer.yaml" 
cfg = OmegaConf.load(MODEL_CONFIG)
online_diar = OnlineClusteringDiarizer(cfg)
filepath = "/workspace/test_60s.wav"
waveform, _ = torchaudio.load(filepath)
vad_model = fvad("/workspace/model/vad_multilingual_frame_marblenet.nemo", "/workspace/diar_infer.yaml")
print(f"waveform.size() {waveform.size())}")
online_diar.buffer_end = 60
scale = 2
for step in range(20):
    start = step * scale
    online_diar.frame_start  = start
    s =  start * 16000
    e = s + scale * 16000  #(step + 1) * scale * 16000
    cur_wav = waveform[:,: e]
    vad_timestamps = vad_model.infer(cur_wav[:, s: ]) + start
    print(f"vad_timestamps: {vad_timestamps}")
    diar_hyp = online_diar.diarize_step(cur_wav.squeeze(0), vad_timestamps)
    print(f"step: {step}, diar_hyp: {diar_hyp}")


```
[log_online1.txt](https://github.com/NVIDIA/NeMo/files/12179113/log_online1.txt)
Include the scripts for preprocessing OAST and unit tests for chat sft datasets
State: closed | Pull request: yes
Labels: name: NLP

In this PR.

1. include scripts for preprocess OAST datasets.
2. annotate the datasets using reward model.
3. clean up the data for training.
4. include the unit tests to make sure the Chat SFT datasets produced expected masks. 
5. add support in the UI to run attribute condition generations.
6. add HF dataset as JSON dataset backend.
`nemo.collections.asr.models.EncDecSpeakerLabelModel.verify_speakers()` excessive embeddings normalization
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

`nemo.collections.asr.models.EncDecSpeakerLabelModel.verify_speakers` has excessive embeddings normalization. First computed ambeddings are normalized here:
```
        # Length Normalize
        X = embs1 / torch.linalg.norm(embs1)
        Y = embs2 / torch.linalg.norm(embs2)
```
(https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/models/label_models.py#L510)
Then during similarity computing the dot product is divided by `X` and `Y` norms (which are mathematically identical to `1`; computationally very close to `1`):
```
        similarity_score = torch.dot(X, Y) / ((torch.dot(X, X) * torch.dot(Y, Y)) ** 0.5)
```
(https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/models/label_models.py#L513)

**Steps/Code to reproduce bug**

Just use `verify_speakers()`. The result is accurate, however doing something excessive doesn't look good.


**Expected behavior**

Don't normalize embeddings twice. Either normalize them first, or divide the dot product of (not normalized) embeddings by the norms. No sense to do both.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal]
 - Method of NeMo install: `pip install nemo-toolkit[asr]`.

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS: Ubuntu 22.04
- PyTorch version: 2.0.1
- Python version: 3.10.12