renwei2024/nemo-github-issues huggingface dataset:
Support flash decoding
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add flash decoding from `flash-attn>=2.2` in this [repo](https://github.com/Dao-AILab/flash-attention/tree/main). 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* Add `model.use_flash_attention=True` when doing inference
* Support with container `nvcr.io/nvidian/ac-aiapps/nemo:swa101623`

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Replace gpus with devices
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add NLPDDPStrategyNotebook and change trainer gpus to devices
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add NLPDDPStrategyNotebook and change trainer gpus to devices
State: closed | Pull request: yes
Labels: name: ASR

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
FSDP + Tensor Parallelism
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Supports torch FSDP with Tensor Parallelism

# Usage
Set `model.fsdp=True` to enable FSDP. Gradient reduction data type can be set using `model.fsdp_grad_reduce_dtype`.

FSDP mode is not compatible with the `Distributed Optimizer` and `O2 optimizer wrapper`, thus should set `model.megatron_amp_O2=False` and do not use `model.optim.name="distributed_fused_adam"`

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
context parallelism
State: closed | Pull request: yes
Labels: name: core

name: NLP

# What does this PR do ?

GPT training with long-context input (e.g., sequence length of 16, 32K, 64K) can easily overflow GPU memory with huge activations. Context parallelism splits long-context input along the dimension of sequence length, parallelizes partitioned sequence segments among multiple GPUs. In this way, each GPU only needs to store activations of a part of sequence length, so that we can avoid memory overflow.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Snake act
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

# What does this PR do ?

Add snake activation to Audio Codec

**Collection**: TTS/ASR

# Changelog 
- Add Snake Activation to activation file
- Optionally choose between ELU/Snake while building model
- Note: Prior checkpoints don't load as is without modifying variable names 


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
update text server to support compute logprobs
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

update text server to support compute logprobs

**Collection**: [nlp]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add fix for max time to quit trainer gracefully, without running validation
State: closed | Pull request: yes
Labels: name: bug

# What does this PR do ?

When max time is reached in training, we'll terminate gracefully without running validation. This allows us to quit as fast as possible after saving a last checkpoint.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add fix for max time to save checkpoint and quit trainer gracefully without running validation.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add nemo to mcore GPT conversion script 
State: closed | Pull request: yes
Labels: name: feature

# What does this PR do ?

Add a conversion script to convert legacy (non-mcore path) nemo checkpoint into mcore-path checkpoint for GPT models

Verification between legacy and converted mcore checkpoints: 
- Weights match exactly
- Model forward pass outputs are close (barring precision differences in TE).
- GPT eval produces identical results
- LoRA training converges with the same training and validation loss curves.  

**Collection**: NLP

# Changelog 
- Add conversion script

# Usage
See top of the script for instructions

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Multimodal merge
State: closed | Pull request: yes
Labels: name: core

name: NLP

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.
Add multimodal models in NeMo

**Collection**: [Note which collection this PR will affect]
Multimodal, Vision, NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Draft] Multimodal Merge
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: Speaker Tasks

name: CI

name: common

# What does this PR do ?

Add multimodal models in NeMo

**Collection**: [Note which collection this PR will affect]

NLP, Vision, Multimodal

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Need full cmudict  to ipa mappings
State: closed | Pull request: no
Labels: name: TTS

**Is your feature request related to a problem? Please describe.**

 in the NeMo/scripts/tts-dataset_files, there is a cmudict phoneme to ipa phoneme mapping file : cmudict-arpabet_to_ipa_nv22.08.tsv,  but this file contains only 42 cmudict phonemes, this seems not enough to translate all cmudict phonemes to ipa, is there any full version of this mapping file?

**Describe the solution you'd like**

A full mapping file so that I can convert  cmudict phonemes into IPA

**Describe alternatives you've considered**


**Additional context**
multimodal/speech_cv collection, second pull request for Audio-Visual Speech Recognition
State: closed | Pull request: yes

# What does this PR do ?

Update the speech_cv branch to support audio-visual speech recognition

**Collection**: [multimodal/speech_cv]

# Changelog 
- Update speech_cv branch
- Add audio-visual datasets
- Add audio-visual models: (AudioVisualEncDecCTCModelBPE, AudioVisualEncDecCTCModel, AudioVisualEncDecHybridRNNTCTCBPEModel, AudioVisualEncDecHybridRNNTCTCModel, AudioVisualEncDecRNNTBPEModel, AudioVisualEncDecRNNTModel)
- Add audio-visual modules (FusionModule, InterCTCMixin)

# Usage

```python
import nemo.collections.multimodal.speech_cv as nemo_cv
from omegaconf import OmegaConf

config_path = "model_config.yaml"
params = OmegaConf.load(config_path)

# hybrid CTC/RNN-T BPE audio-visual speech recognition model
model = nemo_cv.models.AudioVisualEncDecHybridRNNTCTCBPEModel(cfg=params.model)
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix code block typo in docs
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fix typo in code block in docs

**Collection**: Docs

# Changelog 
- Add indendation after `with open_dict(decoding_cfg):` in code block

# Usage
N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
merge modular speechllm implementation from modular_speechllm branch to main
State: closed | Pull request: yes
Labels: name: ASR

name: NLP

name: common

# What does this PR do ?

Include Modular SpeechLLM implementation for Sept. 2023 submission (SALM). The implementation integrates a pretrained FC and a frozen LLM, and can do ASR, AST, and in-context learning for keyword boosting.

The implementation is ptn2.0 compatible.

**Collection**: multimodal

# Usage

refer to `examples/multimodel/modular_speechllm/run_sft_audio_lm.py` and `examples/multimodel/modular_speechllm/eval_audio_gpt_lora.py`

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
fix ptl_bugs in slu_models.py
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

This PR added fixes for PTL2.0 related SLU bugs in r1.21.0: Val metrics logging, None dataloader issue

Collection:
ASR

# Changelog 
- correct logging of validation metrics and test_dataloader() function in nemo/collentions/asr/models/slu_models.py updated to handle cases where dataloader is None as done in #7531 

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?
@stevehuang52 @titu1994

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #7450
Expose --user-defined-symbols and --control-symbols in scripts/tokenizers/process_asr_text_tokenizer.py
State: closed | Pull request: no

`scripts/tokenizers/process_asr_text_tokenizer.py` creates the tokenizer with create_spt_model (`from nemo.collections.common.tokenizers.sentencepiece_tokenizer import create_spt_model`) which already includes those flags. can you just expose those options in the top level script for creating the tokenizer?
EncDecSpeakerLabelModel to have full/close to full reproducibility
State: closed | Pull request: no
Labels: name: stale

## Problem statement
I am training EncDecSpeakerLabelModel for speaker identification model , however I am **having difficulty on reproducibility** on model training.

Below code is part of the example from [HERE](https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb) with modification on seed.

## Main library 
Nemo version : v1.20.0
pytorch-lightning==1.9.4
torch==2.0.1


## Code
```python
import torch
import pytorch_lightning as pl
from pytorch_lightning.callbacks import ModelCheckpoint
from pytorch_lightning import seed_everything


accelerator = 'gpu' if torch.cuda.is_available() else 'cpu'
seed = 42

MODEL_CONFIG = os.path.join(NEMO_ROOT,'conf/titanet-large.yaml')
config = OmegaConf.load(MODEL_CONFIG)
checkpoint_callback = ModelCheckpoint(monitor='loss')
config.model.train_ds.manifest_filepath = train_manifest
config.model.validation_ds.manifest_filepath = validation_manifest
config.model.decoder.num_classes = 3
config.trainer.devices = 1
config.trainer.accelerator = accelerator
config.trainer.max_epochs = 5
config.trainer.strategy = None
config.model.train_ds.augmentor = None
config.trainer.deterministic = True

trainer_config = OmegaConf.create(dict(
    devices=1,
    accelerator=accelerator,
    max_epochs=5,
    max_steps=-1,  # computed at runtime if not set
    num_nodes=1,
    accumulate_grad_batches=1,
    enable_checkpointing=False,  # Provided by exp_manager
    logger=False,  # Provided by exp_manager
    log_every_n_steps=1,  # Interval of logging.
    val_check_interval=1.0,  # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
    deterministic=True,
))
print(OmegaConf.to_yaml(trainer_config))

# Fix random seed in torch , numpy and dataloader.
torch.use_deterministic_algorithms(True)
seed_everything(seed, workers=True)

# Training config setting into pytorch lightning trainer
trainer = pl.Trainer(**config.trainer)


speaker_model = nemo_asr.models.EncDecSpeakerLabelModel(cfg=config.model, trainer=trainer)

trainer.fit(speaker_model)
trainer.validate(speaker_model)
```


## Output results 
**1st run :** 
![image](https://github.com/NVIDIA/NeMo/assets/13691221/5dc5daf8-8350-430b-bae0-531fa55b2c5c)

**2nd run:**
![image](https://github.com/NVIDIA/NeMo/assets/13691221/3ed7b250-6ab3-4f8e-ab39-03ade1f1d66e)

## Expected output 

Both runs should produce identical validation macro or close to identificatal validation macro scores.
How to make checkpoints self-contained per step instead of per rank?
State: closed | Pull request: no
Labels: name: stale

In megatron-lm checkpoints are self-contained, so if I have a checkpoint for step 1000, all its files are in the same directory. However in nemo they are grouped by rank, rather than step.

This causes a huge problem when one needs to sync checkpoints to cloud and be able to delete them once this is done.

Is there a way to invert the checkpoint structure, so that instead of:

```
checkpoints/mp_rank_00/step=02250.ckpt
checkpoints/mp_rank_00/step=02500.ckpt

checkpoints/mp_rank_01/step=02250.ckpt
checkpoints/mp_rank_01/step=02500.ckpt
```
it'll save checkpoints like:
```
checkpoints/02250/mp_rank_00.ckpt
checkpoints/02250/mp_rank_01.ckpt

checkpoints/02500/mp_rank_00.ckpt
checkpoints/02500/mp_rank_01.ckpt
```
So now I can copy+delete `checkpoints/02250` and automate this process easily by discovering new checkpoint folders, rather than trying to scan multiple leaf points, which makes it much more complicated and not lending to simple manual deleting.

I guess that since PTL's Experiment Manager is used this is an artifact of having each gpu saving into its unique rank path.

Any suggestions to how to solve it to bring back megatron-lm style checkpoints?

Thank you!
move core install to /workspace
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

move core install to /workspace

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add few-shot in-context learning and MLP modality adapter
State: closed | Pull request: yes
Labels: name: NLP

name: common

# What does this PR do ?

Add few-shot in-context learning for inference and MLP modality adapter.

An example of manifest is:
```
{
"audio_filepath": ["1.wav","2.wav","3.wav"], 
"duration": [1.05,1.05,2.0],
"answer": "this was her dream as nearly as she could recall it", 
"question": "Following are examples of speech audios and their transcriptions. 
    Example 1: audio is |audio|, transcription is 'I have a dream'. 
    Example 2: audio is |audio|, transcription is ' I don't have a dream'. 
    Given the following audio |audio|, transcribe the audio into words."
}
``` 
The `question` field has special audio locator strings (`|audio|` in this case) to indicate the locations of audios specified by the list of audio files in the field `audio_filepath`. In the case of multiple audios, `duration` and `offset` should also be a list of float numbers.  The program will load the three audios and insert their extracted features into the corresponding locations specified by the audio locator in the order they appear in the list. The dataset config should have a field `audio_locator: |audio|`.



**Collection**: [multimodal]
Fix a nlp nb onnx
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Same issue and fix as in https://github.com/NVIDIA/NeMo/pull/7671

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix a nlp nb onnx
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Same issue and fix as in https://github.com/NVIDIA/NeMo/pull/7671

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add steerlm paper
State: closed | Pull request: yes

# What does this PR do ?

Add steerlm paper to publications

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Tkonuk/joint peft
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Added knob fot user buffer/tensor parallel communication overlap for …
State: closed | Pull request: yes
Labels: name: stale

name: NLP

…MCORE pass

# What does this PR do ?

Added a knob to pass from the Nemo Launcher to the Mcore pass model for the ub_tp_comm_overlap switch.

Reference PR: https://github.com/NVIDIA/Megatron-LM/pull/532
This fix will be used by this PR in the Megatron-LM repository.
Fix multi rank finetune for ASR
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

# What does this PR do ?

Fix multi node fine-tuning script by adding rank based delay

**Collection**: [ASR]

# Changelog
- Add some delay to other ranks for the pretrained model call

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Fix typo in audio codec config, encoder target
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

This PR fixes a typo in encoder target in `audio_codec_24000.yaml`.

**Collection**: TTS

# Changelog 
- Changed encoder target in the config file

# Usage

N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Replace strategy='dp'/None with 'auto'
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

name: NLP

name: Speaker Tasks

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Remove PUBLICATIONS.md, point to github.io NeMo page instead
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Remove PUBLICATIONS.md file and add a link to https://nvidia.github.io/NeMo/publications/ instead.

**Collection**: README

# Changelog 
- Remove PUBLICATIONS.md file
- add link to https://nvidia.github.io/NeMo/publications/ instead.

# Usage
N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Remove PUBLICATIONS.md, point to github.io NeMo page instead
State: closed | Pull request: yes

# What does this PR do ?

Remove PUBLICATIONS.md file and add a link to https://nvidia.github.io/NeMo/publications/ instead.

**Collection**: README

# Changelog 
- Remove PUBLICATIONS.md file
- add link to https://nvidia.github.io/NeMo/publications/ instead.

# Usage
N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
bugfix: pin nemo-text-process to fix Chinese normalizer error.
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Related to [bug report](https://nvbugswb.nvidia.com/NvBugs5/SWBug.aspx?bugid=4312419&cmtNo=), 

Chinese normalizer outputted wrong normalized text for the input, '双辽境内除东辽河、西辽河等**5**条河流', where we expect '双辽境内除东辽河、西辽河等**五**条河流'.

**Evidence**

- On the latest version, i.e. nemo-text-processing=0.2.0rc0, it always normalized it as '双辽境内除东辽河、西辽河等5条河流';

- On the previous version, i.e. nemo-text-processing=0.1.8rc0, it normalized it as expected text, '双辽境内除东辽河、西辽河等五条河流'.

**Detailed Log**

- 0.2.0rc0
```
In [1]: from nemo_text_processing.text_normalization.normalize import Normalizer

In [2]: normalizer = Normalizer(lang='zh', input_case='cased', overwrite_cache=True, cache_dir="./cache_dir")

In [3]: text = '双辽境内除东辽河、西辽河等5条河流'

In [4]: normalizer.normalize(text)
Out[4]: '双辽境内除东辽河、西辽河等5条河流'
```
- 0.1.8rc0
```
Downloading nemo_text_processing-0.1.8rc0-py3-none-any.whl (2.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 12.5 MB/s eta 0:00:00
Installing collected packages: nemo-text-processing
  Attempting uninstall: nemo-text-processing
    Found existing installation: nemo-text-processing 0.2.0rc0
    Uninstalling nemo-text-processing-0.2.0rc0:
      Successfully uninstalled nemo-text-processing-0.2.0rc0
Successfully installed nemo-text-processing-0.1.8rc0


In [1]: from nemo_text_processing.text_normalization.normalize import Normalizer

In [2]: normalizer = Normalizer(lang='zh', input_case='cased', overwrite_cache=True, cache_dir="./cache_dir")

In [3]: text = '双辽境内除东辽河、西辽河等5条河流'

In [4]: normalizer.normalize(text)
Out[4]: '双辽境内除东辽河、西辽河等五条河流'
```
**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix ptl_bugs in slu_models.py
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

This PR added fixes for PTL2.0 related SLU bugs in r1.21.0: Val metrics logging, None dataloader issue

Collection:
ASR

# Changelog 
- correct logging of validation metrics and test_dataloader() function in nemo/collentions/asr/models/slu_models.py updated to handle cases where dataloader is None as done in #7531 

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?
@stevehuang52 @titu1994

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #7450