renwei2024/nemo-github-issues huggingface dataset:
Test C++ runtime on demand in nemo_export.py to avoid possible OOMs
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

- Fix OOM issues that may happen for large models when both Python and C++ runtime are tested at once. The latter test is only launched when explicitly required by specifying `--test_cpp_runtime True`.

- Also, the MR enables C++ runtime test by passing `use_python_runtime=False` to `TensorRTLLM` exporter.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
SDXL improvements (and support for Draft+)   [DRAFT PR]
State: closed | Pull request: yes
Labels: name: core

name: NLP

name: Multi Modal

name: Run CICD

# What does this PR do ?

This PR adds additional functionality to SDXL class of models to support Draft+, and:
* bug fix for adapter control in LoRA (LinearParallelAdapter)
* conversion script for SDXL from huggingface weights to nemo (existing converter script doesnt work)
* additional inference config file to load converted hf weights
* added FSDP strategy for multimodal models
* function to return adapter config, similar to function that returns the adapter itself

Code definitely needs cleanup.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add support to change Multi task model prompt
State: closed | Pull request: yes
Labels: name: ASR

name: common

name: Run CICD

# What does this PR do ?

Adds the ability to change prompt format for multi task models

**Collection**: [ASR]

# Changelog 
- Add function change_prompt().
- Add tests for changing prompt format of function

# Usage

```python
from nemo.collections.asr.models import ASRModel
from nemo.collections.common.prompts.canary import CanaryPromptFormatter

model = ASRModel.from_pretrained("nvidia/canary-1b")

class CanaryPromptFormatterSubclass(CanaryPromptFormatter):
    NAME = "canary2"

prompt_defaults = asr_model.prompt.get_default_dialog_slots()
user_prompt = prompt_defaults[0]
slots = user_prompt['slots']
slots['source_lang'] = 'en'
slots['target_lang'] = 'en'
slots['task'] = 'asr'
slots['pnc'] = 'no'
model.change_prompt(prompt_format='canary2', prompt_defaults=prompt_defaults)
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[NeMo-UX] Switch to torch_dist as default distributed checkpointing backend
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[PyTorch] Add context parallel support for packed dataset in THD format
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

This PR adds context parallel support for packed dataset in THD format in NeMo in response to this TE PR: https://github.com/NVIDIA/TransformerEngine/pull/641. Currently, the TE PR requires each individual sequence length is divisible by (2*context_parallel_size).

# Changes

- Add support to split packed dataset across different CP ranks in a load balanced way
- Add necessary paddings to dataset during packing stage to make sure the individual sequence length is a multiple of 2*cp_size

**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation



You code indentation is not consistent, some places have 4 spaces, and other places have 2 spaces.
NeMo code has 4 spaces always, so please make sure all your code have 4-space indentation.

Thanks for fixing the comments, it looks much better now.
The total sequence length (size of t in THD format) is a constant, right? If so, we should have some padded tokens at the end? how those padded tokens are split across different CP ranks?

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

> # What does this PR do ?
> This PR adds context parallel support for packed dataset in THD format in NeMo in response to this TE PR: [NVIDIA/TransformerEngine#641](https://github.com/NVIDIA/TransformerEngine/pull/641). Currently, the TE PR requires each individual sequence length is divisible by (2*context_parallel_size).
> 
> # Changes
> * Add support to split packed dataset across different CP ranks in a load balanced way
> * Add necessary paddings to dataset during packing stage to make sure the individual sequence length is a multiple of 2*cp_size
> 
> **PR Type**:
> 
> * [x]  New Feature
> * [ ]  Bugfix
> * [ ]  Documentation

Hello, I have some problems with cp need seqlen divisibility world_size * 2. I see that you are padding data in the code, but this will cause the pad token id to enter the flash attention calculation. I don't know if this is correct.
[TTS][ja-JP] add g2p, dict, tokenizer.
State: closed | Pull request: yes
Labels: name: TTS

name: common

name: Run CICD

name: audio

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


The files that had updates are:
nemo/collections/common/tokenizers/text_to_speech/ipa_lexicon.py
nemo/collections/common/tokenizers/text_to_speech/tokenizer_utils.py
nemo/collections/common/tokenizers/text_to_speech/tts_tokenizers.py
nemo/collections/tts/g2p/models/i18n_ipa.py
tests/collections/common/tokenizers/text_to_speech/test_tokenizer_utils.py
tests/collections/common/tokenizers/text_to_speech/test_tts_tokenizers.py
nemo/collections/tts/g2p/models/ja_jp_g2p.py
scripts/tts_dataset_files/ja_JP/ja_JP_wordtoipa.txt
Add Python AIStore SDK to container and bump min Lhotse version
State: closed | Pull request: yes
Labels: name: CI

name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update TransformerEngine initialize_ub parameters
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

# What does this PR do ?

Update TransformerEngine and update paramters of initialize_ub function

**Collection**: [nlp]

# Changelog 
- Update parameter tp_size to tp_group in initialize_ub

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


TE PR that updated the parameters: https://github.com/NVIDIA/TransformerEngine/pull/901

CI passed (test that failed flaky & is now fixed): https://github.com/NVIDIA/NeMo/actions/runs/9702489435

This change is no longer needed with new TE changes: https://github.com/NVIDIA/TransformerEngine/pull/986
PL: Delete precision if using plugin. TODO switch to MegatronTrainerB…
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

Remove precision if using a plugin to the parameters of PL trainer. 
Need to switch to MegatronTrainerBuilder, but this week is tough.
# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix SDXL incorrect name in Docs
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
extend get_gpt_layer_modelopt_spec to support MoE
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

# What does this PR do ?

Extends modelopt spec to support MoE/Dense.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix mock data generation for legacy dataset
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

# What does this PR do ?

Fixes mock data generation for legacy dataset.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[NeMo-UX] minor logging bug fixes
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Makes two small improvements to the refactored exp_manager code
- Computes `files_to_move` within nemo logger setup to avoid erroneously moving files belonging to the existing run
- Fixes logger version to match that of the experiment so everything gets written to the same directory

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[NeMo-UX] Llama and Gemma
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Inflight nemo model export support
State: closed | Pull request: yes
Labels: name: feature

name: Run CICD

# What does this PR do ?
Adds inflight NeMo to TRTLLM v10 model conversion and engine refitting using device weights

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add page context fmha option in TensorRTLLM export
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Add page context fmha option in TensorRTLLM export

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[NeMo-UX] Adding context- & expert-parallelism to MegatronStrategy
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Speaker Diarization goes haywire due to small segments of audio
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I have a long audio of around 3 hours that spans multiple speakers. The speaker diarization label a single speaker when this audio is passed. When I break down into this audio in parts and pass each part separately, some of the parts get assigned speakers correctly but the rest of the portion has the same bug. I identified some 1 min chunks that when added in this audio cause the model to behave this way. I'm seeking possible explanations or solutions to this behavior since I believe that the model should be resilient enough.

I think this might be related to having lots of overlap and a good number of speakers resulting in exceeding Nemo's limit of 20 max speakers.

**Steps/Code to reproduce bug**

Test Speaker Diarization on the [audio]( https://21133.mc.tritondigital.com/OMNY_DAVECHUCKTHEFREAKFULLSHOW_P/media-session/ba27cae7-9871-409f-b206-2183ad4628fe/d/clips/58a22c8d-1810-4cd1-bce2-a849004eeede/f545c88c-6f30-414d-b85a-aff10125b149/17c6c4c0-70cb-4c39-b16b-b18000f52f46/audio/direct/t1717081258/Thursday,_May_30th_2024_Dave_Chuck_the_Freak_Full_Show.mp3?t=1717081258&utm_source=Podcast%26in_playlist%3D9857b359-5905-4672-a201-aff10125b173)

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: AWS
 - Method of NeMo install: pip install

**Environment details**

- AWS Linux 2
- PyTorch version: 2.3.1
- Python version: 3.10

**Additional context**

GPU model


Please note that clustering based speaker diarization is a type of self-supervised machine learning system, not a rule-based software. Thus, speaker diarization can generate incorrect results and such behavior should be regarded as the limitation in accuracy, not a type of bug.

Especially, there is no guarantee that the speaker diarization system would generate the same speaker assignment to the truncated shorter audio clips from the original audio clips.

Closing since there is no clear ways to avoid this case.
Unable to reproduce cache aware streaming results for Conformer that were there for Fastconformer.
State: closed | Pull request: no
Labels: name: stale

I trained Conformer-CTC using cache aware method. but unable to reproduce the results that were there with "Fastconformer cache aware streaming". By using Conformer-CTC cache aware method i get partial sub-words, words combined together like ("hello world" ==> herld) , in streaming output.

the same was the case when i deployed it in a RIVA pipeline.

but when i use the "transcribe" function i get proper transcription for the audio file.
@titu1994 @VahidooX
can you please enlighten on this.
When to use and how to use streaming models in .nemo format and in RIVA pipeline.
My question has some relevance with these issues
[https://github.com/https://github.com/NVIDIA/NeMo/discussions/7010]
[https://github.com/https://github.com/NVIDIA/NeMo/discussions/5284]

I will try to look into this issue next week. It can be due to something broken in the cache-aware streaming script. Have you tried some of the already pretrained cache-aware models? Do they work properly?

You may use cache-aware with nemo but as far as I know, they are not supported in Riva yet. It is a work in progress. For now, you may use nemo for streaming cache-aware models.

You may take a look at this tutorial on how to use cache-aware models:
https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/Online_ASR_Microphone_Demo_Cache_Aware_Streaming.ipynb

Would you please try this tutorial and let me know if it works properly or not for your model?



@VahidooX , I went through the tutorial already. I fine-tuned a model using the streaming config. When I infer the model offline it works, but when I put that model in a streaming setup I get the above results. I will attach videos in a moment

Is it a confirmer or fastconformer?
Have you tried this script?

https://github.com/NVIDIA/NeMo/blob/main/examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Can you support DoRA?
State: closed | Pull request: no
Labels: name: feature

name: stale

Hi Team,
Do you have any plans to support DoRA?

Paper: https://arxiv.org/abs/2402.09353
peft: https://github.com/huggingface/peft/pull/1474

Thank you


Hi @SJmdy 
Dora is a planned feature in the near future. We plan on supporting it once to transition to NeMo 2.0 is mature.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Alit/jamba
State: closed | Pull request: yes
Labels: name: TTS

name: stale

name: ASR

name: NLP

name: Multi Modal

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Nemo container for Nemotron 340B inference fails pytorch_lightning import
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Nvidia recently released Nemotron 340B. The [README on Huggingface provides a guide](https://huggingface.co/nvidia/Nemotron-4-340B-Instruct) for performing inference with this model. I followed the guide using the recommended container `nvcr.io/nvidia/nemo:24.01.framework`. 

However, this container is unable to import `pytorch_lightning` without errors. The inference script tries to import `pytorch_lightning`, which in turn attempts to verify it can import `torchvision` (which it fails to do). I suspect the error is due to the version of `torchvision` installed not being compatible with the `torch` version (`RuntimeError: operator torchvision::nms does not exist`). 

I have encountered similar errors related to importing `pytorch_lightning` in your older containers previously as well. It happens more frequently when alpha/nightly versions of python packages are installed. A lot of containers tend to have alpha/nightly versions of either `torch`, `torchtext` or `torchvision` installed. 



The error was due to a different pytorch version being installed outside of the container in `~/.local/lib/python3.10/site-packages`. Since my home directory was mounted the container prioritized this version over the one in the container and thus there was a mismatch between `torch` and `torchvision`. 

The local Pytorch version was installed by mistake... So make sure to delete or unmount any python installations that might interfere with packages installed in the container.
Support Canary parallel inference
State: closed | Pull request: yes
Labels: name: ASR

name: Run CICD

# What does this PR do ?

Support Canary  at transcribe_speech_parallel.py script

**Collection**: ASR

```python
#python3 ./examples/asr/transcribe_speech_parallel.py model=./canary-1b.nemo predict_ds.manifest_filepath=./manifest.json output_path=/tmp trainer.devices=-1
```

**PR Type**:
- [ V] New Feature
- [ ] Bugfix
- [ ] Documentation


## Who can review?

@pzelasko 

`+` like we talked offline, we have to work around global_rank being set incorrectly before `trainer.predict()` is called - the dataloader has to be initialized with the correct global_rank (and world_size) in order for lhotse's distributed sampler to work correctly

trainer.global_rank is apriori set by PTL in slurm environment. It does not require model to be built or it's functions to be called c

Is this a case where PTL cannot apriori detect global rank ?

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

Jenkins

[🤖]: Hi @karpnv 👋,

I just wanted to let you know that, you know, a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11073753509) for this PR just finished successfully ✨

So it might be time to merge this PR or like to get some approvals 🚀

But I'm just a 🤖 so I'll leave it you what to do next.

Have a great day! 

//cc @ko3n1g

[🤖]: Hi @karpnv 👋,

I just wanted to let you know that, you know, a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11073753509) for this PR just finished successfully ✨

So it might be time to merge this PR or like to get some approvals 🚀

But I'm just a 🤖 so I'll leave it you what to do next.

Have a great day! 

//cc @ko3n1g

[🤖]: Hi @karpnv 👋,

I just wanted to let you know that, you know, a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11073753509) for this PR just finished successfully ✨

So it might be time to merge this PR or like to get some approvals 🚀

But I'm just a 🤖 so I'll leave it you what to do next.

Have a great day! 

//cc @ko3n1g
issue about self attention with mask
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When I tried to use the self-attention module, I encountered an issue with the mask.

In the file 'multi_head_attention.py', the code appears to handle the mask as follows when it is not None:

```python
if mask is not None:
    mask = mask.unsqueeze(1)  # (batch, 1, time1, time2)
    scores = scores.masked_fill(mask, -10000.0)
    attn = torch.softmax(scores, dim=-1).masked_fill(mask, 0.0)  # (batch, head, time1, time2)
else:
    attn = torch.softmax(scores, dim=-1)  # (batch, head, time1, time2)
```

I was wondering if the code should be adjusted, or if we should convert the mask before using it:

```python
...
mask = mask.unsqueeze(1).eq(0)  # (batch, 1, time1, time2)
...
```

Appreciate your response.
How to export SLUIntentSlotBPEModel to ONNX 
State: closed | Pull request: no
Labels: name: stale

Hello guys, I'm dealing with this issue for days but still unable to find the key. I have followed export steps as this [link](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/core/export.html). It works for ASR models, but when I need to export SLUIntentSlotBPEModel model, it shows this following error. Have anyone tried to export this model?
```jupyter-notebook
slu_model.export('/home4/khanhlt/nemo_api/onnx/nemo_slu.onnx')
``` 
```
{
	"name": "AttributeError",
	"message": "'SLUIntentSlotBPEModel' object has no attribute 'output_names'",
	"stack": "---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[3], line 1
----> 1 slu_model.export('/home4/khanhlt/nemo_api/onnx/nemo_slu.onnx')

File //home4/khanhlt/slu/Nemo_SLU_2/NeMo/nemo/core/classes/exportable.py:114, in Exportable.export(self, output, input_example, verbose, do_constant_folding, onnx_opset_version, check_trace, dynamic_axes, check_tolerance, export_modules_as_functions, keep_initializers_as_inputs)
    112 model = self.get_export_subnet(subnet_name)
    113 out_name = augment_filename(output, subnet_name)
--> 114 out, descr, out_example = model._export(
    115     out_name,
    116     input_example=input_example,
    117     verbose=verbose,
    118     do_constant_folding=do_constant_folding,
    119     onnx_opset_version=onnx_opset_version,
    120     check_trace=check_trace,
    121     dynamic_axes=dynamic_axes,
    122     check_tolerance=check_tolerance,
    123     export_modules_as_functions=export_modules_as_functions,
    124     keep_initializers_as_inputs=keep_initializers_as_inputs,
    125 )
    126 # Propagate input example (default scenario, may need to be overriden)
    127 if input_example is not None:

File //home4/khanhlt/slu/Nemo_SLU_2/NeMo/nemo/core/classes/exportable.py:191, in Exportable._export(self, output, input_example, verbose, do_constant_folding, onnx_opset_version, check_trace, dynamic_axes, check_tolerance, export_modules_as_functions, keep_initializers_as_inputs)
    189 input_list, input_dict = parse_input_example(input_example)
    190 input_names = self.input_names
--> 191 output_names = self.output_names
    192 output_example = tuple(self.forward(*input_list, **input_dict))
    194 if check_trace:

File ~/miniconda3/envs/nemo/lib/python3.10/site-packages/torch/nn/modules/module.py:1709, in Module.__getattr__(self, name)
   1707     if name in modules:
   1708         return modules[name]
-> 1709 raise AttributeError(f\"'{type(self).__name__}' object has no attribute '{name}'\")

AttributeError: 'SLUIntentSlotBPEModel' object has no attribute 'output_names'"
}

``` 



We don't support export of the SLU model 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.