renwei2024/nemo-github-issues huggingface dataset:
Integration of Turn-Taking Models into Nemo Framework for Enhanced Realistic Conversations
State: closed | Pull request: no
Labels: name: stale

Since Nemo is a language-focused framework, I was wondering if it's on the roadmap or if there is a possibility to work with turn-taking models.

There is a lot of literature on this, and according to the current state of the art, I believe these kinds of models are indispensable for achieving realistic conversations. These models are based on predicting whose turn it is in the conversation, which allows for more realistic interactions. Specifically:

- It enables the decision of when to give a response to the user, and this range is dynamic, whether the user is thinking or immediately after they have finished a sentence.
- It can detect when the user intends to interrupt what the bot is saying.
- It also offers the possibility of backchannels, such as saying phrases like "yeah" or "uh-huh" while the user is speaking, which has been proven to result in longer conversations by the user.

My questions are as follows:
- Is this on the roadmap?
- There are several open-source models that implement this, for example, https://github.com/ErikEkstedt/VoiceActivityProjection. In this particular case, it is a simple PyTorch model. 
  * Can this model be converted to a Nemo model?
  * In our case, we also have a commercial version of Nvidia Riva. If we convert it to Nemo, could we then convert it to Riva? Could we deploy it on the Riva server, and would it be consumable with Riva clients?


Thank you and regards.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Akoumparouli EP nemo docs
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Will be merged as part of @yaoyu-33 's PR https://github.com/NVIDIA/NeMo/pull/9184. Closing.
Adding 'use_dynamo' option for export to use onnx.dynamo_export() instead of onnx.export()
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: Multi Modal

name: Run CICD

# What does this PR do ?

Adding 'use_dynamo' option for export to use onnx.dynamo_export() instead of onnx.export()
The options is still experimental, default is to use legacy export.
Also added overloadable dynamic_shapes_for_export method to Exportable - so networks could provide customized dynamic shapes (make dimensions relate to each other etc). 
Some changes are workarounds to accommodate current dynamo_export issues (I added comments where appropriate).
Changes to argument names of some forward_for_export() functions were necessary because dynamo_export uses those names directly instead of externally supplied names, and those names did not match what the rest of the code/tests expected.

# Usage:

```python
model.export(output=filename, use_dynamo=True)
```


# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

CI passed: https://github.com/NVIDIA/NeMo/actions/runs/9688953931
Use DP+CP groups as the FSDP sharding domain
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

# What does this PR do ?

Use `DP+CP` group as the FSDP sharding domain to incorporate the gradient reduction in CP group.

# Changelog 
- Add `CP` group when setting the proc group of FSDP

# Usage
n/a

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
RAG Pipeline
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add TN/ITN link in speech tools list
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add link to TN/ITN section of the docs in speech tools list and fix some other issues in (I)TN part of the docs

**Collection**: Docs

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
N/A

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ N/A ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ N/A ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix missing func for T5 model
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

# What does this PR do ?

Fix the missing `setup_mcore_distributed_parallel` for T5 m odel

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Pipeline passed (2 test that failed are flaky): https://github.com/NVIDIA/NeMo/actions/runs/9023302152
training config used for training stt_en_quartznet15x5
State: closed | Pull request: no
Labels: name: stale

This is not a feature request per se. It's more about seeking help :-)

Right now I am trying the `stt_en_quartznet15x5` in nemo and I can load it and run inference without any issues. I can also see the model config which includes the layers, post and preprocessing parts without any issue.

Now, I am trying to train such a model from scratch. I used the default config that comes with the nemo repo like the ones located [here](https://github.com/NVIDIA/NeMo/tree/main/examples/asr/conf/quartznet). However, this won't produce a model with a performance on par with the model that ships with nemo(`stt_en_quartznet15x5`). 

While I can see and find the model configs located [here](https://github.com/NVIDIA/NeMo/tree/main/examples/asr/conf/quartznet), I was wondering if there is a way to see the actual config used by NVIDIA to train the aforementioned model. Things like training configs, batches, epochs, learning rate, etc. that might have been changed in the updated config.

Thank you.

@ROZBEH generally we try to put config to suit for training on small number of GPUs but for training these models we might have used large number of GPUs, so parameters might not be the same. 

@sam1373 do you know the difference between current config and model trained?

I see - thanks @nithinraok 
Yeah knowing the actual config can be really beneficial. 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Precision Problem between nemo model and hugging face model
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

We are using nemo to training our large vision language model. When converting models from nemo format to hugging face format, we found that given the same inputs and weights, we get different outputs.

We test that after layer normalization, even using the same weights and input, outputs are different. 
Nemo use transformers engine and below code to calculate:
 
![飞书20240508-190258](https://github.com/NVIDIA/NeMo/assets/128022807/de2f959b-b6ce-4e13-ba81-272cbd5baaab)

And hugging face using pytorch
![image](https://github.com/NVIDIA/NeMo/assets/128022807/534d4b73-f991-47c2-9654-a6ed3d07f3f9)
 
And I also found there also exists little precision gap in rotational positional embedding、attention and FFN. 


**Expected behavior**

Is the precision gap caussed by different calculation operator? How can I fix that?

Thank you!


Hi, we are aware that some TE implementations won't generate identical results to those of HF (which uses native PyTorch). We use our fused version of operations to speed up training. It seems you are using the Llama model as a foundation model. NeMo thoroughly tests Llama models to ensure that even though the results are not bit-wise matching, the overall performance (benchmarks) is on par.

If you have more concerns about the behavior, please provide us with more details. What model are you converting, what command are you using, and how large is the gap? We can check whether the gap is reasonable.

> Hi, we are aware that some TE implementations won't generate identical results to those of HF (which uses native PyTorch). We use our fused version of operations to speed up training. It seems you are using the Llama model as a foundation model. NeMo thoroughly tests Llama models to ensure that even though the results are not bit-wise matching, the overall performance (benchmarks) is on par.
> 
> If you have more concerns about the behavior, please provide us with more details. What model are you converting, what command are you using, and how large is the gap? We can check whether the gap is reasonable.

Thanks for your patient reply. We also test that the precission gab doesn't affect perfoemance.
fix
State: closed | Pull request: yes
Labels: name: CI

name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Not found file "convert_mistral_hf_to_nemo.py" in /opt/NeMo/scripts/checkpoint_converters/ for Convert Mistral
State: closed | Pull request: no
Labels: name: bug

name: stale

I running from this document https://docs.nvidia.com/nemo-framework/user-guide/latest/playbooks/mistralsft.html
Step 1 pass
Step 2 Run the container using the following command --> pass
           Convert the huggingface model to .nemo model --> not pass

Remark:
My computer:
OS: Ubuntu 22.04LTS
Memory:32GB
GPU:RTX3090(24GB)



This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
(do not merge )Extend multimodal/speech_llm with lhotse, t5 and bestow supports
State: closed | Pull request: yes
Labels: name: NLP

name: Multi Modal

# What does this PR do ?

In multimodal/speech_llm, add lhotse dataloader support and two models, SALM-T5 and Bestow-GPT. Include example configs.


###  Main features under speech_llm
- Lhotse dataloader support for speech SFT in speech_llm
- SALM-style architecture with T5 LLM backbone
- Bestow-style architecture (cross-attention based) with GPT LLM backbone
### Minor edit in nlp collection: 
- megatron_base_model.py: handle the case tokenizer.type is not set
- megatron_lm_encoder_decoder_model.py: hanlde the case encoder_input is used 
- megatron_base_prompt_learning_model.py: group the llm init code under init_model function (follow the pattern from megatron_gpt_prompt_learning_model.py) so that it can be overwritten by subclass when needed
- megatron/utils.py: in gradient accumulation, handle the case where the batch size from dynamic bucketing is not divisible. This happens when using lhotse dataloader with batch_duration

**Collection**: [common,nlp,multimodal]


**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
[NeMo-UX] Add data module
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


CI (Nemo_CICD_Test ) passed: https://github.com/NVIDIA/NeMo/actions/runs/8993756408
[do-not-merge] Illustration how to run audio+text dataloading
State: closed | Pull request: yes
Labels: name: stale

name: ASR

name: common

# What does this PR do ?

The key config options to set are:

```
 +model.train_ds.input_cfg=/path/to/speech_text_input.yaml \
 +model.train_ds.use_multimodal_sampling=true \
 +model.train_ds.batch_tokens=65536 \
 +model.train_ds.token_equivalent_duration=0.08 \
 +model.train_ds.quadratic_factor=4096 \
 +model.train_ds.tarred_audio_filepaths=null \
 model.train_ds.manifest_filepath=null \
 model.train_ds.batch_size=null \
 model.train_ds.batch_duration=null \
 model.train_ds.use_lhotse=true \
 model.train_ds.use_bucketing=true \
 model.train_ds.num_buckets=30 \
```

The values of batch_tokens, token_equivalent_duration, and quadratic_factor need tuning.

where contents of `speech_text_input.yaml` are:

```yaml
 - type: group
    weight: 1.0
    input_cfg:
    - type: nemo_tarred
      manifest_filepath: /path/to/manifest__OP_0..127_CL_.json
      tarred_audio_filepaths: /path/to/audio__OP_0..127_CL_.tar
      tags:
        pnc: 'no'
        source_lang: en
        target_lang: en
        taskname: asr
      weight: 1.0
    - type: nemo_sft_jsonl
      paths: /data/nemo_megatron_sft_chat_qa_data.shuf.jsonl
      language: en
      weight: 1.0
```

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
enable matryoshka embedding learning
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

# What does this PR do ?

allows the option to learn embeddings using matryoshka embedding learning.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Nemo readme revisions
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Titanet-L Augmentation
State: closed | Pull request: no
Labels: name: stale

**Is your feature request related to a problem? Please describe.**

I want to train TitaNet with augmentation.
However, I do not know how can I prepare the rir_noise_manifest.json, file for training Titanet-L with rir noise augmentation.

**Describe the solution you'd like**

Provide a code snippet on how to add augmentation for TitaNet-L training.



I added the following to augmentor, and used the following snippet from online augmentation tutorial
```python
rir_data_path = f'{data_dir}/dataset'
!python {NEMO_ROOT}/scripts/dataset_processing/get_openslr_rir_data.py --data_root {rir_data_path}
rir_manifest_path = os.path.join(rir_data_path, 'processed', 'rir.json')
!head -n 3 {rir_manifest_path}
```
Then to use the augmentation I applied the following 
```python
audio_augmentations = dict(
    speed = dict(
        sr=16000,
        prob=0.3,
        resample_type='kaiser_fast',
        min_speed_rate=0.95,
        max_speed_rate=1.05,
    ),
    noise = dict(
        manifest_path=rir_manifest_path,
        prob=0.5,
        min_snr_db=0,
        max_snr_db=15,
    ),
)
finetune_config.model.train_ds.augmentor = audio_augmentations
```
Am I correct and thanks @okuchaiev 

Yes, code looks fine to me. But for impulse you should use impulse pertubation not noise pertubation. 
Sample can be found here: https://github.com/NVIDIA/NeMo/blob/6442bb67275759f5ece6bd5e366966216e050cfe/examples/speaker_tasks/recognition/conf/titanet-small.yaml#L14 


@nithinraok that's what I thought, However in Titanet-Large they use noise instead of impulse, and it says we are using impulse perturbation. So, does that mean in their training they made an error using RIR corpora for noise instead of pulse perturbation.
https://github.com/NVIDIA/NeMo/blob/6442bb67275759f5ece6bd5e366966216e050cfe/examples/speaker_tasks/recognition/conf/titanet-large.yaml#L14-L26

The paper statement:

![image](https://github.com/NVIDIA/NeMo/assets/18738089/6b21ef9e-c761-4ab9-98a3-163acc0fdfdf)

(just realized you are the first author x.x)
Thank you @nithinraok 

I don;t remember details exactly but as far I remember RIR corpora also has noise samples as well along with impulse responses, and  I have not added impulse section to this config file but was added to titanet-small config. 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
ci: Refactor `cicd-main`
State: closed | Pull request: yes
Labels: name: CI

name: Run CICD

# What does this PR do ?

Split our testsuite into smaller chunks and DRY our `cicd-main` workflow file.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add knob for load_directly_on_device
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

# What does this PR do ?

Add knob for `load_directly_on_device` distributed checkpoint loading flag in GPT config.

**Collection**: NLP

# Changelog 
- Add `model.dist_ckpt_load_on_device` knob for `load_directly_on_device` distributed checkpoint loading flag

# Usage
* set `model.dist_ckpt_load_on_device=False` to disable loading checkpoint weights directly on GPU

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Tokenizer suggestion for fine tuning cache aware streaming model
State: closed | Pull request: no
Labels: name: stale

Hi
I want to fine tune "stt_en_fastconformer_hybrid_large_streaming_multi" on custom data. 
In my dataset I have things like "Vitamin B12", "Code: c12r5", "hb1ac" etc
For these alphanumeric words:
1. Should I convert the above to "vitamin b twelve", "code c one two r five", "h b one a c" ..... for using the default tokenizer
2. Should I create a custom/new tokenizer for this?

If there is any other suggestion, please let me know.
Thank You

If you want to finetune using the original tokenizer, yes you'll need to normalize all numbers to spoken words. 

Changing tokenizer means you'll need a large amount of data to retrain the model, that is not suggested unless you have several thousand hours of speech to reach best results 

> If you want to finetune using the original tokenizer, yes you'll need to normalize all numbers to spoken words.
> 
> Changing tokenizer means you'll need a large amount of data to retrain the model, that is not suggested unless you have several thousand hours of speech to reach best results

How to use the original tokenizer? 
I also created [a discussion](https://github.com/NVIDIA/NeMo/discussions/9217) for this.


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Fix blogs redirects paths
State: closed | Pull request: yes

# What does this PR do ?

Fix redirects from blogs (fix of https://github.com/NVIDIA/NeMo/pull/9120)

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add MoE dispatcher type and token drop args in NeMo
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

# What does this PR do ?

Add MoE dispatcher type and token drop support in NeMo

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
            moe_token_dispatcher_type: alltoall( or allgather)
            moe_pad_expert_input_to_capacity: true
            moe_expert_capacity_factor: 1.0
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Hi @ericharper can I get a review on this PR? Thanks!
Add redirects to research.nvidia.com/labs/conv-ai/
State: closed | Pull request: yes

# What does this PR do ?
Use `mkdocs-redirects` plugin to redirect the gh pages website to https://research.nvidia.com/labs/conv-ai/. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- adds `mkdocs-redirects` to reqs
- add list of redirects (following usage guidance here https://github.com/mkdocs/mkdocs-redirects?tab=readme-ov-file#using)

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update gpt dataset config parameter for mock
State: closed | Pull request: yes
Labels: name: NLP

name: CI

name: Run CICD

# What does this PR do ?

Update parameter for gpt dataset config

**Collection**: [nlp]

# Changelog 
- mock parameter is auto set by mcore

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
chore: hello, world!
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


/cicd

/cicd

/cicd
video neva null image+video folder path fix
State: closed | Pull request: yes
Labels: name: Multi Modal

name: Run CICD

# What does this PR do ?

Fixing error when image or video loader attempts to load image/video folder that is set to null. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


CICD passed, only optional test did not pass. Can go ahead & merge