renwei2024/nemo-github-issues huggingface dataset:
Issue with pip installing on windows due to youtokentome dependency 
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Youtokentome was archived in April 2024. It has several issues when installing on windows related to cython and visual studio C++ MSVC.

**Steps/Code to reproduce bug**
Try workaround described on issues of youtokentome:
```pip install git+https://github.com/gburlet/YouTokenToMe.git@dependencies```

Encounter the following even though c++ msvc is installed:
     "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -Iyoutokentome/cpp "-IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\include" "-IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /EHsc /Tpyoutokentome/cpp/bpe.cpp /Fobuild\temp.win-amd64-cpython-311\Release\youtokentome/cpp/bpe.obj -std=c++11 -pthread -O3
      cl : Command line warning D9002 : ignoring unknown option '-std=c++11'
      cl : Command line warning D9002 : ignoring unknown option '-pthread'
      cl : Command line warning D9002 : ignoring unknown option '-O3'
      bpe.cpp
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\include\cstddef(11): fatal error C1083: Cannot open include file: 'stddef.h': No such file or directory
      error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.39.33519\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2

**Expected behavior**

Should be clean to install on windows using ```pip install nemo_toolkit['all']```

**Environment overview (please complete the following information)**

 - Environment location: Windows 11
 - Method of NeMo install: pip

[all] was never guaranteed to work on Windows. 
[asr] can and should work on windows just fine. 
Yttm has long been removed from NeMo #8322
Add assert for max_steps to be positive in MegatronGPTSFTModel
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Lhotse dataloading: RIR augmentation and nemo/tarred input support for RIR and noise aug
State: closed | Pull request: yes
Labels: name: common

name: Run CICD

# What does this PR do ?

Extensions in Lhotse dataloading augmentations:
* RIR augmentation support (currently requires lhotse RecordingSet input, see e.g. https://github.com/lhotse-speech/lhotse/blob/master/lhotse/recipes/rir_noise.py )
* Broader input type support for noise augmentation. 
  * When a path is provided, it auto-detects
    * YAML input spec (.yaml)
    * lhotse non-tarred CutSet (.jsonl/.jsonl.gz)
    * NeMo non-tarred manifest (any other extension)
  * Also possible to provide a dict with the same keys as used for specifying the main training data, e.g.:

```yaml
noise_path:
  manifest_filepath: my_noises/manifest__OP_0..4_CL_.json
  tarred_audio_filepaths: my_noises/audio__OP_0..4_CL_.tar
```

**Collection**: ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Added 2 papers
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Alit/griffin perf
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Efficient streaming decoding for RNN-T and TDT (support partial hypotheses)
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


@titu1994 if you could rereview at your convenience, that would be appreciated.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

@galv I reopened this PR. It's not yet ready for review (I will let you know once I finalize it), but the code is now working, and the performance matches the "greedy" approach.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Adding deprecation warning to native NeMo RETRO
State: closed | Pull request: yes
Labels: name: stale

name: NLP

name: Run CICD

# What does this PR do ?

Adding deprecation warning to native NeMo RETRO

# Changelog 
Adding deprecation warnings to: 

- examples/nlp/language_modeling/megatron_retro_eval_legacy.py
- examples/nlp/language_modeling/megatron_retro_pretraining_legacy.py
- nemo/collections/nlp/models/language_modeling/megatron_retrieval_model.py

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
nemo2riva export doesn't work with FastPitch
State: closed | Pull request: no
Labels: name: bug

name: stale

**Description**

When I try to export a model to Riva (using **nemo2riva==2.14.0**), the **FastPitch** model export doesn't work (meanwhile, the **HiFiGAN** export works wonderfully). 
I've tried various combinations of **onnx** and **onnxruntime**, combinations of **NeMo** and **nemo2riva** versions, but in the end, nothing works and the following error always pops up:

`onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Failed to load model with error: Invalid tensor data type 0.`

**Steps/Code to reproduce bug**

Using **NeMo** image (1.21.0, 1.22.0, 1.23.0) + installed **nemo2riva==2.14.0**:

`nemo2riva --out $(OUT_RIVA).riva $(NEMO_MODEL) --key tlt_encode --runtime-check`

**Additional info**

- Without `--runtime-check` you can get exported **.riva** model, but it doesn't work by the end (and the model size is huge, also)
- Connected issue without provided solution (was automatically closed) - https://github.com/NVIDIA/NeMo/issues/8569
- Connected issues from **nemo2riva** - https://github.com/nvidia-riva/nemo2riva/issues/38, https://github.com/nvidia-riva/nemo2riva/issues/36

**The failed results mentioned by the authors of the issue above can be obtained by**:
1. Using exported .riva FastPitch from **nemo2riva** without a `runtime_check` 
2. Downgrading **onnx_graphsurgeon** to version **0.3.27**

In such an environment, the build and deploy of rmir will occur, but at the start of the Riva server, there will be either an:
- **IR** compatibility issue (version 9, not 8)
- OR `-UNAVAILABLE: Internal: onnx runtime error 1: Load model from /data/models/riva-onnx-fastpitch_encoder-test/1/model.onnx failed:Invalid tensor data type 0.`

I've tried trained models from previous version of NeMo and newly trained within version 1.23.0 and etc.

As I understand, problem is in: [nemo2riva/nemo2riva/patches/tts/fastpitch.py at r2.15.0 · nvidia-riva/nemo2riva](https://github.com/nvidia-riva/nemo2riva/blob/r2.15.0/nemo2riva/patches/tts/fastpitch.py). Runtime didn't catch this additional params and runtime-check failing

hey?

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Flashlight decoder for ASR
State: closed | Pull request: no
Labels: name: stale

[At this page](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/asr_language_modeling.html#flashlight-flashlight) you describe how to use flashlight decoder by launching your python script. But I can't find any parameters it should have. In further NeMo versions you have [installation guide](https://github.com/NVIDIA/NeMo/blob/2331b063d6f0282fa79c32e4e780db6cc7ef19f2/scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh#L53) but no python script to use it.

Can you please help me with that? This feature (flashlight decoder) is still in progress or deprecated?



Yes it's supported, please find installation instructions here - https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/asr_language_modeling.html#n-gram-language-modeling

That bash script has to be run by users as is, rather than python to install flashlight 

I followed this guide but it refers to scrips that don't exist anymore. For example the following script is not relevant already:
```
python eval_beamsearch_ngram.py nemo_model_file=<path to the .nemo file of the model> \
       input_manifest=<path to the evaluation JSON manifest file \
       kenlm_model_file=<path to the binary KenLM model> \
       beam_width=[<list of the beam widths, separated with commas>] \
       beam_alpha=[<list of the beam alphas, separated with commas>] \
       beam_beta=[<list of the beam betas, separated with commas>] \
       preds_output_folder=<optional folder to store the predictions> \
       probs_cache_file=null \
       decoding_mode=beamsearch_ngram \
       decoding_strategy="<Beam library such as beam, pyctcdecode or flashlight>"
```
because there is no eval_breamsearch_ngram.py [here](https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py).

@karpnv could you update docs ?

NeMo of previous versions has it but there are no such parameters as `decoding_strategy`. I mean it can't be used as flashlight. And even in older versions of NeMo it's absent. For example in [1.18.0](https://github.com/NVIDIA/NeMo/blob/v1.18.0/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py)

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
scripts to convert HF lora to nemo
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

# What does this PR do ?

scripts to convert hugging face lora model into nemo.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
#  convert NeMo to HF:
# the script expects a adapter_config.json file which is standard in HF:
python scripts/checkpoint_converters/lora_converters/convert_nemo_to_canonical.py \
    --nemo_lora_path nemo_style_lora_model.nemo \
    --output_path ./canonical_style_lora_model.nemo \
    --hf_format --hf_config checkpoints/bin/adapter_config.json


#  convert HF to NeMo:
# /checkpoints/bin/ is a folder containing the HF lora checkpoint (usually named adapter_model.bin)
# and a HF lora config file (usually named adapter_config.json)
python scripts/checkpoint_converters/lora_converters/convert_hf_to_canonical.py \
    --hf_lora_path /checkpoints/bin/ \
    --output_path output_dir/converted_lora.nemo \
    --nemo_config model_config.yaml
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Tests passed: https://github.com/NVIDIA/NeMo/actions/runs/8977317099
Let me know if you want to merge
Add llama3 and distributed checkpoint support in NeVA
State: closed | Pull request: yes
Labels: name: NLP

name: Multi Modal

name: Run CICD

# What does this PR do ?

Add llama3 and distributed checkpoint support in NeVA

**Collection**: [multimodal]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add Apex import Guard for MM collection
State: closed | Pull request: yes
Labels: name: stale

name: Multi Modal


> Please capture the error and include in the ImportError so users will know exactly what's not being imported properly.

ImportError is raised at the model level not layer level. Most of the cases here are using Apex's `groupnorm`, I added the ImportError when instantiate the model, not for each layer we invoke groupnorm.

@ericharper do you think this is sufficient ?

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Include param ids in mutex timeout warning
State: closed | Pull request: yes
Labels: name: core

name: stale

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
CUDA memory profile
State: closed | Pull request: yes
Labels: name: core

name: NLP

name: Multi Modal

name: Run CICD

# What does this PR do ?

Profile CUDA memory breakdown. The reference is available in the below link.
https://pytorch.org/blog/understanding-gpu-memory-1/

The memory profile will generate a `pickle file` and a user can load the file in the below page.
https://pytorch.org/memory_viz

# Changelog 
- Added the option to profile a range of training steps to profile the memory usage
- Adjust the training step by the number of micro-batches steps per global step.

# Usage
Add the below knobs to the yaml run config.

```yaml
  memory_profile:                                                                      
     enabled: true                                                                      
     start_step: 2                                                                      
     end_step: 5                                                                        
     rank: 0                                                                            
     output_path: <path/to/out_file>
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Video Neva Pretraining + Inference Implementation
State: closed | Pull request: yes
Labels: name: NLP

name: Multi Modal

name: Run CICD

# What does this PR do ?

This PR enables users to train NeVa models on video data by slicing videos into specific number of frames. 

Co-authored-by: Pratyush Muthukumar pmuthukumar@nvidia.com, Vivian Chen xuanzic@nvidia.com, Slyne Deng slyned@nvidia.com.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Nemo CICD] Run on comment
State: closed | Pull request: yes
Labels: name: stale

name: CI

[Nemo CICD] Run on comment

cicd

cicd

cicd

cicd

/deploy

cicd

/deploy

/deploy

/deploy

cicd

/deploy

/deploy

/deploy

/deploy

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
[Nemo CICD] update pytorch 24.04
State: closed | Pull request: yes
Labels: name: stale

name: CI

name: Run CICD

[Nemo CICD] update pytorch 24.04

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Add option for mutex timeout in distributed optimizer backward hook
State: closed | Pull request: yes
Labels: name: core

name: cherry-pick

name: Run CICD

# What does this PR do ?

Modified version of https://github.com/NVIDIA/NeMo/pull/9084, to debug a hang at https://github.com/NVIDIA/NeMo/blob/f658b6f0445403c338c7371941b1fe644832df48/nemo/core/optim/distributed_adam.py#L131

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
Run GPT, e.g. with the config at https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml.

Enable the distributed optimizer with `model.optim.name=distributed_fused_adam` and set the timeout with `model.optim.lock_timeout=<seconds>`.

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
DataLoaderPrefetcher for quicker starts of training loops
State: closed | Pull request: yes
Labels: name: ASR

name: common

name: Run CICD

# What does this PR do ?

Removes/reduces the waiting on step 0 of a training loop for dataloading worker initialization. The waiting duration is specific to a given training setup and can range from a few seconds to a minute or two. An obvious factor is faster/slower disk/network, but also some settings e.g. using larger bucketing or shuffling buffers requires more time to pre-load data into memory. 

The main point is to trigger the actual initialization (which happens on `DataLoader.__iter__()`) at the time the dataloader is constructed, without blocking. Then it can pre-load the data concurrently to other stuff being initialized in training script and ideally it's ready to yield a mini-batch by the time the training loop starts.

**Collection**: ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


It causes a segmentation fault in GPU tests (but works fine in casual settings -- probably too many daemonic threads spawned in tests when enabled by default, or otherwise some global state conflict). Maybe I'll revisit this later but don't want to invest more time into it right now: closing.
Llama3 Conversion Script Update
State: closed | Pull request: yes
Labels: name: CI

name: Run CICD

# What does this PR do ?

Support Llama3 Hf <-> NeMo conversion

**Collection**: NLP/Llama


# Usage


```python
python scripts/checkpoint_converters/convert_llama_hf_to_nemo.py --input_name_or_path PATH_TO_HF_FOLDER --output_path converted_llama3_nemo.nemo

python scripts/checkpoint_converters/convert_llama_nemo_to_hf.py --input_name_or_path llama3.nemo --output_path converted_llama3_hf.bin
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [X] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Merging since only test failure is 1 known problematic test.
Cleanup deprecated files and temporary changes
State: closed | Pull request: yes
Labels: name: NLP

name: CI

name: Multi Modal

name: Run CICD

# What does this PR do ?

- Remove deprecated files from #8201
- Remove temporary changes due to #8590

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Pipeline passed: https://github.com/NVIDIA/NeMo/actions/runs/9018587937
Add option for mutex timeout in distributed optimizer backward hook
State: closed | Pull request: yes
Labels: name: core

# What does this PR do ?

Modified version of https://github.com/NVIDIA/NeMo/pull/9084, to debug a hang at https://github.com/NVIDIA/NeMo/blob/f658b6f0445403c338c7371941b1fe644832df48/nemo/core/optim/distributed_adam.py#L131

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
Run GPT, e.g. with the config at https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml.

Enable the distributed optimizer with `model.optim.name=distributed_fused_adam` and set the timeout with `model.optim.lock_timeout=<seconds>`.

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


@ShriyaPalsamudram Tested on Eos single-node, the updated code spits out a RuntimeError when the lock is not obtained within the given timeout period.

How were you able to create a timeout situation in one node? Did you set the timer to very very small value?
Comment baichuan test and update pr template
State: closed | Pull request: yes
Labels: name: CI

name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add option for mutex timeout in distributed optimizer backward hook
State: closed | Pull request: yes
Labels: name: feature

name: core

name: stale

# What does this PR do ?

This is to help debug a hang at https://github.com/NVIDIA/NeMo/blob/f658b6f0445403c338c7371941b1fe644832df48/nemo/core/optim/distributed_adam.py#L131

Add a one line overview of what this PR aims to accomplish.

**Collection**: NLP

# Changelog 
- Add option for mutex timeout in distributed optimizer backward hook

# Usage

Run GPT, e.g. with the config at https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml.

Enable the distributed optimizer with `model.optim.name=distributed_fused_adam` and set the timeout with `model.optim.lock_timeout=<seconds>`.

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information



Could we replace `self._lock` itself with the timeout-enabled one? The parent Apex distributed optimizer class also uses `self._lock`  (e.g., [here](https://github.com/NVIDIA/apex/blob/master/apex/contrib/optimizers/distributed_fused_adam.py#L832)) and we want to catch those as well if they take too long.

Testing a modified version in this draft PR: https://github.com/NVIDIA/NeMo/pull/9087

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.