renwei2024/nemo-github-issues huggingface dataset:
Add safe extraction of nemo tar files
State: closed | Pull request: yes
Labels: name: core

name: Run CICD

# What does this PR do ?

Enables safe extraction of nemo tar files in the SaveRestoreConnector class. To avoid security risks of arbitraty file writes via maliciously crafted paths in the tar file,  the safe extraction checks:

1) The member paths within tar archive do not contain any absolute paths or path traversal components (like ../..)
2) The member files are extracted to the intended directory.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins
Update Llama Conversion Script to Support Llama3
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: CI

name: common

name: Multi Modal

# What does this PR do ?

support Llama3 model conversion.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage

```python
python scripts/checkpoint_converters/convert_llama_hf_to_nemo.py --input_name_or_path PATH_TO_HF_FOLDER --output_path converted_llama3_nemo.nemo

python scripts/checkpoint_converters/convert_llama_nemo_to_hf.py --input_name_or_path llama3.nemo --output_path converted_llama3_hf.bin
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.
Update gemma for trt-llm 0.9
State: closed | Pull request: yes

# What does this PR do ?

Updates gemma export for trt-llm 0.9

**Collection**: None

# Changelog 
- Update gemma export for trt-llm 0.9
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation



LGTM

@meatybobby Could you please approve the PR on the code changes tab? We need one approval to be able to merge the PR.

jenkins
`megatron_gpt_finetuning.py` does not work `max_epochs`
State: closed | Pull request: no
Labels: name: bug

name: stale

`examples/nlp/language_modeling/tuning/megatron_gpt_finetuning.py` ignores `trainer.max_epochs`.
Always refer to `trainer.max_steps` only.

Tested on: `nvcr.io/nvidia/nemo:24.03.framework`

Test case:
`trainer.max_steps=200` and `trainer.max_epochs=1` (187 steps)
　- 200 step job finished
`trainer.max_steps=200` and `trainer.max_epochs=5` (935 steps)
　- 200 step job finished

Am I missing something?

It is not possible to set `trainer.max_epochs` when you set `trainer.max_steps`, due to by default `trainer.max_epochs` is ignored.

To set `trainer.max_epochs` you must do it without `trainer.max_steps` but by default `trainer.max_steps=20000` is set.

You should find a `trainer.max_steps` setting that suits your needs.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Error while exporting to TensorRTLLM format - AttributeError: 'NoneType' object has no attribute 'get'
State: closed | Pull request: no
Labels: name: bug

name: stale

I could finetune the gemma model and merge the adapter with base model by following this official notebook
https://github.com/NVIDIA/GenerativeAIExamples/blob/main/models/Gemma/lora.ipynb
Now Tried the given code for TensorRTLLM model conversion
```
from nemo.export import TensorRTLLM
trt_llm_exporter = TensorRTLLM(model_dir="/llmmodels/gemma_pubmedqa_merged_trt_llm")
trt_llm_exporter.export(nemo_checkpoint_path="/llmmodels/merged_model/gemma_lora_pubmedqa_merged.nemo", model_type="gemma", n_gpus=1)
```
But it is not working and gives this error
![Screenshot from 2024-04-18 20-05-52](https://github.com/NVIDIA/NeMo/assets/41164884/5e828643-f23c-4f79-b3a1-bdd6287df6ca)
I am using the docker container for the fine-tuning task

@yaoyu-33 @pradeepdev-1995 Can I work on this issue?

It seems the **model_config.yaml** file is being created inside temp directory during exporting ,but the created model_config.yml file is empty. that's why the Nontype object is returning
I am using the given image  ```nvcr.io/nvidia/nemo:24.01.gemma``` .
@SarthakNikhal 

@okuchaiev  can you please check on this?

I am not using docker directly to run the container (```nvcr.io/nvidia/nemo:24.01.gemma```) on the GPU machine.
Instead, running these in Kubernetes managed cluster(version 1.27.9) in the following way

Download the docker image (```nvcr.io/nvidia/nemo:24.01.gemma```).
Push it to our container Registry.
deploy that image as a jupyter notebook.
work using that notebook.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Error in coverting Mixtral-7B hf checkpoint to Nemo
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
I am trying to run the example run PEFT with Mistral 7B provided in NEVIDI DOCS HUB. When trying to convert  Mistral-7B in huggingface format to nemo format. Got the error message

 ```
python3 /opt/NeMo/scripts/checkpoint_converters/convert_mixtral_hf_to_nemo.py --input_name_or_path=./mistral-7B-hf/ --output_path=mistral.nemo
/usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py:124: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
[NeMo I 2024-04-18 09:56:50 convert_mixtral_hf_to_nemo:150] loading checkpoint ./mistral-7B-hf/
Loading checkpoint shards: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 2/2 [07:21<00:00, 220.59s/it]
Traceback (most recent call last):
  File "/opt/NeMo/scripts/checkpoint_converters/convert_mixtral_hf_to_nemo.py", line 346, in <module>
    convert(args)
  File "/opt/NeMo/scripts/checkpoint_converters/convert_mixtral_hf_to_nemo.py", line 153, in convert
    nemo_config = load_config(model_args, tokenizer.vocab_file)
  File "/opt/NeMo/scripts/checkpoint_converters/convert_mixtral_hf_to_nemo.py", line 115, in load_config
    nemo_config.num_moe_experts = int(mixtral_config['num_local_experts'])
KeyError: 'num_local_experts'
```

**Steps/Code to reproduce bug**
I am using the ngc container [24.03
](nemo:24.03.framework.sif) . And following the steps listed on this page https://docs.nvidia.com/nemo-framework/user-guide/latest/playbooks/mistralpeft.html


Hi, which one do you want to use the mistral-7b or the mixtral-7b? if you want to use the mistral-7b please use convert_mistral_hf_to_nemo.py instead of convert_mixtral_hf_to_nemo.py. Thanks.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Update chatglm converter
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Use the `args.input_name_or_path` to set `tokenizer.type` .

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Why use two types of names? spe refers to the Google sentencepiece library tokenizer. bpe for SentencePiece tokenizer 
State: closed | Pull request: no
Labels: name: bug

We can see this annotation (scripts/tokenizers/process_asr_text_tokenizer.py) :
![image](https://github.com/NVIDIA/NeMo/assets/20372403/f6e6fc5b-a7d6-4444-95b7-2c566e001f59)

and than (nemo/collections/asr/parts/mixins.py):
![image](https://github.com/NVIDIA/NeMo/assets/20372403/518aa7ea-4e7f-42c8-97f4-e82c0f51d517)

spe refers to the Google sentencepiece library tokenizer
bpe for SentencePiece tokenizer 

Are they same tokenizer  ?

The process tokenizer script decides between libraries - spe for sentencepiece, wpe for hugging face wpe. Spe is chosen cause it is possible to have different libraries implement the BPE algorithm - for example we used yttm for a long time which supported BPE for machine translation. 

The Mixin class one refers to the algorithm and not the library. Here, we support two algorithms, BPE (sentencepiece) and WPE (HF). We could potentially habe supported yttm as a BPE algorithm provider, but chose not too since it's a deprecated library. 

As of now we have no intention to support Any tokenizer other than sentencepiece for ASR. Even Hugging face WPE is only used for academic purposes, not for production models 

> The process tokenizer script decides between libraries - spe for sentencepiece, wpe for hugging face wpe. Spe is chosen cause it is possible to have different libraries implement the BPE algorithm - for example we used yttm for a long time which supported BPE for machine translation.
> 
> The Mixin class one refers to the algorithm and not the library. Here, we support two algorithms, BPE (sentencepiece) and WPE (HF). We could potentially habe supported yttm as a BPE algorithm provider, but chose not too since it's a deprecated library.
> 
> As of now we have no intention to support Any tokenizer other than sentencepiece for ASR. Even Hugging face WPE is only used for academic purposes, not for production models

Got it, thank you for your reply!
Any2any inference fix langid
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: stale

name: ASR

name: NLP

name: Speaker Tasks

name: CI

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Hi, @Davood-M This request is aimed to fix the language id mismatch issue when conducting inference on any2any model. Could you please review and merge it to a proper branch?

jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Any2any training load lang
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: stale

name: ASR

name: NLP

name: Speaker Tasks

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Hi, @Davood-M (Neglect all previous pull request) This is the latest code for fixing language if mis-match issue for training any2any. Could you please help to review and merge it to a proper branch?


jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
change the condition for get qkv tensor from linear_qkv output in mcoremixin
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

linear_qkv could be fused or non-fused modules. Check the number of output from linear_qkv to grab qkv tensor. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Gemma bug
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix item 3 described in https://github.com/huggingface/transformers/pull/29402

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Load lang id any2any training mengruw
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: stale

name: ASR

name: NLP

name: Speaker Tasks

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Hi, @Davood-M I suggested some fix for the language ids mis-match issue when training any2any NMT model. Could you please help to review and merge it to a proper branch? We need the code for April release. Thank you in advance!

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Fix External CLIP Converter
State: closed | Pull request: yes
Labels: name: Multi Modal

# What does this PR do ?

Fix the key mapping for CLIP converter, and add a docstring.

**Collection**: [Multimodal]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add MegatronGPTDeployable for serving .nemo models in pytriton
State: closed | Pull request: yes
Labels: name: stale

name: Run CICD

# What does this PR do ?

We currently only support deploying TRTLLM models in the Triton inference server.  This MR adds the necessary support to let us deploy .nemo models to Triton as well.

**Collection**: deploy

# Changelog 
- Adds MegatronLLMDeployable class which implements ITritonDeployable

# Usage
* Usage is shown in the new tests/deploy/test_triton_deployable.py

```python
    megatron_deployable = MegatronLLMDeployable(args.nemo_checkpoint, args.num_gpus)
    nm = DeployPyTriton(
        model=megatron_deployable,
        triton_model_name=model_name,
        triton_model_version=1,
        max_batch_size=8,
        port=8000,
        address="0.0.0.0",
        streaming=False,
    )
    nm.deploy()
    nm.run()
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Enable Sequence Packing and Pipeline Parallel in NeVA
State: closed | Pull request: yes
Labels: name: NLP

name: Multi Modal

name: Run CICD

# What does this PR do ?

Enable Sequence Packing and Pipeline Parallel in NeVA

**Collection**: [Multimodal]

# Changelog 
- Added Sequence Packing dataset preparation script
- Update Neva Model and dataset to support packed dataset, also add args to turn on packed dataset
- Enable Pipeline parallelism

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins
Fix union type annotations for autodoc+mock-import rendering
State: closed | Pull request: yes
Labels: name: ASR

name: common

# What does this PR do ?

Sphinx's autodoc with mocked imports cannot handle `a | b` typing syntax when one of the arguments is a forward reference (used by autodoc to substitute imported classes). Introducing forward references also brakes the type dispatch mechanism in `@singledispatch`. I used the opportunity to fix a few other obvious errors that I saw in the long list of sphinx warnings.

**Collection**: ASR

# Changelog 
- Fix for documentation build of ASR collection.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Changes to enable CUDA graph for LLM
State: closed | Pull request: yes
Labels: name: core

name: cherry-pick

name: NLP

name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

CI Passed https://github.com/NVIDIA/NeMo/actions/runs/9387719229

CI passed https://github.com/NVIDIA/NeMo/actions/runs/9387719229
Update te install for jenkins
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Add config option for FP32 embedding grads
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

name: Run CICD

# What does this PR do ?

Add a config option to control whether the distributed Adam optimizer stores embedding grads in FP32.

**Collection**: NLP

# Changelog 
- Add config option for FP32 embedding grads

# Usage

Run GPT, e.g. with the config at https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml.

Enable the distributed optimizer with `model.optim.name=distributed_fused_adam`. FP32 embedding grads are enabled by default and can be disabled with `model.with_fp32_embedding_grads=False`.

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Currently only supported with GPT and T5.
* Support for FP32 embedding grads was recently refactored in https://github.com/NVIDIA/NeMo/pull/8792 and https://github.com/NVIDIA/NeMo/pull/8867.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

pipeline passed: https://github.com/NVIDIA/NeMo/actions/runs/9387711476
a2a fix removed tp world size and group from init
State: closed | Pull request: yes
Labels: name: cherry-pick

name: stale

name: NLP

name: Run CICD

# What does this PR do ?

a2a fix removed tp world size and group from init

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
mcore ds updates
State: closed | Pull request: yes
Labels: name: NLP

name: CI

name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Error response from daemon: unauhorized: authentication required
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**
![image](https://github.com/NVIDIA/NeMo/assets/28260618/74b49ff2-3f2e-4536-b136-3732d6bb3600)

When pulling Nemo mirror for training: 23.08.03 image, there is an issue with authentication required.

**Steps/Code to reproduce bug**


**Expected behavior**

How to solve this bug?



![image](https://github.com/NVIDIA/NeMo/assets/28260618/adda37c7-1620-4035-bd81-6b2ca02b399c)


After verifying the API key, the image still cannot be downloaded successfully！

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Conflict between precision and plugins arguments in Trainer
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When attempting to train a model using Nemo 2403, a `ValueError` is raised indicating a conflict between the `precision` argument and the `plugins` argument being passed to the Trainer. The error message states that both `precision=bf16-mixed` and `plugins=PipelineMixedPrecisionPlugin` were received and cannot be used together.

```python
Traceback (most recent call last):
  File "/opt/NeMo/examples/nlp/language_modeling/megatron_gpt_continue_training.py", line 167, in main
    trainer = Trainer(plugins=plugins, strategy=strategy, **cfg.trainer, callbacks=callbacks)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/argparse.py", line 70, in insert_env_defaults
    return fn(self, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 401, in __init__
    self._accelerator_connector = _AcceleratorConnector(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 134, in __init__
    self._check_config_and_set_final_flags(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 271, in _check_config_and_set_final_flags
    raise ValueError(
ValueError: Received both `precision=bf16-mixed` and `plugins=<nemo.collections.nlp.parts.nlp_overrides.PipelineMixedPrecisionPlugin object at 0x1551053e0d90>`. Choose one.
```

**Steps/Code to reproduce bug**

1. megatron_gpt_continue_training mixtral 8x22b with Nemo 2403 image.

**Expected behavior**

- This issue seems to be related to [Issue #8848](https://github.com/NVIDIA/NeMo/issues/8848), where a similar error was encountered when converting Mistral/Mixtral models to the Nemo format.
- [PR #8908](https://github.com/NVIDIA/NeMo/pull/8908) partially addressed this issue by removing precision args in the trainer due to a PyTorch Lightning (PTL) update. However, the changes were not applied to the training script.

@adamlin120 thanks for reporting the issue.

Can you please try https://github.com/NVIDIA/NeMo/compare/main...akoumparouli/update_megatron_gpt_cont_training and let me know if that fixes the issue for you? thanks.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Feature Normalization in the ASR preprocessor is too slow.
State: closed | Pull request: no
Labels: name: bug

Another issue I discovered while working on #8673 . 

Right now, when running Parakeet CTC 1.1B at batch size 32, the preprocessor takes about 10milliseconds (of course, this depends on the lengths of the inputs, but this was with librispeech test other). 90% of the time is taken by feature normalization, and the GPU is hardly active during that time.

<img width="580" alt="image" src="https://github.com/NVIDIA/NeMo/assets/4767568/26ccbafb-5810-437a-9bf4-4f44787aa379">

You can reproduce via running this:

```
nsys profile -c cudaProfilerApi $(which python) examples/asr/speech_to_text_eval.py  pretrained_name=nvidia/parakeet-ctc-1.1b dataset_manifest=/home/dgalvez/scratch/data/test_other_sorted_downward.json  \
batch_size=$batch_size  output_filename=test_other_decoded.jsonl  amp=$amp  amp_dtype=bfloat16  use_cer=false num_workers=1
```

The culprit is this code that sequentially does a for loop over the batch: https://github.com/NVIDIA/NeMo/blob/468d5b6d369733909524d42b80a514f33bc19263/nemo/collections/asr/parts/preprocessing/features.py#L65-L73

First of all, it's actually copying a single scalar value of seq_len to CPU three times per loop. If you have to do a GPU to CPU copy, please do it just once via `seq_len = seq_len.cpu()`. But the actual kernels being launched are likely to go faster if we just do a single batched operation. Note that the slowness of this sequential for loop operation only increases as you run at larger batch sizes. An extra 10ms of latency during streaming inference is not really acceptable. The person who wrote the code initially probably wasn't sure how to deal with a ragged tensor for doing mean and std reductions, but we can do it straightforwardly with torch.where() and a mask built from seq_len, without ever communicating with the CPU (which is important for cuda stream capture to a graph).

I will either fix this as part of #8673 or make a separate PR, if the change is larger and riskier than I initially think.

You can download the .nsys-rep file I took a screenshot of here:
[report6.nsys-rep.gz](https://github.com/NVIDIA/NeMo/files/15003685/report6.nsys-rep.gz)


Could you put it to a separate pr ? Let's keep prs small / medium sized and focussed so we can debug them if they cause issues later. 

Also, don't make the assumption that a GPU will always be there (or that cuda graphs are supported). We can have fast paths for those cases when GPU is available or cudapython 12.3 and above is installed via conda / pip but they aren't guaranteed. 

> [don't make the assumption] that cuda graphs are supported

It is good practice to avoid synchronizing with the CPU in general, with GPU code. If we do that, we can get cuda graphs "for free". No part of that is building in cuda graphs as a requirement to do execution.
fsdp tp > 1 bug fix
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins
Add config option for FP32 embedding grads
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a config option to control whether the distributed Adam optimizer stores embedding grads in FP32.

**Collection**: NLP

# Changelog 
- Add config option for FP32 embedding grads

# Usage

Run GPT, e.g. with the config at https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml.

Enable the distributed optimizer with `model.optim.name=distributed_fused_adam`. FP32 embedding grads are enabled by default and can be disabled with `model.with_fp32_embedding_grads=False`.

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Currently only supported with GPT and T5.
* Support for FP32 embedding grads was recently refactored in https://github.com/NVIDIA/NeMo/pull/8792 and https://github.com/NVIDIA/NeMo/pull/8867.

jenkins