renwei2024/nemo-github-issues huggingface dataset:
Fix CLIP transformer layer api
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Fix CLIP transformer layer api for a ci failure

**Collection**: [multimodal]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.collections.vision.modules.common.megatron.vision_transformer
nemo/collections/vision/modules/common/megatron/vision_transformer.py:168:0: C0301: Line too long (151/119) (line-too-long)
nemo/collections/vision/modules/common/megatron/vision_transformer.py:72:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/vision/modules/common/megatron/vision_transformer.py:85:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/vision/modules/common/megatron/vision_transformer.py:91:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/vision/modules/common/megatron/vision_transformer.py:351:0: C0115: Missing class docstring (missing-class-docstring)

-----------------------------------
Your code has been rated at 9.71/10
```

---

Thank you for improving NeMo's documentation!

[🤖]: Hi @yaoyu-33 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11938034904) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
Leave target_module as default in PEFT Recipes
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Leave target_module as default in PEFT Recipes. Only specify target_module as linear_qkv in performance recipes to match with published performance numbers. Otherwise leave it as default to match with other models.


**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


<!-- pylint-output-strict-mode-true -->

beep boop 🤖: 🚨 The following files must be fixed before merge!

---

Your code was analyzed with PyLint. The following annotations have been identified:

```

------------------------------------
Your code has been rated at 10.00/10
```

---

Thank you for improving NeMo's documentation!

<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```

------------------------------------
Your code has been rated at 10.00/10
```

---

Thank you for improving NeMo's documentation!

[🤖]: Hi @cuichenx 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11925171236) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
Add BERT Model To NeMo2.0
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


<!-- pylint-output-strict-mode-true -->

beep boop 🤖: 🚨 The following files must be fixed before merge!

---

Your code was analyzed with PyLint. The following annotations have been identified:

```

------------------------------------
Your code has been rated at 10.00/10
```

---

Thank you for improving NeMo's documentation!

<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```

------------------------------------
Your code has been rated at 10.00/10
```

---

Thank you for improving NeMo's documentation!

<!-- pylint-output-strict-mode-true -->

beep boop 🤖: 🚨 The following files must be fixed before merge!

---

Your code was analyzed with PyLint. The following annotations have been identified:

```

------------------------------------
Your code has been rated at 10.00/10
```

---

Thank you for improving NeMo's documentation!

[🤖]: Hi @suiyoubi 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12149267703) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
chore(beep boop 🤖): Bump `MCORE_TAG=b6866ae...` (2024-11-20)
State: closed | Pull request: yes
Labels: name: stale

name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=b6866aed34d28fc37e352f3bae53fa48ad150952`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.
Default eps
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


for testing, do not merge

<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.collections.llm.recipes.optim.adam
nemo/collections/llm/recipes/optim/adam.py:24:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/llm/recipes/optim/adam.py:64:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/llm/recipes/optim/adam.py:91:0: C0116: Missing function or method docstring (missing-function-docstring)

-----------------------------------
Your code has been rated at 8.00/10
```

---

Thank you for improving NeMo's documentation!

[🤖]: Hi @maanug-nv 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11922703396) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
Add dora recipes
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.collections.llm.api
nemo/collections/llm/api.py:599:0: C0116: Missing function or method docstring (missing-function-docstring)

-----------------------------------
Your code has been rated at 9.99/10
```

---

Thank you for improving NeMo's documentation!

<!-- pylint-output-strict-mode-true -->

beep boop 🤖: 🚨 The following files must be fixed before merge!

---

Your code was analyzed with PyLint. The following annotations have been identified:

```

------------------------------------
Your code has been rated at 10.00/10
```

---

Thank you for improving NeMo's documentation!

[🤖]: Hi @cuichenx 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11959837478) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
Huvu/t5 nemo2.0 recipes update
State: closed | Pull request: yes
Labels: name: stale

name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.collections.llm.recipes.t5_11b
nemo/collections/llm/recipes/t5_11b.py:232:0: C0301: Line too long (121/119) (line-too-long)
************* Module nemo.collections.llm.recipes.t5_220m
nemo/collections/llm/recipes/t5_220m.py:230:0: C0301: Line too long (121/119) (line-too-long)
************* Module nemo.collections.llm.recipes.t5_3b
nemo/collections/llm/recipes/t5_3b.py:232:0: C0301: Line too long (121/119) (line-too-long)

-----------------------------------
Your code has been rated at 9.76/10
```

---

Thank you for improving NeMo's documentation!

[🤖]: Hi @huvunvidia 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11980970621) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
move more deps
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
ci: Pass-through of `workflow_event`
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Inference code request
State: closed | Pull request: no
Labels: name: stale

Hi there,

I gone through the repo, and i ran the inference for speaker diarization on toy sample using system VAD, its giving the output. But for the recorded audio file its giving errors. can you provide the code for custom samples inference for system VAD.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
chore(beep boop 🤖): Bump `MCORE_TAG=f214627...` (2024-11-19)
State: closed | Pull request: yes
Labels: name: stale

name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=f214627ac3fb228b4079e3eb897302337e1e577f`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.
Update llama32 vision (mllama) use attention bias
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Fix a previous workaround in llama32 vision to use attention bias in attention calculation.

**Collection**: [multimodal]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


<!-- pylint-output-strict-mode-true -->

beep boop 🤖: 🚨 The following files must be fixed before merge!

---

Your code was analyzed with PyLint. The following annotations have been identified:

```

------------------------------------
Your code has been rated at 10.00/10
```

---

Thank you for improving NeMo's documentation!

[🤖]: Hi @yaoyu-33 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11979011761) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
Merge multiple "distributed LoRA checkpoints"
State: closed | Pull request: no
Labels: name: stale

**Is your feature request related to a problem? Please describe.**

TensorRT-LLM only accepts a single rank .nemo LoRA checkpoint (in the case of Llama 3.1 8b). Therefore, the only way to use my fine-tuned model with TensorRT-LLM backend is to merge my distributed LoRA checkpoints into the base model using the scripts/nlp_language_modeling/merge_lora_weights/merge.py script. However, that results in a lot of big models, if I want to do that for multiple downstream tasks/fine-tuned models.

More specifically, my checkpoints after training with TP=PP=2 look like (the contents of the megatron_gpt_peft_lora_tuning.nemo LoRA checkpoint file):
```
./                                                                                                                                                                           
./model_config.yaml                                                                                                                                                          
./tp_rank_00_pp_rank_000/
./tp_rank_00_pp_rank_000/model_weights.ckpt 
./tp_rank_00_pp_rank_001/
./tp_rank_00_pp_rank_001/model_weights.ckpt 
./tp_rank_01_pp_rank_000/
./tp_rank_01_pp_rank_000/model_weights.ckpt 
./tp_rank_01_pp_rank_001/
./tp_rank_01_pp_rank_001/model_weights.ckpt
```

**Describe the solution you'd like**

It would be nice if we could merge the distributed LoRA weights to a .nemo LoRA checkpoint file that only contains weights for a single rank. That way, the LoRA would be compatible with TensorRT-LLM even if training on multiple GPUs.

Thanks in advance!

Best regards,
John


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
P2p chunk size setting in nemo 2.0
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Add the env var (`NCCL_P2P_NET_CHUNKSIZE`) that adjusts P2P communication chunk size to nemo 2.0, which is critical to unload the buffering time from NCCL kernel execution time.

# Changelog 
Added `NCCL_P2P_NET_CHUNKSIZE` nemo 2.0 env var plugin

# Usage
Set `nccl_pp_comm_chunksize` in `PerfEnvPlugin`.


# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.lightning.run.plugins
nemo/lightning/run/plugins.py:33:0: C0301: Line too long (144/119) (line-too-long)
nemo/lightning/run/plugins.py:75:0: C0301: Line too long (145/119) (line-too-long)
nemo/lightning/run/plugins.py:127:0: C0301: Line too long (174/119) (line-too-long)
nemo/lightning/run/plugins.py:203:0: C0301: Line too long (150/119) (line-too-long)
nemo/lightning/run/plugins.py:65:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/run/plugins.py:105:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/run/plugins.py:150:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/run/plugins.py:213:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/run/plugins.py:265:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/run/plugins.py:284:4: C0116: Missing function or method docstring (missing-function-docstring)

-----------------------------------
Your code has been rated at 9.16/10
```

---

Thank you for improving NeMo's documentation!
chore(beep boop 🤖): Bump `MCORE_TAG=6c88bfc...` (2024-11-18)
State: closed | Pull request: yes
Labels: name: stale

name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=6c88bfce1c11b4fc80d2e6a3b13d5d80a66c0371`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.
chore(beep boop 🤖): Bump `MCORE_TAG=ce507ee...` (2024-11-17)
State: closed | Pull request: yes
Labels: name: stale

name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=ce507eeafd6a40027f623734d78a4d9595ee50a2`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.
Japanese model name for Forced Aligner
State: closed | Pull request: no
Labels: name: stale

Somebody tell me the name of the model to be used with the Forced Aligner for Japanese Language, the english one works but, I'm looking for a japanese one and latest with more accuracy, Thanks.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
chore(beep boop 🤖): Bump `MCORE_TAG=63b8520...` (2024-11-16)
State: closed | Pull request: yes
Labels: name: stale

name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=63b85200e970664c109c8e21e17803370547d354`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.
OOM with RAM with Lhotse
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

When training a model consuming more memory, I noticed that my training would stop after a constant number of epochs. Upon further investigation, I found that during training / validation, the memory usage of the CPU memory (RAM) continues to rise and never get released, which results in OOM after a number of epochs. This happens while I was using the Lhotse dataloader, while I was able to train a small fast-conformer CTC model for hundreds of epochs, the same version of NeMo only is able to run the training for an XL fast-conformer CTC model for ~20 epochs. ~110 epochs if I use 1/4 of the works for the dataloader. So somehow the dataloader is not releasing memory for the data loaded.

**Steps/Code to reproduce bug**

- shows up in XL fast-conformer CTC and medium fast-conformer CTC RNNT hybrid
- standard configs for these models in terms of hyperparameters
- other lhotse related configs:

```
trainer:
  devices: -1
  num_nodes: 1
  max_epochs: 150
  max_steps: 150000
  val_check_interval: 1000
  accelerator: auto
  strategy: ddp
  accumulate_grad_batches: 1
  gradient_clip_val: 1.0
  precision: bf16-mixed
  log_every_n_steps: 200
  enable_progress_bar: true
  num_sanity_val_steps: 1
  check_val_every_n_epoch: 1
  sync_batchnorm: true
  enable_checkpointing: false
  logger: false
  benchmark: false
  use_distributed_sampler: false
  limit_train_batches: 1000
 train_ds:
  manifest_filepath: null
  sample_rate: 16000
  batch_size: null
  shuffle: true
  num_workers: 8
  pin_memory: true
  max_duration: 45
  min_duration: 1
  is_tarred: false
  tarred_audio_filepaths: null
  shuffle_n: 2048
  bucketing_strategy: synced_randomized
  bucketing_batch_size: null
  shar_path:
  - xxxxx
  use_lhotse: true
  bucket_duration_bins:
  - xxx
    batch_duration: 600
    quadratic_duration: 30
    num_buckets: 30
    bucket_buffer_size: 10000
    shuffle_buffer_size: 10000
    num_cuts_for_bins_estimate: 10000
    use_bucketing: true
```

**Expected behavior**

Training should continue until specified stop.

**Environment overview (please complete the following information)**

 - GCP
 - Method of NeMo install: poetry: nemo-toolkit = {version = "2.0.0rc1", extras = ["asr"]}

**Environment details**

- nemo-toolkit 2.0.0rc1 / 2.0.0
- PyTorch-lightning 2.4.0
- PyTorch 2.2.2+cu121
- Python 3.11.8

**Additional context**

GPU: A100 40G.

@nithinraok ~~has told me to try using limit_validation_batches, use smaller duration audios, use fully_randomized which I haven't fully tested. I'll report back when these are tested but the issue persists so far~~

Edit: tried these suggestions, still OOM.


@pzelasko fyi

Is this related to https://github.com/lhotse-speech/lhotse/issues/518? @pzelasko 

I changed it to 1 worker and it is much more stable

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
remove redundant docs
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Remove redundant documentation.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Revert "fix(export): GPT models w/ bias=False convert properly"
State: closed | Pull request: yes
Labels: name: Run CICD

Reverts NVIDIA/NeMo#11255

This PR broke other models and was unnecessarily fixing gptnext when I should have just used llama

<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.export.trt_llm.tensorrt_llm_build
nemo/export/trt_llm/tensorrt_llm_build.py:31:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/trt_llm/tensorrt_llm_build.py:19:0: W0611: Unused Builder imported from tensorrt_llm.builder (unused-import)
nemo/export/trt_llm/tensorrt_llm_build.py:23:0: W0611: Unused add_lora imported from tensorrt_llm.models.modeling_utils (unused-import)

-----------------------------------
Your code has been rated at 9.32/10
```

---

Thank you for improving NeMo's documentation!

[🤖]: Hi @terrykong 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11863365040) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
ci: Exclude CPU machines from scan
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
ci: Deploy containers
State: closed | Pull request: yes
Labels: name: stale

name: CI

name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


[🤖]: Hi @ko3n1g 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/11894945859) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.
Missing BOS tokens for HF tokenizer
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

In the current implementation of HF AutoTokenizer, the tokenizer does not behave in the same way as the HF version. I noticed the problem with the BOS token (which is extremely important for the Gemma 2 model). Specifically, the problem occurs with `text_to_ids` function, https://github.com/NVIDIA/NeMo/blob/ed244d943948b105982f5cb17b51991e8acebfb7/nemo/collections/common/tokenizers/huggingface/auto_tokenizer.py#L222-L225 which is relevant for `preprocess_data_for_megatron.py`. The problem is that this function first transforms text to tokens and then tokens to IDs and consequently does not append the BOS token at the beginning as the HF encode method does (I assume the problem applies to other special tokens as well).

**Steps/Code to reproduce bug**

```
from nemo.collections.common.tokenizers.huggingface.auto_tokenizer import AutoTokenizer as ATNemo
from transformers import AutoTokenizer as ATHF


tokenizer_path = "google/gemma-2-9b"
hf_tokenizer = ATHF.from_pretrained(tokenizer_path)
nemo_tokenizer = ATNemo(tokenizer_path)

text = "Text to tokenize"

# Common HF tokenization
ids = hf_tokenizer.encode(text)
print("HF tokenization:", ids)

# NeMo tokenization
ids = nemo_tokenizer.text_to_ids(text)
print("NeMo tokenization:", ids)

# HF tokenization using NeMo steps
tokens = hf_tokenizer.tokenize(text)
ids = hf_tokenizer.convert_tokens_to_ids(tokens)
print("HF tokenization using NeMo steps:", ids)
```

The outputs for provided code snippet are:
```
HF tokenization: [2, 1637, 577, 223491]
NeMo tokenization: [1637, 577, 223491]
HF tokenization using NeMo steps: [1637, 577, 223491]
```

Notice how HF tokenization have token ID 2 at the beginning, which corresponds to the BOS token.

**Expected behavior**

I expect NeMo HF wrapper to behave the same way as the HF tokenizer. The solution for this case would be to change the `text_to_ids` function to:
```
 def text_to_ids(self, text):
     ids = self.tokenizer.encode(text) 
     return ids 
```

Is there any reason why it isn't implemented this way?

**Environment overview (please complete the following information)**

I was using official NeMo container (24.09). However, I see that code for HF wrapper is still the same on main branch, so the issue should still apply.


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Draft: Interface for asymmetric pipeline schedule
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Interface for asymmetric pipeline schedule, where the first and the last Transformer layers use a dummy layer so they have only embedding and logit+loss functions.

# Changelog 
- Add `standalone_embedding_stage` and `standalone_loss_stage` to enable dummy Transformer layer in the first and last Transformer layers.
- Support both nemo 1.0 and 2.0

# Usage
Set `standalone_embedding_stage=True` and `standalone_loss_stage=Trrue`.


# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Fix broken links
State: closed | Pull request: yes

# What does this PR do ?

Fix broken links


# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

LGTM
chore(beep boop 🤖): Bump `MCORE_TAG=4c4215f...` (2024-11-15)
State: closed | Pull request: yes
Labels: name: stale

name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=4c4215fd7570975ef817757d8bfd466d29d51dca`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.