renwei2024/nemo-github-issues huggingface dataset:
Add null tokenizer
State: closed | Pull request: yes
Labels: name: NLP

name: common

name: Run CICD

name: skip-docs

name: r2.1.1

# What does this PR do ?

Add a null tokenizer to ease the runs for performance benchmark and debugging

# Changelog 
Add a null tokenizer

# Usage

```python
tokenizer=run.Config(                                  
    get_nmt_tokenizer,                                  
    library="null",                                   
    model_name = "NullTokenizer",                            
    vocab_size = 256000,
)
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


[🤖]: Hi @erhoo82 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12684452138) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
Fix Mistral Conversion Issue
State: closed | Pull request: yes
Labels: name: Run CICD

name: r2.1.0

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


[🤖]: Hi @suiyoubi 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12678713987) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
chore(beep boop 🤖): Bump `MCORE_TAG=4dc8977...` (2025-01-08)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=4dc8977167d71f86bdec47a60a98e85c4cfa0031`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.
NeMo Llama 2 70B Pre-Training 20TB Dataset OOM
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

NeMo Llama 2 70B Pre-Training 20TB Dataset out-of-memory (CPU memory)

**Steps/Code to reproduce bug**

Using the default YAML file here: https://github.com/NVIDIA/NeMo-Framework-Launcher/blob/main/launcher_scripts/conf/training/llama/llama2_70b.yaml

The dataset is the pile dataset replicated 35 times such that its size is 20TB

We run this on a GKE cluster with 2 A3 Mega nodes

**Expected behavior**

We expect the training to run successfully with num_workers=4. But this runs into OOM error. Reducing num_workers to 2 worked. But this is problematic because we are using GCSFuse and the latency is high, so num_workers=2 is not able to saturate the training.

So we want to know if this is a known problem on Nvidia's end.

**Environment overview (please complete the following information)**

 - Environment location: GCP GKE

**Environment details**

NeMo 24.07



Increasing num_workers can increase the host memory usage. Is my understanding correct that these A3 mega nodes have ~1.8TB of host memory? Do you have data points of how the process mem increase as you increase num_workers?

Does increasing the shared memory like in [this summary comment](https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662) help in your case?

Also, if you're not using [Nemo 2.0](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemo-2.0/index.html), I think that would be worth a try. I have seen anecdotes of host OOMs going away after switching

> Is my understanding correct that these A3 mega nodes have ~1.8TB of host memory?

Correct, our two-node cluster has a total of 3,760 GB

>  Do you have data points of how the process mem increase as you increase num_workers?

What would be the most helpful format to share this data?

Reducing num_nodes or reducing the dataset size will reduce the total amount of CPU memory used. We will experiment increasing shared memory.

> Also, if you're not using [Nemo 2.0](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemo-2.0/index.html), I think that would be worth a try. I have seen anecdotes of host OOMs going away after switching

Hi Terry, thank you for your help. We did try NeMo 2.0, and found that it does indeed resolve the OOM issue.

However, it appears that NeMo 2.0 does not have Kubernetes support for it yet. Do you have any ideas on when that might happen?

> Hi Terry, thank you for your help. We did try NeMo 2.0, and found that it does indeed resolve the OOM issue.

Actually there is k8s support via skypilot: https://github.com/NVIDIA/NeMo-Run/blob/main/docs/source/guides/execution.md#execute-nemo-run

Please give that a try and feel free to leave feedback for the nemo-run team if something is missing.

Closing this issue as the original OOM has a resolution. Please open another issue for any issues/requests for k8s.
Cherry pick llava next changes that were missed for the release
State: closed | Pull request: yes
Labels: name: CI

name: Multi Modal

name: Run CICD

# What does this PR do ?

Cherry pick llava next changes that were missed for the release

**Collection**: [Note which collection this PR will affect]
Multimodal 
# Changelog 
- Cherry pick following PRs

https://github.com/NVIDIA/NeMo/pull/11563
https://github.com/NVIDIA/NeMo/pull/11661
https://github.com/NVIDIA/NeMo/pull/11424

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```

------------------------------------
Your code has been rated at 10.00/10
```

---

Thank you for improving NeMo's documentation!

<!-- pylint-output-strict-mode-true -->

beep boop 🤖: 🚨 The following files must be fixed before merge!

---

Your code was analyzed with PyLint. The following annotations have been identified:

```

------------------------------------
Your code has been rated at 10.00/10
```

---

Thank you for improving NeMo's documentation!

[🤖]: Hi @yashaswikarnati 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12669601008) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
Use empty dict instead of none to load only metadata from dist ckpt
State: closed | Pull request: yes
Labels: name: Run CICD

due to change in mcore commit https://github.com/NVIDIA/Megatron-LM/commit/31e8bfa926ca05a3b70a48d3ed6b86410a85a262

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```

------------------------------------
Your code has been rated at 10.00/10
```

---

Thank you for improving NeMo's documentation!

[🤖]: Hi @ko3n1g 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12659990780) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
only install bitsandbytes on x86
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


[🤖]: Hi @akoumpa 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12659774131) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
remove tensorstore pin in requirements*.txt
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Removes an outdated pin to tensorstore 0.1.45

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Removes version pins in requirements_infer.txt and requirements_nlp.txt

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Llama 405B NeMo version
State: open | Pull request: no

In the models catalog [here](https://catalog.ngc.nvidia.com/models) I could only find Llama 8B and 70B. Is there a 405B version available somewhere? 

I tried converting the HF model to NeMo using the `convert_llama_hf_to_nemo.py` script but the code runs out of memory even on a 1TB system. Is there a workaround for this?
fix llama-3.1 hf model_id
State: open | Pull request: yes

# What does this PR do ?

fix llama-3.1 hf model_id

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update requirements_lightning.txt
State: open | Pull request: yes
Labels: name: Run CICD

PTL to 2.5.0.post0 (used in 24.12 release)

Could you also please update **lightning** version here: https://github.com/NVIDIA/NeMo/blob/main/requirements/requirements_vllm.txt#L8

(It's a separate requirements file used for creating virtual environment for vLLM; for verification we're going to add a vLLM export test to Github Actions soon in https://github.com/NVIDIA/NeMo/pull/11543)
[pre-commit.ci] pre-commit suggestions
State: open | Pull request: yes

<!--pre-commit.ci start-->
updates:
- [github.com/pre-commit/pre-commit-hooks: v4.3.0 → v5.0.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.3.0...v5.0.0)
- [github.com/psf/black-pre-commit-mirror: 24.3.0 → 24.10.0](https://github.com/psf/black-pre-commit-mirror/compare/24.3.0...24.10.0)
<!--pre-commit.ci end-->

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.
chore(beep boop 🤖): Bump `MCORE_TAG=4dc8977...` (2025-01-07)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=4dc8977167d71f86bdec47a60a98e85c4cfa0031`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.

[🤖]: Hi @ko3n1g 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12661151842) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
Pipeline-parallel support for Knowledge Distillation (NeMo 2)
State: open | Pull request: yes

# What does this PR do ?

Enable Pipeline-Parallelism in conjunction with student-teacher distillation in NeMo 2

**Collection**: [LLM]

# Changelog 
- Create new script to enable KD in NeMo 2
- Modify the `MegatronParallel` forward pass to run teacher in addition to student

# Usage
* TBA

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update requirements_lightning.txt
State: open | Pull request: yes
Labels: name: Run CICD

PTL to 2.5.0.post0 (used in 24.12 release)

[🤖]: Hi @pablo-garay 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12763746325) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
Add additional arguments to TransformerLayer subclasses
State: open | Pull request: yes
Labels: name: NLP

name: Run CICD

name: skip-docs

# What does this PR do ?

Add `sequence_len_offset` to the signature of subclasses of TransformerLayer for compatibility with inference updates.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add nsys profiling to ESM-2 benchmarking and apply rope
State: open | Pull request: yes


<!-- pylint-output-strict-mode-true -->

beep boop 🤖: 🚨 The following files must be fixed before merge!

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.collections.llm.bert.model.esm
nemo/collections/llm/bert/model/esm.py:7:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/llm/bert/model/esm.py:16:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/llm/bert/model/esm.py:25:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/llm/bert/model/esm.py:3:0: W0611: Unused HuggingFaceBertModel imported from nemo.collections.llm.bert.model.bert (unused-import)
************* Module nemo2_esm2_tp
nemo2_esm2_tp.py:52:0: C0301: Line too long (153/119) (line-too-long)
nemo2_esm2_tp.py:53:0: C0301: Line too long (179/119) (line-too-long)
nemo2_esm2_tp.py:28:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo2_esm2_tp.py:22:0: W0611: Unused AutoTokenizer imported from nemo.collections.common.tokenizers (unused-import)

-----------------------------------
Your code has been rated at 9.04/10
```

---

Mitigation guide:

* Add sensible and useful docstrings to functions and methods
* For trivial methods like getter/setters, consider adding `# pylint: disable=C0116` inside the function itself
* To disable multiple functions/methods at once, put a `# pylint: disable=C0116` before the first and a `# pylint: enable=C0116` after the last.

By applying these rules, we reduce the occurance of this message in future.

Thank you for improving NeMo's documentation!
chore(beep boop 🤖): Bump `MCORE_TAG=2d7c521...` (2025-01-06)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=2d7c521de7c5878501cf7e366e7d8e13267c8eb6`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.
chore(beep boop 🤖): Bump `MCORE_TAG=30ffe88...` (2025-01-05)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=30ffe880d15df0d72d5d0cb90707283ce977835d`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.
llm.import_ckpt cannot run directly
State: open | Pull request: no
Labels: name: bug

When I run the following code directly, it seems there is a deadlock inside with spawn multiple processes. The follow code is exactly the same as just use llm.import_ckpt directly as a main function. 

```
from nemo.collections import llm
import nemo_run as run

ckpt_import = run.Partial(
    llm.import_ckpt,
    model=run.Config(llm.LlamaModel, config=run.Config(llm.Llama31Config8B)),
    source=f'hf://{model_path}',
    overwrite=True,  
)
local_executor = run.LocalExecutor()
run.run(ckpt_import, executor=local_executor, direct=True)
```

However, if I remove direct=True, and use the local_executor to run the importer, the checkpoint can be successfully transformed. 
```
run.run(ckpt_import, executor=local_executor) # can sucess
```

Can you let me know whether this behavior is expected or there is a bug within the ModelConnector? 

Thank you!
Allow configuration of PP communication backend to UCC in nemo2
State: open | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Allow configuration of PP communication backend to UCC (default NCCL) in nemo2

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
chore(beep boop 🤖): Bump `MCORE_TAG=f682bd0...` (2025-01-04)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=f682bd0ee639b04d3d46a3f8f1881a543c7b653d`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.
What kind of manifest does NEST require?
State: open | Pull request: no

Hi! I want to train my own pretrained Fast-Conformer model using the NEST framework, but I'm not sure what structure I need for this script: [masked_token_pred_pretrain.py](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/speech_pretraining/masked_token_pred_pretrain.py). Additionally, I don't know what type of validation dataset (validation_ds) I need if the data is unlabeled. Please help me figure this out!
Do not load context for model transform in llm inference
State: closed | Pull request: yes
Labels: name: Run CICD


[🤖]: Hi @hemildesai 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12626573102) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

[🤖]: Hi @hemildesai 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12765152503) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
Cherry pick `Update changelog for `r2.1.0` (11745)` into `r2.1.0`
State: closed | Pull request: yes
Labels: name: cherry-pick

name: Run CICD

beep boop [🤖]: Hi @github-actions[bot] 👋,
    
    we've cherry picked #11745 into  for you! 🚀
    
    Please review and approve this cherry pick by your convenience!
ci: Fix changelog generator
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add BERT Embedding Models
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


[🤖]: Hi @suiyoubi 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12616668182) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
NeMo/examples/slu/speech_intent_slot   0 files were filtered totalling 0.00 hours
State: open | Pull request: no
Labels: name: bug

I am trying to train a model on the SLURP dataset using the NeMo framework by following the steps outlined in the `README.md` file under `NeMo/examples/slu/speech_intent_slot`. 

- **Data Preparation** and **Building Tokenizers** completed successfully.
- The `.json` files (ending with `slu.json`) were correctly generated under the `slurp_data` folder.

However, when I run the training script, I encounter the following error:

```
self._train_dl = self._setup_dataloader_from_config(config=train_data_config)
  File "/home/ddyang/miniconda3/envs/nemo/lib/python3.10/site-packages/nemo/collections/asr/models/slu_models.py", line 420, in _setup_dataloader_from_config
    return torch.utils.data.DataLoader(
  File "/home/ddyang/miniconda3/envs/nemo/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 350, in __init__
    sampler = RandomSampler(dataset, generator=generator)  # type: ignore[arg-type]
  File "/home/ddyang/miniconda3/envs/nemo/lib/python3.10/site-packages/torch/utils/data/sampler.py", line 143, in __init__
    raise ValueError(f"num_samples should be a positive integer value, but got num_samples={self.num_samples}")
ValueError: num_samples should be a positive integer value, but got num_samples=0
```

It seems like the data was not read correctly. Could you please guide me on how to resolve this issue?

---
chore(beep boop 🤖): Bump `MCORE_TAG=076972e...` (2025-01-02)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=076972e37420b5325c5fe06e7131be7d96f05b53`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.
chore(beep boop 🤖): Bump `MCORE_TAG=076972e...` (2025-01-01)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=076972e37420b5325c5fe06e7131be7d96f05b53`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.
Cherry pick `Bug fix minor bug in TRT-LLM deployment (11714)` into `r2.1.0`
State: closed | Pull request: yes
Labels: name: cherry-pick

name: Run CICD

beep boop [🤖]: Hi @oyilmaz-nvidia 👋,
    
    we've cherry picked #11714 into  for you! 🚀
    
    Please review and approve this cherry pick by your convenience!

<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```

------------------------------------
Your code has been rated at 10.00/10
```

---

Thank you for improving NeMo's documentation!

<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```

------------------------------------
Your code has been rated at 10.00/10
```

---

Thank you for improving NeMo's documentation!

[🤖]: Hi @ko3n1g 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12667707388) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
chore(beep boop 🤖): Bump `MCORE_TAG=076972e...` (2024-12-31)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=076972e37420b5325c5fe06e7131be7d96f05b53`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.
NeMo intermittent start-up failure with OMPI temp directory error on k8s
State: open | Pull request: no
Labels: name: bug

**Describe the bug**

NeMo intermittent start-up failure with OMPI temp directory error.

**Steps/Code to reproduce bug**

* Launch multinode NeMo training job on k8s with privilege mode without HostPID
* Observe that in about 3~4% of the time, NeMo job fails to launch with error:

```
ERROR 2024-11-13T14:41:01.114829205Z [resource.labels.containerName: megatron] --------------------------------------------------------------------------
ERROR 2024-11-13T14:41:01.114859401Z [resource.labels.containerName: megatron] A call to mkdir was unable to create the desired directory:
ERROR 2024-11-13T14:41:01.114863131Z [resource.labels.containerName: megatron] {}
ERROR 2024-11-13T14:41:01.114865257Z [resource.labels.containerName: megatron] Directory: /tmp/ompi.machine-name-redacted.0/pid.2284793
ERROR 2024-11-13T14:41:01.114867380Z [resource.labels.containerName: megatron] Error: No such file or directory
ERROR 2024-11-13T14:41:01.114871655Z [resource.labels.containerName: megatron] Please check to ensure you have adequate permissions to perform
ERROR 2024-11-13T14:41:01.114874427Z [resource.labels.containerName: megatron] the desired operation.
ERROR 2024-11-13T14:41:01.114876515Z [resource.labels.containerName: megatron] --------------------------------------------------------------------------
ERROR 2024-11-13T14:41:01.114878903Z [resource.labels.containerName: megatron] [machine-name-redacted:2284793] [[47352,0],0] ORTE_ERROR_LOG: Error in file util/session_dir.c at line 107
ERROR 2024-11-13T14:41:01.114895028Z [resource.labels.containerName: megatron] [machine-name-redacted:2284793] [[47352,0],0] ORTE_ERROR_LOG: Error in file util/session_dir.c at line **346**
```

**Expected behavior**

NeMo training job to launch without hiccup

**Environment overview (please complete the following information)**

 - Environment location: GKE
 - Method of NeMo install: Used pre-built docker image from `nvcr.io/nvidia/nemo:24.07`
chore(beep boop 🤖): Bump `MCORE_TAG=2da43ef...` (2024-12-30)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=2da43ef4c1b9e76f03b7567360cf7390e877f1b6`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.
chore(beep boop 🤖): Bump `MCORE_TAG=2da43ef...` (2024-12-29)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=2da43ef4c1b9e76f03b7567360cf7390e877f1b6`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.
chore(beep boop 🤖): Bump `MCORE_TAG=2da43ef...` (2024-12-28)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=2da43ef4c1b9e76f03b7567360cf7390e877f1b6`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.
chore(beep boop 🤖): Bump `MCORE_TAG=2da43ef...` (2024-12-27)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=2da43ef4c1b9e76f03b7567360cf7390e877f1b6`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.
convert_llama_hf_to_nemo.py use llama31
State: open | Pull request: no
Labels: name: bug

python /opt/NeMo/scripts/checkpoint_converters/convert_llama_hf_to_nemo.py --input_name_or_path=/workspace/model_ckpts/Meta-Llama-3.1-8B-Instruct/ --output_path=llama3.1-8b-instruct.nemo --llama31 True

Does llama31 not pack embedding into nemo file?
Generated
![Image](https://github.com/user-attachments/assets/b8eccff6-d3e5-4626-8c54-8b0525ab9b6e)


Downloa

![Image](https://github.com/user-attachments/assets/d3f4094e-f845-4a6a-b882-423be6553a80)


Hi @yangzhipeng1108, We hit the same problem with the checkpoint conversion, Just wondering if you could leave your contact information in your github, Could we send you a email regarding this. We would love to explore the potential cooperation opportunity
chore(beep boop 🤖): Bump `MCORE_TAG=2da43ef...` (2024-12-26)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=2da43ef4c1b9e76f03b7567360cf7390e877f1b6`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.
Bug fix minor bug in TRT-LLM deployment
State: closed | Pull request: yes
Labels: name: bug

name: NLP

name: Run CICD

name: r2.1.0

# What does this PR do ?

Fixes a minor bug deploying trt-llm models


[🤖]: Hi @oyilmaz-nvidia 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12488550311) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
chore(beep boop 🤖): Bump `MCORE_TAG=2da43ef...` (2024-12-25)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=2da43ef4c1b9e76f03b7567360cf7390e877f1b6`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.
Cherry pick `added required instalation for sox to process mp3 file (11709)` into `r2.1.0`
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

name: Run CICD

beep boop [🤖]: Hi @Ssofja 👋,
    
    we've cherry picked #11709 into  for you! 🚀
    
    Please review and approve this cherry pick by your convenience!
removed the line which caused a problem in nfa_tutorial
State: closed | Pull request: yes
Labels: name: bug

name: ASR

name: Run CICD

name: r2.1.0

# What does this PR do ?

This PR fixes the bug which was noticed during the usage of tutorials/tools/NeMo_Forced_Aligner_Tutorial.ipynb
**Collection**: [ASR]
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation



[🤖]: Hi @Ssofja 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12484285474) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.collections.asr.models.hybrid_rnnt_ctc_models
nemo/collections/asr/models/hybrid_rnnt_ctc_models.py:716:0: C0301: Line too long (141/119) (line-too-long)
nemo/collections/asr/models/hybrid_rnnt_ctc_models.py:717:0: C0301: Line too long (139/119) (line-too-long)
nemo/collections/asr/models/hybrid_rnnt_ctc_models.py:16:0: W0611: Unused import json (unused-import)
nemo/collections/asr/models/hybrid_rnnt_ctc_models.py:17:0: W0611: Unused import os (unused-import)
nemo/collections/asr/models/hybrid_rnnt_ctc_models.py:18:0: W0611: Unused import tempfile (unused-import)
nemo/collections/asr/models/hybrid_rnnt_ctc_models.py:24:0: W0611: Unused tqdm imported from tqdm.auto (unused-import)

-----------------------------------
Your code has been rated at 9.82/10
```

---

Mitigation guide:

* Add sensible and useful docstrings to functions and methods
* For trivial methods like getter/setters, consider adding `# pylint: disable=C0116` inside the function itself
* To disable multiple functions/methods at once, put a `# pylint: disable=C0116` before the first and a `# pylint: enable=C0116` after the last.

By applying these rules, we reduce the occurance of this message in future.

Thank you for improving NeMo's documentation!

[🤖]: Hi @Ssofja 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12495141516) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
added required instalation for sox to process mp3 file
State: closed | Pull request: yes
Labels: name: bug

name: Run CICD

name: r2.1.0

# What does this PR do ?

This PR fixes the bug which was noticed during the usage of tutorials/tools/CTC_Segmentation_Tutorial.ipynb
**Collection**: [ASR]
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation