renwei2024/nemo-github-issues huggingface dataset:
Fix memory leak at loss func
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Merges #8803, #8810, #8853 into one.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Re #8803: `nemo/collections/nlp/models/information_retrieval/megatron_sbert_model.py` was removed with [this commit](https://github.com/NVIDIA/NeMo/commit/b273fa6caf41bde383e93adfaa10349a3f361807).

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
Move logic for distopt FP32 grads to models
State: closed | Pull request: yes
Labels: name: bug

name: NLP

# What does this PR do ?

https://github.com/NVIDIA/NeMo/pull/8792 introduced some runtime failures in T5 since it adds GPT-specific logic to `MegatronBaseModel`. This PR moves the distopt FP32 grad logic to the specific models.

**Collection**: NLP

# Changelog 
- Move logic for distopt FP32 grads to models

# Usage

Run T5, e.g. with the config at https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/megatron_t5_config.yaml.

Enable the distributed optimizer with model.optim.name=distributed_fused_adam.

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Fixes bug from https://github.com/NVIDIA/NeMo/pull/8792
* Closes https://github.com/NVIDIA/NeMo/pull/8857
* Closes https://github.com/NVIDIA/NeMo/pull/8810

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
Avoid unpacking NeMo checkpoints before exporting to TRT-LLM
State: closed | Pull request: yes

# What does this PR do ?

Replaces the unpacking of .nemo files into a temporary directory before use with direct use of files inside .nemo (which is just a TAR archive), thereby saving many seconds on each export operation.

**Collection**: NLP

# Changelog 
- Replaced unpacking of nemo checkpoints on export with a VFS-like TarPath object.

# Usage
```python
with TarPath('/path/to/archive.tar') as archive:
    myfile = archive / 'filename.txt'
    if myfile.exists():
        data = myfile.read()
        ...
```

**PR Type**:
- [X] New Feature (improvement, rather)
- [ ] Bugfix
- [ ] Documentation


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
dehardcode test string
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

By default `to_word_list_format` uses '<extra_id_1>' as its test string to map a string to its tokens, but using '<extra_id_1>' may not map correctly, so add option to specifiy the test string.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Can this be merged @oyilmaz-nvidia ?

jenkins
TRT-LLM export P-tuning related fixes
State: closed | Pull request: yes

# What does this PR do ?

Fixes the implementation of PEFT for TRT-LLM export.

**Collection**: NLP

# Changelog 

- Add the bos token to LLAMA based models (not just P-tuning related, improves LLAMA results in general).
- Remember the vtoken counts for each p-tuning table when the tables are added.
- Prepend the right number of vtokens to each query based on its task_id.
- Preserve the dtype of the p-tuning table when it is padded.
- Validate that all p-tuning tables fit into max_prompt_embedding_table_size limit.

 
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

# Additional Information
* Related to bug 4350064
Fix transcription utils function for duration check
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fix for case, when duration is `null` in manifest

**Collection**: ASR


# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Huvu/mcore retro
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
Added audio codec checkpoint to docs
State: closed | Pull request: yes
Labels: name: documentation

name: TTS

# What does this PR do ?

This PR adds a pointer to audio codec checkpoint to docs.

**Collection**: Docs

# Changelog 
- Added a table with info about the available checkpoints

# Usage
n/a

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins
Quantized checkpoint support in export and deploy modules
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Add support for testing, deploying and running quantized "qnemo" checkpoints in `nemo.deploy` and `nemo.export` modules.

**Collection**: NLP

# Changelog 
- qnemo support
- bump ammo version to 0.9.4
- update code for trt-llm 0.9.0
- Lambada evaluation setup
- timing of evaluation (with minor improvements)
- documentation

# Usage

Building TensorRT-LLM engine and running a basic prompt:
```python
from nemo.export import TensorRTLLM


trt_llm_exporter = TensorRTLLM(model_dir="/path/to/trt_llm_engine_folder")
trt_llm_exporter.export(
    nemo_checkpoint_path="/path/to/qnemo/ckpt.qnemo",
    model_type="llama",
    n_gpus=1,
)
output = trt_llm_exporter.forward(["Hi, how are you?", "I am good, thanks, how about you?"])
```

Running Lambada test locally:
```python
python tests/export/test_nemo_export.py \
  --model_name local \
  --model_type llama \
  --checkpoint_dir /path/to/qnemo/ckpt.qnemo \
  --trt_llm_model_dir /path/to/trt_llm_engine_folder \
  --min_gpus 1 \
  --run_accuracy \
  --test_deployment True \
  --test_data_path lambada.json
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to https://github.com/NVIDIA/NeMo/pull/8743


jenkins
vad finetune on my dataset
State: closed | Pull request: no
Labels: name: stale

Hi, I am trying to finetune vad and speaker diarization model on korean dataset. However, I cannot find tutorial related to vad. I've found [File not found (github.com)](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/06_Voice_Activiy_Detection.ipynb) tutorial from previous issues, but it seems that the tutorial is deleted. How can I finetune vad model with my dataset?
Also, Is there any way to train vad&speaker diarization model together?

Here is the tutorial for VAD: https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/Voice_Activity_Detection.ipynb 

Currently we don;t support training VAD and SD model together

> Here is the tutorial for VAD: https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/Voice_Activity_Detection.ipynb
> 
> Currently we don;t support training VAD and SD model together

Thank you for your answer! Additionally, I have a question about training dataset. Can one example of dataset contain more than 2 speech chunks?

I didn;t understand your question well: `Can one utterance contain more than 2 speech chunks?` 
Do you mean, if VAD is frame or segment based? 



This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Check if model has position embed before accessing param
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

Looks like all tests passed, please merge @ericharper thank you.

Merged this to attempt fix the broken CI pipeline
Fix precision of output model in conversion scripts
State: closed | Pull request: yes

# What does this PR do ?

`--precision bf16` was not functional in some conversion scripts and converted checkpoints are twice the size as they should be. This PR fixes the precision of the output model.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Disable gradient reduction for Lora Adapters when SP is enabled
State: closed | Pull request: yes
Labels: name: core

name: ASR

name: NLP

name: CI

name: Multi Modal

# What does this PR do ?

Disable gradient reduction for the Lora Adapter when SP is enabled. The first ColumnParallelLinear layer for the Lora adapter should not reduce the gradients when SP is enabled.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

I think we need to add an mcore version guard and/or change the megatron-lm commit in jenkinsfile
Fix memory leak at loss func
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Resolve the memory lease from loss function

# Changelog 
- Do not grep values from config

# Usage
N/A


# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This has been transferred to https://github.com/NVIDIA/NeMo/pull/8868 to avoid being bottlenecked by Jenkins.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Add deploy triton and query scripts
State: closed | Pull request: yes

# What does this PR do ?

Adds deploy to triton script for llm models.

# Changelog 
- Deployment and query scripts for LLM models.
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation
- [x] Script

If you haven't finished some of the above items you can still open "Draft" PR.



jenkins

jenkins

jenkins
upgrade to trtllm0.9
State: closed | Pull request: yes

# What does this PR do ?

Update export module to support trtllm0.9



All the combined updates for TRT-LLM 0.9 is here https://github.com/NVIDIA/NeMo/pull/8873 . That PR includes @abharwani 's commits. We can close this since all the commits are already pulled in the other PR.

Closing this since @abharwani 's commits were merged in with this PR https://github.com/NVIDIA/NeMo/pull/8873
Overlap allreduce_sequence_parallel_gradient
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

This PR aims to overlap the process of [allreduce_sequence_parallel_gradient](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py#L867) with [allreduce_first_last_embeddings](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py#L902C9-L902C40). 

# Changelog 
The following two changes enable overlapping:
- Change the order in which these two functions are called to enable overlapping. Since the gradients all-reduced in these two functions are independent of each other, it is safe to change the order.
- allreduce_sequence_parallel_gradient should execute in a non-blocking stream to enable overlapping.

# Usage
This optimization only works when applying sequence_parallel and pipeline_model_parallel.

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
TransformerLayer MLP parameters are not being set during model initialization
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

**Fix is implemented here: https://github.com/NVIDIA/NeMo/pull/8845**

Transformer layer MLP always uses default values for bias, activation, and normalization if model.mcore_gpt=False, model.transformer_engine=True, model.megatron_amp_O2=True.

**Steps/Code to reproduce bug**

Add the following lines to NeMo/examples/nlp/language_modeling/megatron_gpt_pretraining.py after model initialization:
```
logging.warning(f"DEBUG: layernorm_mlp.activation={model.model.module.language_model.encoder.layers._modules['0'].layernorm_mlp.activation}")
logging.warning(f"DEBUG: layernorm_mlp.use_bias={model.model.module.language_model.encoder.layers._modules['0'].layernorm_mlp.use_bias}")
logging.warning(f"DEBUG: layernorm_mlp.normalization={model.model.module.language_model.encoder.layers._modules['0'].layernorm_mlp.normalization}")
logging.warning(f"DEBUG: layernorm_mlp.layernorm_mlp.fc1_weight.shape={model.model.module.language_model.encoder.layers._modules['0'].layernorm_mlp.fc1_weight.shape}")
logging.warning(f"DEBUG: layernorm_mlp.layernorm_mlp.fc2_weight.shape={model.model.module.language_model.encoder.layers._modules['0'].layernorm_mlp.fc2_weight.shape}")
```

Run the following script with and without the changes:
```
#!/bin/bash

python /opt/NeMo/examples/nlp/language_modeling/megatron_gpt_pretraining.py \
    model.mcore_gpt=False \
    model.transformer_engine=True \
    trainer.precision=bf16 \
    model.megatron_amp_O2=True \
    model.activation=fast-swiglu \
    model.bias=false \
    model.normalization=rmsnorm
```


**Expected behavior**

The values for layernorm_mlp are set correctly:
```
[NeMo W 2024-04-08 09:30:49 megatron_gpt_pretraining:42] DEBUG: layernorm_mlp.activation=fast-swiglu
[NeMo W 2024-04-08 09:30:49 megatron_gpt_pretraining:43] DEBUG: layernorm_mlp.use_bias=False
[NeMo W 2024-04-08 09:30:49 megatron_gpt_pretraining:44] DEBUG: layernorm_mlp.normalization=RMSNorm
[NeMo W 2024-04-08 09:30:49 megatron_gpt_pretraining:45] DEBUG: layernorm_mlp.layernorm_mlp.fc1_weight.shape=torch.Size([3072, 768])
[NeMo W 2024-04-08 09:30:49 megatron_gpt_pretraining:46] DEBUG: layernorm_mlp.layernorm_mlp.fc2_weight.shape=torch.Size([768, 3072])
```

**Environment overview (please complete the following information)**

 - Docker
 - nvidia/nemo:24.03 + git pull
 - docker run --rm -it --entrypoint /bin/bash --network=host --runtime=nvidia --shm-size=2g nvcr.io/nvidia/nemo:24.01.01.framework

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Fix for TransformerLayer MLP parameters not being set with specific hyperparameters
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

**Detailed description here: https://github.com/NVIDIA/NeMo/issues/8846**
This PR fixes Transformer layer MLP hyperparameters not being set with model.mcore_gpt=False, model.transformer_engine=True, and model.megatron_amp_O2=True.

**Collection**: NLP

# Changelog 
- Passing missing arguments to layer constructors.

# Usage
* Add the following lines to opt/NeMo/examples/nlp/language_modeling/megatron_gpt_pretraining.py after model initialization to debug:

```
logging.warning(f"DEBUG: layernorm_mlp.activation={model.model.module.language_model.encoder.layers._modules['0'].layernorm_mlp.activation}")
logging.warning(f"DEBUG: layernorm_mlp.use_bias={model.model.module.language_model.encoder.layers._modules['0'].layernorm_mlp.use_bias}")
logging.warning(f"DEBUG: layernorm_mlp.normalization={model.model.module.language_model.encoder.layers._modules['0'].layernorm_mlp.normalization}")
logging.warning(f"DEBUG: layernorm_mlp.layernorm_mlp.fc1_weight.shape={model.model.module.language_model.encoder.layers._modules['0'].layernorm_mlp.fc1_weight.shape}")
logging.warning(f"DEBUG: layernorm_mlp.layernorm_mlp.fc2_weight.shape={model.model.module.language_model.encoder.layers._modules['0'].layernorm_mlp.fc2_weight.shape}")
```

Test with the following script with and without the changes:
```
#!/bin/bash

python /opt/NeMo/examples/nlp/language_modeling/megatron_gpt_pretraining.py \
    model.mcore_gpt=False \
    model.transformer_engine=True \
    trainer.precision=bf16 \
    model.megatron_amp_O2=True \
    model.activation=fast-swiglu \
    model.bias=false \
    model.normalization=rmsnorm \
```
The bias, normalization, and activation should be correctly set with the fix.


# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
remove sbert since we have bert_embedding_model
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Jenkins

Jenkins
zarr ckpt to torch_dist ckpt converter
State: closed | Pull request: yes
Labels: name: feature

name: stale

name: Run CICD

# What does this PR do ?

Adds a script to convert zarr ckpt format to torch distributed ckpt format.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

jenkins
Enable DGRAD RS overlap
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Corresponding TE/MCore PRs:
- TE: https://github.com/NVIDIA/TransformerEngine/pull/754
- MCore: https://gitlab-master.nvidia.com/ADLR/megatron-lm/-/merge_requests/1320

jenkins

Addressed @timmoon10's comments. Please review

jenkins
Set TP overlap flag in ModelParallelConfig, fix TP overlap for LoRA
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

~I see we are setting `return_layernorm_output_gathered=False` when UB is enabled. Won't that change the numerics? If the LoRA adapter has tensor parallelism enabled, then don't we require the input to be gathered?~

Nevermind, I see that we are all-gathering the layernorm output at https://github.com/NVIDIA/NeMo/blob/ec662bfa871918741f41698d3b86ff8ab3bf4e94/nemo/collections/nlp/modules/common/megatron/adapters/parallel_adapters.py#L277-L282
Add clip conv layer
State: closed | Pull request: yes
Labels: name: Multi Modal

# What does this PR do ?

Replace the linear layer in the VIT backbone clip with the convolution layer.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Changes in both vit_backbone.py and convert_external_clip_to_nemo.py

# Usage
More similarity with the existing Clip models in the literature.

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
Attaching the successful run of the [convert_external_clip_to_nemo.py]
<img width="1494" alt="Screenshot 2024-04-08 at 1 04 33 PM" src="https://github.com/NVIDIA/NeMo/assets/49656670/d69a95ff-290f-4f1f-8aaa-cab792e32f3c">


I have verified the conversion works and this new converted ckpt works with SD. Thanks!!

jenkins
Update Latest News
State: closed | Pull request: yes

# What does this PR do ?

Adds links to articles on
* NeMo framework on GKE
* Responsible Gen AI using NeMo and Picasso
* NeMo powering Amazon Titan foundation models


Can you please put it into a droppable menu as shown here, this is too much unrelated things before even beginning to discuss the actual toolkit. If users are interested they can  open the menu. 

For reference from an old version of NeMo - https://github.com/NVIDIA/NeMo/blob/r1.20.0/PUBLICATIONS.md
Alit/draft
State: closed | Pull request: yes
Labels: name: NLP

name: Multi Modal

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Jenkins
LoRA merge script clean up
State: closed | Pull request: yes

# What does this PR do ?

1. Remove references to `pipeline_model_parallel_split_rank` since it is only used for enc-dec models, and the merge script is for decoder only models. This enables users to not have to specify this in the run command.
2. Remove setting `apply_rope_fusion` to False since bug #8590 is fixed. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Natural language prompting for Canary
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Fix packed seq doc math rendering issue
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

minor fix for rendering issue with "#"

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Use Label-Looping algorithm for RNN-T decoding by default
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Enable Label-Looping algorithm introduced in #8286 and #7926 (`loop_labels=True`) by default for RNN-T greedy decoding.

**Collection**: [ASR]

# Changelog 
- Enable Label-Looping algorithm by default (`loop_labels=true`)
- fix Label Looping algorithm for RNN-T/TDT + Stateless network
- fix tests with custom RNNT Decoder
- parametrize tests for batched greedy decoding to test both algorithms (Frame-/Label-Looping)

# Usage

Label-Looping algorithm is used by default now for batched greedy decoding.

For Frame-Looping algorithm one can use:
```shell
python examples/asr/speech_to_text_eval.py  <...> \
 rnnt_decoding.greedy.loop_labels=false
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Not ready yet

jenkins

> Late review with a few FYI comments. Do we test the cuda graphs implementation with stateless transducers yet?

Thanks for the comments, @galv! I will make fixes in next PRs. 
Cuda graphs implementation with stateless prediction network is not tested on CI for now (no production model), but I tested loop labels + cuda graphs locally with a custom model, and it works.