renwei2024/nemo-github-issues huggingface dataset:
fix issues with convert_nemo_llama_to_hf.py
State: closed | Pull request: yes

# What does this PR do ?

fix issues with convert_nemo_llama_to_hf.py

**Collection**: nlp

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Do we have a test for `convert_nemo_llama_to_hf.py` anywhere?

Like we have, for example, for the script `convert_hf_llama_to_nemo.py` working the other way around [here](https://github.com/NVIDIA/NeMo/blob/521cfb4c9359c4033f34776da4b73fabf3e94240/Jenkinsfile#L131-L138).

@janekl I'm not sure if there's such a test since I'm not the original author of `convert_nemo_llama_to_hf.py`. The changes here are made based on trying to load the files into hf (which didn't work previously if you use the pytorch_model.bin directly, since all model weights are contained within ['state_dict'] which the corresponding pytorch_model.bin from HF do not, and instead expose the keys directly). The only other change here is ensuring `model_config.tensor_model_parallel_size = 1`, which makes the conversion fail otherwise when running with `--cpu-only ` Maybe Utkarsh Uppal, the author of https://github.com/NVIDIA/NeMo/pull/7770 might know.

jenkins
Reworked MegatronPretrainingRandomBatchSampler to correctly handle epochs > 1
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fixes MegatronPretrainingRandomBatchSampler to correctly instantiate when consumed_samples >= total_samples

**Collection**: NLP

# Changelog 
- nemo/collections/nlp/data/language_modeling/megatron/megatron_batch_samplers.py: slightly rewrote the MegatronPretrainingRandomBatchSampler class

# Usage
No change to how it was used before, however, I have run many training runs using this fixed version in Nemo-Aligner and it appears to work correctly with no issues.

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins
ModuleNotFoundError: No module named 'nemo.deploy'
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I am trying to experiment with Nemotron model using the example in Huggingface.

https://huggingface.co/nvidia/nemotron-3-8b-qa-4k

I installed nemo but I get the following error.

ModuleNotFoundError                       Traceback (most recent call last)
Cell In[3], line 1
----> 1 from nemo.deploy import NemoQuery
      2 # In this case, we run inference on the same machine
      3 nq = NemoQuery(url="localhost:8000", model_name="Nemotron-3-8B-QA-4K")

ModuleNotFoundError: No module named 'nemo.deploy'

I am able to import nemo itself.

**Expected behavior**

Import nemo.deploy

**Environment overview (please complete the following information)**

 - Environment location: Bare-metal
 - Method of NeMo install: pip install in a conda env
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: MacOS
- PyTorch version: 2.1.1
- Python version: 3.10.12




This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Remove deprecated arguments from TE's TransformerLayer
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Transformer engine doesn't use this arguments since v0.7 and now in v1.0 they removed them completely.

**Collection**: [Note which collection this PR will affect]
NLP

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x ] Bugfix
- [ ] Documentation

## Who can review?

@ericharper 



the PR changes look fine but we might want to note that this only changes things for NeMo local GPT models, not m-core based ones which are the ones we majorly use now

I think the main issue is that user are used to configuring this from the config. Now TE expects an environment variable to be set. Mcore makes sure that the env var and the config are consistent. Can NeMo set the env var automatically so that the user doesn't have to set the env var?

yes, thats right. It would probably be better if we automatically set the TE env variable if someone passes it in the config

Ideally, I would prefer if this env var setting happens in m-core since m-core also relies on arg passing format and is the place where we actually interface with TE and setup TE/local layers. But if we feel that it might need broader discussion it might make sense to add this in nemo for now so that it unblocks the issue?

@ericharper @aklife97 can you approve this for now in nemo so it unblocks? 

jenkins

jenkins
[NLP] Access scaler only in FP16 case
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

When trying to load FP16 model with `trainer.precision != 16` (or `16-mixed`) a user gets a mysterious error message like:
```
  File "/opt/NeMo/nemo/collections/nlp/models/language_modeling/megatron_base_model.py", line 122, in __init__
    self.model_parallel_config: ModelParallelConfig = self.build_model_parallel_config()
  File "/opt/NeMo/nemo/collections/nlp/models/language_modeling/megatron_base_model.py", line 782, in build_model_parallel_config
    "grad_scale_func": self.trainer.precision_plugin.scaler.scale
AttributeError: 'NoneType' object has no attribute 'scale'
```
I'm changing the code to access `trainer.precision_plugin.scaler.scale` only when `trainer` is configured with the relevant precision: "16" or "16-mixed".

**Collection**: NLP

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
How to get the data_process code in NeMo doc for common crawl?
State: closed | Pull request: no

**Is your feature request related to a problem? Please describe.**

I read the doc for NeMO about data processing [here,](https://docs.nvidia.com/nemo-framework/user-guide/latest/modelguide/pretrainingdatasets/textcleaningandlanguageseparation.html) and I want to use these scripts for data preprocessing. But I can't find a way to get them.

**Describe the solution you'd like**

A website or a github URL to download all these scripts.



@SefaZeng there is a issue with links, its not open sourced through NeMo. 
Currently you can request for access to NeMo framework at https://developer.nvidia.com/nemo-framework and run scripts

@SefaZeng closing this issue, feel free to reopen if you are unable to run the scripts after downloading the nemo-framework container.

> @SefaZeng there is a issue with links, its not open sourced through NeMo. Currently you can request for access to NeMo framework at https://developer.nvidia.com/nemo-framework and run scripts

Thank you for your reply. Could you please tell what is the difference between NeMo and NeMo-Framework?

(As of current)
[NeMo](https://nvidia.github.io/NeMo/) is open source conversational toolkit for training ASR, TTS, NLP and Speaker Diarization models. 
[NeMo framework](https://developer.nvidia.com/nemo-framework) is a open access container based framework for building, customizing and deploying LLM/multi-modal models.
added missing torch import
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Added torch import for this error:

```
  File "/code/nemo/collections/nlp/modules/common/megatron/adapters/mcore_mixins.py", line 85, in get_query_key_value_tensors
    (query, key, value) = torch.split(
NameError: name 'torch' is not defined
```

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins
v1.21.0 is incompatible with nemo2riva 2.13.1
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

`nemo_toolkit[all]` has dependencies that conflict with `nemo2riva`:

```
29.39 The conflict is caused by:
29.39     nemo-toolkit[all] 1.21.0 depends on black==19.10b0; extra == "all"
29.39     nvidia-eff 0.6.5 depends on black==22.3.0
29.39     nemo-toolkit[all] 1.21.0 depends on isort<6.0.0 and >5.1.0; extra == "all"
29.39     nvidia-eff 0.5.7 depends on isort<5
```

Not sure why `black` and `isort` are listed as dependencies...

**Steps/Code to reproduce bug**

```sh
pip install nemo_toolkit[all]==1.21.0 nemo2riva==2.31.1
```


**Expected behavior**

Support nemo2riva

**Environment overview (please complete the following information)**

`nvcr.io/nvidia/riva/riva-speech:2.31.1-servicemaker`



I think `test` should not be in `all`.
For now, a workaround is:

```
pip install 
    nemo_toolkit[core]==1.21.0 \
    nemo_toolkit[common]==1.21.0 \
    nemo_toolkit[asr]==1.21.0 \
    nemo_toolkit[nlp]==1.21.0 \
    nemo_toolkit[tts]==1.21.0 \
    nemo_toolkit[slu]==1.21.0 \
    nemo2riva==2.31.1
```

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

Still not resolved

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Fix for megatron_amp_O2 + MegatronGPTSFTModel
State: closed | Pull request: yes
Labels: name: stale

name: NLP

Currently MegatronGPTSFTModel inherits sharded_state_dict from NLPAdapterModelMixin, but when restoring weights from the distributed checkpoint it's expected that MegatronGPTModel's version of sharded_state_dict has been run, resulting in:
```...
  File "/root/llm/NeMo/examples/nlp/language_modeling/tuning/megatron_gpt_peft_tuning.py", line 63, in main
    model = MegatronGPTSFTModel.restore_from(cfg.model.restore_from_path, model_cfg, trainer=trainer)
  File "/root/llm/NeMo/nemo/collections/nlp/models/nlp_model.py", line 465, in restore_from
    return super().restore_from(
  File "/root/llm/NeMo/nemo/core/classes/modelPT.py", line 442, in restore_from
    instance = cls._save_restore_connector.restore_from(
  File "/root/llm/NeMo/nemo/collections/nlp/parts/nlp_overrides.py", line 743, in restore_from
    checkpoint = dist_checkpointing.load(
  File "/opt/Megatron-LM/megatron/core/dist_checkpointing/serialization.py", line 72, in load
    sharded_objects, sharded_state_dict = load_sharded_objects(sharded_state_dict, checkpoint_dir)
  File "/opt/Megatron-LM/megatron/core/dist_checkpointing/serialization.py", line 123, in load_sharded_objects
    return dict_list_map_inplace(load_sharded_object, sharded_objects), sharded_state_dict
  File "/opt/Megatron-LM/megatron/core/dist_checkpointing/dict_utils.py", line 155, in dict_list_map_inplace
    x[k] = dict_list_map_inplace(f, v)
  File "/opt/Megatron-LM/megatron/core/dist_checkpointing/dict_utils.py", line 155, in dict_list_map_inplace
    x[k] = dict_list_map_inplace(f, v)
  File "/opt/Megatron-LM/megatron/core/dist_checkpointing/dict_utils.py", line 159, in dict_list_map_inplace
    return f(x)
  File "/opt/Megatron-LM/megatron/core/dist_checkpointing/serialization.py", line 120, in load_sharded_object
    loaded_obj = torch.load(load_path)
  File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 988, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 437, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 418, in __init__
    super().__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '/pretrained_models/Llama-2-70b-bf16-mcore/model_weights/model.module.decoder.layers.self_attention.linear_proj._extra_state/shard_0_80.pt'
```

This code change overrides sharded_state_dict in MegatronGPTSFTModel to call that method instead.  Changes may also be needed to reconcile MegatronT5SFTModel.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

Closing in favor of more-compatible change on the NLPAdapterModelMixin side in #7971.
upgrade to latest mcore and TE
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Tools to parallelize Megatron tokenization
State: closed | Pull request: no
Labels: name: stale

**Is your feature request related to a problem? Please describe.**

I recently heard about the parallel data curation tools in NeMo, which allows one to run download, clean, and de-dup data using multiple nodes.

https://developer.nvidia.com/blog/curating-trillion-token-datasets-introducing-nemo-data-curator/

That's very exciting.  Thanks for putting this together.

The NeMo script to tokenize data for Megatron supports running multiple workers on a single node:

https://github.com/NVIDIA/NeMo/blob/main/scripts/nlp_language_modeling/preprocess_data_for_megatron.py

For large datasets, I'd like to go even further and parallelize this task across multiple nodes.

**Describe the solution you'd like**

About a year ago, I wrote up some tools to do this.  I opened a similar ticket on Megatron-LM with more info:

https://github.com/NVIDIA/Megatron-LM/issues/492

Would you be willing to accept something like this in NeMo?

Those existing files would need to be refreshed, because the indexed_dataset mmap file format has since changed.  I am willing to take some time to do that if that would be helpful.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
add_support_for_swiglu_fusion
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Upgrade to PTL 2.1
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Chat method for TextGeneration API
State: closed | Pull request: no
Labels: name: feature

name: stale

I propose ``class TextGeneration`` (from which ``MegatronGPTModel`` inherits) to add ```.chat(dict)``` method.
In addition to that **``MegatronGPTModel`` should have member method or properties which would allow to set or get model's template.** That template should be serialized/retrieved from ``model._cfg`` object and saved to .yaml file during model serialization.

Context. Base models do not require this method. But aligned models require setting the correct template for how user, assistant turns as well as system prompt need to be presented to the model. This means that the user of aligned .nemo checkpoint need to chase model's developer or documentation to understand what template they should set.

Example usage:
```python
messages = [
    {"role": "system", "content": "You are a helpful AI assistant well trained in astrophysics. You produce helpful and insightful responses."},
    {"role": "user", "content": "Does Earth revolve around the Sun?"},
    {"role": "assistant", "content": "Yes, but to be more precise, Earth-Sun system revolves around a common center of mass which is few miles off Sun's center."},
    {"role": "user", "content": "How many miles is it away from Sun's center?"}
]

model.eval()
responses=model.chat(messages)
print(responses)
```
If user calls ```.chat``` method on base model it should raise an exception saying chat template is not set.


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Update manifest.py to speedup loading tarred datasets
State: closed | Pull request: yes
Labels: name: common

A simple trick to skip checking for relative filepaths when using tarred datasets. On my local desktop, the modification reduces the manifest loading time for tarred LS data from 4.98s to 3.28s, with 34% relative speedup. The trick is ugly...so please let me know of better ways to do it...

jenkins

> Is there any update on this ? @stevehuang52 @anteju

@titu1994 Nope, I think it's ready to merge if there're no other comments.

jenkins
FSDP + Tensor Parallelism
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

This supports torch FSDP training with Tensor-parallelism.

Set `model.fsdp=True` to enable FSDP. Gradient reduction data type can be set as `model.fsdp_grad_reduce_dtype=bf16`.

FSDP mode is not compatible with the Distributed Optimizer and O2 optimizer wrapper, thus should set `model.megatron_amp_O2=False` and not use `model.optim.name=distributed_fused_adam`.

To store checkpoint in FSDP sharded format, set model.fsdp_sharded_checkpoint=True.

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

@erhoo82 Thanks for hard work. Do you have timeline in mind when the PR will get merged? Thanks!

@stu1130 Have been caught up by high priority work. Trying to close soon.

@athitten,  @ericharper 
- I have removed the torch FSDP patch file as the bug was fixed at PyT23.09 container.
- I have fixed some confusion in the mixed precision definition. Currently, NeMo sets the master param data type as FP32 when `megatron_O2=False`, which is the case of FSDP. I removed explicit model data type casting to FP32 after FSDP because it is already FP32 when `megatron_O2=False`. Also, `param_dtype` in FSDP sets only the compute data type in forward and backward propagation, which matches the original intention of FSDP design.

jenkins

@athitten Is the below error relevant?
`ModuleNotFoundError: No module named 'transformer_engine'`

jenkins

jenkins

jenkins

jenkins
[ASR] Online Speaker Diarization with ASR: Part-1
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

This PR addresses utility functions (not core functions) for online diarization with ASR (multi-speaker ASR).
Streaming ASR and diarization utility functions and classes are added with unit tests.

This is Part1 of the clustering online diarization integration.
In Part2, example file, yaml file and tutorial notebook will be uploaded.


**Collection**: ASR

# Changelog 
`nemo/collections/asr/models/__init__.py`
Adding `OnlineClusteringDiarizer` class

`nemo/collections/asr/parts/utils/decoder_timestamps_utils.py`
adding `FrameBatchASRLogitsSample` and `FrameBatchASRLogits` 
for ASR with timestamp generation

`nemo/collections/asr/parts/utils/diarization_utils.py`
`OnlineDiarWithASR` and a few utility functions

`nemo/collections/asr/parts/utils/streaming_utils.py`
Adding `FrameBatchVAD` class which handles VAD streaming.


# Usage
Usage will be included in part2 with example file and yaml file.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in NeMo ASR.

# Additional Information

N.A.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.
Add support for non-FP32 distributed optimizer state
State: closed | Pull request: yes
Labels: name: core

name: stale

name: NLP

# What does this PR do ?

The distributed Adam optimizer can now be configured with BF16 or FP16 optimizer state.

**Collection**: NLP

# Changelog 
- Add support for non-FP32 distributed optimizer state

# Usage

Run a language model, e.g. with the config at https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml.

Enable the distributed optimizer with `model.optim.name=distributed_fused_adam` and set the optimizer state dtype with `+model.optim.dtype=bfloat16`.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information



This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Change FP8 Defaults
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Change the certain FP8 parameter defaults to more recommended values so that new users can find the optimal FP8 recipe faster.

**Collection**: NLP

# Changelog 
- Change all defaults of `fp8_amax_history_len` from 1 to 1024
- Change all defaults of `fp8_amax_compute_algo` from 'most_recent' to 'max'
- Change config defaults of `fp8_hybrid` to True to mark that hybrid is the preferred fp8 mode (note `fp8` is still False so by default FP8 is still off.)

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
add Dutch P&C FC model info
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Update scores table for Fast Conformer Dutch models

**Collection**: [ASR]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation


jenkins

jenkins
FSDP + Tensor Parallelism with Patch
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

This supports torch FSDP training with Tensor-parallelism.

Set `model.fsdp=True` to enable FSDP. Gradient reduction data type can be set as `model.fsdp_grad_reduce_dtype=bf16`.

FSDP mode is not compatible with the Distributed Optimizer and O2 optimizer wrapper, thus should set `model.megatron_amp_O2=False` and not use `model.optim.name=distributed_fused_adam`.

To store checkpoint in FSDP sharded format, set `model.fsdp_sharded_checkpoint=True`.

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add support for swiglu fusion
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: Speaker Tasks

name: CI

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix CPU initialization of GPT models
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add `use_cpu_initialization` check in MegatronGPTModel.
Allows conversion scripts such as `convert_nemo_gpt_to_mcore.py` to run for bigger models in cpu mode.

**Collection**: NLP

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Fix k2 installation: update for latest PyTorch, move script to `installers` dir
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

- fix k2 installation: use latest release compatible with the latest PyTorch version
- move k2 installer to `scripts/installers`directory

**Collection**: [CI, ASR]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Codec] Finite scalar quantizer
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

This PR adds finite scalar quantizer as used in
Mentzer et al., Finite Scalar Quantization: VQ-VAE Made Simple (https://arxiv.org/abs/2309.15505v1)

**Collection**: TTS

# Changelog 
- Added `FiniteScalarQuantizer`
- Added unit tests

# Usage
This can be used as a drop-in replacement for `ResidualVectorQuantizer`, for example using the following configuration

```yaml
model:
  ...
  commit_loss_scale: 0.0 # need to be set to zero, since FSQ does not include commit loss
  ...
  vector_quantizer:
    _target_: nemo.collections.tts.modules.audio_codec_modules.FiniteScalarQuantizer
    num_levels: [8, 5, 5, 5]
```

For using larger vector, `GroupFiniteScalarQuantizer` can be used to repeat quantization pattern several times, for example using the following configuration

```yaml
model:
  ...
  commit_loss_scale: 0.0 # need to be set to zero, since FSQ does not include commit loss
  ...
  vector_quantizer:
    _target_: nemo.collections.tts.modules.audio_codec_modules.GroupFiniteScalarQuantizer
    num_groups: 3
    num_levels_per_group: [3, 4]
```

This corresponds to FSQ with `num_levels: [3, 4, 3, 4, 3, 4]`.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins
Distributed optimizer support for experimental FP8 tensors
State: closed | Pull request: yes
Labels: name: core

name: stale

name: NLP

name: CI

# What does this PR do ?

This PR integrates with Transformer Engine's experimental `Float8Tensor`s. This allows the model to only store FP8 weight matrices. The distributed optimizer stores an FP32 master copy of the weights and performs param all-gathers in FP8.

**Collection**: NLP

# Changelog 
- Adds distributed optimizer support for FP8 parameters
- Adds the option to initialize GPT with FP8 parameters

# Usage

Run GPT, e.g. with the config at https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml.

Enable FP8 support with `model.fp8=True`, FP8 parameters with `model.fp8_params=True`, and the distributed optimizer with `model.optim.name=distributed_fused_adam`.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [x] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Depends on https://github.com/NVIDIA/apex/pull/1749
* Uses Transformer Engine `Float8Tensor` added in https://github.com/NVIDIA/TransformerEngine/pull/452
* Closes https://github.com/NVIDIA/NeMo/pull/7469 and https://github.com/NVIDIA/NeMo/pull/7565

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
[TTS] Add period discriminator and feature matching loss to codec recipe
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Add the feature matching loss and period discriminator used in most traditional vocoders. These will be used in my next PR which adds a configuration for creating a codec from mel spectrogram.

**Collection**: [TTS]

# Changelog 
- Add absolute feature matching loss.
- Add period discriminator
- Add wrapper for supplying multiple discriminators.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
different truncation types for long context truncation, addition of longbench tasks, preprocessing bug fixs
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Adding able to truncate from left/middle/right portions of context. 
Adding LongBench tasks to dataset preparation script.
Changed placement of remove_newline_tab flag in prepare_truncated_data.py so there these characters are removed prior to checking if the context's sequence length is greater than the max sequence length. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add back import guard
State: closed | Pull request: yes
Labels: name: core

name: NLP

# What does this PR do ?

Add back the import guard that was deleted in a previous PR

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #7750 


jenkins

jenkins