renwei2024/nemo-github-issues huggingface dataset:
Adding method back that was removed accidentally
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

@ericharper  should we move this to the base model instead ?

jenkins

> @ericharper should we move this to the base model instead ?

Yes, let's do it in another pr.
Cache Aware Streaming tutorial notebook [old PR, closed]
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: CI

name: common

Update: this PR is closed, replaced with https://github.com/NVIDIA/NeMo/pull/8296

cc @vsl9, @VahidooX 

# What does this PR do ?

Adding the notebook that allows us to do real-time streaming using a cache-aware fastconformer model (supports single-lookahead and multi-lookahead models).

**Collection**: `tutorials/asr`

# Changelog 
- Add cache-aware streaming notebook in `tutorials/asr`
- Add call to `self.setup_streaming_params` inside `set_default_att_context_size` method of `ConformerEncoder`.



# Usage

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Restore O2 FP16/BF16 wrapper for GPT
State: closed | Pull request: yes
Labels: name: bug

name: NLP

# What does this PR do ?

When running GPT with O2 enabled, I am seeing errors at the following line since `_wrap_model_for_O2` is not defined:

https://github.com/NVIDIA/NeMo/blob/852338459c78c257a2b5f6688c0b1a5d2b048366/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py#L266

This function was removed in https://github.com/NVIDIA/NeMo/pull/7814. I see that this PR added the function to BERT, so could this be a bug?

**Collection**: NLP

# Changelog 
- Restore `_wrap_model_for_O2` function in GPT

# Usage

Run GPT, e.g. with the config at https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml, and set `model.megatron_amp_O2=True`

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

Pinging @shanmugamr1992 

# Additional Information
* Bug from https://github.com/NVIDIA/NeMo/pull/7814

Closed by https://github.com/NVIDIA/NeMo/pull/8038.
Nvidia nemo blog and Apples to Oranges comparison of H200 vs A100
State: closed | Pull request: no
Labels: name: bug

name: stale

Can someone from nemo fix this performance comparison and have A100 also on the same nemo version as the H200? The same blog is posted in this repo readme as indication of nemo performance numbers.

https://developer.nvidia.com/blog/new-nvidia-nemo-framework-features-and-nvidia-h200-supercharge-llm-training-performance-and-versatility/

A100: Nemo 23.08
H200: Nemo 24.01-alpha

I do not understand why Nvidia would publish this chart using different versions of nemo. Is A100 not supported in the latest nemo alpha/git release? Please help the community by doing apples to apples comparison, or provide the code/data to reproduce the data. 

Moreover, most of us do not have access to 8xA100 or 8XH100 to verify the results so it is even more important we, as potential end users, see true apples to apples comparisons.  Thanks.

![image](https://github.com/NVIDIA/NeMo/assets/417764/209d4151-b0f0-4222-831f-4aa0536b7a39)



This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

Touch.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Handle float limit_val_batches
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


@athitten Can you write unit tests for this? It would be nice to have at leas a docsting (it could be in the test itself) describing what is the expected behaviour of this function and what purpose does it serve.

If this code is here to fix a bug in PTL or cover some case not covered by PTL please describe what it is and write a comment in the code. This looks like a hack to me and should be properly commented to help future developers understand what's going on in this function and when potentially can it be removed.

jenkins

jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Upgrade to PTL 2.1 and 2.2
State: closed | Pull request: yes
Labels: name: core

name: NLP

name: CI

name: Multi Modal

# What does this PR do ?

Upgrades NeMo to use PTL 2.1 and 2.2. NeMo now uses PTL 2.2.1 (the current latest release).

Major changes that are seen across a lot of files in the PR:

1. The function signature of `train/validation/test_step` is changed while using the `dataloader_iter` API. The `batch_idx` arg is removed from the function signature and `next(dataloader_iter)` now returns `batch, batch_idx, dataloader_idx`. 

With this change the support for multiple dataloaders which was removed with PTL 2.0 is added back. [More details here](https://lightning.ai/docs/pytorch/stable/upgrade/from_2_0.html#:~:text=predict_step(self%2C%20dataloader_iter%2C%20batch_idx)%20in%20LightningModule-,remove%20batch_idx%20from%20the%20signature%20and%20expect%20dataloader_iter%20to%20return%20a%20triplet%20(batch%2C%20batch_idx%2C%20dataloader_idx),-PR18390)

2. PTL 2.1 and above does not support having both precision and  precision plugin in the trainer. It can be observed in the changes that if precision plugin is created, then precision is made None.

3. Some of the other simple changes are explained with comments in the PR.


**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

jenkins


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
Jiaqiz/option to disable adapters & merge all lora layers
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a condition for all adapters modules: run through the adapter only when it is enabled

Add merging script for all lora layers.

# Usage
To enable or disable an adapter, we can use

```python
AdapterModuleMixin.set_enabled_adapters(name, enabled = True)
AdapterModuleMixin.set_enabled_adapters(name, enabled = False)
```
If ```name``` is not provided, then all adapters will be enabled/disabled. For example, one can disable all adapters using the following codes, which is equivalent to use only the base model.
```python
from nemo.core.classes.mixins.adapter_mixins import AdapterModuleMixin
for _, module in model.named_modules():
    if isinstance(module, AdapterModuleMixin) and module.is_adapter_available():
        module.set_enabled_adapters(enabled=False)
```

**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


@HeyyyyyyG is this WIP?

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

> LGTM! but i think we should merge #7988 before this one

Yes totally agree

jenkins

jenkins

jenkins

jenkins

jenkins
NeMo Multimodal Docs and Tests Initial PR
State: closed | Pull request: yes
Labels: name: documentation

name: CI

# What does this PR do ?

Add Docs and Tests to NeMo Multimodal

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
Support torch jit script
State: closed | Pull request: yes
Labels: name: core

# What does this PR do ?

Fix compatibility of the core classes (`NeuralModule`, `Exportable`, etc.) with `torch.jit.script`.
Currently inheriting from `NeuralModule` instead of `nn.Module` breaks `torch.jit.script` compiler.

**Collection**: [Core]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage


```python
class SimpleLinear(NeuralModule):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(10, 20, bias=False)

    def forward(self, x):
        return self.linear(x)

module = torch.jit.script(SimpleLinear())
x = torch.zeros(2, 10)
result = module(x)
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

jenkins

Tests seem to have failed, could you check ?

jenkins

jenkins

jenkins

> Tests seem to have failed, could you check ?

All tests are passed. There were some weird issues with connection unrelated to this PR.
Fix defer_setup issue of failing to create validation and test dataloaders
State: closed | Pull request: yes
Labels: name: core

name: stale

# What does this PR do ?

After #7455 , `ModelPT.val_dataloader()` and `ModelPT.test_dataloader()` return empty lists instead of None.
This modification causes `ModelPT.setup()` failed to pass the 'if' statements at line 835 and 844, and the dataloaders are therefore remaining empty lists without construction:
https://github.com/NVIDIA/NeMo/blob/8216178cfaf89228884bd1e4205776a2667c6e51/nemo/core/classes/modelPT.py#L829-L845

This PR changes the `self.xxx_dataloader() is None` statements into `not self.xxx_dataloader()`, such that both "None" and empty lists could fit the condition.


**Collection**: All

# Changelog 
- change `self.xxx_dataloader() is None` into `not self.xxx_dataloader()`, so as to handle both None and empty lists
```
def setup(self, stage: Optional[str] = None):
    ...

    if stage in ('fit', 'validate'):
        val_deferred_setup = ...
        if not self.val_dataloader() and val_deferred_setup:
            self.setup_multiple_validation_data(val_data_config=self._cfg.validation_ds)

    if stage == 'test':
        test_deferred_setup = ...
        if not self.test_dataloader() and test_deferred_setup:
            self.setup_multiple_test_data(test_data_config=self._cfg.test_ds)
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation


# Additional Information
* Related to #5569 , #7455 


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
GPTSFTChatDataset loss_mask becomes all False when prompt length > max_seq_length
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

In the `GPTSFTChatDataset`, if the first prompt length exceeds `max_seq_length`, all following turns are truncated out. Then the `loss_mask` becomes all `False` for the example.

https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_chat_dataset.py#L359

This is problematic because, when the `loss_mask` is all False, the `loss` of the MegatronGPTModel will be `nan`.

https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py#L1015

**Steps/Code to reproduce bug**

This can be reproduced by passing into `GPTSFTChatDataset` an example whose first turn prompt has > 2048 tokens when the `max_seq_length=2048`. Then use the `GPTSFTChatDataset` in a supervised fine-tuning job (e.g., `train_gpt_sft.py` in [NeMo-Aligner](https://github.com/NVIDIA/NeMo-Aligner/blob/main/examples/nlp/gpt/train_gpt_sft.py))

**Expected behavior**

In the `collate_fn` function, check if the loss_masks of all examples are not empty. If not, raise an error.

I ran into the same issue => hacked the code to zero out the loss on problematic micro-batches: https://github.com/odelalleau/NeMo/commit/b383e6a1e55dd315fa194c0a8fafd7ee3fef3437

(obviously not a proper fix but can be useful to get unblocked)

Yeah, this will also fix the problem. 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue should keep active since the same problem has been encountered by others as well.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Can I use a Sentencepiece tokenizer model from my ASR model in order to train a NeuralRescorer?
State: closed | Pull request: no

Hi,

I would like to train a Neural Rescorer model for ASR using the transformer_lm.py script. I have one small question:

In the documentation it says: "Our LMs support all tokenizers available in NeMo, but require special beginning-of-string <bos> and end-of-string <eos> tokens." 

But I imagine that the tokenizer of the TransformerLM must be the same as the tokenizer of the ASR model, right? Can I pass the tokenizer model of the ASR model directly to TransformerLM? Should I add a <bos> and <eos> token and change the size of the tokenizer? ( but I imagine the ASR and Transformer should have the same tokenizer size)

Could you help me on this matter?

Thank you in advance!



Closing this issue as it was already answered here:

https://github.com/NVIDIA/NeMo/issues/4390
Finetuning 
State: closed | Pull request: no
Labels: name: stale

How is it possible to finetune the citrinet model? On nvidias documentation I saw something similar to TAO toolkit but never saw any example on it. On the other hand I saw the japanese finetuning process. And for me I’m spotting for a wer of less 5%. I got a deadline in less than 3 weeks , can someone help ?!

Hi, apologies for late response. 

Have you looked at these sections on finetuning: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/configs.html#fine-tuning-configurations and https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/examples/kinyarwanda_asr.html#training-from-scratch-and-finetuning 

Let us know if you have issues using the scripts. 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Temporary pin Lightning-Utilities version due to broken NamedTuple
State: closed | Pull request: yes

# What does this PR do ?

Pin Lightning-Utitilies package version due to broken NamedTuple in `0.10.0`
https://github.com/Lightning-AI/utilities/pull/160 introduced the issue, 
https://github.com/Lightning-AI/utilities/blob/a00e1144dd300e3bc2fc8746a8e85510ffe35aa7/src/lightning_utilities/core/apply_func.py#L67 here NamedTuple force converted to tuple.

This fixes the bug https://nvbugspro.nvidia.com/bug/4405798 (broken `ASR_TTS_Tutorial`)

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

@titu1994 It seem to be a know issue, hope will be fixed soon https://github.com/Lightning-AI/utilities/issues/196

Merging for now - fyi @ericharper @athitten 

Don't merge this to main for now (unless Eric or @athitten gives the go ahead for main)

@titu1994 The issue is fixed in Lightning-Utilities in `main`, waiting for the next release: https://github.com/Lightning-AI/utilities/pull/210
Check `torchaudio` dependencies in installation script
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Check `torchaudio` dependencies. Exit with the appropriate message if some dependencies are not installed.
Fixes bug: https://nvbugspro.nvidia.com/bug/4403803

**Collection**: [CI]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Check `torchaudio` dependencies in installation script
State: closed | Pull request: yes

# What does this PR do ?

Check `torchaudio` dependencies. Exit with the appropriate message if some dependencies are not installed.
Fixes bug: https://nvbugspro.nvidia.com/bug/4403803

**Collection**: [CI]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


> Ooh so the issue was missing dependencies and not the torchaudio code itself ? Won't we need to install the dependencies anyway inside container?

@titu1994 We install the dependencies when building the container https://github.com/NVIDIA/NeMo/blob/main/Dockerfile#L34
Check `torchaudio` dependencies in installation script
State: closed | Pull request: yes

# What does this PR do ?

Check `torchaudio` dependencies. Exit if not installed with the appropriate message.
Fixes bug: https://nvbugspro.nvidia.com/bug/4403803

**Collection**: [CI]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
NMT lang code fix en to any training
State: closed | Pull request: yes
Labels: name: core

name: stale

name: NLP

# What does this PR do ?

Fix for en to any training / finetuning of model for different language pairs, 
With this fix , the en to any NMT model can be finetuned for a different subset of language pairs than the model was originally pretrained on. During pretraining , we need to set the pretrained_language_list to - null.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Adds a new field pretrained_language_list in the aayn_base_megatron.yaml config file, which specifies the list of languages the model was pretrained on
- Changes to the lang id processing so that the model can finetuned on a small subset of languages of the initial pretrained list

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Enhance flexibility by passing callbacks as method argument
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

This PR enhances the flexibility of CustomProgressBar by modifying the create_trainer method to accept callbacks as a method argument, replacing its previous fixed implementation.

**Collection**: NLP

# Changelog 
Moved callbacks to method arguement.

# Usage
N/A

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] Refactor
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


@titu1994 could you also please help to review this? Short one, focusing on UX

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

@titu1994 could you have another look?
Update link to yaml file in ASR_with_Transducers.ipynb
State: closed | Pull request: yes
Labels: name: ASR

changed link from `../../examples/asr/conf/contextnet_rnnt/contextnet_rnnt.yaml` to `/content/configs/contextnet_rnnt.yaml` because the latter did not exist and threw and error when trying to load the yaml config file.

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix tokenizer argparse in scripts
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Fix missing argument in script argparse

**Collection**: [ASR, NLP]

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Fix tokenizer argparse in scripts
State: closed | Pull request: yes

# What does this PR do ?

Fix missing argument in script argparse

**Collection**: [ASR, NLP]

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Added VectorQuantizer base class
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

This PR adds an abstract base class for vector quantizers.

**Collection**: TTS

# Changelog 
- Added `VectorQuantizerBase`

# Usage
n/a

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] Refactor
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

jenkins
Logging changes tested for gpt_pretraining
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Enables use of megatron timers in NeMo GPT pretraining and megatron_base based models.

**Collection**: collections/nlp/models/language_modeling

# Usage

```
 model.enable_megatron_timers=True
 model.megatron_timer_kwargs.log_every_n_steps= 10
 model.megatron_timer_kwargs.log_mode=minmax
 model.megatron_timer_kwargs.barrier= False 
```

```python
torchrun --nnodes=1 --nproc-per-node=8 \
    /gsw/code/NeMo/examples/nlp/language_modeling/megatron_gpt_pretraining.py  \
    --config-path=/gsw/code/NeMo/examples/nlp/language_modeling/conf \
    --config-name=megatron_gpt_config \
    trainer.devices=8 \
    trainer.num_nodes=1 \
    model.enable_megatron_timers=True \
    ++model.megatron_timer_kwargs.log_option=max \

```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

Created a PR to release branch: https://github.com/NVIDIA/NeMo/pull/8354/files
Update asr_language_modeling.rst: Add a missing word
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Simply add a missing word in the doc `Update asr_language_modeling.rst`.

# Changelog 
NA

# Usage
NA

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Thanks !
Make pipelined TP comm overlap available with mcore
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Enable the use of pipelined tensor-parallel communication overlap with mcore path.
Removed the condition that enables the use of `ub_tp_comm_overlap` only with `transformer_engine=True`.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

jenkins

jenkins
.ctm in data simulator annotator compliant with RT-09 specification
State: closed | Pull request: yes
Labels: name: ASR

name: Speaker Tasks

Redid this PR from https://github.com/NVIDIA/NeMo/pull/7999

An attempt to fix https://github.com/NVIDIA/NeMo/issues/7445 so that the data simulator .ctm are compliant with RT-09 specification (see https://web.archive.org/web/20170119114252/http://www.itl.nist.gov/iad/mig/tests/rt/2009/docs/rt09-meeting-eval-plan-v2.pdf): 
```
<SOURCE><SP><CHANNEL><SP> <BEG-TIME><SP><DURATION><SP><TOKEN><SP>
<CONF><SP><TYPE><SP><SPEAKER><NEWLINE>
```

I have put <NA> for fields unknown e.g. <TOKEN>. 
This makes it also easy to use the generated sessions with https://github.com/lhotse-speech/lhotse

NeMo contributors, please do not merge this PR until I make sure all the CTM in NeMo is following the official CTM format.

Seems that also this is not compliant (speaker id instead of channel): 

https://github.com/NVIDIA/NeMo/blob/fa8d416793d850f4ce56bea65e1fe28cc0d092c0/scripts/speaker_tasks/create_alignment_manifest.py#L74C79-L74C79


This is instead kinda compliant (but lacks <NA> for missing fields):
https://github.com/NVIDIA/NeMo/blob/fa8d416793d850f4ce56bea65e1fe28cc0d092c0/tools/nemo_forced_aligner/utils/make_ctm_files.py#L108

@erastorgueva-nv Elena, I have found a line that renders CTM and I replaced with the `get_ctm_line` to remove the duplication when users create CTM files with NeMo. Please check if there is anything wrong happening and this does not cause any errors. (There is no unit tests for the make_ctm_files.py)

@stevehuang52 We are also making slight changes to data simulator. Please review and approve.

the data simulation parts look good to me

jenkins

@tango4j / reviewers, merge when ready. 

Also reminder, NeMo devs need to explicitly write "jenkins" in order to execute the CI 

> @tango4j / reviewers, merge when ready.
> 
> Also reminder, NeMo devs need to explicitly write "jenkins" in order to execute the CI

Oh, when did it change the protocol? 
Thanks for the heads up. From now on, I will do "jenkins".


jenkins

jenkins

jenkins

jenkins

jenkins