renwei2024/nemo-github-issues huggingface dataset:
NeMo ASR Demo
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Hopefully fix the microphone access error for the transcription demo on the ASR docs main page.

**Collection**: docs

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Script is failing at "punctuator = PunctuationCapitalizationModel.from_pretrained("punctuation_en_bert")"
State: closed | Pull request: no
Labels: name: bug

name: stale

**I am getting the following error at "punctuator = PunctuationCapitalizationModel.from_pretrained("punctuation_en_bert")"
I am using Ubuntu 22.04.2 LTS**

 File "/home/deb/.local/lib/python3.10/site-packages/nemo/core/classes/common.py", line 852, in from_pretrained
    instance = class_.restore_from(
  File "/home/deb/.local/lib/python3.10/site-packages/nemo/core/classes/modelPT.py", line 435, in restore_from
    instance = cls._save_restore_connector.restore_from(
  File "/home/deb/.local/lib/python3.10/site-packages/nemo/core/connectors/save_restore_connector.py", line 248, in restore_from
    self.load_instance_with_state_dict(instance, state_dict, strict)
  File "/home/deb/.local/lib/python3.10/site-packages/nemo/core/connectors/save_restore_connector.py", line 203, in load_instance_with_state_dict
    instance.load_state_dict(state_dict, strict=strict)
  File "/home/deb/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for PunctuationCapitalizationModel:
	Unexpected key(s) in state_dict: "bert_model.embeddings.position_ids".
add bloom support
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Add BLOOM model support to NeMo.

**Collection**: [Note which collection this PR will affect]

[NLP]

# Changelog 
- Add BLOOM model architecture.
  -  Add input embedding layernorm to the language model.
  - Add model configurations.
- A conversion script to convert HuggingFace checkpoints to `.nemo` file.
- Configurations and scripts for pertaining, evaluation, and SFT.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix syntax error introduced in PR-7079
State: closed | Pull request: yes
Labels: name: cherry-pick

name: stale

name: NLP

# What does this PR do ?

Fix quotes causing syntax error in notebook json

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
bug fix - alibi causal: for ith query, bias is m · [−(i − 1), ..., −2, −1, 0] for the first i keys
State: closed | Pull request: yes

# What does this PR do ?

Fix alibi position embedding for causal attention. 


**Collection**: NLP

# Changelog 
Existing: returns (1, num_heads, 1, key_length)
Fixed: returns (1, num_heads, query_length, key_length), where for ith query, the bias is m · [−(i − 1), ..., −2, −1, 0] for the first i keys, as per the alibi paper
  
**PR Type**:
- [ ] Bugfix


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
When importing nemo.collections.asr: ImportError: cannot import name 'get_num_classes' from 'torchmetrics.utilities.data'
State: closed | Pull request: no
Labels: name: bug

name: stale

I cannot start my app due to an import error when using NeMo ASR. The relevant stack trace: 

```
  (...)
  File "/asr/asr_solution.py", line 34, in <module>
    import nemo.collections.asr as nemo_asr # NeMo's ASR library
  File "/usr/local/lib/python3.10/site-packages/nemo/collections/asr/__init__.py", line 15, in <module>
    from nemo.collections.asr import data, losses, models, modules
  File "/usr/local/lib/python3.10/site-packages/nemo/collections/asr/losses/__init__.py", line 15, in <module>
    from nemo.collections.asr.losses.angularloss import AngularSoftmaxLoss
  File "/usr/local/lib/python3.10/site-packages/nemo/collections/asr/losses/angularloss.py", line 18, in <module>
    from nemo.core.classes import Loss, Typing, typecheck
  File "/usr/local/lib/python3.10/site-packages/nemo/core/__init__.py", line 16, in <module>
    from nemo.core.classes import *
  File "/usr/local/lib/python3.10/site-packages/nemo/core/classes/__init__.py", line 16, in <module>
    from nemo.core.classes.common import (
  File "/usr/local/lib/python3.10/site-packages/nemo/core/classes/common.py", line 31, in <module>
    from nemo.utils import logging
  File "/usr/local/lib/python3.10/site-packages/nemo/utils/__init__.py", line 22, in <module>
    from nemo.utils.lightning_logger_patch import add_memory_handlers_to_pl_logger
  File "/usr/local/lib/python3.10/site-packages/nemo/utils/lightning_logger_patch.py", line 18, in <module>
    import pytorch_lightning as pl
  File "/usr/local/lib/python3.10/site-packages/pytorch_lightning/__init__.py", line 20, in <module>
    from pytorch_lightning import metrics  # noqa: E402
  File "/usr/local/lib/python3.10/site-packages/pytorch_lightning/metrics/__init__.py", line 15, in <module>
    from pytorch_lightning.metrics.classification import (  # noqa: F401
  File "/usr/local/lib/python3.10/site-packages/pytorch_lightning/metrics/classification/__init__.py", line 14, in <module>
    from pytorch_lightning.metrics.classification.accuracy import Accuracy  # noqa: F401
  File "/usr/local/lib/python3.10/site-packages/pytorch_lightning/metrics/classification/accuracy.py", line 18, in <module>
    from pytorch_lightning.metrics.utils import deprecated_metrics, void
  File "/usr/local/lib/python3.10/site-packages/pytorch_lightning/metrics/utils.py", line 22, in <module>
    from torchmetrics.utilities.data import get_num_classes as _get_num_classes
ImportError: cannot import name 'get_num_classes' from 'torchmetrics.utilities.data' (/usr/local/lib/python3.10/site-packages/torchmetrics/utilities/data.py)
```

This does not happen in my local virtual environment, but in a docker container, so I am guessing that somehow not all necessary files have been installed when building the container. However, `nemo_toolkit[asr]` is part of the libs being pipped via Dockerfile, and I thought all necessary requirements were contained in that installation. Should I add anything specific to the requirements?
skip all reduce fix
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

In certain scenarios(ex: single stage embeddings) there are no gradients for a rank with sequence parallel.  This PR will skip the all reduce on those ranks.

**Collection**: [NLP]

# Changelog 
- skip seqparallel all reduce on no grads

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix syntax error introduced in PR-7079
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix quotes causing syntax error in notebook json

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
MSDD inference is too slow
State: closed | Pull request: no
Labels: name: stale

I run the MSDD model on Nvidia A10 (24GB), but the inference is too slow, I looked on the code and there is a lot of traffic between the CPU and GPU and vice versa.

most of the time GPU utilization is on 0%

First the data is split into short segments according to the number of scales (I have 5 scales).
After each scale splitting the embedding extraction is applied and save the embedding to pkl file.
Then the the clustering is applied and finally the MSDD is applied.

Is there something that can be done in order to speed up the inference?
Is there any flag for parallelism the embedding extraction stage?

please help.
Fix absolute path in path join call
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

This fixes a regression introduced in 39aff5ca3d52247e136a86429d567255c02a5d44 when joining paths was changed from string concatenation to `os.path.join`. At the moment the call disregards the first part (`frame_pred_dir`) and always returns a path starting with `/overlap_smoothing_output`.

**Collection**:  asr

# Changelog 

- Remove leading slash in argument to `os.path.join`

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [-] Did you write any new necessary tests?
- [-] Did you add or update any necessary documentation?
- [-] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [-] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
No code for SLU metrics computation
State: closed | Pull request: no
Labels: name: stale

Hi!
I was trying to find a place where SLU metrics are computed. AFAIU this is going to be [here](https://github.com/NVIDIA/NeMo/blob/main/examples/slu/speech_intent_slot/eval_utils/evaluator.py#L92). But I can't find where ```ErrorMetric ``` class is defined. Are you planning to make this code publicly available?
[feature request] RNN-T loss to support arbitrary padding
State: closed | Pull request: no
Labels: name: stale

I think currently, the RNN-t loss supports any padding for every example in batch except the longest example. What is the nature of such limitation? (as it seems that the underlying code is already able to handle padding, so not clear why the longest example is different)

Original discussion: https://github.com/NVIDIA/NeMo/issues/6797#issuecomment-1589149555
Add GQA config to megatron gpt model
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add GQA config in GPT config file.
__Note__ that this PR merge into `mcore_gpt_path` not `main`.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* Set `num_query_groups` in `MegatronGPTModel` `cfg.model`. If `None`, normal (multi-head) attention is used. If `1`, multi-query attention is used.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Disable distopt contiguous param buffer by default
State: closed | Pull request: yes
Labels: name: bug

name: NLP

# What does this PR do ?

This fixes a bug reported by @wdykas, where https://github.com/NVIDIA/NeMo/pull/5684 significantly increased the size of GPT checkpoints.

I think this is because this PR configures the model parameters so they are all views into a single large buffer. However, PyTorch tensors save their underlying memory buffer when pickled, so checkpointing the model results in multiple redundant copies of all the parameters. This PR disables this behavior by default, at the expense of adding an extra copy after each param all-gather.

**Collection**: NLP

# Changelog 
- Disable distopt contiguous param buffer by default.

# Usage
This fix is applied by default. To restore the old behavior, configure the optimizer with `contiguous_param_buffer=True`.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

Pinging @wdykas.

# Additional Information
* Fixes bug from https://github.com/NVIDIA/NeMo/pull/5684.
Adding docs and models for multiple lookahead cache-aware ASR
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

# What does this PR do ?

This PR adds the following items related to multi-lookahead cache-aware streaming models:
+ documentation
+ support of multi-lookahead for transcribe_speech.py, transcribe_speech_parallel.py, and speech_to_text_eval.py, 
+ one pre-trained model which is uploaded on NGC and supports 0s, 80ms, 480ms and 1040ms:
https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_fastconformer_hybrid_large_streaming_multi

# Changelog 
- Added a multi-lookahead cache-aware streaming model which supports multiple look-ahead in one single model.

# Usage
You may switch between different supported look-aheads by:
1- calling the set_default_att_context_size method of the model's encoder like this:

```python
asr_model.encoder.set_default_att_context_size(att_context_size=[70,1])
```

2- set att_context_size when using  transcribe_speech.py, transcribe_speech_parallel.py, or speech_to_text_eval.py.

**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [X] Documentation
Use GPTModel from mcore
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

This PR adds a path to use GPTModel from Megatron Core.

Requirements:
Make sure that megatron core is installed from a recent commit:

```
git clone https://github.com/NVIDIA/Megatron-LM.git && \
            cd Megatron-LM && \
            git checkout 3316e811cc5335ee24c2d203416d864edcf2f7a8 && \
            pip install -e .
```

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage

Set `mcore_gpt=True` in the `MegatronGPTModel` config.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Added script to extract ASR CTC and RNNT models from ASR hybrid models
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Adds a script which extracts pure CTC or pure RNNT models from ASR hybrid models in .nemo format

**Collection**: ASR

# Changelog 
- Added new script in examples/asr/asr_hybrid_transducer_ctc/helpers/convert_nemo_asr_hybrid_to_ctc.py

# Usage
* You can potentially add a usage example below

```shell
python convert_nemo_asr_hybrid_to_ctc.py -i /path/to/hybrid.nemo -o /path/to/saved_ctc_model.nemo -m 'ctc'|'rnnt'
```

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix tensor_parallel function calls to megatron-lm
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Megatron-LM has changed its function APIs in tensor_parallel, where a lot of params (e.g., `use_cpu_initialization`) are moved into a config object (e.g., `megatron.core.model_parallel_config.ModelParallelConfig`), for example [here](https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/tensor_parallel/layers.py#L148)

This PR fixes such errors when trying to use those functions with the latest Megatron-LM codebase, but there may be some other missed functions. 

**Collection**: [NLP]
update TTS readme
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add paths to labeler.
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

found some paths not covered by our labeler.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Fix] load_state_dict in nlp_model.py
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix error in loading checkpoints. The pytorch [load_state_dict()](https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=load_state_dict#torch.nn.Module.load_state_dict) function has return variable that was not returned by this function in this file. 


**Collection**: [NLP]
Merge main to asr_normalize
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: Speaker Tasks

name: CI

name: common

# What does this PR do ?

Merging main to asr_normalize branch

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix caching bug in causal convolutions for cache-aware ASR models
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

# What does this PR do ?

Fixed the caching bug in the convolution layers for cache-aware models

# Changelog 
- Fixed the caching bug in the convolution layers for cache-aware models

**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation
small Bugfix
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: ASR NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
migrated class CSVFieldsMemmapDataset from BioNeMo
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
small Bugfix
State: closed | Pull request: yes
Labels: name: ASR

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: ASR NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
RuntimeError: cannot sample n_sample > prob_dist.size(-1) samples without replacement
State: closed | Pull request: no
Labels: name: bug

name: stale

@titu1994 @meghmak13 
Dear Team,
      I am training **conformer based SSL** model where I have configured the parameters 
**min_duration : 3.2
patch_size: 16
mask_patches: 0.5
num_negatives: 40
sample_from_same_utterance_only: true
sample_from_non_masked: false
codebook_size: 300
num_groups: 2
num_classes: 90000**

 but I am getting following error
**RuntimeError: cannot sample n_sample > prob_dist.size(-1) samples without replacement**  

I am using **Nemo-22.05 images(ngc) and 48, A100-SXM4-40GB GPU** i.e 6 nodes for the training. Kindly suggest me what is need to be done ? Thanks in advance
Fix documentation for Numba
State: closed | Pull request: yes
Labels: name: core

name: cherry-pick

name: ASR

name: NLP

# What does this PR do ?

Fix import of numba to prevent doc breakage.  

importlib cannot actually import a module that is mocked, trying to call `module.__version__` will return a mock module object - not a string and then Version() will fail to parse that mock object.

This PR defers the attempt to find the version compatibility of FP16 to the moment the library is resolved dynamically rather than at static import time.

**Collection**: [Core]

# Changelog 
- Correct numba import to be dynamic only to avoid mock import failing
- Fix NeMo version in docs and readme.md

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Update Frame-VAD doc and fix onnx export
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Update some VAD doc and fix onnx exportable.

Had to overwrite the `forward_for_export` function because the MLP class in `collections.common` doesn't have neural types and the param list of MLP doesn't have `encoder_output` which is used in ASR models

**Collection**: [ASR]
fix pos id - hf update
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
File not found error while loading titanet_small
State: closed | Pull request: no
Labels: name: bug

Hello,

I am trying to load titanet small using the following line of code: 
```
nemo_speaker_model_small = EncDecSpeakerLabelModel.from_pretrained(model_name="titanet_small")
```

Here is the error:
```FileNotFoundError: Model titanet_small was not found. Check cls.list_available_models() for the list of all available models.```

I am using the latest nemo_toolkit
support llama
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Support LLaMa model: checkpoint conversion, pretraining, SFT, PEFT

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Support llama
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Support LLaMa model: checkpoint conversion, pretraining, SFT, PEFT

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
remove pos emb from state dict for old models
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Adding docs and models for multiple lookahead cache-aware ASR
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

This PR adds the following items related to multi-lookahead cache-aware streaming models:
+ documentation
+ support of multi-lookahead for transcribe_speech.py, transcribe_speech_parallel.py, and speech_to_text_eval.py, 
+ one pre-trained model which is uploaded on NGC and supports 0s, 80ms, 480ms and 1040ms:
https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_fastconformer_hybrid_large_streaming_multi

# Changelog 
- Added a multi-lookahead cache-aware streaming model which supports multiple look-ahead in one single model.

# Usage
You may switch between different supported look-aheads by:
1- calling the set_default_att_context_size method of the model's encoder like this:

```python
asr_model.encoder.set_default_att_context_size(att_context_size=[70,1])
```

2- set att_context_size when using  transcribe_speech.py, transcribe_speech_parallel.py, or speech_to_text_eval.py.

**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [X] Documentation
Refined export_config
State: closed | Pull request: yes
Labels: name: TTS

name: cherry-pick

name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix documentation for Numba
State: closed | Pull request: yes
Labels: name: core

name: ASR

name: NLP

# What does this PR do ?

Fix import of numba to prevent doc breakage.  

importlib cannot actually import a module that is mocked, trying to call `module.__version__` will return a mock module object - not a string and then Version() will fail to parse that mock object.

This PR defers the attempt to find the version compatibility of FP16 to the moment the library is resolved dynamically rather than at static import time.

**Collection**: [Core]

# Changelog 
- Correct numba import to be dynamic only to avoid mock import failing
- Fix NeMo version in docs and readme.md

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Standalone Embedding Stage
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

This PR adds support for standalone embeddings in NeMo FW

**Collection**: [NLP]

# Changelog 
- Adds standalone embedding stage support

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Support for flash attention 2.0
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Adds naive support for flash attention 2.0. 

[TODO]: Add support for the fixed seq length implementation which should be faster.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
memmap worker arg
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

GPTSFTDataset uses memmap data loading which uses python mp to create index files. The number of workers for the index file creation defaults to os.cpu_count() // 2 which might be too high for shared nodes. 

This PR adds an arg to specify the number of workers that the creation of index files will use.

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)