renwei2024/nemo-github-issues huggingface dataset:
st standalone model
State: closed | Pull request: yes
Labels: name: ASR

name: common

# What does this PR do ?

Adds support for ConformerEnc-TransformerDec speech translation (ST) model.

**Collection**: asr

# Changelog 
- Added baseline ST model
- Tested to work with both yttm and SentencePiece (SP) tokenizers, SP is considered default

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Inverse Normalization has some issue
State: closed | Pull request: no
Labels: name: bug

name: stale

Hi Team,

I am trying latest itn model t5 for inference.(https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/itn_en_t5) 

Getting error as the following:
Error executing job with overrides: ['lang=en', 'mode=itn', 'tagger_pretrained_model=itn_en_t5', 'decoder_pretrained_model=itn_en_t5', 'inference.interactive=False', 'inference.from_file=[sample.txt]']
Traceback (most recent call last):
  File "NeMo/examples/nlp/duplex_text_normalization/duplex_text_normalization_infer.py", line 78, in main
    normalizer_electronic = ElectronicNormalizer(input_case="cased", lang=lang, deterministic=True)
  File "/home/ubuntu/NeMo/examples/nlp/duplex_text_normalization/nn_wfst/en/electronic/normalize.py", line 46, in __init__
    self.tagger = ClassifyFst(
  File "/home/ubuntu/NeMo/examples/nlp/duplex_text_normalization/nn_wfst/en/electronic/tokenize_and_classify.py", line 66, in __init__
    electonic_graph = ElectronicFst(deterministic=deterministic).fst
TypeError: __init__() missing 1 required positional argument: 'cardinal'

Is it the issue with the "nemo_text_processing" package?or version mismatch of other packages?
Speaker Diarization - Extracting speaker embeddings for labels from files with multiple speakers from MSDD models
State: closed | Pull request: no
Labels: name: stale

**Is your feature request related to a problem? Please describe.**

One problem I come across often is, that I can't verify already known speakers from other files, since inference only yields labels for a **MSDD model** approach.
For example: I have file **A** with speakers Paul, Tom and Charlie and File **B** with speaker Bert, Charlie and Johann. To determine if **Charlie is in File A and B**, I have to get his speaker embedding. At the moment I have to infer speaker diarization on file A to do this, get the timestamps for Charlie, cut the file at the marks, stitch it together again and get the corresponding feature embedding which I can then compare to file B, by doing the same. However for a production setup I would need the embedding from the MSDD model, but from my understanding the MSDD model only produces a probability vector?

**Describe the solution you'd like**

A solution would be to extract the feature embeddings with some kind of method, after diarization and associate them with the labels. I found the get_cluster_avg_embs_model() method for MSDD models, but I am not sure how to use it and if it yields the correct results (to me its not clear which parameters to use here, since there are a lot of internal states in the model after diarization). So the question would, which parameter yields the best representation of a speaker to recognize him again?
Allow `data_prefix` to be `None` when mocking dataset
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Allow `data_prefix` to be None when mocking dataset, i.e., allow the following config
```yaml
    data_prefix: null
    data_impl: mock
```

Without this change, it will throw en error saying that `None` type doesn't support the `len()` method.

**Collection**: [NLP]

# Changelog 
- nemo/collections/nlp/data/language_modeling/megatron/gpt_dataset.py

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)

cc @titu1994 Could you kindly review it? Thanks.
RuntimeError: torch.cat(): expected a non-empty list of Tensors when sequence parallel is enabled
State: closed | Pull request: no
Labels: name: bug

name: stale

Hi,

I found there are some bugs when sequence parallel is enabled and transformer engine is not used. Specifically, I received 

```
  File "/fsx/NeMo/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py", line 551, in training_step
    self.allreduce_sequence_parallel_gradients()
  File "/fsx/NeMo/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py", line 654, in allreduce_sequence_parallel_gradients
    coalesced = torch._utils._flatten_dense_tensors(grads)
  File "/usr/local/lib/python3.8/dist-packages/torch/_utils.py", line 459, in _flatten_dense_tensors
    return torch._C._nn.flatten_dense_tensors(tensors)
RuntimeError: torch.cat(): expected a non-empty list of Tensors
```
After some debugging, I found attribute `sequence_parallel` is not added to the layer norm because the apex layer norm implementation that NeMo uses does not support sequence parallel: https://github.com/NVIDIA/NeMo/blob/b0e5bf3627dbcfb3f4a72d73d3c5e92184d8b1f6/nemo/collections/nlp/modules/common/megatron/fused_layer_norm.py#L18-L19

Besides, is there any plan to add sequence parallel to `rmsnorm`?
No module named 'nemo.collections' 
State: closed | Pull request: no
Labels: name: bug

I have followed the steps as mentioned in git repo for nemo, but each time I'm installing nemo_toolkit['all'] two packages pynini and mecab-python3 is failing to build wheels, also tried nemo_toolkit['asr'] which was successful but as soon as i run the code, i got the same module not found error.
btw, I'm using windows 10.
Fix incorrect embedding grads with distopt BF16 grad reductions
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: Speaker Tasks

name: CI

name: common

# What does this PR do ?

In the case where we are training GPT with distopt, BF16 grads, and pipeline parallelism, we are not properly synchronizing the embedding grads between the first and last pipeline stage. The embeddings were being accumulated into BF16 buffers and the pipeline-parallel reduction was on FP32 buffers (so just all-reducing zeros). Note that we synchronize these buffers before the data-parallel reduction, so that is still being done properly in FP32.

This PR fixes the issue by eagerly accumulating the embedding grads into FP32 buffers.

**Collection**: NLP

# Changelog 
- Distopt accumulates grads for FP32 params into FP32 buffers.

# Usage

In the config file, set the optimizer with `name: distributed_fused_adam` and `grad_sync_dtype: bf16`.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Depends/includes https://github.com/NVIDIA/NeMo/pull/6886
Fix requirements for pydantic + inflect
State: closed | Pull request: yes

# What does this PR do ?

Upperbound pydantic

**Collection**: [ASR,NLP,TTS]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Add ASR with TTS Tutorial. Fix enhancer usage.
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [ASR]

# Changelog 
- Tutorial for using hybrid ASR-TTS Models (`ASRWithTTSModel`)
- fix enhancer loading in `ASRWithTTSModel`

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
peft folder name
State: closed | Pull request: yes
Labels: name: stale

name: NLP

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Draft: Torch-native FSDP support
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Add the support for PyTorch-native FSDP for GPT models.

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [*] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Merge release r1.19.0 into main
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Makes sure main and release have not diverged.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix num_workers=0 issue for gpt sft
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
GPT eval script error out due to unknown restore_path
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**
megatron_gpt_eval.py script error out if using checkpoint+hparams instead of nemo

**Steps/Code to reproduce bug**
Following the instruction:
`python megatron_gpt_eval.py \ checkpoint_dir=PATH_TO_CHECKPOINT_FILE \ checkpoint_name=CHECKPOINT_FILE_NAME \ hparams_file=HPARAMS_FILE \ inference.greedy=True \ inference.add_BOS=True \ trainer.devices=1 \ trainer.num_nodes=1 \ tensor_model_parallel_size=-1 \ pipeline_model_parallel_size=-1 \ prompts=[prompt1,prompt2]`
The script will error out:
`Traceback (most recent call last): File "megatron_gpt_eval.py", line 176, in main model_config = MegatronGPTModel.restore_from( File "/workspace/NeMo-1.18.0/nemo/core/classes/modelPT.py", line 425, in restore_from restore_path = os.path.abspath(os.path.expanduser(restore_path)) File "/usr/lib/python3.8/posixpath.py", line 231, in expanduser path = os.fspath(path) TypeError: expected str, bytes or os.PathLike object, not NoneType`
**Expected behavior**

A clear and concise description of what you expected to happen.
The root cause is that "gpt_model_file=null" check is at the higher level, hence the current script will error out due to restore path unknown. Solution will be Move https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/megatron_gpt_eval.py#L172-L189 to after https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/megatron_gpt_eval.py#L191


**Environment overview (please complete the following information)**
23.02 docker

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model
Enable SFT script to do multi-round/epoch training 
State: closed | Pull request: no
Labels: name: stale

**Is your feature request related to a problem? Please describe.**

In the current SFT training script, it runs for 1 epoch with the given max_step number, the max_epoch does not matter as the script generates a temp data file according to the max_step and runs through it. But in some applications, it is desirable to build a multi-epoch baseline for comparison purpose (e.g. for comparing a 5-round (1 epoch per round) federated learning result against a 5-epoch centralized training with the same number of steps per epoch). Currently this cannot be done, since the data used are not the same: 1-epoch-1000-step is not equivalent to 5-epoch-200-step due to the difference in the actual data being sampled.

**Describe the solution you'd like**

I wonder if we could enable multi-epoch training for SFT: the per-epoch data will be specified by max_step (or call it step_per_epoch). 

**Describe alternatives you've considered**

An "explicit" implementation of multi-round training will further be helpful: instead of calling .fit() once, something like below will really help easy-transition to FL:
for round in range(K):
    #necessary state loading
    trainer.fit(model)
    #necessary state saving

**Additional context**

Add any other context or screenshots about the feature request here.
Add more arithmetic resolvers
State: closed | Pull request: yes
Labels: name: core

name: stale

# What does this PR do ?

Add arithmetic resolvers other than `multiply`

# Changelog 
- No high level changes
- This PR contains custom resolvers for arithmetic, comparison and ternary expression

# Usage
Now in hydra configs you can use more arithmetic and ternary expressions.

```yaml
hydra_config_field: ${if:${condition},0,${ceil_div:123400,${batch_size}}}}
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
These resolvers are solution to already implemented [here](https://github.com/NVIDIA/NeMo/blob/main/nemo/core/config/hydra_runner.py#L27). It would be beneficial to implement a single resolver for evaluating arithmetic. The approach could be to use ast module to make a custom parser that allows only literals and binary operators.
Blank free transduction
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Adding cache-aware streaming ASR checkpoints.
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

It adds the following cache-aware checkpoints for streaming ASR:
stt_en_fastconformer_hybrid_large_streaming_80ms
stt_en_fastconformer_hybrid_large_streaming_480ms
stt_en_fastconformer_hybrid_large_streaming_1040ms
Update core commit for CI
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Update core commit for CI

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Upgrade base pytorch container to 23.06
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Waiting for 23.06 container to be published.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix confidence ensembles RNNT logprobs selection logic for exclude_blank scenario
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

- Fixes the algorithm for filtering blank frames while building confidence ensembles
- Adds checks for confidence value calculation to handle the edge case of all blank frames

**Collection**: [Note which collection this PR will affect]
ASR 

# Changelog 
- changes to get_filtered_logprobs() and compute_confidence() functions in nemo/collections/asr/models/confidence_ensembles.py


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Frame VAD eval not working?
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
I prepared a test manifest in the required format (i.e. with rttm label, because it doesn't work with the format indicated in the documentation), and tried to run the inference with the default parameters of the frame vad inference. I changed `evaluate` to `True` and here is my output:
```
[NeMo I 2023-06-28 13:40:46 frame_vad_infer:181] Evaluating VAD results
Evaluating VAD results:   0%|                                                                                                                                                                                                     | 0/1508 [00:00<?, ?it/s][NeMo W 2023-06-28 13:40:46 nemo_logging:349] .../Tools/miniconda3/envs/nemo_official/lib/python3.8/site-packages/pyannote/metrics/utils.py:200: UserWarning: 'uem' was approximated by the union of 'reference' and 'hypothesis' extents.
      warnings.warn(
    
Evaluating VAD results: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1508/1508 [00:08<00:00, 173.89it/s]
[NeMo I 2023-06-28 13:40:58 frame_vad_infer:192] AUROC: 0.9776
[NeMo I 2023-06-28 13:40:58 frame_vad_infer:193] DetER=100.0000, False Alarm=0.0000, Miss=100.0000
[NeMo I 2023-06-28 13:40:58 frame_vad_infer:194] with params: {'onset': 0.3, 'offset': 0.3, 'pad_onset': 0.2, 'pad_offset': 0.2, 'min_duration_on': 0.2, 'min_duration_off': 0.2, 'filter_speech_first': True}
[NeMo I 2023-06-28 13:40:58 frame_vad_infer:195] Done!
```

When I look at the output in:
```
vad_frame_outputs/seg_output_onset0.3-offset0.3-pad_onset0.2-pad_offset0.2-min_duration_on0.2-min_duration_off0.2-filter_speech_firstTrue-/file_id_1.rttm
```
there is definitely some overlap with the reference segments. So Miss=100 is unlikely. There are also some segments detected which are not in the reference, so False Alarm=0 is unlikely.
Could it be that the evaluation is missing something?

Thanks for your help!
Add missing save restore connector to eval scripts
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix missing save restore connector to eval scripts to allow dynamically determining the values of model parallelism

**Collection**: [NLP]


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Removed optional optimize_for_inference
State: closed | Pull request: yes
Labels: name: core

# What does this PR do ?

Removed optional optimize_for_inference call - causes problems with some nets and does not really help otherwise.
Fixed verbose flag passing.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Adding tutorial for confidence ensembles
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do?

There are a couple of bugfixes to the confidence computation and also a tutorial.

It currently misses a few links:
- to the paper (already submitted to arxiv, waiting to get the link)
- to the confidence tutorial from Aleksandr, which is not merged yet
- to the documentation of the confidence ensembles, which I hope to update in the next few days either in this PR or in the follow-up

**Collection**: ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
lora inference ci
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

adds a CI test to check inference with tp2 lora models.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix require_grad typos
State: closed | Pull request: yes
Labels: name: ASR

name: NLP

# What does this PR do ?

Fix `require_grad` typos (should be `requires_grad`).
Before the fix, the code doesn't cause any errors but doesn't do what it's supposed to do.

**Collection**: ASR, NLP

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [X] Did you write any new necessary tests?
- [X] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation
Fix FP8 scales conversion issue from non-nemo checkpoint to nemo checkpoint (try 2) (failed try 1: #6911)
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

TLDR; 
Converting a non-nemo checkpoint to NeMo with Transformer Engine modules results in truncating of scales in _extra_state. This WAR temporarily fixes the issue by manually toggling the fp8 attribute in Transformer Engine modules.

More context:
1. When exporting TE module data for checkpointing (fp8 scale info, etc.), `fp8` attribute needs to be set (see here: https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/module/base.py#L344) but this attribute is only set during the forward (and when using `fp8_autocast` context manager). If the `fp8` attribute isn't set, the relevant scale information isn't included in the pickled `_extra_state` tensor (and this is the "truncation of scales" issue referred in the TLDR, basically an empty tensor is pickled which makes it seem like the scale information or `_extra_state` tensor is truncated). (Note that I tried adding the `fp8_autocast` context manager around the `model.save_to()` call and it didn't work as expected)
2. Until, there's a cleaner way to enable the `fp8` atribute for the Transformer Engine modules, this WAR allows externally enabling it before the `model.save_to()` call in the checkpoint converter script.


**Collection**: NLP

# Changelog 
- Toggle the `fp8` attribute in a module if it exists (if it exists, its probably a TE module)

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
tied weights for adapters
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add rampup bs assertion
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix AN4 dataset links
State: closed | Pull request: yes
Labels: name: ASR

name: Speaker Tasks

# What does this PR do ?

Fix AN4 dataset link in docs. Remove broken links to AN4 data (no longer available).

**Collection**: [Docs, Tutorials]

# Changelog 
- fix AN4 dataset link in documentation
- remove broken AN4 links from tutorials

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Is it a bug in rotary embedding implementation?
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

The figure below is how rotary embedding is applied to a query vector as described in the RoFormer paper, but the current implementation in NeMo seems not implementing this formula. Specifically in the [`_rotate_half` function](https://github.com/NVIDIA/NeMo/blob/82044837a7bcfa1cfd3a91f6b0014e052bdfcaf0/nemo/collections/nlp/modules/common/megatron/position_embedding/rotary_position_embedding.py#L39-L47), for an input `[A, B, C, D] -> [-C, -D, A, B]`, shouldn't  the output be `[-B, A, -D, C]` instead (as the [RoFormer](https://github.com/huggingface/transformers/blob/8e164c5400b7b413c7b8fb32e35132001effc970/src/transformers/models/roformer/modeling_roformer.py#L328-L331) implemented in the Transformers library)? 

![image](https://github.com/NVIDIA/NeMo/assets/4978794/9344e805-793d-4e9b-bd3d-30da9a98abac)
[TTS] Add tutorial for TTS data prep scripts
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Add a tutorial demonstrating how to do the end to end data preparation and training with the new TTS preprocessing scripts and data loader.

**Collection**: [TTS]

# Changelog 
- Create tutorial

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation
Fastpitch e2e
State: closed | Pull request: yes
Labels: name: TTS

name: stale

# What does this PR do ?

FastPitch + HiFi-GAN end-2-end model


**Collection**: [TTS]

# Changelog 
- Add specific line by line info of high level changes in this PR.


# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Caching conformer fix to not mutate input tensors
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Mutating inputs are, strictly speaking, not allowed by ONNX spec.
Fixes ONNXruntime issue: 
https://github.com/microsoft/onnxruntime/issues/14956

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)