renwei2024/nemo-github-issues huggingface dataset:
Remove broadcast from T5 prompt learning inference
State: closed | Pull request: yes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

# What does this PR do ?

Removes a broadcast call that causes hanging during T5 prompt learning inference.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add wer details - insertion, deletion, substitution rate 
State: closed | Pull request: yes
Labels: name: ASR

Signed-off-by: fayejf <fayejf07@gmail.com>

# What does this PR do ?

Add wer details - insertion, deletion, substitution rate 

**Collection**: ASR

# Changelog 
- Add insertion, deletion, and substitution rate for evaluating ASR result
- (Note the original wer has not been deprecated at this time, and might be deprecated once we confirm the new package is reliable and faster. )

# Usage
* You can potentially add a usage example below

```python
wer, words, ins_rate, del_rate, sub_rate = word_error_rate_detail(hypotheses=["GPU"],  references=["GPU GPU"], use_cer=False)
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Bert interleaved
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix log-rank-0-only logic
State: closed | Pull request: yes

# What does this PR do ?

Fix logic behind `trainer.log_global_rank_0_only` and `trainer.log_local_rank_0_only` flags (currently they do nothing).

**Collection**: all

# Changelog 
- Fix `trainer.log_*_rank_0_only` flags logic

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Dual streaming/non-streaming Conformer with adaptive look-ahead
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix ASR Buffered inference scripts
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [ASR]

# Changelog 
- Deepcopy the alignments before compute of alignments
- Do proper class level check of log level instead of incorrect config checks (older model doesn't have in config because the log=True is set by default inside of the function)


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Add kwargs to TE full activation recompute path
State: closed | Pull request: yes
Labels: name: NLP

Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

# What does this PR do ?

`te_checkpoint` now has support for kwargs. This is necessary for the `is_first_microbatch` keyword argument during forward pass without which the numerics of the forward pass in the normal vs recompute phase differ.
Upgrade to 22.11
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Upgrade to the latest NGC PyTorch 22.11 container

**Collection**: [Core]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Switch order of args in optimizer_step override
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

PTL 1.8 swapped the order of the positional args for the optimizer_step method that we are overriding.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Remove notebook
State: closed | Pull request: yes

# What does this PR do ?

Remove notebook which is no longer supported.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Data parallel collect results
State: closed | Pull request: yes

# What does this PR do ?

Added gather_objects to easily collect results generated using data parallel.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
added set_start_method + function param bugfix
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

# What does this PR do ?

- adds set_start_method for t5, gpt prompt learning and t5 fine-tuning.
- includes a bugfix for function parameters

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[BugFix] Adding task variable for torchmetrics Accuracy
State: closed | Pull request: yes

Signed-off-by: Taejin Park <tango4j@gmail.com>

# What does this PR do ?

In the latest `torchmetrics==0.11.0`, Accuracy class' `__init__()` requires `task` variable to be specified.
Adding task variable for torchmetrics Accuracy



**Collection**: [Note which collection this PR will affect]
ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
This is bugfix for label_models.py

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anybody in ASR


# Additional Information
* Related to # (issue)
Support non-standard padding token id
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Read the id of the padding token from the tokenizer when creating the embedding, rather than always defaulting to 0. This allows use of (admittedly bizarre) non-standard tokenizer models that don't give the padding token <PAD> the id 0.

Opening this as a draft PR because I'd like to get feedback on how the `TabularTokenizer` plays into this, as it does not provide a `TabularTokenizer.pad_id` property. In this case, I just default to 0, which is identical to the previous implementation's behavior. It could also be that `TabularTokenizer` is not meant to be used with this class, in which case this doesn't matter. I also need to test this out one last time before converting it to a normal PR.

**Collection**: NLP

# Changelog 

- Remove hard-coded `token_idx = 0` when creating an `nn.Embedding` in the class `nemo.modules.common.transformer.transformer_modules.TransformerEmbedding`
- Get the padding ids for the `encoder_tokenizer` and `decoder_tokenizer` in `MTEncDecModel.__init__` and pass these down to all necessary submodules, until they get to the `TransformerEmbedding` objects

# Usage

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
[TTS][ZH] Disambiguate polyphones with augmented dict and Jieba segmenter for Chinese FastPitch
State: closed | Pull request: yes
Labels: name: TTS

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

To improve the G2P accuracy for Chinese polyphone, this PR first splits Chinese sentencs into words using jieba, then call pypinyin with cc_cedict.txt loaded.

Below is the test result for [g2pM test set](https://github.com/kakaobrain/g2pM/blob/master/data/test.sent)
![image](https://user-images.githubusercontent.com/28823245/205200025-9da54dea-b939-45bf-acbb-3c4a13df61bf.png)


@XuesongYang  Would you mind having a look at this PR? Thx.
Can't reproduce machine translation bleu score
State: closed | Pull request: no
Labels: name: bug

Hi, I'm trying to evaluate the [NMT En Fr Transformer12x2 model ](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/nmt_en_fr_transformer12x2) with sacrebleu, and cannot reproduce the bleu score reported in the doc - reported WMT13 and WMT14 bleu scores are 35.3/41.3 and I got 31.1/36.1. Is there anything I'm missing? I did get the warning from sacrebleu about 
```
sacreBLEU: That's 100 lines that end in a tokenized period ('.')
sacreBLEU: It looks like you forgot to detokenize your test data, which may hurt your score.
sacreBLEU: If you insist your data is detokenized, or don't care, you can suppress this message with the `force` parameter.
```

Thank you.
added set_start_method + function param bugfix
State: closed | Pull request: yes

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

# What does this PR do ?

- adds set_start_method for t5, gpt prompt learning and t5 fine-tuning.
- includes a bugfix for function parameters

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
ASR_with_SpeakerDiarization Notebook Doesn't Work
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Error:
```
[NeMo E 2022-12-01 21:47:24 common:503] Model instantiation failed!
    Target class:	nemo.collections.asr.models.label_models.EncDecSpeakerLabelModel
    Error(s):	__new__() missing 1 required positional argument: 'task'
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/nemo/core/classes/common.py", line 482, in from_config_dict
        instance = imported_cls(cfg=config, trainer=trainer)
      File "/usr/local/lib/python3.8/dist-packages/nemo/collections/asr/models/label_models.py", line 168, in __init__
        # self._macro_accuracy = Accuracy(num_classes=num_classes, average='macro')
    TypeError: __new__() missing 1 required positional argument: 'task'
```

Looks like there's something wrong with how the Accuracy class is instantiated

**Steps/Code to reproduce bug**

1. Load [notebook](https://github.com/NVIDIA/NeMo/blob/stable/tutorials/speaker_tasks/ASR_with_SpeakerDiarization.ipynb#L766) in collab
2. Run all cells
3. Fails on the `Run diarization with the extracted word timestamps` step

**Expected behavior**

The notebook works

**Environment overview (please complete the following information)**

 - Environment location: Collab w/GPU
Upperbound the torchmetrics version
State: closed | Pull request: yes

# What does this PR do ?

Fixes current failures around torchmetrics new release. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Upper bound torchmetrics version.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add StreamingFeatureBufferer class for real-life streaming decoding
State: closed | Pull request: yes
Labels: name: ASR

Signed-off-by: Taejin Park <tango4j@gmail.com>

# What does this PR do ?

Add StreamingFeatureBufferer class for real-life streaming decoding


**Collection**: [Note which collection this PR will affect]
ASR

# Changelog 

Added StreamingFeatureBufferer() in streaming_utils.py.


# Usage

Will add usage when the example code is ready and pushed.


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in ASR

# Additional Information
* Related to # (issue)
Draft: latent model support
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Adds support to token_level_encoder_decoder module to accept Hidden and Loss transformations and apply them. 

# Usage
* You can add the following to your config : 
hidden_transforms: [MegatronGaussianHiddenTransform()] and it will apply a gaussian transform to the hiddens.
TTS tutorial update: use speaker 9017 instead of 6097
State: closed | Pull request: yes
Labels: name: TTS

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

# What does this PR do ?

Replaces speaker 6097's data with speaker 9017's since the latter has given explicit permission.

**Collection**: TTS
# Changelog 
- Updated link to data and recalculated pitch stats for the new speaker.

# Usage
* Tutorial should run as usual.
Exporting Megatron GPT
State: closed | Pull request: yes
Labels: name: stale

name: NLP

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

# What does this PR do ?

Adds export functionality for Megatron GPT

**Collection**: [Note which collection this PR will affect]

# Changelog 
- export related functions for Megatron GPT model

# Usage
* exportable using export.py with --autocast option

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
NLP docs fixes
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

- Fixes invalid image URL in `Entity Linking` page
- Fixes title format in `Question Answering` page (to resolve issues in NLP Tasks TOC page)

**Collection**: NLP docs

# Changelog 
- Resolves issues in NLP docs

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [X] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [X] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Multiblank Transducer
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add support for multi-blank transducers described in https://arxiv.org/pdf/2211.03541

**Collection**:

ASR

# Changelog 
- implementation of multi-blank both with numba kernel code as well as pytorch.
- add relevant test code.
- add example .yaml file for multi-blank ASR model configuration.

# Usage

Invoke transducer model training with examples/asr/conf/conformer/conformer_multiblank_transducer_bpe.yaml file. This will train a multi-blank model with big blank durations [2,4,8], and sigma=0.05. 

To run multi-blank with custom configurations, e.g. using [2,3,5,8], change the options by,

```
model.loss.multiblank_rnnt_kwargs.big_blank_durations=[2,3,5,8] # used for loss computation in training.
model.loss.multiblank_rnnt_kwargs.sigma=0.05 # 0.05 is the recommended value.
model.decoding.big_blank_durations=[2,3,5,8] # used for decoding. The length of the list should be consistent with above.
model.joint.num_extra_outputs=4 # this needs to be consistent with the duration list you set
```

You could set model.decoding.big_blank_durations to have values smaller than what's used in training (model.loss.multiblank_rnnt_kwargs.big_blank_duration), e.g. [1,1,1,1]. This makes decoding slower but usually gives better accuracy.

To run inference, add option rnnt_decoding.big_blank_durations when calling NeMo/examples/asr/transcribe_speech.py, to specify the big blank durations, e.g.

```
python NeMo/examples/asr/transcribe_speech.py \
      model_path=your_big_blank_model.nemo \
      dataset_manifest=your_data.json  \
      ++rnnt_decoding.big_blank_durations=[2,3,5,8] \
      output_filename=your_output_file
```


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Conformer local attention
State: closed | Pull request: yes
Labels: name: ASR

Signed-off-by: Samuel Kriman <samuelkriman@gmail.com>

# What does this PR do ?

This PR introduces the option of using local attention in Conformer with relative positional encoding, following the approach from Longformer (https://arxiv.org/abs/2004.05150), based on Ankur Kumar's branch https://github.com/iankur/NeMo/tree/longformer_rel_pos. This is useful for running on long audio files, as it can be done faster than with buffered inference. It may also improve performance on audio with long segments of noise that may confuse models with full attention.

This method can be used with checkpoints trained with regular attention. For example, Conformer-Large checkpoints (such as https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_conformer_ctc_large and https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_conformer_transducer_large) can be run on ~30 minute audio files on a single 32GB GPU after switching to local attention. The duration can be further increased to ~2 hours by switching to stacking subsampling (this won't work without additional training). Please note that the WER may increase slightly on our main evaluation sets when using small context local attention with models trained with full context (see table below for some WER(%) comparisons). This effect can be mitigated by either increasing the context size or fine-tuning the model using local attention.

| Model | LS test-other | LS test-clean | People Speech | SLR |
|----------|----------|----------|----------|----------|
| Conformer-Large-CTC   | 4.53   | 2.19   | 23.04   | 5.94   |
| Conformer-Large-CTC with local attention (64, 64)  | 5.39   | 2.57   | 23.89   | 6.2  |
| Conformer-Large-CTC with local attention (192, 192)  | 4.53   |  2.19 | 23.2   | 5.96  |
| Conformer-Large-RNNT  | 3.73  | 1.8  | 21.69  | 5.4  |
| Conformer-Large-RNNT with local attention (64, 64) | 4.6  | 2.14  | 21.85  | 5.67  |
| Conformer-Large-RNNT with local attention (192, 192) | 3.75  | 1.83 | 21.64  | 5.41 |


**Collection**: ASR

# Changelog 
- adds option to use local attention for Conformer
- adds option for overriding attention parameters when running transcribe or eval scripts, or after loading model

# Usage

With an eval script can be used like this to switch to local attention with any Conformer checkpoint

```
python speech_to_text_eval.py \
  (...other params...)  \
  ++model_change.conformer.self_attention_model="rel_pos_local_attn" \
  ++model_change.conformer.att_context_size=[64, 64]
```

Or after loading model 

```python
asr_model = ASRModel.from_pretrained('stt_en_conformer_ctc_large')
asr_model.change_attention_model(
                self_attention_model="rel_pos_local_attn",
                att_context_size=[64, 64]
)
```

**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation
Conformer local attention
State: closed | Pull request: yes
Labels: name: core

name: ASR

# What does this PR do ?

Sliding window local attention for Conformer.

**Collection**: ASR

# Changelog 
- adds option to use local attention for Conformer
- adds option for overriding/merging parameters into model config when using transcribe/eval scripts or loading model

# Usage

With an eval script can be used like this to switch to local attention with any Conformer checkpoint

```
python speech_to_text_eval.py \
  (...other params...)  \
  ++merge_into_model_config.encoder.att_context_size=[64,64] \
  ++merge_into_model_config.encoder.self_attention_model="rel_pos_local_attn"
```

Or when loading model 

```python
merge_into_model_config = {
  'encoder': dict({
    'self_attention_model': "rel_pos_local_attn",
    'att_context_size': [64, 64]
    }),
}
merge_into_model_config = DictConfig(merge_into_model_config)
asr_model = ASRModel.from_pretrained('stt_en_conformer_ctc_large', map_location='cuda',
  merge_into_model_config=merge_into_model_config)
```

**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
prompt tuning fix for unscale grad errors
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?
Fixes the unscale gradient error.

**Collection**: [NLP]

# Changelog 
- Adds CI test for simple prompt tuning
- Initializes the prompt embeddings with full prec 

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
T5 Eval bugfix
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

minor bugfix

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Added function arg

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
T5 Eval bugfix
State: closed | Pull request: yes

# What does this PR do ?

minor bugfix

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Added function arg

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Way to use `prefetch_factor` in ASR training
State: closed | Pull request: no
Labels: name: feature

name: stale

**Is your feature request related to a problem? Please describe.**
The PyTorch [dataloader](https://pytorch.org/docs/stable/data.html) supports pre-fetching the batch of data so that GPU doesn't wait for data processing. But I could not find any options to set the  `prefetch_factor` parameter in the example config.YAML file

**Describe the solution you'd like**
Like others dataloader parameter, we can also set this prefetch_factor param in the config YAML file and use this during training 
A clear and concise description of what you want to happen.

Thanks
ssl conformer learning rate & warmup
State: closed | Pull request: no
Labels: name: bug

name: stale

Hi, I've trained a **SSL conformer** model. Now I wanna **fine-tune** it with labeled data.
I encountered two problems:

1. I want the learning rate decrease linearly.
I don't want warmup. So I set it 0.
But in this situation, it first increase and then decrease.

2. the second problem is I want to control the learning rate decrease speed. For this one I changed the weight_decay and also min_lr but it decreased very fast. How can I control it?
I want it be constant.
Multi-blank Transducers
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add support for multi-blank transducers described in https://arxiv.org/pdf/2211.03541

**Collection**: [Note which collection this PR will affect]

ASR

# Changelog 
-  implementation of multi-blank both with numba kernel code as well as pytorch.
-  add example .yaml file for multi-blank ASR model configuration. 

# Usage

Multi-blank model training could be invoked by using the .yaml file included in this PR and call.

```
python NeMo/examples/asr/asr_transducer/speech_to_text_rnnt_bpe.py \\
[your options here]
```

with your options for data/hyperparameters/etc. This trains a multi-blank model with big-blank durations [2, 4, 8].

To make changes in the big blank durations, change the options by,

```
model.loss.multiblank_rnnt_kwargs.big_blank_durations=[2,3,5,8] # or any other set of durations you want
model.decoding.big_blank_durations=[2,3,5,8] # used for decoding
model.joint.num_extra_output=4 # this needs to be consistent with the duration list you set
```


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Conformer_Transducer's encoder exported to ONNX fails to instantiate an `onnxruntime.InferenceSession`
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Creating an onnx `InferenceSession` using the exported encoder from a conformer_transducer model (using the default parameters of `export`) leads to an `ONNXRuntimeError`.

```
[ONNXRuntimeError] : 1 : FAIL : Load model from /tmp/tmp_ssi0eq1/encoder-test.onnx failed:Type Error: Type parameter (Tind) of Optype (Slice) bound to different types (tensor(int32) and tensor(int64) in node (/pos_enc/Slice).
```

**Steps/Code to reproduce bug**

Here's a minimal example to reproduce the bug.

```python
#! /usr/bin/env python3
# -*- coding: utf-8 -*-

import tempfile
import onnxruntime as ort
from nemo.collections.asr.models import EncDecRNNTBPEModel


def main():
    model = EncDecRNNTBPEModel.from_pretrained(
        "stt_fr_conformer_transducer_large"
    )

    with tempfile.TemporaryDirectory() as temp_dir:
        model.export(temp_dir + "/test.onnx")

        try:
            ort.InferenceSession(temp_dir + "/encoder-test.onnx")
            print("encoder has been sucessfully initialized")
        except Exception as e:
            print("Error on encoder")
            print(e)

        try:
            ort.InferenceSession(temp_dir + "/decoder_joint-test.onnx")
            print("decoder_joint has been sucessfully initialized")
        except Exception as e:
            print("Error on decoder_joint")
            print(e)


if __name__ == "__main__":
    main()
```

**Expected behavior**

Well... The `InferenceSession` should be correctly initialized from the exported onnx file, shouldn't it ?

**Environment overview and details**

 - Environment location:

 - Method of NeMo install:
```
> conda create -n NeMo python==3.9
> pip install Cython
> pip install "nemo_toolkit[all]"
```

- OS version
```
> uname -srvpio
Linux 6.0.9-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 16 Nov 2022 17:01:17 +0000 unknown unknown GNU/Linux
```
- PyTorch version
```
> python
>>> import torch
>>> torch.__version__
'1.13.0'
```
- Python version
```
> python --version
Python 3.9.0
```
Cannot finetune FastPitch on a custom dataset
State: closed | Pull request: no

I followed the finetuning tutorial and tried to finetune the pretrained LJSpeech model with the command
`python examples/tts/fastpitch_finetune.py --config-name=fastpitch_align_v1.05.yaml train_dataset=/root/autodl-tmp/ryanspeech.train.json validation_datasets=/root/autodl-tmp/ryanspeech.val.json sup_data_path=/root/autodl-tmp/nemo-training/ryan_22k/sup_data exp_manager.exp_dir=/root/autodl-tmp/nemo-training/ryan_22k +init_from_nemo_model=/root/autodl-tmp/tts_en_fastpitch_align.nemo  model.train_ds.dataloader_params.batch_size=24 pitch_mean=150.32728576660156 pitch_std=42.45840835571289 pitch_fmin=65.4063949584961 pitch_fmax=1760.0  model.optim.lr=2e-4 ~model.optim.sched`
But I got the error:
```
Traceback (most recent call last):
  File "examples/tts/fastpitch_finetune.py", line 33, in main
    model.maybe_init_from_pretrained_checkpoint(cfg=cfg)
  File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/utilities/rank_zero.py", line 32, in wrapped_fn
    return fn(*args, **kwargs)
  File "/root/NeMo/nemo/core/classes/modelPT.py", line 1066, in maybe_init_from_pretrained_checkpoint
    self.load_state_dict(restored_model.state_dict(), strict=False)
  File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1482, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for FastPitchModel:
        size mismatch for fastpitch.encoder.word_emb.weight: copying a param with shape torch.Size([115, 384]) from checkpoint, the shape in current model is torch.Size([114, 384]).
```
-------------------
EDIT:
seems like it has something to do with the `+model.text_tokenizer.add_blank_at=true` in the tutorial.
Problem solved.
TTS Speech marks
State: closed | Pull request: no
Labels: name: TTS

**Is your feature request related to a problem? Please describe.**

In TTS, if one wants to provide timing hints of the currently spoken word to the user, one needs to map the timing information (beginning/end time offset in generated pcm) to a word or characters or phonemes. This is necessary/applicable for e.g. screen readers or language learning apps.

**Describe the solution you'd like**

The acoustic model should be able to output timing information of each spoken word / phoneme related to the input phonemes/words. Ideally, the information could be provided together with the output MEL spectrograms as an extra tensor, so that inferencing doesn't need to be done twice. Alternatively, providing a function, that takes the generated MEL spectrograms and maps these to the aforementioned timing information would also be a very good solution and could possibly be implemented generically for all TTS models that output MEL spectrograms.

**Describe alternatives you've considered**

Alternatively, of course without any lip-synchronization accuracy, one could just devide the overall generated audio time of the synthesized PCM by the number of input characters/phonemes, naively assuming every character/phoneme needs an equal amount of time to speak. Although easily to implement, timing results are however off most of the time.

**Additional context**

Add any other context or screenshots about the feature request here.