renwei2024/nemo-github-issues huggingface dataset:
[BugFix] decoder timestamp count has a mismatch when <unk> is decoded
State: closed | Pull request: yes
Labels: name: bug

name: ASR

name: Speaker Tasks

Signed-off-by: Taejin Park <tango4j@gmail.com>

# What does this PR do ?


decoder timestamp count has a mismatch when <unk> is decoded.
Instead of having a temporary solution to ignore <unk>, the script is changed to 
output <unk> as decoded output with a correct timestamps.

https://github.com/NVIDIA/NeMo/pull/5481/files

In this previous PR, the <unk> symbol is coded to be ignored, 
but it still causes mismatch errors when <unk> is in the middle of two isolated one-token words.

This PR fixes such errors.


**Collection**: [Note which collection this PR will affect]
ASR

# Changelog 

`ctc_decoder_predictions_tensor_with_ts()` in decoder_timestamps_utils.py
is changed to output the actual decoded text with the matching word timestamps.

# Usage

```python
python offline_diar_with_asr_infer.py \
    diarizer.manifest_filepath=<path to manifest file> \
    diarizer.out_dir='demo_asr_output' \
    diarizer.speaker_embeddings.model_path=<pretrained modelname or path to .nemo> \
    diarizer.asr.model_path=<pretrained modelname or path to .nemo> \
    diarizer.asr.parameters.asr_based_vad=True \
    diarizer.speaker_embeddings.parameters.save_embeddings=False
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation


## Who can review?

ASR contributors

# Additional Information
* Related to # (issue)
Update files for lightning 1.9.0
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Update our code so that we can support Lightning 1.8.x and 1.9.x

**Collection**: Common

# Changelog 
- Update NeMo for Lightning 1.9.0.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.
Pin lightning version less than 1.9.0
State: closed | Pull request: yes

Signed-off-by: SeanNaren <snarenthiran@nvidia.com>

Fixes https://github.com/NVIDIA/NeMo/issues/5821

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
ImportError PL Logger
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Getting `ImportError` bug when importing e.g. ASR model when installing NeMo from main. Presumably because of PL update yesterday? [(pl-release)](https://github.com/Lightning-AI/lightning/releases/tag/1.9.0)

**Steps/Code to reproduce bug**

`python -m pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[asr]`
`from nemo.collections.asr.models import EncDecSpeakerLabelModel `

yields:
![image](https://user-images.githubusercontent.com/37636054/213150477-9b91c652-80e2-49e2-99ae-d37c2aeb6c0b.png)


**Expected behavior**

..
**Environment overview (please complete the following information)**
Google Colab

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Google Colab
- PyTorch version: 1.13.1+cu116
- Python version: Python 3.8.10
add apt-get upgrade -y in dockerfile
State: closed | Pull request: yes
Labels: name: cherry-pick

name: CI

# What does this PR do ?

add apt-get upgrade -y in dockerfile

**Collection**: core

# Changelog 
- add apt-get upgrade -y in dockerfile to upgrade packages for vulns patch


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add apt-get upgrade -y in dockerfile
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

add apt-get upgrade -y in dockerfile

**Collection**: core

# Changelog 
- add apt-get upgrade -y in dockerfile to upgrade packages for vulns patch


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add container info to main page
State: closed | Pull request: yes

# What does this PR do ?

Add container in for to main page

**Collection**: core

# Changelog 
- Add built container info to main readme for better visibility. 


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update description for question answering tutorial
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Placed a note to remove support for multiple GPU in  question answering tutorial, redirecting them to use examples scripts instead, if multiple GPU is needed

**Collection**: nlp

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update description for question answering tutorial
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Placed a note to remove support for multiple GPU in  question answering tutorial, redirecting them to use examples scripts instead, if multiple GPU is needed

**Collection**: nlp

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix transducer and question answering tutorial bugs bugs
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

name: NLP

# What does this PR do ?

Fix bugs in Intro to Transducer and Question Answering tutorials

**Collection**: nlp, asr

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix transducer and question answering tutorial bugs bugs
State: closed | Pull request: yes
Labels: name: ASR

name: NLP

# What does this PR do ?

Fix bugs in Intro to Transducer and Question Answering tutorials

**Collection**: nlp, asr

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Megatron positional encoding alibi fix
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

This PR fixes a bug in ALiBi implementation.

ALiBi paper: https://arxiv.org/abs/2108.12409

Results shows that ALiBi is comparable in performance to RPE, and is faster by 7% from RPE and 8% slower than learnable abs embeddings during training. 
See plots below. ALiBi (this PR) in green, ALiBi (before fix) in pink, learned absolute embeddings in red, and RPE in cyan.


![Screen Shot 2023-01-25 at 17 32 00](https://user-images.githubusercontent.com/7453913/214607208-fcc652b2-4ae0-42b5-864c-d01eb62e7996.png)
![Screen Shot 2023-01-25 at 17 32 50](https://user-images.githubusercontent.com/7453913/214607215-c41b7bc8-33e6-4378-8356-f811f22e3c02.png)
![Screen Shot 2023-01-25 at 17 33 01](https://user-images.githubusercontent.com/7453913/214607219-6bd338e0-e088-4891-ac4e-edd1fa314d16.png)
![Screen Shot 2023-01-25 at 17 33 27](https://user-images.githubusercontent.com/7453913/214607227-1480cd98-c17e-453d-bf56-5ae6f1e3d053.png)
![Screen Shot 2023-01-25 at 17 36 03](https://user-images.githubusercontent.com/7453913/214607229-f395a390-ffe8-4fd8-b55e-090623fd0b34.png)
![Screen Shot 2023-01-25 at 17 36 17](https://user-images.githubusercontent.com/7453913/214607232-6a87f3c8-aee7-4f2a-81bc-f38167932ff5.png)



**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add build action
State: closed | Pull request: yes
Labels: name: CI

Signed-off-by: SeanNaren <snarenthiran@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
How to choose number of epochs in Conformer-Transducer (RNNT) pretrained model finetuning
State: closed | Pull request: no
Labels: name: stale

Hi All,

I have taken conformer-transducer (RNNT) & conformer CTC pretrained English medium models, and finetuned them for Tamil dataset. Finetuned CTC has decent WERs, whereas RNNT models have unacceptable WERs. Both were trained for 18 epochs. The CTC finetuning has consistently exponentially decreasing Validation loss. However, validation loss of RNNT consistently increased after 5th epoch. PFA RNNT val loss snapshot.

![image](https://user-images.githubusercontent.com/6142443/212663699-62081f70-cf12-418d-9e4d-52b1464aa57b.png)

So, in this case, 
Question 1: Do you suggest me to train only for 5 epochs while using RNNT and use the saved model for decoding?
Question 2: Is there any way like any parameter to be set either in exp_manager or trainer, so that it would automatically save the check point with best WER
Question 3: Am I doing anything wrong here. Should I need to check/change anything to make RNNT validation error to decrease.

Note: We have done several experiments with Conformer-CTC and they seem to work without any issues with same setup (like no. of epochs etc). However, we are exploring the RNNT for the first time. So, not sure if something is missing.
Please suggest @titu1994  Thanks.
[TTS][DE] Augment tokenization/G2P to preserve capitalization of words and mix phonemes with word-level graphemes for an input text.
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: common

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

# What does this PR do ?

```
1. support grapheme case options with either "upper", "lower", or "mixed".
2. automatically prepend a symbol "#" or any customized one to each grapheme of a word if both phonemes and graphemes are mixed in order to distinguish graphemes from phoneme symbols if there are overlaps in between.
3. normalize texts with "NFC" form to represent the letter with diacritics as a single letter. Similar reason as https://github.com/NVIDIA/NeMo/blob/main/nemo_text_processing/g2p/data/data_utils.py#L140.
4. add de IPA dictionary and de heteronym set.
5. fix typos. 
```

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Decoder model [Duplex Text Normalization] train failing, using TarredDataset
State: closed | Pull request: no
Labels: name: bug

name: stale

**Unable to start training Decoder** model **using tarred dataset** obtained from script [create_tarred_dataset.py](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/duplex_text_normalization/data/create_tarred_dataset.py).

The error occurs on the line:
https://github.com/NVIDIA/NeMo/blob/dd921e792427e3788ad9fe3a1e568afc85a8bc93/nemo/collections/nlp/data/text_normalization/decoder_dataset.py#L537

and looks like:

`ValueError: didn't find ['pkl'] in ['__key__', 'tsv-batch-3649.pkl', 'tsv-batch-3650.pkl', 'tsv-batch-3651.pkl', 'tsv-batch-3652.pkl', ....]` (where by "...."  I mean enumeration of other .pkl files).

As I understand it, it would be more correct to do the following in [decoder_dataset.py](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/data/text_normalization/decoder_dataset.py#L537):

1. Obtain every filename with ".pkl" extension and create `pkls` (List). After, create a string like:
```
pkls = ';'.join(pkls)
```

2. Rename as follows (using every key, separated by ";" - **supports by WebDataset**)
```
self._dataset = self._dataset.rename(pkl=pkls, key='__key__').to_tuple('pkl', 'key').map(f=self._build_sample)
```

However, this method also does not allow you to start training.
update torchmetrics to latest version
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

name: CI

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

# What does this PR do ?

update torchmetrics to latest version 

**Collection**: ASR

# Changelog 
- update to latest API of torch metrics 
- Timer issue with dataloader is because of wrong num of classes (when only 2 present). This behavior got updated with latest torchmetrics. 

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
update torchmetrics to latest version
State: closed | Pull request: yes
Labels: name: ASR

name: CI

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

# What does this PR do ?

update torchmetrics to latest version 

**Collection**: ASR

# Changelog 
- update to latest API of torch metrics 
- Timer issue with dataloader is because of wrong num of classes (when only 2 present). This behavior got updated with latest torchmetrics. 

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
RETRO model finetuning
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

1. Add RETRO model NQA finetune scripts.
2. Make RETRO model compatible with MegatronLM RETRO model.
    - Can turn off the RoPe positional embedding.
    - Add version to RETRO model.
3. Add stride to the RETRO indexed memory map file.
4. during inference, host multiple BERT models for retrieval service to use.
Fix setuptools to usable version
State: closed | Pull request: yes
Labels: name: cherry-pick

Signed-off-by: smajumdar <titu1994@gmail.com>

# What does this PR do ?

Setuptools 65.x is causing issues with 

**Collection**: [Core]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Fix setuptools to usable version
State: closed | Pull request: yes

Signed-off-by: smajumdar <titu1994@gmail.com>

# What does this PR do ?

Setuptools 65.x is causing issues with 

**Collection**: [Core]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
CommonVoice support for `convert_hf_dataset_to_nemo` script
State: closed | Pull request: yes

# What does this PR do ?

Related #5638 #5772  

Renames `sentence` column to `text` before creating manifest. Exposes `ensure_ascii` for saving the transcripts + fix naming of config class.

**Collection**: ASR

# Changelog 
- CommonVoice support for `convert_hf_dataset_to_nemo` script.

# Usage
* You can potentially add a usage example below

```bash
python scripts/speech_recognition/convert_hf_dataset_to_nemo.py \
        output_dir=/data/commonvoice/ \
        path="mozilla-foundation/common_voice_11_0" \
        name="eo" \ # esperanto
        split="validation" \
        ensure_ascii=False \
        use_auth_token=True
```

**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation
[DOC] move installation guide to doc intro from README
State: closed | Pull request: yes
Labels: name: stale

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

# What does this PR do ?

This PR was cherry-picked from https://github.com/NVIDIA/NeMo/pull/5484, which attempted to merge r1.13.0. Rebased to r1.15.0.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Lexicon support for multiple pronunciations of the same word
State: closed | Pull request: no
Labels: name: stale

Hi All,

Does nemo has lexicon support for handling multiple pronunciations of the same word. 

I understand this is supported in Riva framework as mentioned at https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-python-advanced-customize-vocabulary-and-lexicon.html#customizing-pronunciation-with-lexicon-mapping . But, just wanted to know, is it also supported during normal decoding.

Hi @titu1994 , Is there any further update on https://github.com/NVIDIA/NeMo/issues/2903 ? Whether any functionality is added for offline support than Riva support? Thanks
Domain language adaptation functionality in Nemo
State: closed | Pull request: no
Labels: name: stale

Hi All, 
I wanted to adapt some specific domain text to a model trained on Generic datasets. In Kaldi, this is done by adding more domain data to LM and use it during decoding. 

Is it this similar in Nemo too, like building a LM by adding domain specific text data & generic training text data together, then use this LM at the time of decoding? Or any other specific Nemo feature/ functionality that provide better support to accomplish this?
Thanks
TS-Exported SpeakerNet model has vastly different outputs than non-exported model.
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I have exported the pre-trained SpeakerNet model into TorchScript. With the same data input, I am calling this exported model and its original non-exported model but the results are vastly different.

**Steps/Code to reproduce bug**

```
processed_signal = ...         # torch.Size([1, 64, 416])
processed_signal_len = ...     # torch.Size([1])


# Load original pre-trained model.
from nemo.collections.asr.models import EncDecSpeakerLabelModel
original_encoder = EncDecSpeakerLabelModel.from_pretrained(
    model_name="speakerverification_speakernet"
)
original_encoder.eval()


# Export pre-trained model to TorchScript.
original_encoder.export('exported_encoder.ts')


# Original model's inference.
original_logits, original_embs = original_encoder.forward_for_export(processed_signal, processed_signal_len)


# Exported model's inference.
exported_encoder = torch.jit.load('exported_encoder.ts')
exported_logits, exported_embs = exported_encoder.forward(processed_signal=processed_signal, processed_signal_len=processed_signal_len)
```

However, I can see that `original_embs` and `exported_embs` is vastly different. This is very confusing to me.

**Expected behavior**

I would expect that the original model's and exported model's prediction to be the same (or at least very close to same).

**Environment overview (please complete the following information)**

Environment location is GCP. Method of install is from pip.

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version = Debian GNU/Linux
- PyTorch version = 1.11.0+cu113
- Python version = 3.7.12
Adithyare/adapter new placement
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Flashlight Decoder for Nemo
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Adds Flashlight decoding to Nemo

**Collection**: ASR

# Changelog 
- Added flashlight_decoder.py to collections/asr/modules
- Changed collections/asr/metrics/wer.py and wer_bpe.py to support new setter methods for ctc_decoding object
- Changed collections/asr/parts/submodules/ctc_beam_decoding.py to include all of the new logic for Flashlight decoding which uses the new beam search API
- Added script to scripts/asr_language_modeling/ngram_lm which creates Flashlight format lexicon files from KenLM arpa files

# Usage
Works for lexicon and lexicon-free flashlight decoding

```python
import torch

from nemo.collections.asr.models import ASRModel
from nemo.collections.asr.metrics.wer import CTCDecodingConfig

device = torch.device('cuda')
asr_model = ASRModel.restore_from(...).to(device)
files = ['/path/to/audio1.wav', '/path/to/audio2.wav']

# Lexicon-based
decoding_cfg = CTCDecodingConfig()

decoding_cfg.strategy = "flashlight"
decoding_cfg.beam.search_type = "flashlight"
decoding_cfg.beam.kenlm_path = '/path/to/kenlm.bin'
decoding_cfg.beam.flashlight_cfg.lexicon_path='/path/to/lexicon.lexicon'
decoding_cfg.beam.beam_size = 32
decoding_cfg.beam.beam_alpha = 0.2
decoding_cfg.beam.beam_beta = 0.2
decoding_cfg.beam.flashlight_cfg.beam_size_token = 32
decoding_cfg.beam.flashlight_cfg.beam_threshold = 25.0

# Lexicon-free (for char based languages like Japanese)
decoding_cfg = CTCDecodingConfig()

decoding_cfg.strategy = "flashlight"
decoding_cfg.beam.search_type = "flashlight"
decoding_cfg.beam.kenlm_path = '/path/to/kenlm.bin'
decoding_cfg.beam.flashlight_cfg.lexicon_path=None
decoding_cfg.beam.beam_size = 32
decoding_cfg.beam.beam_alpha = 0.2
decoding_cfg.beam.beam_beta = 0.2
decoding_cfg.beam.flashlight_cfg.beam_size_token = 32
decoding_cfg.beam.flashlight_cfg.beam_threshold = 25.0
decoding_cfg.beam.flashlight_cfg.unit_lm = True


asr_model.change_decoding_strategy(decoding_cfg)

transcripts = asr_model.transcribe(paths2audio_files=files, batch_size=2, return_hypotheses=False, logprobs=False)
```

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [X] Did you write any new necessary tests?
- [X] Did you add or update any necessary documentation?
- [X] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [X] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update numpy requirements from 1.21 to 1.22
State: closed | Pull request: yes
Labels: name: cherry-pick

Seeking to overcome this issue  UserWarning: Failed to initialize NumPy: module compiled against API version 0xf but this version of numpy is 0xe based on https://iq.opengenus.org/module-compiled-against-api-version-0x10-but-this-version-of-numpy-is-0xe/

Note: this warning actually causes a crash later one when trying to import things from numpy typing

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: NLP, ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update numpy requirements from 1.21 to 1.22
State: closed | Pull request: yes

Seeking to overcome this issue  UserWarning: Failed to initialize NumPy: module compiled against API version 0xf but this version of numpy is 0xe based on https://iq.opengenus.org/module-compiled-against-api-version-0x10-but-this-version-of-numpy-is-0xe/

Note: this warning actually causes a crash later one when trying to import things from numpy typing

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: NLP, ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Megatron export triton update
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- updating biased gelu 
- minor updates to mimic nemo nmt

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update torchaudio dependency version for tutorials
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

Signed-off-by: smajumdar <titu1994@gmail.com>

# What does this PR do ?

Update torchaudio dependency in tutorials to avoid issue with torch.stft signature.

**Collection**: [ASR]

# Changelog 
- Update tutorial dependency for torchaudio

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Update torchaudio dependency version for tutorials
State: closed | Pull request: yes
Labels: name: ASR

Signed-off-by: smajumdar <titu1994@gmail.com>

# What does this PR do ?

Update torchaudio dependency in tutorials to avoid issue with torch.stft signature.

**Collection**: [ASR]

# Changelog 
- Update tutorial dependency for torchaudio

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
set max_steps for lr decay through config
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Allows setting lr decay steps through config that could be different from the trainer.max_steps

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
DALI fails in mulit-node setup
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

In a multi-node setup, DALI pipelines' fail because data is only visible on one node.

**Steps/Code to reproduce bug**

1. Download the data on the main node (rank 0), place it in `/scratch/audio-data`
2. Launch training
3. DALI raises error on other nodes :
`[/opt/dali/dali/operators/reader/loader/nemo_asr_loader.cc:80] Assert on "fstream" failed: Could not open NEMO ASR manifest file: "/scratch/audio-data/manifest.json`


**Expected behavior**

If `ddp` is enabled, DALI should run only on rank 0 node and load the data from there.

**Environment overview (please complete the following information)**

 - Environment location: AWS (HPC)
 - Method of NeMo install: `pip install git+https://github.com/NVIDIA/NeMo.git@9bc1220132cca7ba71550105d98dd12cfc7fefbe#egg=nemo_toolkit[asr]`

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- PyTorch version 1.13.1
- Python version: 3.8.10

**Additional context**

Any other suggestions on how to setup multi-node training are welcome.
Port Riva's mel cepstral distortion w/ dynamic time warping notebook
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Ports the Riva Mel Cepstral Distortion - Dynamic Time Warping notebook (nvidia-riva/tutorials#96) to NeMo.

HTML version here for ease of viewing:
https://drive.google.com/file/d/1E7bk95HQXZE_LyCfryr1r9LEKPzm8J9D/view?usp=sharing

**Collection**: TTS

# Changelog 
- Minor changes to ensure the notebook runs both locally and in Colab (tarred the example data, changed the way the audio is displayed)
- Additional docstrings and function explanations
- Minor text updates
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.