renwei2024/nemo-github-issues huggingface dataset:
ARP to IPA mapping, g2p_encode for IPATokenizer
State: closed | Pull request: yes

Signed-off-by: ekmb <ebakhturina@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
MLP Prompt Learning  Encoder
State: closed | Pull request: yes

# What does this PR do ?

Adds MLP option for prompt learning encoder

**Collection**: NLP

# Changelog 
- Updates Prompt Encoder class
- Updates GPT Prompt Learning Model Class
- Updates Base Prompt Learning Model Class

# Usage
These can optionally be added to the prompt learning config

```
p_tuning: # P-tuning specific params
    encoder_type: "mlp" 
    encoder_hidden: 512
    num_layers: 2
    dropout: 0.0
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix decoding for T5 models with RPE
State: closed | Pull request: yes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

# What does this PR do ?

Fixes a bug with decoding from Enc-Dec models with RPE.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Spoken Language Identification
State: closed | Pull request: yes

# What does this PR do ?

Add training script and config (titanet) for spoken language identification. 

**Collection**:ASR

# Changelog 
- Calculate weight for using weighted cross entropy for imbalanced training set.
- Add val_acc_macro and val_auroc for model selection. Support test_acc_macro and test_aucro as well.
- Extend speech_to_label.py to support language identification.
- Add model config file which achieves SOTA performance on voxlingua107.


# Usage
```python
python speech_to_label.py  --config-path="../conf/lang_id" --config-name="titanet_large" \
model.train_ds.manifest_filepath=<train_manifest> \
model.validation_ds.manifest_filepath=<dev_manifest> \
model.train_ds.augmentor.noise.manifest_path=<noise_manifest>  \
model.train_ds.augmentor.impulse.manifest_path=<impulse_manifest>  \
exp_manager.wandb_logger_kwargs.name="titanet_large" \
exp_manager.wandb_logger_kwargs.project="lang_id" \
+exp_manager.checkpoint_callback_params.monitor="val_acc_macro" \
+exp_manager.checkpoint_callback_params.mode="max" 

```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation? 
        [doc will be updated in next PR]
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Tutorial][ASR][Fix] Data paths in ASR with NeMo tutorial
State: closed | Pull request: yes


# What does this PR do ?

This PR aims to fix a few small issues in the ASR with NeMo tutorial.

**Collection**: ASR

# Changelog 
- check if `data_dir` exists and create if necessary
- prepend `data_dir` to `paths2audio_files`
- check if the expected config file exists and download if necessary

# Usage
n/a

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
n/a
Fix providing glue in seq2seq eval
State: closed | Pull request: yes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

# What does this PR do ?

Fixes a bug that prevented doing GLUE fine-tuning eval.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Updates to adapter training
State: closed | Pull request: yes

# What does this PR do ?

Parallel adapter did not perform as well initially. 

**Collection**: [NLP]

# Changelog 
- Config now lets you select the type of initialization for ColumnLinear.

# Usage
* See  #4717

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
G2P docs
State: closed | Pull request: yes

Signed-off-by: ekmb <ebakhturina@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ X] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Why `.float()` is in `conformer_modules.py`?
State: closed | Pull request: no
Labels: name: stale

https://github.com/NVIDIA/NeMo/blob/eae1684f7f33c2a18de9ecfa42ec7db93d39e631/nemo/collections/asr/parts/submodules/conformer_modules.py#L259

Hi! I'm wondering why `.float()` is here. It seems like it will break code if both model parameters and input in float16
Fix small spelling mistakes
State: closed | Pull request: yes
Labels: name: documentation

Signed-off-by: SeanNaren <snarenthiran@nvidia.com>

# What does this PR do ?

When going through the examples I saw these small spelling mistakes.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Check if conda is active rather than if it is present - reinstall.sh
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
https://github.com/NVIDIA/NeMo/blob/main/reinstall.sh#L27
This should  check if conda is active (`  [ -n "${CONDA_PREFIX}" ] `) rather than if conda is present (`   [ -x "$(command -v conda)" ]
 `). 
Ref: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html
If conda is installed on a system, but a standard python virtual environment is being, this fails to install nemo correctly. 

**Steps/Code to reproduce bug**

- Deactivate conda
- Create a python virtual environment
- Install Nemo in it
Updated inference code and squad scripts
State: closed | Pull request: yes

Signed-off-by: Virginia Adams <vadams@nvidia.com>

# What does this PR do ?

1. Updates squad preprocessing script and adds squad metric calculation script
2. Updates prompt learning inference script to write prediction to file
3. Turns off sequence parallel for GPT models trained with it

**Collection**: NLP

# Changelog 
- Update squad preprocessing script
- Add squad metric calculation script
- Updates GPT prompt learning model class
- Updates prompt learning GPT eval script
-  Updates T5 prompt learning model class
- Updates prompt learning T5 eval script

# Usage
* You can potentially add a usage example below

Updated gpt prompt learning eval command
```
python megatron_gpt_prompt_learning_eval.py \
            virtual_prompt_model_file=PATH_TO_NEMO_PROMPT_LEARNING_MODEL_FILE \
            gpt_model_file=PATH_TO_FROZEN_GPT_MODEL_FILE \
            inference.greedy=True \
            inference.add_BOS=False \
            trainer.devices=1 \
            trainer.num_nodes=1 \
            tensor_model_parallel_size=1 \
            pipeline_model_parallel_size=1 \
            pred_file_path=PATH_WHERE_PRED_TEXT_FILE_WILL_BE_SAVED \
            data_paths=[path/to/dataset1.jsonl, path/to/dataset2.jsonl]

```

Squad metric calculation script

```
python squad_metric_calc.py --ground-truth squad_test_gt.jsonl --preds squad_20B_preds.txt
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add chinese to language doc and fix bug
State: closed | Pull request: yes

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Need Info on train dataset 'augmentor' key in the config.yml file.
State: closed | Pull request: no

Found this  lines in the stt_es_conformer_ctc_large config  file 

'augmentor': {'noise': {'prob': 0.1, 'manifest_path': '/rir_noises_tarred/noises_20s_tarred/tarred_audio_manifest.json', 'audio_tar_filepaths': '/rir_noises_tarred/noises_20s_tarred/audio_{0..255}.tar', 'min_snr_db': 0, 'max_snr_db': 30}, 'transcode_aug': {'prob': 0.2}}, 'is_tarred': False, 'tarred_audio_filepaths': None, 'shuffle_n': 2048}

could not find reference of this key  in any notebooks in the tutorial dir.

how to obtain these nose manifest ?  (is this related to the online_noise_augmentation.ipynb notebook)?,  
will setting this to none affect the new model being trained?
Riva itn 2.0
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Update TN and ITN for ES to 2.0, add en_in (EN International, current Riva EN ITN 2.0) 

**Collection**: nemo_text_processing

# Changelog 
Add en_in ITN  (current Riva EN ITN 2.0)
Update ES for TN and ITN to  2.0


**PR Type**:
- [X ] New Feature
- [X ] Bugfix
clean warnings from tests and CI runs, and prepare for upgrade to PTL 1.8
State: closed | Pull request: yes

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

# What does this PR do ?

Remove warnings from core and ASR code. NLP part needs more rework than expected. 

**Collection**: Core, ASR

# Changelog 
- [x] weights_save_path has been removed
- [x] Some changes on ptl self.log sync_dist = True 
- [x] torch.qr -> torch.linalg.qr 
- [x] on_pretrain_routine_start -> on_fit_start
- [x] Update most metrics to have full_state_update = True variable
- [x] Pytest  remove with_downloads marker warning 
- [x] regex escape sequence and np float deprecation


# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation
adding loss normalization options to rnnt joint 
State: closed | Pull request: yes

# What does this PR do ?

Adds "mean" and "mean_volume" loss normalization options to RNNT Joint

**Collection**: 
ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
In the config file, set `rnnt_reduction` to `mean`, `mean_batch` (same as the default today), or `mean_volume` in the `model` section

```python
# Add a code snippet demonstrating how to use this 
rnnt_reduction: mean_batch
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
adding instancenorm as an option for conv normalization
State: closed | Pull request: yes

# What does this PR do ?

Adds  instancenorm as an option for conv normalization

**Collection**: ASR

# Changelog 
- nemo/collections/asr/parts/submodules/conformer_modules.py - accept `instance_norm` as an accepted config parameter value for ```model.encoder.conv_norm_type```

# Usage
* In config file, set ```model.encoder.conv_norm_type=instance_norm```

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Chinese text normalization]Chinese TN part in text_normalization
State: closed | Pull request: yes

This PR continues https://github.com/NVIDIA/NeMo/issues/4543 & https://github.com/NVIDIA/NeMo/pull/4638 & https://github.com/NVIDIA/NeMo/pull/4683
What does this PR do ?
Add Chinese Text Normalization Tools in NeMo

Collection: 
[NeMo/norm_text_processing/text_normalization]
[NeMo/tools/text_processing_deployment]
[NeMo/tests/nemo_text_processing/zh]
Prompt Learning Eval Updates
State: closed | Pull request: yes

Signed-off-by: Virginia Adams <vadams@nvidia.com>

# What does this PR do ?

1. Updates squad preprocessing script and adds squad metric calculation script
2. Updates prompt learning inference script to write prediction to file
3. Turns off sequence parallel for GPT models trained with it

**Collection**: NLP

# Changelog 
- Update squad preprocessing script
- Add squad metric calculation script
- Updates GPT prompt learning model class
- Updates prompt learning GPT eval script

# Usage
* You can potentially add a usage example below

Updated gpt prompt learning eval command
```
python megatron_gpt_prompt_learning_eval.py \
            virtual_prompt_model_file=PATH_TO_NEMO_PROMPT_LEARNING_MODEL_FILE \
            gpt_model_file=PATH_TO_FROZEN_GPT_MODEL_FILE \
            inference.greedy=True \
            inference.add_BOS=False \
            trainer.devices=1 \
            trainer.num_nodes=1 \
            tensor_model_parallel_size=1 \
            pipeline_model_parallel_size=1 \
            pred_file_path=PATH_WHERE_PRED_TEXT_FILE_WILL_BE_SAVED \
            data_paths=[path/to/dataset1.jsonl, path/to/dataset2.jsonl]

```

Squad metric calculation script

```
python squad_metric_calc.py --ground-truth squad_test_gt.jsonl --preds squad_20B_preds.txt
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update diarization folder structure
State: closed | Pull request: yes

# What does this PR do ?

This PR updates the folder structure of example/speaker_task/diarization  folder.

<img width="369" alt="image" src="https://user-images.githubusercontent.com/13211483/187006224-6ac5993b-e6b2-4ac2-85d5-c66208850983.png">


**Collection**: [Note which collection this PR will affect]

ASR (speaker tasks)

# Changelog 
- yaml files in diarization example folder
- changed the names in the diarization example folder.
- changed related parts in docs
- updated yaml paths in tutorials 

# Usage

Usage can be found in README.md file in example diarization folder


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix type error in rnnt_wer.py, rnnt_wer_bpe.py, wer_bpe.py
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Fix float/integer type error in wer computation. Similar to PR https://github.com/NVIDIA/NeMo/pull/4816 but on different files.

# Changelog 
Fix type errors in file rnnt_wer.py, rnnt_wer_bpe.py, wer_bpe.py. 

# Usage
N/A



# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Asr codeswitch
State: closed | Pull request: yes

# What does this PR do ?

Enables manifest processing / multilingual model training where there is more than one language ID per sample

**Collection**: [Note which collection this PR will affect]
ASR

# Changelog 
- [audio_to_text.py] modified to parse the manifest in the new format
- [collections.py] modified to not require global lang id per sample in this case (since lang id will be specified on each text span)

# Usage
Use manifest format as shown below. No other code changes required.

```json
{"audio_filepath": "_data_cs_constructed_data_v1_0.wav", "shard_id": 168, "duration": 17.715125, "text": [{"lang": "es", "str": "es probablemente la amiga de puck que m\u00e1s aparece y m\u00e1s presencia tiene "}, {"lang": "en", "str": "she might have gone on but that she caught kinraid's eyes looking at her with kindly admiration "}, {"lang": "es", "str": "o las que tienes de estar con alguien que no lo es porque"}]}
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add kab language asr models
State: closed | Pull request: yes

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

# What does this PR do ?

Add kabyle language ASR models 

**Collection**: ASR

# Changelog 
- Add kabyle language ASR models  

# Usage

```python
import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.EncDecRNNTBPEModel.from_pretrained(model_name="stt_kab_conformer_transducer_large")
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation
Speaker Diarization Inference - unable to detect number of speakers when oracle_num_speakers set to False
State: closed | Pull request: no

I am new to nemo and following this Colab notebook - https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Diarization_Inference.ipynb

When I try set num_speakers = None in manifest file and set config.diarizer.clustering.parameters.oracle_num_speakers=False , and run it without oracle vad diarization,  it only detects one speaker even for the example file provided in the notebook


This is the code I amended:

```
an4_audio_url = "https://nemo-public.s3.us-east-2.amazonaws.com/an4_diarize_test.wav"
an4_audio = wget.download(an4_audio_url, data_dir)

`import json
meta = {
    'audio_filepath': an4_audio, 
    'offset': 0, 
    'duration':None, 
    'label': 'infer', 
    'text': '-', 
    **'num_speakers': None,** 
    'rttm_filepath': None, 
    'uem_filepath' : None
}
with open('data/input_manifest.json','w') as fp:
    json.dump(meta,fp)
    fp.write('\n')



**config.diarizer.clustering.parameters.oracle_num_speakers=False**

```

Can someone help me with why it is not able to predict two speakers?
SQuAD Script and Prompt Learning Eval Updates
State: closed | Pull request: yes

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Updates squad specific scripts for data set pre-processing and evaluation. Make improvements to prompt learning inference script and fixes bug with batch prompt learning inference. 

Add a one line overview of what this PR aims to accomplish.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix float/integer type error in WER.update()
State: closed | Pull request: yes




# What does this PR do ?

Without this, `WER.update()` fails with the following exception:

    TypeError: 'float' object cannot be interpreted as an integer

This is because the `score` property is initialized as `tourch.int64`,
but actually is a float value. Fix the initialization so that the
property has the correct type.

*Technical Note:* the actual failing line is line 780 in `nemo/collections/asr/metrics/wer.py`:

```python3
self.scores = torch.tensor(scores, device=self.scores.device, dtype=self.scores.dtype)
```

Since `self.scores.dtype` is `torch.int64` in the current version, this line ended
up something like `torch.tensor(0.5, dtype=torch.int64)`  and fails.

**Collection**: [Note which collection this PR will affect]

ASR

# Changelog 

- Fix TypeError in WER.update()

# Usage

N/A

# Before your PR is "Ready for review"

**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [N/A] Did you write any new necessary tests?
- [N/A] Did you add or update any necessary documentation?
- [N/A] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [N/A] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Megatron NMT Adapters
State: closed | Pull request: yes
Labels: name: stale

Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

# What does this PR do ?

Megatron NMT Adapters

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update Apex commit
State: closed | Pull request: yes

# What does this PR do ?

Updates Apex commit used for building and testing to include distributed Adam.

**Collection**:  NLP

# Changelog 
- Update Apex commit in Dockerfile and Jenkinsfile

# Usage

```bash
cd /path/to/nemo
DOCKER_BUILDKIT=1 docker build .
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Enables https://github.com/NVIDIA/NeMo/pull/4487
* The Apex commit corresponds to https://github.com/NVIDIA/apex/pull/1463
* CI fails for https://github.com/NVIDIA/NeMo/pull/4799 since the existing Docker container uses an old Apex commit
Fine tuning stt_en_conformer_transducer_xxlarge gives a dtype error on GPU ( A100) 
State: closed | Pull request: no
Labels: name: bug

Getting a  "TypeError: __init__() got an unexpected keyword argument 'dtype'" 
at `File python3.8/site-packages/nemo/collections/asr/parts/submodules/subsampling.py:144, in ConvSubsampling.forward(self, x, lengths)
    140 x = x.unsqueeze(1)
    141 if self._subsampling == 'striding':
    142     # added in order to prevent slowdown in torch.nn.Conv2d with bfloat16 / CUDNN v8 API
    143     # to be removed once the above is fixed in cudnn
--> 144     with torch.cuda.amp.autocast(dtype=torch.float32):
    145         x = self.conv(x)`

**Steps/Code to reproduce bug**
```
pretrained =None
try:
    pretrained = nemo_asr.models.EncDecRNNTBPEModel.from_pretrained(model_name="stt_en_conformer_transducer_xxlarge")
except:
    pretrained = nemomodels.EncDecRNNTBPEModel.restore_from(restore_path= model_loc_path)
pretrainedConfig = DictConfig(pretrained.cfg)
pretrainedConfig.train_ds.manifest_filepath = TRAIN_MANIFEST
pretrainedConfig.validation_ds.manifest_filepath = VALIDATION_MANIFEST
pretrainedConfig.test_ds.manifest_filepath = TEST_MANIFEST
pretrainedConfig['train_ds']['is_tarred'] = False
pretrainedConfig['train_ds']['tarred_audio_filepaths'] = None
pretrainedConfig['validation_ds']['is_tarred'] = False
pretrainedConfig['validation_ds']['tarred_audio_filepaths'] = None
pretrained.set_trainer(trainer)
pretrained.setup_training_data(pretrainedConfig['train_ds'])
pretrained.setup_validation_data(pretrainedConfig['validation_ds'])


from nemo.utils import exp_manager
from omegaconf import OmegaConf

# Environment variable generally used for multi-node multi-gpu training.
# In notebook environments, this flag is unnecessary and can cause logs of multiple training runs to overwrite each other.
os.environ.pop('NEMO_EXPM_VERSION', None)

exp_config = exp_manager.ExpManagerConfig(
    exp_dir=f'experiments/',
    name=f"Transducer-Model",
    checkpoint_callback_params=exp_manager.CallbackParams(
        monitor="val_wer",
        mode="min",
        always_save_nemo=True,
        save_best_model=True,
    ),
)

exp_config = OmegaConf.structured(exp_config)
logdir = exp_manager.exp_manager(trainer, exp_config)

# Train the model
trainer.fit(pretrained)
```

The same code works on a CPU , but fails on A100 GPU



nemo-toolkit              1.10.0    
torch                     1.9.0+cu111             
torch-stft                0.1.4                
torchaudio                0.9.0                  
pytorch-lightning         1.7.2

Linux cloud-amd64 #1 SMP Debian 4.19.249-2 (2022-06-30) x86_64 GNU/Linux

A100 GPU on GCP gace this dtype error , but worked okay on CPU
Fix MixerTTS data loading index error
State: closed | Pull request: yes

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

# What does this PR do ?

Fixes indexing error in MixerTTS-X data loading introduced by adding items to the TTS datasets

**Collection**: TTS

# Changelog 
- Changed indexing from 13 to 15 due to two additional items `voiced_mask` and `p_voiced`

**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
s2s dialog generation custom data processor
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Added data processor to support custom data for DialogueS2SGenerationModel.

**Collection**: [nlp]

# Changelog 
- Added a data processor for a custom dataset for the DialogueS2SGenerationModel.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
What data was used to train the German Conformer Transducer?
State: closed | Pull request: no

Hello,

in the [model's page](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_de_conformer_transducer_large), it is said that the model was trained on:

- 567 hours of MCV7
- 1524 hours of MLS
- 214 hours of VoxPopuli

I am wondering how the subsets for each dataset were chosen, especially for MCV, where there is accented data. Unfortunately, I could not find the details of this dataset online. Could you please clarify how the subsets were chosen?

Kind regards,
Carlos
Fix initializing weights from ptl ckpt with exclude
State: closed | Pull request: yes

Signed-off-by: sam1373 <samuelkriman@gmail.com>

# What does this PR do ?

# Changelog 
- fix typo "model_path" to "ckpt_path"
- also makes the "load_from_string" parameter optional

**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation


# Additional Information
* Related to [#4805](https://github.com/NVIDIA/NeMo/issues/4805)
avoid data clipping after convolution with rir samples
State: closed | Pull request: yes

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

# What does this PR do ?

Fix nan issue with precision=16

**Collection**: ASR

# Changelog 
- Clip max value of data samples after convolution with rir impulses to 1 

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Initializing weights from checkpoint with configs - init_from_ptl_ckpt
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
Hi! When I tried to initialize weights of the model from checkpoint using configs and excluding some parts of it I get an error that model_path is not defined
I looked a the code at https://github.com/NVIDIA/NeMo/blob/main/nemo/core/classes/modelPT.py and indeed there is  an initialized variable  ckpt_path, but model_path  is initialized for init_from_nemo_model
![image](https://user-images.githubusercontent.com/53433879/186564873-4faebdc0-a4bc-4f65-b9d3-7e0024025780.png)

Could you check if it was just accidental copy-paste and there should be ckpt_path instead of model_path  
                        self.load_part_of_state_dict(
                            ckpt['state_dict'], include, exclude, f'nemo file with path `**{model_path**}`'
                        )

in this snippet of  the code 

        if 'init_from_ptl_ckpt' in cfg and cfg.init_from_ptl_ckpt is not None:
            with open_dict(cfg):
                if isinstance(cfg.init_from_ptl_ckpt, str):
                    # Restore checkpoint
                    ckpt_path = cfg.pop('init_from_ptl_ckpt')
                    ckpt = torch.load(ckpt_path, map_location=map_location)

                    # Restore checkpoint into current model
                    self.load_state_dict(ckpt['state_dict'], strict=False)
                    logging.info(
                        f'Model checkpoint restored from pytorch lightning chackpoint with path : `{ckpt_path}`'
                    )

                    del ckpt
                elif isinstance(cfg.init_from_ptl_ckpt, (DictConfig, dict)):
                    model_load_dict = cfg.init_from_ptl_ckpt
                    for model_load_cfg in model_load_dict.values():
                        ckpt_path = model_load_cfg.path
                        # Restore model
                        ckpt = torch.load(ckpt_path, map_location=map_location)

                        include = model_load_cfg.pop('include', [""])
                        exclude = model_load_cfg.pop('exclude', [])

                        self.load_part_of_state_dict(
                            ckpt['state_dict'], include, exclude, f'nemo file with path `{model_path}`'
                        )