renwei2024/nemo-github-issues huggingface dataset:
Fixing Megatron BERT output dimensions to [batch x sec x hidden]
State: closed | Pull request: yes

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Enable mlflow logger
State: closed | Pull request: yes

Signed-off-by: whrichd <trabeitwrq@gmail.com>

# What does this PR do ?

MLFlow is already a dependancy in Pytorch Lightning, so some small changes enables MLFlow tracking, a free alternative to WnB.

**Collection**: utils

# Changelog 
- Changed exp_manager.py

# Usage
* Add the MLflow logger just as how you would add the weighs and biases logger, in the config files.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
added utf8 encoding
State: closed | Pull request: yes

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Writes T5 prompt learning prediction file with utf-8 encoding. 

Add a one line overview of what this PR aims to accomplish.

**Collection**: NLP

# Changelog 
- T5 prompt learning eval script

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Deploying to Riva with self-trained ASR model, but unable to infer audio 
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I used NeMo to train a Mandarin ASR model (QuartzNet), and then I converted it to .rmir format through the tutorial.
I deployed .rmir to riva (I only have this model enabled), but the client cannot successfully call the ASR model on the server side, there will be the following error
![error](https://user-images.githubusercontent.com/56921951/188802743-ba9db1e3-d518-45df-8704-80b7954c571a.png)


**Steps/Code to reproduce bug**

- build environment
docker pull nvcr.io/nvidia/riva/riva-speech:2.4.0-servicemaker
sudo docker run --gpus all -it --rm -v /NeMo/nemo_experiments/QuartzNet15x5/rmir_fold:/servicemaker-dev -v /NeMo/nemo_experiments/QuartzNet15x5:/data --entrypoint="/bin/bash" nvcr.io/nvidia/riva/riva-speech:2.4.0-servicemaker

- convert to .riva file
nemo2riva --out Nemo_CTC.riva QuartzNet15x5.nemo

- convert to .rmir file
riva-build speech_recognition -f /servicemaker-dev/NeMo_CTC.rmir /data/2022-08-19_07-43-11/Nemo_CTC.riva --decoder_type=greedy

- start riva-speech
ngc registry resource download-version nvidia/riva/riva_quickstart:2.4.0
cd riva_quickstart_v2.4.0
bash riva_init.sh
bash riva_start.sh
**get a result like this =>**
**c1106038@DGX-Station-62:~/NeMo/riva_quickstart_v2.4.0$ sudo bash riva_start.sh
Starting Riva Speech Services. This may take several minutes depending on the number of models deployed.
2cfde38fd430b4ce346b125850848e187cc399422c59470f5eaef05bc00ea22b
Waiting for Riva server to load all models...retrying in 10 seconds
Riva server is ready...**

(In order to use a custom rmir file, only modify riva_model_loc and use_existing_rmirs=true in config.sh)
1. riva_model_loc="/home/c1106038/NeMo/nemo_experiments/QuartzNet15x5/rmir_fold"
2. use_existing_rmirs=true

- start client
bash riva_start_client.sh
(In order to use GPU modify the riva_start_client.sh as follows)
![image](https://user-images.githubusercontent.com/56921951/188805307-511da5b5-c8c0-43ba-9bc0-57c1eb5b977b.png)
**get a result like this =>**
![修改後](https://user-images.githubusercontent.com/56921951/188804990-d812dfd3-2287-4613-8c22-89b491bbc6b1.png)

- Execute riva_asr_client
riva_asr_client --audio_file=/test/morning_conference_000001.wav -language_code=zh-CN
**get a Error=>**
![error (1)](https://user-images.githubusercontent.com/56921951/188806188-a118efc9-5dad-4821-baee-943604bf21af.png)


**Expected behavior**

Custom model inference audio available on Riva.

**Environment overview (please complete the following information)**

- DGX V100 GPU
- Method of NeMo install:
docker pull nvcr.io/nvidia/nemo:22.05
docker run --gpus all -it --name Nemo_container -v NeMo:/workspace --shm-size=8g -p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit stack=67108864 --device=/dev/snd nvcr.io/nvidia/nemo:22.05

**Environment details**
OS version: Ubuntu 20.04
PyTorch version: 1.12.0a0+8a1a93a
Python version: 3.8.13
[Speaker Recognition and Verification] got runtime error on training
State: closed | Pull request: no
Labels: name: bug

name: stale

I tried to learn training speaker embedding extractor from [https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/starthere/tutorials.html](url)
Domain: ASR
title: Speaker Recognition and Verification
It's no problem before cell [20]
cell [20]:
`trainer.fit(speaker_model)`
bug:
```
INFO:pytorch_lightning.accelerators.gpu:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
[NeMo I 2022-09-07 02:27:14 modelPT:587] Optimizer config = SGD (
    Parameter Group 0
        dampening: 0
        foreach: None
        lr: 0.006
        maximize: False
        momentum: 0
        nesterov: False
        weight_decay: 0.001
    )
[NeMo I 2022-09-07 02:27:14 lr_scheduler:914] Scheduler "<nemo.core.optim.lr_scheduler.CosineAnnealing object at 0x7f682c63a690>" 
    will be used during training (effective maximum steps = 140) - 
    Parameters : 
    (warmup_ratio: 0.1
    min_lr: 0.0
    max_steps: 140
    )
INFO:pytorch_lightning.callbacks.model_summary:
  | Name              | Type                              | Params
------------------------------------------------------------------------
0 | preprocessor      | AudioToMelSpectrogramPreprocessor | 0     
1 | encoder           | ConvASREncoder                    | 19.4 M
2 | decoder           | SpeakerDecoder                    | 2.8 M 
3 | loss              | AngularSoftmaxLoss                | 0     
4 | _accuracy         | TopKClassificationAccuracy        | 0     
5 | spec_augmentation | SpectrogramAugmentation           | 0     
------------------------------------------------------------------------
22.1 M    Trainable params
0         Non-trainable params
22.1 M    Total params
88.546    Total estimated model params size (MB)
[NeMo I 2022-09-07 02:27:15 label_models:312] val_loss: 13.214
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-22-f8bb6804e159>](https://localhost:8080/#) in <module>
----> 1 trainer.fit(speaker_model)

14 frames
[/usr/local/lib/python3.7/dist-packages/nemo/utils/timers.py](https://localhost:8080/#) in start(self, name)
     90 
     91         if "start" in timer_data:
---> 92             raise RuntimeError(f"Cannot start timer = '{name}' since it is already active")
     93 
     94         # synchronize pytorch cuda execution if supported

RuntimeError: Cannot start timer = 'train_step_timing' since it is already active
Epoch 0: 0%
0/15 [00:00<?, ?it/s]

```
**Additional context**
Before the above Runtime error, I got `TypeError: unhashable type: 'list'` from `/usr/local/lib/python3.7/dist-packages/numba/cuda/dispatcher.py`. But I fixed it.
Fixed output of BERT to be [batch x seq x hidden]
State: closed | Pull request: yes

Signed-off-by: Micha Livne <mlivne@cs.toronto.edu>

# What does this PR do ?

This PR Fixes the output of BERT to be [batch x seq x hidden] instead of [se x batch x hidden]

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] add missing WikiHomograph data entries to CMUdict, updates to match new ipa set
State: closed | Pull request: yes

Signed-off-by: ekmb <ebakhturina@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Convert rnnt asr model to tensorrt
State: closed | Pull request: no
Labels: name: feature

I am working on converting the RNNT model to tensorrt for my application and I have convert the torch model to ONNX as suggested. However, I am stuck on converting ONNX to trt model right now. I tried lots of methods and still failed. Is it possible to convert? Did anyone succeed on converting this model to tensorrt? Which tensorrt version should I use or how can I do the conversion? Any tutorials?
Fix Megatron NMT consumed samples and ckpt_to_nemo split rank
State: closed | Pull request: yes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

# What does this PR do ?

Fixes bugs in Megatron NMT due to incorrect split rank in the checkpoint file and split rank in ckpt->nemo conversion.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Q: How to reset optimizer/LR to finetune?
State: closed | Pull request: no

Hi,

I trained a MT model following the example from `enc_dec_nmt.py`, so far all good. Now I'd like to load one of the checkpoints generated from this training, keep training the model weights, but restarting the optimizer / LR / global trainer step / etc. I've tried different approaches, but in all cases the training continues from where it was (i.e. from the previous global step, epoch and everything). 

What's the correct way of achieving this?

Approaches I tried:

### Approach 1

Setting `resume_if_exists: true` in my config, so NeMo will take the last checkpoint. The in the model file (`mt_enc_dec_model.py`) I added a `on_train_start` hook, where I create a new optimizer and assign it back to the model. It looks something like this

```python
    def on_train_start(self) -> None:
        opt: LightningOptimizer = self.optimizers()
        new_pt_opt: Optimizer = opt.optimizer.__class__(opt.optimizer.param_groups, **opt.optimizer.defaults)
        self.trainer.strategy.optimizers = [new_pt_opt]

```

### Approach 2

With `resume_if_exists: false`. When calling `trainer.fit` in `enc_dec_nmt.py`, setting `ckpt_path` to one of the checkpoints. Basically something like 

```python
trainer.fit(mt_model, ckpt_path="path/to/checkpoint.cpkt")
```
Conformer Transducer - Call to cuLaunchKernel results in CUDA_ERROR_INVALID_VALUE
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Error while training conformer transducer bpe ASR model.

**Steps/Code to reproduce bug**
Use the attached config and initiate the training. After a few steps before epoch completion the following 
`Call to cuLaunchKernel results in CUDA_ERROR_INVALID_VALUE` error occurs and the training get stopped.

`Call to cuLaunchKernel results in CUDA_ERROR_INVALID_VALUE
Error executing job with overrides: []
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/parts/numba/rnnt_loss/rnnt_pytorch.py", line 174, in forward
    return self.loss(
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/parts/numba/rnnt_loss/rnnt_pytorch.py", line 64, in forward
    loss_func(
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/parts/numba/rnnt_loss/rnnt.py", line 223, in rnnt_loss_gpu
    status = wrapper.cost_and_grad(
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/parts/numba/rnnt_loss/utils/cuda_utils/gpu_rnnt.py", line 249, in cost_and_grad
    return self.compute_cost_and_score(acts, grads, costs, pad_labels, label_lengths, input_lengths)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/parts/numba/rnnt_loss/utils/cuda_utils/gpu_rnnt.py", line 161, in compute_cost_and_score
    gpu_rnnt_kernel.compute_alphas_kernel[self.minibatch_, self.maxU_, self.stream_, 0](
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/compiler.py", line 729, in __call__
    return self.dispatcher.call(args, self.griddim, self.blockdim,
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/compiler.py", line 919, in call
    kernel.launch(args, griddim, blockdim, stream, sharedmem)
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/compiler.py", line 576, in launch
    driver.launch_kernel(cufunc.handle,
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 2550, in launch_kernel
    driver.cuLaunchKernel(cufunc_handle,
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 319, in safe_cuda_api_call
    self._check_ctypes_error(fname, retcode)
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 387, in _check_ctypes_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [1] Call to cuLaunchKernel results in CUDA_ERROR_INVALID_VALUE
`
**Expected behavior**

Run training without any error.

**Environment overview (please complete the following information)**

 - SLURM Managed cluster
 - DGX-1 8x V100 GPU
 - Singularity environment - nvcr.io/nvidia/nemo:22.05

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**
DGX-1 8x V100 GPU x 4 Nodes
**Training Config**
[cmr_en_rnnt_v2_n4.txt](https://github.com/NVIDIA/NeMo/files/9497280/cmr_en_rnnt_v2_n4.txt)
Changes to MSDD code after review, fix test log call
State: closed | Pull request: yes

# What does this PR do ?

Going through the MSDD code I noticed a few off the bat code changes to be made + fix the `self.log` call in the multi_test function.

cc @tango4j @nithinraok

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Add sync for logging calls to ensure aggregation across devices
State: closed | Pull request: yes
Labels: name: bug

# What does this PR do ?

After speaking to @nithinraok a few places do not have sync calls when logging (some are taken care off by the underlying Metrics, or they are already synced across processes like global steps).

I've been a bit cautious and not touched the return objects from `validation_step` or `test_step`. There seems to be some parsing of logic in the ModelPT file so need to investigate further as to what to do with those metrics, and how they are logged.

# Changelog 
- Add `sync_dist` to logging metric calls to ensure aggregation across all devices.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
How to enable kenlm weight accounting with Conformer-CTC acoustic model on BPE tokens?
State: closed | Pull request: no
Labels: name: bug

Trained Conformer-CTC on BPE tokens, tried to make a prediction based on kenlm. But all the predictions turned out to be just nonsense from one word.

As I understand it, this is due to the fact that models on BPE tokens return an underscore instead of a space.
Tell me, can this problem be somehow elegantly solved in order to use kenlm with an acoustic model on BPE?

From thoughts, can kenlm be trained on texts where there is an underscore instead of a space? But that doesn't seem to work...
ipa bug fix
State: closed | Pull request: yes

Signed-off-by: ekmb <ebakhturina@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[ASR] Generate multichannel noise
State: closed | Pull request: yes

# What does this PR do ?

This PR adds tools for generating multi-channel microphone signal approximately matching the desired spatial coherence

**Collection**: ASR

# Changelog 
- Adds functions for generating an approximate noise field
- Unit tests for spatial coherence calculation and noise generation

# Usage

```python
noise_field = generate_approximate_noise_field(mic_positions, noise_signal, sample_rate)
```

where `mic_positions` are positions of the `M` microphones in 3D Cartesian coordinates, `noise_signal` is an `M`-channel noise signal, where channels are statistically independent, sampled at `sample_rate` Hz. Currently, only `spherical` noise field is supported.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
n/a
IPA G2P bugfixes
State: closed | Pull request: yes

Fix uppercasing bug that led to heteronyms list mismatch, bad path in IPA FastPitch config

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

**Collection**: TTS

# Changelog 
- The IPA G2P class uppercases words by default (to avoid IPA symbol conflicts), but they would not match anything in the  lowercased heteronyms file. Fixed by uppercasing heteronyms when loading the file.
- IPA FastPitch config fixed to point to the updated location of the IPA G2P class.

**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

# Additional Information
* The first fix is identical to the one merged into the r1.11.0 branch (#4860), reproduced here since the file has been moved in `main`.
[TTS][German][Single Speaker] Generated audio far away from original Thorsten voice  
State: closed | Pull request: no
Labels: name: bug

name: TTS

**Describe the bug**

Using the main branch of Nemo and TTS inference with Fastpitch and Hifigan  /  German single speaker model results in bad audio. 

**Steps/Code to reproduce bug**

git clone https://github.com/NVIDIA/NeMo/
cd NeMo
python3.8 -m venv venv
source venv/bin/activate
pip3 install a-lot-of-missing-requirements when executing
python3 tts_german.py where inference code is as follows tts_german.py

```
import soundfile as sf
from nemo.collections.tts.models.base import SpectrogramGenerator, Vocoder

# Download and load the pretrained fastpitch model
spec_generator = SpectrogramGenerator.from_pretrained(model_name="tts_de_fastpitch_singlespeaker")#.cuda()
# Download and load the pretrained hifigan model
vocoder = Vocoder.from_pretrained(model_name="tts_de_hui_hifigan_ft_fastpitch_multispeaker_5")#.cuda()

# All spectrogram generators start by parsing raw strings to a tokenized version of the string
parsed = spec_generator.parse("In den alten Zeiten, wo das Wünschen noch geholfen hat, lebte ein König, dessen Töchter waren alle schön.")
# They then take the tokenized string and produce a spectrogram
spectrogram = spec_generator.generate_spectrogram(tokens=parsed)
# Finally, a vocoder converts the spectrogram to audio
audio = vocoder.convert_spectrogram_to_audio(spec=spectrogram)

# Save the audio to disk in a file called speech.wav
# Note vocoder return a batch of audio. In this example, we just take the first and only sample.
sf.write("speech.wav", audio.to('cpu').detach().numpy()[0], 44100)
```

**Expected behavior**

Good audio

**Environment overview (please complete the following information)**

See above

**Environment details**

See above

**Additional context**

Audio sample [https://sndup.net/hxnz/](https://sndup.net/hxnz/)
Learning rate does not change
State: closed | Pull request: no
Labels: name: bug

name: stale

Can you please tell me why the learning rate does not change when the fine-tuning model is launched?
```python
cfg = OmegaConf.load("conformer_ctc/config.yaml")
trainer = pl.Trainer(
    logger=logger,
    **cfg.trainer,
    callbacks=[
        model_checkpoint_callback
    ]
)
model = nemo_asr.models.EncDecCTCModel(cfg=cfg.model, trainer=trainer)
model.set_trainer(trainer)

trainer.fit(model, ckpt_path="pretrained/epoch=39.ckpt")
```
Log fit:
> 610/1200 [06:32<-1:37:02, -0.43it/s, loss=61.6, v_num=103]

But according to the graphs learning_rate = 1-e6 throughout the training.


Half "conformer_ctc/config.yaml":
```
name: "Conformer-CTC-Char"
model:
  sample_rate: 8000
  optim:
    name: adamw
    lr: 2.0
    betas: [0.9, 0.98]
    # less necessity for weight_decay as we already have large augmentations with SpecAug
    # you may need weight_decay for large models, stable AMP training, small datasets, or when lower augmentations are used
    # weight decay of 0.0 with lr of 2.0 also works fine
    weight_decay: 0.0

    sched:
      name: NoamAnnealing
      d_model: ${model.encoder.d_model}
      # scheduler config override
      warmup_steps: 15000
      warmup_ratio: null
      min_lr: 1e-6

trainer:
  devices: -1
  num_nodes: 1
  max_epochs: 100
  max_steps: -1
  val_check_interval: 1.0
  accelerator: auto
  strategy: dp
  accumulate_grad_batches: 1
  gradient_clip_val: 1.0
  precision: 32 # Should be set to 16 for O1 and O2 to enable the AMP.
  # amp_level: O2
  # amp_backend: apex
  log_every_n_steps: 50
  progress_bar_refresh_rate: 10
  resume_from_checkpoint: null
  num_sanity_val_steps: 0 
  check_val_every_n_epoch: 1 # number of evaluations on validation every n epochs
  sync_batchnorm: true
  enable_checkpointing: true
  benchmark: false # needs to be false for models with variable-length speech input as it slows down training


exp_manager:
  exp_dir: null
  name: ${name}
  use_datetime_version: True
  create_tensorboard_logger: true
  create_checkpoint_callback: true
  checkpoint_callback_params:
    monitor: "val_wer"
    mode: "min"
    save_top_k: 3
    always_save_nemo: True
  resume_if_exists: False
  resume_ignore_no_checkpoint: False

  create_wandb_logger: false
  wandb_logger_kwargs:
    name: null
    project: null
```
Fix error with class method reference in msdd
State: closed | Pull request: yes
Labels: name: bug

# What does this PR do ?

Small error when referring to the available models. This isn't tested yet since no models are yet available for MSDD, but I think soon @tango4j will introduce them!

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Add AMI dataset script
State: closed | Pull request: yes

# What does this PR do ?

Adds a script to streamline downloading/parsing AMI. @tango4j suggested adding microphone array audio as well, just need to figure out how to format this.

**Collection**: ASR

# Changelog 
- Add script to download the AMI dataset for Speaker Diarization

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Error when training ASR with Transducers on GPU
State: closed | Pull request: no
Labels: name: bug

![image](https://user-images.githubusercontent.com/53433879/188069192-c4997fe2-43fb-4c72-aa8e-a3e6163e07bd.png)

when training ASR with Transducers and leaving 

**freq_masks: 2
time_masks: 10** 

instead of zeroing them out an error occurs

![image](https://user-images.githubusercontent.com/53433879/188069618-69d91380-d074-4054-a610-3ef0f9e378ed.png)
TypeError                                 Traceback (most recent call last)
[<ipython-input-29-1bf927b35c34>](https://localhost:8080/#) in <module>
      1 # Train the model
----> 2 trainer.fit(model)

42 frames
[/usr/local/lib/python3.7/dist-packages/numba/cuda/dispatcher.py](https://localhost:8080/#) in __getitem__(self, args)
    567         if len(args) not in [2, 3, 4]:
    568             raise ValueError('must specify at least the griddim and blockdim')
--> 569         return self.configure(*args)
    570 
    571     def forall(self, ntasks, tpb=0, stream=0, sharedmem=0):

TypeError: unhashable type: 'list'

It happens both in google colab and  when training on local machine.
To reproduce this I skipped  

**config.model.spec_augment.freq_masks = 0
config.model.spec_augment.time_masks = 0**

in colab tutorial **ASR_with_Transducers**

if I zero spec_augment everything works. Could you check if it's a possible bug or it is something I do wrong
Speaker Diarization - Audio Length Limit
State: closed | Pull request: no
Labels: name: stale

Hi,

Thanks for the prompt reply on helping me out with diarization with unknown speakers.

I just had a question - is there a limit on the length of audio file that can be processed for speaker diarization , or any speaker model in general with nemo.

If I have files with 1.5 hrs length, can they be effectively diarized?

Thanks!!
Update Speaker Diarization notebooks with unknown oracle_num_speakers
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: ASR

# Changelog 
- Update Speaker Diarization notebooks with unknown oracle_num_speakers.
- Use multiscale diarization in tutorials.
- Add flexibility for vad result plotting

# Usage
* Notebook change

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix uppercasing mismatch for IPA heteronyms
State: closed | Pull request: yes

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

# What does this PR do ?

Fixes bug where lowercase heteronyms wouldn't match with uppercased graphemes in IPA G2P. (Graphemes were uppercased to avoid conflicts with IPA symbols, but the heteronyms list is lowercased, so they would never match.)

**Collection**: TTS

**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Missing types in clustering
State: closed | Pull request: yes

# What does this PR do ?

When going through the code saw a couple of missing types in the class, doesn't hurt to add them.


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Kenlm Language model with citrinet model(Subword Tokenization) is not working (Issue in deployment on riva).
State: closed | Pull request: no
Labels: name: bug

I am trying to deploy my asr system in Riva and facing issue in deploying custom Kenlm language model. Steps followed - :

**Nemo model training** **:**
Model       : citrinet_512
Tokenizer : subword (bpe of sentence piece tokenizer)
Language model : Kenlm 3 gram 
language : Filipino

**Issue :** Nemo provides us the script(/NeMo/scripts/asr_language_modeling/ngram_lm/train_kenlm.py) to train a kenlm language model taking manifest as input. It seems like this script converts the training manifest in to character level format using offset and tokenizer. After that language model is trained as character level language model. 

**Issue in riva :** I passed nemo model, language model binary, decoding vocab to riva build command but getting empty transcripts during Inference.
how riva is going to know that language model is trained as character level model and how riva will decode it.


**Decoding Vocab analysis:** Considering language model is trained on character level so I tokenized my corpus using subword tokenizer used in nemo model then convert each word(new_char) in to character level by :
offset = 100
vocab_number = convert each word of tokenized corpus into id.
new_char = chr(vocab_number + offset)

```
unique_vocabs = set()
offset = 100
with open(corpus) as crp:
    for i in crp:
        text = i.split(" ")
        for t in text:
            curr_vocab = ""
            for j in tokenizer.text_to_ids(t):
                curr_vocab = curr_vocab + chr(j+offset)

            unique_vocabs.add(curr_vocab)

with open(decoder_vocab_path, "w") as vcb:
    for i in unique_vocabs:
        vcb.write(i)
        vcb.write('\n')
```
I Passed decoder_vocab_path to riva build command. Sample vocab ->
[vocab.txt|attachment](upload://xkiOPpsQSBRY95y7xh6RMb1mee9.txt) (111.6 KB)
**Is this the right way to make decoding vocab for a character level language model to deploy it in riva?**


Any help would be appreciated.
What all parameters need to be changed and how in config, if i want to change min_duration in SSL_CONFORMER_PRETRAINING
State: closed | Pull request: no
Labels: name: bug

link to config: https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/ssl/conformer/conformer_ssl.yaml

when I change min duration to let's say 1.6 sec 
I get errors like 
RuntimeError: shape '[8, 44, -1]' is invalid for input of size 143360
RuntimeError: cannot sample n_sample > prob_dist.size(-1) samples without replacement


what all parameters I need to change in order to run the experiment successfully
these are the few parameters I think need some updation.  
stride_layers
combine_time_steps
patch_size
mask_patches
num_negatives
CUSOLVER_STATUS_INTERNAL_ERROR
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
For few audio file has this error when calling speaker diariation function, Error Message as below:
Error:cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling cusolverDnSsyevd(handle, jobz, uplo, n, A, lda, W, work, lwork, info)

**Steps/Code to reproduce bug**
The error might be occuring  in the function clustering: nmsec , when call eigh(laplacian) with cuda (cuda device =1/0).

def eigDecompose(laplacian, cuda, device=None):
print(len(laplacian))
if TORCH_EIGN:
if cuda:
if device is None:
device = torch.cuda.current_device()
A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**
A diarised audio file

**Environment overview (please complete the following information)**
Linux environment, 4 gpus

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version :Microsoft Windows [Version 10.0.19044.1889]
- PyTorch version : 1.12.0+cu102
- Python version: Python 3.7.13
- -Nemo :1.8
- Cuda : 11.6

**Additional context**
Re-opening wrt :https://github.com/NVIDIA/NeMo/issues/3757#issue-1150037680
Joint rnnt/ctc training
State: closed | Pull request: yes
Labels: name: stale

Signed-off-by: Ankur Kumar <ankukumar@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.
- Adds support to optionally train CTC decoder jointly with RNN-T model (encoder is shared)

**Collection**: [ASR]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below
Need to add following flags to RNN-T training command and set `D_MODEL` to encoder output dim
```python
++model.ctc.ctc_reduction=mean_batch \
++model.ctc.ctc_loss_weight=0.3 \
++model.ctc.decoder._target_=nemo.collections.asr.modules.ConvASRDecoder \
++model.ctc.decoder.feat_in=$D_MODEL \
++model.ctc.decoder.num_classes=-1 \
++model.ctc.decoder.vocabulary=[]
```

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add rnnt be
State: closed | Pull request: yes

# What does this PR do ?
adds rnnt model
Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add new models 
State: closed | Pull request: yes

Signed-off-by: George Zelenfroind [gzelenfroind@nvidia.com](mailto:gzelenfroind@nvidia.com)
# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.
Adds links to Belarusian models (ctc and rnnt conformer large)
**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [yes] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add new features to SDE and transcribe speech
State: closed | Pull request: yes

Signed-off-by: George Zelenfroind <gzelenfroind@nvidia.com>

# What does this PR do ?
Adds feature of comparing to models to SDE 
And adds new way of transcribing audio - add new transcripts to existing JSON file 

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]
asr

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below
* in readme

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)