renwei2024/nemo-github-issues huggingface dataset:
WIP: Cache aware streaming nfa
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

Add cache aware streaming model in nemo force aligner

Should be easy for ctc segmentation tool to call and get the log probs as well.

Most of the codes are copied from ```speech_to_text_cache_aware_streaming_infer.py```.


**Collection**: [Note which collection this PR will affect]
ASR

# Changelog 
Added two functions in ASRModuleMixin (mixin.py)  (Not sure if this is the right place to add. Please comment and suggest.)

# Usage
* You can potentially add a usage example below

```python
nemo_model="nemo_models/cache_aware_ctc.nemo"
cd NeMo/tools/nemo_forced_aligner/
python3 align.py \
        batch_size=2 \
        model_path=$nemo_model \
        model_downsample_factor=4 \
        manifest_filepath=$manifest \
        output_dir=./streaming_ctm_with_pred \
        align_using_pred_text=true \
        simulate_cache_aware_streaming=true
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

@VahidooX  @erastorgueva-nv

# Additional Information
* Related to # (issue)
[TTS] bugfix for fastpitch German tutorial
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

fixed tutorial errors and update the instructions.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add flag to get attention from fusion
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Currently GPT gets the attention_mask from the fused softmax kernel.

If, the kernel isn't used, then the fallback to pytorch softmax breaks.

This PR makes it configurable to get attention_mask from the kernel.

**Collection**: NLP

# Changelog 
- Update forward and loss func

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Spectrogram Enhancer: correct dim for length when loading data
State: closed | Pull request: yes
Labels: name: bug

name: TTS

# What does this PR do ?

Makes DataLoader for Spectrogram enhancer (#5565) use correct dimension to determine length of the spectrogram.

Currently it uses number of mel bands as length, so in practice enhancer trains on 80x80 patches, which is obviously suboptimal and a **bug**.

**Collection**: TTS

# Changelog 
- Fixes just one single dimension index

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #6007 but there's more work to do
Question regarding training MSDD.
State: closed | Pull request: no

I am trying to train MSDD on audios containg 3-5 speakers. After preparing the manifest file that points to the pairwise rttms and setting max_speakers to 5, I get the following error:

```
, line 3086, in binary_cross_entropy
    raise ValueError(
ValueError: Using a target size (torch.Size([102, 2])) that is different to the input size (torch.Size([102, 5])) is deprecated. Please ensure they have the same size.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
```
I've also seen this comment in 
```max_num_of_spks: 2 # Number of speakers per model. This is currently fixed at 2.```

Does this mean that msdd can be used only on audios with 2 speakers? Or does it mean that only 2 speakers overlaps are considered?
Micro batch loader for bert model
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update align.py
State: closed | Pull request: yes
Labels: name: cherry-pick

Add model.eval() to ensure the accuracy.

# What does this PR do ?
fix a small bug

**Collection**: [Note which collection this PR will affect]
no collection would be affected

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?
@erastorgueva-nv 

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix links, add missing file
State: closed | Pull request: yes
Labels: name: TTS

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update align.py
State: closed | Pull request: yes

Add model.eval() to ensure the accuracy.

# What does this PR do ?
fix a small bug

**Collection**: [Note which collection this PR will affect]
no collection would be affected

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?
@erastorgueva-nv 

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Change perturb rng for reproducing results easily
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do?

Change perturb rng for reproducing results easily

**Collection**: ASR

# Changelog 
- Remove Random number generator and use int rng and seed. 
- update tutorial. 

# Usage
You can use rng (int) for Perturbations.

```python
 augmentor:
          silence:
            prob: 0.8
            min_start_silence_secs: 0
            max_start_silence_secs: 5
            min_end_silence_secs: 0
            max_end_silence_secs: 5
            rng: 42 # default is None**
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Pr doc tn
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
RNNT loss Umax mismatch error
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Hi! I'm having an issue while fine-tuning `stt_en_conformer_transducer_large`. After a while I get an error stating:
`Output length mismatch! Given U: 35, Expected max U from target lengths: 35 + 1` triggered [here](https://github.com/NVIDIA/NeMo/blob/3e7499599a13869e772612d286d660b574715a5e/nemo/collections/asr/parts/numba/rnnt_loss/rnnt_pytorch.py#L405).

In my mind this is strange because I was already 4 epochs in and I'd imagine the loss has already been calculated for all samples. My understanding of the RNNT loss itself is limited though so I might be missing something.

The maximum number of tokens in the dataset labels is 39 (using the sp tokenizer). Training was not resumed from a previous checkpoint. The architecture of the model has stayed the same during training.

**Steps/Code to reproduce bug**

Normal training with own dataset and custom tokenizer. Max token length in the dataset: 39 (a clue?).

Train config:
```yaml
  loss:
    loss_name: default
    warprnnt_numba_kwargs:
      fastemit_lambda: 0.0
      clamp: -1.0
  variational_noise:
    start_step: 0
    std: 0.0
  optim:
    name: adamw
    lr: 5.0
    betas:
    - 0.9
    - 0.98
    weight_decay: 0.001
    sched:
      name: NoamAnnealing
      d_model: 512
      warmup_steps: 1000
      warmup_ratio: null
      min_lr: 1.0e-06
```


**Expected behavior**

Train loss should be calculated normally.

**Environment overview (please complete the following information)**

 - Environment location: GCP VM
 - Method of NeMo install: `pip install nemo-toolkit` (v1.14.0)

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS: Ubuntu 20.04
- PyTorch version: 1.12.1+cu113
- Python version: 3.7.12

**Additional context**

Add any other context about the problem here.
GPU: 1xV100

EDIT: Added NeMo version
Update data_prep.py
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?
A bug fix

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]
nemo tools
coolections will not be affected

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
python NeMo/tools/nemo_forced_aligner/align.py \
        pretrained_name="stt_en_citrinet_1024_gamma_0_25" \
        model_downsample_factor=8 \        
        batch_size=2
        manifest_filepath=./test_manifest.json \
        output_dir=./ctm_test_manifest
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update data_prep.py
State: closed | Pull request: yes

When batch_size >= 2, the align.py doesn't work on the batch_size examples. This function should return [start, end] (inclusive) examples.

# What does this PR do ?

Fix a function logic:  https://github.com/NVIDIA/NeMo/blob/main/tools/nemo_forced_aligner/utils/data_prep.py#L78

**Collection**: [Note which collection this PR will affect]
nemo neural aligner
no collections will be affected

# Changelog 
- 

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
python NeMo/tools/nemo_forced_aligner/align.py \
         pretrained_name="stt_en_citrinet_1024_gamma_0_25" \
       model_downsample_factor=8 \
       manifest_filepath=./test_manifest.json \
      output_dir=./ctm_test_manifest
      batch_size=2
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?
Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Add Spanish IPA dictionaries and heteronyms
State: closed | Pull request: yes

# What does this PR do ?

Add Spanish pronunciation dictionaries following the same format and naming convention as the German dictionary.

**Collection**: [TTS]

# Changelog 
- Add dictionary and heteronym files for es_ES (Spain Spanish) and es_LA (Latin American Spanish).

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Disabling radtts tests untin we have real model
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Renaming batch_lengts back to lens, disabling tests for now
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[ASR Training] Loss nan during training
State: closed | Pull request: no
Labels: name: bug

name: stale

**W0215 06:48:41.808522 140192834852672 x2num.py:14] NaN or Inf found in input tensor**


**Pytorch Lighting Parameters**


omit amp_level and amp_backend,only used precision=16 to enable AMP

#amp_level="O1",
precision=16, 
#amp_backend='apex'

though it enables AMP [Using 16bit native Automatic Mixed Precision (AMP)],during training it shows training loss nan with a warning message **W0215 06:48:41.808522 140192834852672 x2num.py:14] NaN or Inf found in input tensor**


**Learning Rate =2**


![image](https://user-images.githubusercontent.com/107035118/218989436-147d27a4-78c5-4d20-a422-ca77d7adb645.png)


**Additional context**
- Nemo Docker version nemo:22.12
- Multi GPU
Added bert micro batch loader
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?
Integrates the microbatch loader for bert
Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]
NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add Customization Dataset Preparation Tool
State: closed | Pull request: yes

Allows users to read data into prompt-and-completion format .jsonl as expected by the Customization service/NeMo LLM P tuning service

Signed-off-by: Zhilin Wang zhilinw@nvidia.com

# What does this PR do ?

Allows users to read data into prompt-and-completion format .jsonl as expected by the Customization service/NeMo LLM P tuning service

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
See tutorial.ipynb

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
adding early stop callback to ptuning
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Adds a early stopping callback to the ptl trainer object for prompt and p-tuning for gpt and t5 models.

**Collection**: [NLP]

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix reinstall.sh dependencies
State: closed | Pull request: yes

# What does this PR do ?

Adds missing dependency for using `build` to build the nemo pip wheel

**Collection**: [Core]

# Changelog 
- Add missing dependency `pytest-runner`


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Add pyctcdecode to high level beam search API
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Adds PyCTCDecode to high level beam search API 

**Collection**: [ASR]

# Changelog 
- Adds pyctcdecode support.
- Approximates the tokenized text output since the engine does not return token indices

# Usage

```python

import nemo.collections.asr as nemo_asr

filepath = "<Path to audio file>"
kenlm_path = "<Path to ARPA or BIN KenLM File built in CHAR / WORD mode; not SUBWORD mode !>"

model = nemo_asr.models.ASRModel.from_pretrained('stt_en_conformer_ctc_large')  # type: nemo_asr.models.EncDecCTCModelBPE
# Create default decoding config
model.change_decoding_strategy(None)
decoding_cfg = model.cfg.decoding
decoding_cfg.strategy = "pyctcdecode"
decoding_cfg.beam.beam_size = 128
decoding_cfg.beam.return_best_hypothesis = False
decoding_cfg.beam.beam_alpha = 1.0
decoding_cfg.beam.beam_beta = 0.0
decoding_cfg.beam.kenlm_path = kenlm_path

# pyctcdecode config modifications
decoding_cfg.beam.pyctcdecode_cfg.beam_prune_logp = -10.
decoding_cfg.beam.pyctcdecode_cfg.token_min_logp = -5.

# Prepare beam search decoding strategy
model.change_decoding_strategy(decoding_cfg)

hypothesis = model.transcribe([filepath], return_hypotheses=True)
print("Num hypothesis per sample :", len(hypothesis[0]))
print("Result type :", hypothesis[0][0].__class__.__name__)
for candidate in range(len(hypothesis[0])):
    print(f"Beam Search {candidate + 1} :", hypothesis[0][candidate].text, hypothesis[0][candidate].score)

```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
correct bash style according to SC2236.
State: closed | Pull request: yes

# What does this PR do ?

replace `! -z` -> `-n`.

https://github.com/koalaman/shellcheck/wiki/SC2236

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add BERT support for overlapping forward compute with distopt communication
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

This PR adds experimental support in BERT for the distributed Adam optimizer to overlap its parameter all-gather with the forward compute.

**Collection**: NLP

# Changelog 
- Add BERT support for overlapping forward compute with distopt communication

# Usage

Set the optimizer to `distributed_fused_adam` in the config file:

https://github.com/NVIDIA/NeMo/blob/a7c2a04db4375ea8669c2c8c33cb4c4c30ffabe9/examples/nlp/language_modeling/conf/megatron_bert_config.yaml#L144

Configure the optimizer with `overlap_param_sync: True`.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

Pinging @shanmugamr1992.

# Additional Information
* Follow-up to https://github.com/NVIDIA/NeMo/pull/5684.
Update TitaNet-L and MSDD models 
State: closed | Pull request: yes
Labels: name: ASR

name: Speaker Tasks

# What does this PR do ?

* Update hydra config on published NGC titanet-model and release as v1 version
* Improve speaker label tests with latest hydra config
* Add a feature to write model labels `labels.txt` to exp directory 
* Update batch inference function to return gt labels and trained model labels (if available) 

**Collection**: ASR

# Changelog 
- Update label models to include latest v1 version of titanet-large
- Added titanet-small config file (model will be published soon)
- Now instead of returning index of gt labels from AudioSpeechLabelDataset, return the speaker label names directly as seen from manifest file

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [x] Documentation
Use a separate communicator for DP AMAX reduction
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?
- Feed a separate communicator to TE for a data-parallel AMAX reduction
- This enables overlapping AMAX reduction and other DP-communications e.g., gradient reduction and parameter gathering

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Test accuracy of 0 with high loss on test set after Finetuning Titanet-large model for Speaker Identification
State: closed | Pull request: no

Hello Team.

I have following three issues:
1. I tried to finetune the titanet-large model using the German dataset with 274 speakers and the total data size is small. THe dataset is around 30h - 40h in total. I tried with few different configurations like - adamw optimizer with 0.0001 learning rate and angularsoftmax loss, adamw optimizer with 0.0001 learning rate and crossentropy loss, novograd optimizer with 0.0001 learning rate and crossentropy loss, sgd optimizer with 0.0001 learning rate and crossentropy loss, adamw optimizer with 0.008 learning rate and crossentropy loss, and so on. 
Issue: I have a small test set with 14 speakers who are common with training set, the total test size is 14minutes. The test accuracy is 0 with high loss (around 30 - 60). Please do provide some inputs.

2. I had earlier tried to finetune the TitaNet-large model with VoxCeleb German subset data. The test accuracy was around 96%. I updated the nemo-toolkit with newer version or from the main branch like two days ago and used the model from earlier to finetune the model with another German dataset with 274 speakers. I am having the following issue:
'''
Error executing job with overrides: [] 
Error in call to target 'nemo.collections.common.losses.cross_entropy.CrossEntropyLoss':                                                                     TypeError("__init__() got an unexpected keyword argument 'scale'")
full_key: loss  
'''
3. After updating the nemo-toolkit to a newer version, I am not able to load the models that were finetuned using the earlier versions of the nemo-toolkit. Receiving the following erroe message. please provide inputs:
'''
RuntimeError: Error(s) in loading state_dict for EncDecSpeakerLabelModel:
Unexpected key(s) in state_dict: "decoder.final.bias".
'''

Regards,
Sreeni...
Ragged batching changes for RadTTS, some refactoring
State: closed | Pull request: yes
Labels: name: TTS

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Tools] Fix ctc segmentation: exclude audacity files
State: closed | Pull request: yes
Labels: name: cherry-pick

Signed-off-by: ekmb <ebakhturina@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Experiment crashes after resuming from checkpoint
State: closed | Pull request: no
Labels: name: bug

name: stale

[multilang_output_6.txt](https://github.com/NVIDIA/NeMo/files/10732583/multilang_output_6.txt)
[multilang_error_6.txt](https://github.com/NVIDIA/NeMo/files/10732586/multilang_error_6.txt)
**Describe the bug**
using 22.07 image from NGC cloud
unable to resume my experiment even after setting ### resume_if_exist=true 
AssertionError: If capturable=False, state_steps should not be CUDA tensors.


i have attache my log file your perusal
@titu1994 @meghmak13
Radtts ragged
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Adding ragged batch input option for RadTTS - with some refactoring
Fixing 2 bugs from previous PR: 
   1. clamp(1,..)
   2. argsort() for unsort_tensor() was in wrong direction  

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add ClearML Logging
State: closed | Pull request: yes

# What does this PR do ?

Adding [ClearML ](https://github.com/allegroai/clearml) integration for logging of training process:

- Enable PyTorch auto logging (cfg parameter)
- TensorBoard auto logging (on default)
- Logging .nemo checkpoint files

**Collection**: [Note which collection this PR will affect]
utils

# Changelog 
- Add new `loggers` module into `nemo.utils`
- Add new `clearml_logger.py` into `nemo.utils.loggers`
- Add new `ClearMLParams` and `ClearMLLogger` classes into `clearml_logger.py`
- Move `dllogger.py` to `nemo.utils.loggers`
- Integrate ClearMLLogger to PL loggers

# Usage
In model config.yml file:
```yml
exp_manager:
  # You may use this section to create a ClearML logger
  create_clearml_logger: true
  clearml_logger_kwargs:
    task: "Test Integration ClearML" # Task name for this experiment 
    project: "Speech/ASR/RUS_OS/Train_NeMo" # Project name for this experiment
    tags: ["Test1", "2"] # Additional tags (default always include "NeMo" tag)
    model_name: "Some name" # Model name in ClearML (default from exp_manager:name)
    log_model: true # If true - logging model .nemo file ClearML
    log_cfg: true # If true - logging model config to model ClearML instance
    log_metrics: true # If true - logging lightning metrics to model ClearML instance
    connect_pytorch: false # If true - enable automatic pytorch logging (all saved checkpoints)
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)