renwei2024/nemo-github-issues huggingface dataset:
Data sampler length caculation error
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

The [`__len__`](https://github.com/NVIDIA/NeMo/blob/dc9dda09ff4fb986cf7653e9384c4ea6ec4202fd/nemo/collections/nlp/data/language_modeling/megatron/data_samplers.py#LL81C8-L81C8) function for BaseMegatronSampler use `self.total_samples - self.consumed_samples` to caculate the sampler length. Is that intentional or just a bug.

The current implementation will make the log confused once after resuming from some checkpoints. The total iters will decrease every time resuming from a newer checkpoint.


**Expected behavior**

The total iters should keep unchanged every time resuming from new checkpoint.

**Environment overview (please complete the following information)**

 - Environment location: Docker
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used
  
    - docker pull: `docker pull nvcr.io/nvidia/nemo:23.02`
    - docker run: `nvidia-docker run -it --shm-size 64g --network host -v /path/on/host:/path/in/docker nvcr.io/nvidia/nemo:23.02 bash`

**Additional context**

Add any other context about the problem here.
Example: GPU model
cardinal argument error
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

When I run default ` python duplex_text_normalization_infer.py \
    tagger_pretrained_model=neural_text_normalization_t5 \
    decoder_pretrained_model=neural_text_normalization_t5 \
    inference.from_file=False \
    lang=en \
    mode=tn` got this error
   
![image](https://github.com/NVIDIA/NeMo/assets/67840077/605922aa-42e5-43aa-86e9-4ab3dfd98a7f)

**Expected behavior**

I want to test how to tn/itn works

**Environment overview (please complete the following information)**

 - Environment location: [local]
 - Method of NeMo install: [pip install ].
 
**Environment details**

Otherwise, please provide:
- OS version - 20.04
- PyTorch version - 2.0.1
- Python version - 3.10.10
ModuleNotFoundError: No module named 'ctc_decoders'
State: closed | Pull request: no
Labels: name: stale

hello
while using eval_beamsearch_ngram.py , i face the following error:
"
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.10/site-packages/nemo/collections/asr/modules/beam_search_decoder.py", line 64, in __init__
    from ctc_decoders import Scorer, ctc_beam_search_decoder_batch
ModuleNotFoundError: No module named 'ctc_decoders'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/azad2/NeMo/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py", line 393, in main
    candidate_wer, candidate_cer = beam_search_eval(
  File "/home/azad2/NeMo/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py", line 185, in beam_search_eval
    _, beams_batch = model.decoding.ctc_decoder_predictions_tensor(
  File "/root/anaconda3/lib/python3.10/site-packages/nemo/collections/asr/metrics/wer.py", line 418, in ctc_decoder_predictions_tensor
    hypotheses_list = self.decoding(
  File "/root/anaconda3/lib/python3.10/site-packages/nemo/collections/asr/parts/submodules/ctc_beam_decoding.py", line 166, in __call__
    return self.forward(*args, **kwargs)
  File "/root/anaconda3/lib/python3.10/site-packages/nemo/core/classes/common.py", line 1087, in __call__
    outputs = wrapped(*args, **kwargs)
  File "/root/anaconda3/lib/python3.10/site-packages/nemo/collections/asr/parts/submodules/ctc_beam_decoding.py", line 280, in forward
    hypotheses = self.search_algorithm(prediction_tensor, out_len)
  File "/root/anaconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/root/anaconda3/lib/python3.10/site-packages/nemo/collections/asr/parts/submodules/ctc_beam_decoding.py", line 329, in default_beam_search
    self.default_beam_scorer = BeamSearchDecoderWithLM(
  File "/root/anaconda3/lib/python3.10/site-packages/nemo/collections/asr/modules/beam_search_decoder.py", line 66, in __init__
    raise ModuleNotFoundError(
ModuleNotFoundError: BeamSearchDecoderWithLM requires the installation of ctc_decoders from scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
"
even though i have run install_bemsearch_decoders.py.
does any one know whats the problem?
is there any other way to install "ctc_decoders"?
[TTS] Added Indic Tokenizers for the Indic-TTS dataset.
State: closed | Pull request: yes
Labels: name: TTS

name: stale

name: common

# What does this PR do ?

Added two syllable tokenizers for Indic Languages.

**Collection**: common

# Changelog 
- Added an IndicSyllableBFTokenizer which performs brute force split of text on vowel signs.
- Added an IndicSyllableTokenizer which splits syllables based on the indic-nlp-library's syllabifier module.

# Usage
- Since providing all possible syllables for all Indic language is nigh impossible, I've added a script which can extract syllables from the manifest. The logic for this resides in `scripts/dataset_preprocessing/tts/indictts` .
- I have also added a config for using the tokenizer with fastpitch under `examples/tts/conf/indictts`

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc) 
  - This requires you to install the indic-nlp-library which is under MIT License and clone the indic-nlp-resources library and save the path under "INDIC_RESOURCES_ROOT". I've added these reqs to the docstring and have made a check to see if this exists in code as well with clear instructions. 
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [+] New Feature
- [ ] Bugfix
- [ ] Documentation
GPT inference long context
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update README.rst about container
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
eval fix
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Allow expanded checkpoints for base model during eval.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Speaker Recognition with clean and phone call audio
State: closed | Pull request: no
Labels: name: stale

Hello everyone,
We are working on a project that identifies if is the same person in different audios, basically speaker recognition. We put 75% of threshold to identify the same person.

We work with clean audios and phone call audios in Portuguese, the correct was have a model for each audio type, or could join these two audio type for a robust model? For example, I do a train only with phone call audios and after that a finetune with clean audios, or the other way around.
We try to do both together, a train with two dataset and a finetune with two dataset, and we had an unsatisfactory result


```
Clean audios, 44959 train, 2351 dev

Predict           Fraud                 Same Person              

Fraud             23060                 13.925                            

Same Person       18                    449                                 


Clean and phone call audios, 11334 train, 1214 dev

Predict           Fraud                Same Person              

Fraud             20921                16.064                            

Same Person       6                    461   


Phone call audios, 6658 train, 975 dev

Predict           Fraud                 Same Person              

Fraud             35271                 1.714                           

Same Person       78                    389                            

```

For parameters we're using 'titanet-large', 32 batch_size for both, 0.011823477534453206 for lr,  0.07095099595312822 for margin, 60 for scale, 'sgd' optimizer and 2 epochs.

Here some audios that we have problem to identify as different persons:
[WrongResults.zip](https://github.com/NVIDIA/NeMo/files/11531858/WrongResults.zip)
All of these audios have a high score to the wrong person, like 85% to 90% of confidence.

With this context, make sense this results? Why I get this high confidence with different persons? Have any way to improve this results? And how many audios are expected to have a good result?
Also we use an issue, https://github.com/NVIDIA/NeMo/issues/5930, as recommendation to train and test.
Add optional index mapping dir in mmap text datasets
State: closed | Pull request: yes
Labels: name: NLP

If datasets are stored on a read-only medium, index files cannot be created into adjacent files and an
alternative directory must be specified for index
mapping files.

# What does this PR do ?

This commit adds an optional flag to specify the directory to write index files to.

**Collection**: nlp

# Changelog 
- Add an optional `index_mapping_dir` to the constructors of `TextMemMapDataset`,  `CSVMemMapDataset` and `JSONLMemMapDataset`.
- Add unit tests.
- Propagate `index_mapping_dir` from `GPTSFTDataset`.
- 
# Usage

```bash
nvidia-docker run --shm-size=4gb  -v ~/llm/data:/llm/data:ro -it sft:latest  python megatron_gpt_sft.py ++model.data.train_ds.index_mapping_dir=/tmp  ++model.data.validation_ds.index_mapping_dir=/tmp
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix a bug, use _ceil_to_nearest instead as _round_to_nearest is not d…
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Fix a bug when training SFT model with dialog dataset when setting `self.pad_to_max_length=False`

**Collection**: [Note which collection this PR will affect]

# Changelog 
```
#gpt_sft_chat_dataset.py

## original code
## max_length = min(self.max_seq_length, self._round_to_nearest(max_length, 8))

## Changed to
max_length = min(self.max_seq_length, self._ceil_to_nearest(max_length, 8))
```
In [gpt_sft_chat_dataset.py](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py#LL270C56-L270C72), we use `_ceil_to_nearest` which is also defined in this file. However, `_round_to_nearest` is not defined in NeMo.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.
Fix a bug, use _ceil_to_nearest instead as _round_to_nearest is not d…
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix a bug when training SFT model with dialog dataset when setting `self.pad_to_max_length=False`

**Collection**: [Note which collection this PR will affect]

# Changelog 
```
#gpt_sft_chat_dataset.py

## original code
## max_length = min(self.max_seq_length, self._round_to_nearest(max_length, 8))

## Changed to
max_length = min(self.max_seq_length, self._ceil_to_nearest(max_length, 8))
```
In [gpt_sft_chat_dataset.py](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py#LL270C56-L270C72), we use `_ceil_to_nearest` which is also defined in this file. However, `_round_to_nearest` is not defined in NeMo.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.
Update all invalid tree references to blobs for NeMo samples
State: closed | Pull request: yes

Update all invalid tree references to blobs for NeMo samples

Currently, most of the TTS tutorials are "broken" because the link to colab is invalid. This fixes those links. I also verified that all the new links work correctly.

**Collection**: NeMo Tutorials

# Changelog 
- Several of the tutorials pointed to an invalid git tree (should be blob). Updated to blob and verified they now work on colab

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests? (none needed)
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc) (none)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [x] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Make Note usage consistent in adapter_mixins.py
State: closed | Pull request: yes
Labels: name: core

Inconsistent usage of the word Note, which includes a broken reading in one case.

I'm just doing some tidying -- not trying to be critical.

**Before**:

![image](https://github.com/NVIDIA/NeMo/assets/7584774/efceb708-a2fe-46e6-aeed-29b43e988fa0)

**After**:
![image](https://github.com/NVIDIA/NeMo/assets/7584774/8a697ba3-ad4d-42db-8616-f5cf5fd407d5)

**Collection**: NeMo Core (documentation)

# Changelog 
- I changed the inconsistent use of the word Note: in documentation to `.. note:: ` as used elsewhere

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests? (no tests needed -- documentation fix)
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc) (no)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix masking bug for TTS Aligner
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Fixes a masking bug where boolean values were inverted from what was expected. Reverted change from PR #5653.

**Collection**: TTS

# Changelog 
- Added check for `== 0` (applied to the input mask) back to the Aligner forward call. Otherwise the mask is applied incorrectly, as the values are flipped from what is expected in the Aligner module.

**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

# Additional Information
* See Issue #6584
merging main to asr_normalize
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: CI

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
update batch size recommendation to min 32 for 43b
State: closed | Pull request: yes

# What does this PR do ?

update batch size recommendation to min 32 for 43b

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Speaker Diarization takes too long
State: closed | Pull request: no
Labels: name: stale

I have finetuned a speaker recognition model with the help of this tutorial : https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb
After that I trained a MSDD Model for Speaker Diarization and replaced the pretrained titanet model with my finetuned Speaker Recognition Model. I followed this tutorial :https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Diarization_Training.ipynb
I tried both, training with a frozen speaker embeddings model and joint training and noticed that the validation loss was much lower when using joint training. Nevertheless, the inference time for Speaker Diarization takes forever when extracting embeddings. How can I tackle that problem?
PE refactor
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[ASR] Adding ssl config for fast-conformer
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?
Adding SSL config for fast-conformer


**Collection**: ASR

# Changelog 
- Added SSL config for fast-conformer (examples/asr/conf/ssl/fast-conformer/fast-conformer.yaml)
- Added boolean flags whether to indicate a particular loss is active or not via config file and associated changes ( nemo/collections/asr/models/ssl_models.py file).


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
minor fix for missing chat attr
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
SDE unt lvl comparison
State: closed | Pull request: yes

# What does this PR do ?
Adds new feature to compare two ASR models on utterance level visually

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]
ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below
```
python3 ./NeMo/tools/speech_data_explorer/data_explorer.py test.json -nc pred_text_citri pred_text_qn
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [+] Did you add or update any necessary documentation?
- [-] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [+] New Feature
- [ ] Bugfix
- [+] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
(RE) Error in Aggregate Tokenizer for multilingual ASR 
State: closed | Pull request: no

** i re-posted in feature request form since there was no reply in bug report

Hello,
I built Multilingual ASR model with Korean and English.
However, I found that tokens are assigned not correctly.

I used 128 tokens for english, and 1641 tokens for korean (Total 1749)
when i checked vocabulary, there were several suspicious points.

When i printed asr_model.tokenizer.langs_by_token_id, it correctly shows tokens. (from 0 to 128 : 'en', from 129 to 1749: 'kor').
However, some alphabet token is assigned in korean. For example, print(asr_model.tokenizer.token_to_id('q', 'kor')) results 1702.

Also, in my knowledge, the aggregated tokenizer must remove duplicated token such as 'unkown'. But, there is two 'unk' tokens in both index 0 and 128

I also saw same phenomonon in pretrained_model, Conformer_transducer_enes_large.

tokens such as 'pre', 'ble' 'is', 'to' etc are assigned in Es, not En. I think these tokens are close to english tokens rather than spanish token.

there were two 'unk' tokens.
print(asr_model.tokenizer.token_to_id('unk', 'en')) results in 0, and
print(asr_model.tokenizer.token_to_id('unk', 'es')) results in 1024.
GPT extrapolatable position embedding (xpos/sandwich/alibi/kerple) and Flash Attention
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?
* Add extrapolatable position embedding, e.g. xpos, sandwich.
* Fix extrapolatable position embedding, e.g. alibi, kerple. 
* Add flash attention (FA)

**Collection**: [NLP]

# Changelog 
* GPT
  - `examples/nlp/language_modeling/conf/megatron_gpt_config.yaml`
  - `nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py`
  - `nemo/collections/nlp/models/language_modeling/megatron/gpt_model.py`
  - `nemo/collections/nlp/modules/common/megatron/language_model.py`
* T5
  - `nemo/collections/nlp/modules/common/megatron/token_level_encoder_decoder.py`
  - `nemo/collections/nlp/modules/common/megatron/megatron_encoders.py`
  - `nemo/collections/nlp/modules/common/megatron/megatron_decoders.py`
  - `nemo/collections/nlp/modules/common/megatron/megatron_transformer_encoder.py`
  - `nemo/collections/nlp/modules/common/megatron/megatron_transformer_decoder.py`
* PE
  - `nemo/collections/nlp/modules/common/megatron/transformer.py`
  - `nemo/collections/nlp/modules/common/megatron/attention.py`
  - `nemo/collections/nlp/modules/common/megatron/kerple_relative_position.py`
  - `nemo/collections/nlp/modules/common/megatron/sandwich_relative_position.py`
  - `nemo/collections/nlp/modules/common/megatron/xpos_relative_position.py`
* FA
  - `nemo/collections/nlp/modules/common/megatron/attention.py`

# Usage
* PE

```python
# GPT
model.position_embedding_type='' # Options ['learned_absolute', 'rope', 'alibi', 'xpos', 'sandwich', 'kerple']
# T5
model.encoder.position_embedding_type='' # Options ['learned_absolute', 'rope', 'alibi', 'xpos', 'sandwich', 'kerple']
model.decoder.position_embedding_type='' # Options ['learned_absolute', 'rope', 'alibi', 'xpos', 'sandwich', 'kerple']
```
* FA

```python
# GPT
model.use_flash_attention=true
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Todo
- [x] Add tests for PE and FA
- [x] Integrate multiple PE classes into one file 
- [x] Move if/else of different attention bias outside CoreAttention
- [ ] Update requirements files
- [x] Update dockerfile
- [ ] Add a warning for FA
[TTS] Add callback for saving audio during FastPitch training
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Add an optional and configurable callback for FastPitch training which allows providing a vocoder checkpoint and saving any combination of model output (spectrogram, alignment, audio, audio with gt duration, prosody features) to (local_disk, tensorboard, wandb).

The callback expects the user to provide a separate manifest with all of the examples to generate the audio for. A user can create this by manually selecting 5-10 rows from their validation manifest.

The callback interface should be generic enough to use it for logging any training recipe, such as vocoder training (which will be added in a future PR).

**Collection**: [TTS]

# Changelog 
- Add generic LoggingCallback interface for logging model predictions
- Add FastPitchArtifactGenerator to log model outputs for FastPitch
- Modify new fastpitch example config to show to configure the logger
- Set FastPitch min_token_duration to 1 to avoid various edge cases during inference where output audio is empty array (all duration predictions are 0). This is also more accurate as the aligner requires the alignment to assign at least 1 audio frame to each input token. I could make this is a config (defaulting to 0) if anyone thinks the change is risky.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Lora/PEFT training script CI test
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix bucketing bug issue for picking new bucket
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fixes issue with bucketing datasets, where loss for new run jumps from a previous run. 
Tested. 

**Collection**: ASR

# Changelog 
- break once bucket finishes, so as to pick next bucket

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

# Additional Information
* Related to #6191
Error in Aggregate Tokenizer for multilingual ASR
State: closed | Pull request: no
Labels: name: bug

Hello, 
I built Multilingual ASR model with Korean and English.
However, I found that tokens are assigned not correctly.

I used 128 tokens for english, and 1641 tokens for korean (Total 1749)
when i checked vocabulary, there were several suspicious points.

When i printed asr_model.tokenizer.langs_by_token_id, it correctly shows tokens. (from 0 to 128 : 'en', from 129 to 1749: 'kor').
However, some alphabet token is assigned in korean. For example, print(asr_model.tokenizer.token_to_id('q', 'kor')) results 1702.

Also, in my knowledge, the aggregated tokenizer must remove duplicated token such as 'unkown'. But, there is two 'unk' tokens in both index 0 and 128

I also saw same phenomonon in pretrained_model, Conformer_transducer_enes_large.

1. tokens such as 'pre', 'ble' 'is', 'to' etc are assigned in Es, not En. I think these tokens are close to english tokens rather than spanish token.

2. there were two 'unk' tokens. 
print(asr_model.tokenizer.token_to_id('unk', 'en')) results in 0, and
print(asr_model.tokenizer.token_to_id('unk', 'es')) results in 1024.
Upgrade to PyTorch 23.04 Container
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
remove upgrading setuptools in reinstall.sh
State: closed | Pull request: yes

# What does this PR do ?

keeping it would make installation log misleading like below. It'd better remove it.
```
(nemo) xueyang@DEV:~/workspace/NeMo:(main)$ ./reinstall.sh
Uninstalling stuff
Found existing installation: nemo-toolkit 1.18.0rc0
Uninstalling nemo-toolkit-1.18.0rc0:
  Successfully uninstalled nemo-toolkit-1.18.0rc0
Found existing installation: sacrebleu 2.3.1
Uninstalling sacrebleu-2.3.1:
  Successfully uninstalled sacrebleu-2.3.1
WARNING: Skipping nemo_asr as it is not installed.
WARNING: Skipping nemo_nlp as it is not installed.
WARNING: Skipping nemo_tts as it is not installed.
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com/
Requirement already satisfied: setuptools in /home/xueyang/miniconda3/envs/nemo/lib/python3.8/site-packages (65.5.1)
Collecting setuptools
  Downloading setuptools-67.7.2-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 50.8 MB/s eta 0:00:00
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 65.5.1
    Uninstalling setuptools-65.5.1:
      Successfully uninstalled setuptools-65.5.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
nemo-text-processing 0.1.7rc0 requires setuptools==65.5.1, but you have setuptools 67.7.2 which is incompatible.
Successfully installed setuptools-67.7.2

[notice] A new release of pip is available: 22.2.2 -> 23.1.2
[notice] To update, run: pip install --upgrade pip
```
**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
handle artifacts when path is an extracted dir
State: closed | Pull request: yes
Labels: name: core

# What does this PR do ?

handle artifacts assumes a compressed `.nemo` this PR checks if model is a `dir` and does not try to untar if it is a `dir`.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Confidence ensembles: fix issues and add tuning functionality
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Adding new functionality for confidence ensembles and fix existing issues. Note that there is a plan with another follow-up PR to fix (partially) existing TODOs in the code + add docs, types and a tutorial on the usage.

**Collection**: ASR

# Changelog 
- In the previous PR I directly used frame confidence returned from NeMo. This is problematic for two reasons. First, it does not filter blanks, even when exclude_blanks is set to True in the config. Second, it can only compute a single confidence, so there is a problem with using it in a tuning mode where we want to get 100s of confidence values at the same time. So to overcome these problems, I wrote `compute_confidence` function, alongside with a few additional utilities that does what we need for confidence ensembles outside of the decoding code.
- Implemented tuning for logistic regression parameters as well as for the confidence parameters. This requires user to specify dev dataset and run transcribe over it.
- Added more tests for tuning of the confidence.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
eval_beamsearch_ngram.py with hybrid ctc
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

- Fix eval_beamsearch_ngram.py with hybrid ctc

- Set default value text_processing.separate_punctuation=false for 
scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py
examples/asr/speech_to_text_eval.py

**Collection**: ASR

# Usage
* You can potentially add a usage example below

```python
python3 /path/to/NeMo/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py \
nemo_model_file=/path/to/hybrid/model.nemo \
input_manifest=/path/to/manifest \
kenlm_model_file=/path/to/kenlm \
decoding_mode=beamsearch_ngram \
beam_width=[32] \
beam_alpha=[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0,1.1] \
beam_beta=[0.0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9] \

```

**PR Type**:
- [V ] Bugfix