renwei2024/nemo-github-issues huggingface dataset:
P-tuning refactor Part 3/N
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

The pre-refactor code saved inference ready `.nemo` files. This PR ensures that the new `.nemo` has a key that mirrors the old inference ready `.nemo` file.

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
remove duplicate definition of manifest read and write func.
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

# What does this PR do ?

remove duplicate definitions of read and write manifest files, and re-use ones defined in `nemo.collections.asr.parts.utils.manifest_utils`

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Adding changes to fix the mv error
State: closed | Pull request: yes
Labels: name: Speaker Tasks

# What does this PR do ?

This PR fixes a line containing bash command "mv".
The line is unnecessary and creating an error.
Also added "rm" command that removes tar file, in case where users run the cell again.


**Collection**: [Note which collection this PR will affect]
ASR

# Changelog 
Removed "mv" line.


# Usage
This is a ipython notebook.
You can run it at `tutorials/speaker_tasks/Speaker_Diarization_Training.ipynb`.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Explicitly check for united embeddings when logging params
State: closed | Pull request: yes
Labels: name: bug

name: NLP

# What does this PR do ?

Fixes a bug with logging params with PP > 1.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix incomplete batches
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Fix error when padding batches.

**Collection**: NLP

# Changelog 
- Fix batches padding in MegatronPretrainingSampler

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Davidm/cherrypick r1.16.0
State: closed | Pull request: yes
Labels: name: core

name: NLP

name: CI

# What does this PR do ?

Adding important main PRs to 1.16 branch
 
**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Make KenLM with PC for AggregateTokenizer and merge it
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

**Collection**: asr

# Changelog 

Make KenLM with punctuation and capitalization (PC) for AggregateTokenizer 
Merge ARPA models with OpenGRM

## For scripts/asr_language_modeling/ngram_lm/train_kenlm.py 

- allow to use multilanguage NeMo model with AggregateTokenizer (nemo_model_file)
- allow to make ngram model from folder with many files (train_paths)
- prun kenlm model (ngram_prune)


```python

python3 /base_path/NeMo/scripts/asr_language_modeling/ngram_lm/train_kenlm.py    \
    nemo_model_file=/path/to/aggregate_model.nemo \
    train_paths=[/path/train_clean_100.json,/path/train_clean_360.json,/path/folder_with_other_files] \
    kenlm_bin_path=/workspace/nemo/decoders/kenlm/build/bin \
    kenlm_model_file=/path/lms/7gram_0001.kenlm \
    preserve_arpa=true \
    ngram_length=7 \
    ngram_prune=[0,0,1]
```

## Use opengrm to merge ngrams
scripts/asr_language_modeling/ngram_lm/opengrm.py

- merge two ngram arpa models

```python
python3 scripts/asr_language_modeling/ngram_lm/ngram_merge.py \
                    --kenlm_bin_path /workspace/nemo/decoders/kenlm/build/bin \
                    --ngram_bin_path /workspace/nemo/decoders/ngram-1.3.14/src/bin \
                    --arpa_a /path/ngram_a.kenlm.tmp.arpa \
                    --alpha 2 \
                    --arpa_b /path/ngram_b.kenlm.tmp.arpa \
                    --beta 1 \
                    --out_path /path/out \
                    --test_file /path/test_manifest.json \
                    --symbols /path/isyms.txt \
                    --nemo_model_file /path/model.nemo
```

##Install scripts for kenlm, decoders and opengrn
- scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh
- scripts/installers/setup_os2s_decoders.py
- scripts/installers/install_opengrm.sh
- scripts/installers/Dockerfile.ngramtools
  
**PR Type**:
- [ V] New Feature
[TTS] Separate TTS tokenization and g2p util to fix circular import
State: closed | Pull request: yes
Labels: name: TTS

name: common

# What does this PR do ?

Split the g2p data_utils into 2 util classes, one containing tokenization functions and the other g2p specific ones.

This fixes a circular import between the the text tokenizers and G2p.

**Collection**: [TTS]

# Changelog 
- Move relevant helper methods to a new tokenizer utility to avoid tts_tokenizers importing tts g2p.
- Update unit tests

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
[CI] Set readthedocs python version to 3.8
State: closed | Pull request: yes

# What does this PR do ?

Offline it was mentioned that API docs were failing due to some syntax available in Python 3.8. This can be seen here: https://readthedocs.com/projects/nvidia-nemo/builds/1305908/

Setting the build python version to 3.8 for readthedocs.

**Collection**: Common

# Changelog 
- Increment readthedocs build python version to 3.8.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation
Mistake in the Cache-aware streaming example script
State: closed | Pull request: no

Hello,

it looks like there's a mistake in the [cache-aware streaming example.](https://github.com/NVIDIA/NeMo/blob/786a8501db2bd3854ed6f629a51fe2792b58bb00/examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py)
[This argument ](https://github.com/NVIDIA/NeMo/blob/0322b324e815c155f90e75dde2eff78d3b9d3dd4/examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py#L238) doesn't have any impact, because by design it applies input features normalization right before return from [this function](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/parts/utils/streaming_utils.py#L1499), but output of this function is never used (because [self.buffer](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/parts/utils/streaming_utils.py#L1311) is the actual place where NN inputs are stored).

I see that code related to this example was moved in December from one place to another, as well as some refactoring was done, so I expect there may be that some changes were done in these 2 files I'm referring to, and docstrings were not updated.

Summarizing all the above, this is my question: 
is there a mistake in the docs [here](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py#L54) and we just don't need to do online normalization, or is it mistake in the [CacheAwareStreamingAudioBuffer](https://github.com/NVIDIA/NeMo/blob/2a6101421f4366ec28bac1165f26fec8eeedea59/nemo/collections/asr/parts/utils/streaming_utils.py#L1298) and normalized features should be stored in the buffer?

Thank you for response!
Add citation
State: closed | Pull request: yes

# What does this PR do ?

Adds CITATION.eff file

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] bugfix Chinese Fastpitch tutorial
State: closed | Pull request: yes
Labels: name: TTS

name: cherry-pick

# What does this PR do ?

fixed broken procedures in the tutorial and minor related code chagnes.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] bugfix for fastpitch German tutorial
State: closed | Pull request: yes
Labels: name: TTS

name: cherry-pick

# What does this PR do ?

fixed tutorial errors and update the instructions.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Spectrogram Enhancer: correct dim for length when loading data
State: closed | Pull request: yes
Labels: name: bug

name: TTS

name: cherry-pick

# What does this PR do ?

Makes DataLoader for Spectrogram enhancer (#5565) use correct dimension to determine length of the spectrogram.

Currently it uses number of mel bands as length, so in practice enhancer trains on 80x80 patches, which is obviously suboptimal and a **bug**.

**Collection**: TTS

# Changelog 
- Fixes just one single dimension index

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #6007 but there's more work to do
Update TitaNet-L and MSDD models 
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

name: Speaker Tasks

# What does this PR do ?

* Update hydra config on published NGC titanet-model and release as v1 version
* Improve speaker label tests with latest hydra config
* Add a feature to write model labels `labels.txt` to exp directory 
* Update batch inference function to return gt labels and trained model labels (if available) 

**Collection**: ASR

# Changelog 
- Update label models to include latest v1 version of titanet-large
- Added titanet-small config file (model will be published soon)
- Now instead of returning index of gt labels from AudioSpeechLabelDataset, return the speaker label names directly as seen from manifest file

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [x] Documentation
Updated data simulator config part in Speaker_Diarization_Training.ipynb
State: closed | Pull request: yes
Labels: name: Speaker Tasks

# What does this PR do ?

This PR fixes the deprecated configuration in speaker diarization training notebook.


**Collection**: [Note which collection this PR will affect]
ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage

Now, create alignment script for libri speech should follow the following commands 
```python
!python NeMo/scripts/speaker_tasks/create_alignment_manifest.py \
  --input_manifest_filepath LibriSpeech/dev_clean.json \
  --base_alignment_path LibriSpeech_Alignments \
  --output_manifest_filepath ./dev-clean-align.json \
  --ctm_output_directory ./ctm_out \
  --libri_dataset_split dev-clean
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Small fixes for flashlight decoder
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fixes two small bugs in the flashlight decoder, one due to superfluous import

**Collection**: ASR

# Changelog 
- Two changes to nemo/collections/asr/modules/flashlight_decoder.py

# Usage
* No changes, just continue to use flashlight decoding as normal

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Convert esperanto into a notebook
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Moves esperanto into a notebook instead of a documentation page.

**Collection**: ASR

# Changelog 
- Move esperanto example to a notebook.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation
Extraneous Import in Flashlight Module with implications for dependencies
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

In the recently added flashlight module, https://github.com/NVIDIA/NeMo/blob/0e70135ccf24d882dcdc549321e747199a232bdf/nemo/collections/asr/modules/flashlight_decoder.py#L102 has a never-used import `from flashlight.lib.sequence.criterion import get_data_ptr_as_bytes.` This import appears to be from a different part of flashlight, not flashlight text, so it introduces a new (huge) dependency.

Edit: The sequence bindings are in https://github.com/flashlight/sequence/.
Provision for Multitask training
State: closed | Pull request: no
Labels: name: stale

**Is your feature request related to a problem? Please describe.**
So while training a conformer model in an unsupervised fashion. I saw usage of two loss, i.e. contrastive loss and mlm loss. 

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
So if i want to train the conformer model for ASR as well as language detection at the same time provided lang_ID/labels. and using to two losses. Multitasking 
**Describe the solution you'd like**
to thave the data like this
{audio_filepath:"English/audio/file1/path", "duration": 2.0, text: "Transcritpion in english ", "lang_id": "English"}
{audio_filepath:"English/audio/file2/path", "duration": 2.0, text: "Transcritpion in english ", "lang_id": "English"}
{audio_filepath:"English/audio/file3/path", "duration": 2.0, text: "Transcritpion in english ", "lang_id": "English"}
{audio_filepath:"Hindi/audio/file/path", "duration": 2.0, text: "Transcritpion in Hindi ", "lang_id": "Hindi"}
{audio_filepath:"Hindi/audio/file/path", "duration": 2.0, text: "Transcritpion in Hindi ", "lang_id": "Hindi"}
{audio_filepath:"Hindi/audio/file/path", "duration": 2.0, text: "Transcritpion in Hindi ", "lang_id": "Hindi"}
A clear and concise description of what you want to happen.
Similarly here also two losses will be taken into account. one can be ctc and other can be cross entropy. can in decoding, if a file is given first 30sec is chunked on which it will first detect the language and then decode the transcription in the aforementioned  language 
@titu1994 @meghmak13
[TTS] Add Chinese bert for polyphone disambiguation
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Add a Chinese bert recipe for polyphone disambiguation.

**Collection**: [Note which collection this PR will affect]

G2P


# Usage

```sh
# prepare manifest
mkdir -p /mnt/samsung-t7/yuekai/tts/cpp_manifest
git clone https://github.com/kakaobrain/g2pM.git
python3 export_zh_cpp_data_to_manifest.py --data_folder g2pM/data/ --output_folder /mnt/samsung-t7/yuekai/tts/cpp_manifest
```

```sh
# model training
python3 heteronym_classification_train_and_evaluate.py \
    --config-name "heteronym_classification_zh.yaml" \
    train_manifest="/mnt/samsung-t7/yuekai/tts/cpp_manifest/train.json" \
    validation_manifest="/mnt/samsung-t7/yuekai/tts/cpp_manifest/dev.json" \
    model.test_ds.dataset.manifest="/mnt/samsung-t7/yuekai/tts/cpp_manifest/test.json" \
    model.wordids="/mnt/samsung-t7/yuekai/tts/cpp_manifest/wordid.tsv" \
    do_training=False \
    do_testing=True
```



# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Change ExpManager's save _best_model and always_save_nemo behavior
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

`save_best_model` and `always_save_nemo` will work with tp/pp > 1 with this PR.

`save_best_model` and `always_save_nemo` will work independently i.e 
- to track best .nemo just set `save_best_model=True` (the other flag can be false)  
- to track last .nemo just set `always_save_nemo=True` (the other flag can be false)
- to distinguish the two .nemo files the suffix is now `.best.nemo` and `.last.nemo`

Training metadata is inserted into the .yaml file in these .nemo files.
**Collection**: [ASR,NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below
Add these flags to your yaml file.
```
exp_manager.checkpoint_callback_params.always_save_nemo=True
exp_manager.checkpoint_callback_params.save_best_model=True
```

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Ngram
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: ASR

# Changelog 
- Add  preprocess() and separate_punctuation() functions for texts with punctuation and capitalization (PC) 
- Run examples/asr/transcribe_speech.py with beamsearch decoder (beam_strategy=beam)
- Save N-best hypothesis into the "beams" field of manifest file
Flashlight decoder is not supported yet

# Usage
* speech_to_text_eval.py

```python
python3 ./examples/asr/speech_to_text_eval.py   \
model_path=/some/path/ctc_conformer_bpe.nemo   \
dataset_manifest=some/path/test_pc.json  \
output_filename=some/path/test_pc_ctc1.json   \
ctc_decoding.strategy=beam \
ctc_decoding.beam.kenlm_path=/some/path/a1.kenlm \
ctc_decoding.beam.return_best_hypothesis=false \
do_lowercase=true \
rm_punctuation=true
```

* eval_beamsearch_ngram.py

```python
python3 ./scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py  \
nemo_model_file=/some/path/ctc_conformer_bpe.nemo   \
input_manifest=some/path/test_pc.json  \
kenlm_model_file=/some/path/a1.kenlm \
decoding_mode=beamsearch_ngram \
decoding_strategy=beam \
do_lowercase=true \
rm_punctuation=true 
```

**PR Type**:
-  New Feature
Avoid multiple warnings for loss mask
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Currently loss masking for GPT (used when `model.data.pad_samples_to_global_batch_size=True`) issues a warning for every masked sample. In typical workflows there might be hundreds of masked samples in 1 batch, which spawns hundreds of warnings every validation. This PR fixes that

**Collection**: NLP

# Changelog 
- Turn loss mask warning into a debug log

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Propagate LayerNorm1P to TE
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?
Enables using LayerNorm1P implementation with TransformerEngine.
So far, only non-TE path allowed using LayerNorm1P.

NOTE: this PR requires TE version >=0.6!

**Collection**: NLP

# Changelog 
- enable LayerNorm1P for TransformerEngine

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Spectrogram Enhancer: support arbitrary input length
State: closed | Pull request: yes
Labels: name: bug

name: TTS

# What does this PR do ?

Make the spectrogram enhancer work on arbitrary length spectrograms irrespective of lengths during training. This a convolutional model (=locally-connected network), really no reason why it should not support arbitrary lengths

Original StyleGAN 2 architecture starts from a fixed-size noise-initialized tensor and generate images. To support arbitrary length spectrograms, the architecture was modified to start from a rather long tensor and chop off part of necessary length early on. This was not a good decision.

This PR makes the move from a long starting tensor to a tensor with a fixed length of 1.

**Collection**: TTS

# Changelog 
- Change length of the starting ("initial") tensor in the enhancer to 1
- Remove `max_spectrogram_length` from the model and the config

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Fixes #6007
Fix enhancer usage
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

# What does this PR do ?

Fix enhancer usage in ASR-TTS models

**Collection**: [ASR, TTS]

# Changelog 
- suppress type checking for enhancer usage

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fast glu activations
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Adds fast versions of geglu, swiglu and reglu. By having a single large matmul and chunking the outputs.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
ST Sampled Softmax
State: closed | Pull request: yes
Labels: name: stale

name: ASR

name: NLP

name: common

# What does this PR do ?

Adds Sampled softmax to Speech Translation preliminary support

**Collection**: [ASR, ST]

# Changelog 
- Update config for adding ctc and transformer sampled softmax
- Make text augmenttion optional
- Update code in transformer model to support ctc and transformer sampled softmax
- Add `SampledConvASRDecoder` and `SampledMultiLayerPerceptron` for CTC and transformer sampled softmax support
- Add `SampledTokenClassifier` to wrap `SampledMultiLayerPerceptron` to support both normal MLP and sampled MLP.
- Auto style fixes

# Usage

```shell

python speech_to_text_transf.py \
    --config-path="<Path to config>" \
    --config-name="speech_translation_with_tts.yaml" \
    ...
    model.ctc_coef=0.3 \
    model.sampled_softmax=True \
    model.n_samples=256 \  # <any value > 1>

```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
P-tuning refactor Part 2/N
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?
Follow up of #6054

Refactor of GPT Prompt Tuning to use the same code structure as P-Tuning.

**Collection**: [NLP]

# Changelog 
- Adds an option to train a PromptEncoder with just the embeddings which is functionally same as prompt-tuning.
- Removes references to prompt_tables in gpt prompt learning model.
- streamlines the training and inference time code

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] bugfix Chinese Fastpitch tutorial
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

fixed broken procedures in the tutorial and minor related code chagnes.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
P-tuning refactor Part 1/N
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

The `PromptEncoder` class has variations like `PromptEncoderMLP` and `BIGLSTMPromptEncoder`. The PR removes the `BIGLSTMPromptEncoder` (it was not used has has straight up compile errors) and moves `PromptEncoderMLP` inside the `PromptEncoder` class as a member var.

Essentially the PR creates a single `PromptEncoder` which houses all the required variations.

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix typo in asr evaluator readme
State: closed | Pull request: yes

# What does this PR do ?

fix typo in asr evaluator readme

**Collection**: ASR

# Changelog 
- fix typo in asr evaluator readme


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)