renwei2024/nemo-github-issues huggingface dataset:
Added rouge monitoring support for T5
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] relax hardcoded prefix for phonemes and tones and infer phoneme set through dict
State: closed | Pull request: yes
Labels: name: TTS

name: common

# What does this PR do ?

1. previously, '#' was hardcoded as a prefix for Chinese phonemes and tones in both g2p and tokenizer. It is apt to raise errors if mismatched prefix symbols are applied for g2p and tokenizer.

2. previously, phoneme set and tone set are hardcoded as well. It would raise errors when the input phoneme dict contains different set of phonemes. Fixed this by inferring phoneme set based on input dictionary.

3. run pytest nightly and old NGC model checkpoint passed.
```
$ pytest --nightly tests/collections/tts/models/test_fastpitch.py::test_inference[tts_zh_fastpitch_sfspeech]

tests/collections/tts/models/test_fastpitch.py::test_inference[tts_zh_fastpitch_sfspeech] PASSED
```


**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
`NeMoModelCheckpoint.on_train_end()` saves the epoch info incorrectly
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

`NeMoModelCheckpoint.on_train_end()` saves the epoch info incorrectly, which would break resuming training from the last checkpoint.

For example, when the maximum time limit is reached on a cluster, we would usually save checkpoints and launch another job to resume training. Assume that the break happens in epoch 0, ideally, the epoch info of the fit loop should be like the following:
```python
In [3]: ckpt['loops']['fit_loop']['epoch_progress']
Out[3]:
{'total': {'ready': 1, 'completed': 0, 'started': 1, 'processed': 0},
 'current': {'ready': 1, 'completed': 0, 'started': 1, 'processed': 0}}
```
Then in the second job, the PTL trainer knows this is a restarting job. 

However, `NeMoModelCheckpoint` save the `-last.ckpt` in `on_train_end()`, where the `completed` and `processed` counters have been increased. So the epoch info is

```python
In [4]: ckpt_last['loops']['fit_loop']['epoch_progress']
Out[4]:
{'total': {'ready': 1, 'completed': 1, 'started': 1, 'processed': 1},
 'current': {'ready': 1, 'completed': 1, 'started': 1, 'processed': 1}}
```

This bug would break resuming training from the last checkpoint in several cases:
1. When `trainer.max_epochs = 1`, PTL thinks the max epoch has been reached and then stops training immediately.
2. When `trainer.max_epochs > 1`, PTL can restart training but starts from epoch 1 instead of 0, and calculate the total number of batches incorrectly. The logs look like:
```
# at the end of the first job.
Epoch 0:  39%|███▉      | 411/1055 [3:43:23<5:50:01, 32.61s/it, loss=1.25, v_num=jxb0, reduced_train_loss=1.180, global_step=390.0, consumed_samples=49920.0, val_loss=1.140]
wandb: Waiting for W&B process to finish... (success).

# at the beginning of the second job.
Training: 391it [00:00, ?it/s]
Training:   0%|          | 0/645 [00:00<00:00, -18221920.71it/s]
Epoch 1:   0%|          | 0/645 [00:00<?, ?it/s]                
Epoch 1:  64%|██████▍   | 412/645 [00:36<00:20, 11.43it/s]
Epoch 1:  64%|██████▍   | 412/645 [00:36<00:20, 11.42it/s, loss=1.13, v_num=25nj, reduced_train_loss=1.130, global_step=391.0, consumed_samples=49920.0]
```
As we can see, it restarts training from Epoch 1, and the total number of batches decreases from 1055 to 645. Ideally, it should restart at Epoch 0, step 412/1055.

**Steps/Code to reproduce bug**

We run SFT using `examples/nlp/language_modeling/tuning/conf/megatron_gpt_sft.yaml`.
[TTS] fix inconsistent type hints for IpaG2p
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

The type hint was mismatched with https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/g2p/models/i18n_ipa.py#L159.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix Codeql config
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

**Collection**: [Docs]

# Changelog 
- Fix codeql config file


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Fix fastpitch test nightly
State: closed | Pull request: yes

# What does this PR do ?

Fix multi-speaker FastPitch test nightly

**Collection**: [TTS]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Gpt alibi fa infer yy
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

* added long prompt in gpt inference config
* temporarily commented model.generate() in gpt eval
* allow to configure max seq length for gpt eval
* allow to set use_flash_attention for gpt eval


**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Gpt alibi fa infer y
State: closed | Pull request: yes
Labels: name: TTS

name: NLP

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
preprocess squad in sft format
State: closed | Pull request: yes

# What does this PR do ?

adds feature to preprocess squad in sft format which is going to be used for tutorials on lora in the near future.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix preprocessing for `--text_file` option.
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

- Currently, the `--text_file` option always looks for .json files, so I've fixed it to look for .txt files.

**Collection**: [NLP]

# Changelog 
- If you use the `--text_file` option, it will look for .txt files.
- Fix bug opening invalid file instead of `json_file`.

# Usage
You can run the preprocessing task as follows:

```python
python NeMo/scripts/nlp_language_modeling/preprocess_data_for_megatron.py \
--input=/datas/raw/ebooks \
--text_file \
--preproc-folder \
--tokenizer-library=huggingface \
--tokenizer-type=/gpt2_bpe_tokenizer \
--merge-file=/gpt2_bpe_tokenizer/merges.txt \
--vocab-file=/gpt2_bpe_tokenizer/vocab.json \
--dataset-impl mmap \
--append-eod \
--output-prefix=hfbpe_gpt_training_data \
--workers=64
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Not finding correct phoneme dictionary file for tts_es_fastpitch_multispeaker
State: closed | Pull request: no
Labels: name: TTS

name: stale

Hello, I want to finetune the tts_es_fastpitch_multispeaker model but I don't know where to find the correct file for the phoneme dictionary. This is the code I’m running for finetuning:

`!(python fastpitch_finetune.py --config-name=fastpitch_align_44100_ipa_multi.yaml \
  train_dataset=./transcript_train.json \
  validation_datasets=./transcript_dev.json \
  sup_data_path=./fastpitch_sup_data \
  pitch_stats_path=tts_dataset_files/pitch_stats.json \
  phoneme_dict_path=tts_dataset_files/es_LA_nv230301.dict \
  exp_manager.exp_dir=./ljspeech_to_9017_no_mixing_5_mins \
  +init_from_nemo_model=./tts_es_fastpitch_multispeaker.nemo \
  trainer.check_val_every_n_epoch=25 \
  model.train_ds.dataloader_params.batch_size=12 model.validation_ds.dataloader_params.batch_size=12 \
  model.n_speakers=174 \
  model.pitch_fmin=30 model.pitch_fmax=512 model.optim.lr=2e-4 \
  ~model.optim.sched model.optim.name=adam trainer.devices=1 trainer.strategy=null \
)`

I tried with the two phoneme files avaliable in this repository:
`
NeMo/scripts/tts_dataset_files/es_ES/es_ES_nv230301.dict
`
`NeMo/scripts/tts_dataset_files/es_LA/es_LA_nv230301.dict`

And with both I have this error:
`size mismatch for fastpitch.encoder.word_emb.weight: copying a param with shape torch.Size([101, 384]) from checkpoint, the shape in current model is torch.Size([103, 384]).`

I was able to by pass this by simple removing some lines from one of the files but, if possible, I would like to use the same dictionary used in the original training to avoid unknown char/phoneme errors. Thanks.
[TTS] fix test fastpitch nightly
State: closed | Pull request: yes

# What does this PR do ?

Fix multi-speaker FastPitch testing

**Collection**: [TTS]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Tutorial fixes
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Notebook link fixes

**Collection**: [Docs]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Tutorial fixes
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

name: NLP

# What does this PR do ?

Notebook link fixes

**Collection**: [Docs]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
[TTS] Filter out silent audio files during preprocessing
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

First version of preprocessing ignored audio files where start/end of speech could not be detected. During testing I find that all such examples are silent audio files with no speech. And that most public datasets have several such files. This updates the script to remove such files from the final training manifest.

**Collection**: [TTS]

# Changelog 
- Update preprocessing script to remove silent/empty audio files from final training manifest.
- Add `overwrite_audio` so that is script fails partway, you can resume where you left off instead of having to reprocess the entire dataset.
- Fix parsing of `overwrite` arguments, as trying to parse them as a bool results in `True` regardless of what value you provide in the CLI.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
TextMemMapDataset assumes shared storage between nodes
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

There is an assumption in [TextMemMapDataset](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/data/language_modeling/text_memmap_dataset.py) that all nodes share the same storage. When this is not the case, the class fails to find index files on ancillary nodes (those on which we the node rank is not zero).

**Steps/Code to reproduce bug**

* Copy a text file F into the local filesystems of two nodes A and B.
* In a multi-node setup comprising nodes A and B, instantiate a `TextMemMapDataset` (or any subclass such as `JSONLMemMapDataset`) with `dataset_paths=[F]`).
* Assuming node B is the one `node_rank>0`, all node B processes fail with an exception is raised with a message like  `ValueError: Memory Map for F is not found, missing one or more of files: F.idx.{.npy,.info}`.

**Expected behavior**

No exception should occur.

**Environment overview (please complete the following information)**

A cluster that supports multi-node jobs and isolated (not shared) filesystems on each node.

**Environment details**

NeMo 23.02.

**Additional context**

This issue appears to stem from these lines in `TextMemMapDataset.__init__()`:

```
        if not is_distributed or (is_distributed and torch.distributed.get_rank() == 0):
            build_index_files(dataset_paths, newline_int, workers=self._worker, build_index_fn=build_index_fn)
```

The test should use the local rank. However if the input storage is indeed shared between nodes, `local_rank=0` processes might collide with one another, as they would write into the same files. In order to avoid this, one suggestion would be to factor in the node rank into the index file path, which cause separate index files to be created on each node. If this is deemed unacceptable on shared filesystems where this is not necessary, other options would include:
* adding a flag to turn on/off that behavior
* adding an extra barrier to automatically detect whether the index files have been created on first pass:

```
        if not is_distributed or (is_distributed and torch.distributed.get_rank() == 0):
            # Creates index files only on global rank 0.
            build_index_files(dataset_paths, newline_int, workers=self._worker, build_index_fn=build_index_fn)

        if is_distributed:
            # Wait for global rank 0 to create index files.
            torch.distributed.barrier()

        if is_distributed and local_rank == 0:
            # Shared file system case: this is a no-op since the index files have already been created.
            # Non-shared file system case: will only create the index files on node_rank>0 and local_rank==0.
            build_index_files(dataset_paths, newline_int, workers=self._worker, build_index_fn=build_index_fn)

        if is_distributed:
            # Wait for all local ranks 0 to create index files.
            torch.distributed.barrier()
```
support torch.jit for maj_vote_spk_count=True
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

support torch.jit.save for offline clustering maj_vote_spk_count=True

**Collection**: [Note which collection this PR will affect]

# Changelog 

- for torch.jit, remove redundant torch.tensor on `est_num_of_spk_list` in `offline_clustering.py`

# Usage

```bash
pytest tests/collections/asr/test_diar_utils.py::TestSpeakerClustering::test_offline_speaker_clustering
```


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

main reviewer: @tango4j 

This is a part of ASR, diarizer.
@titu1994 @redoctopus, @jbalam-nv, or @okuchaiev



# Additional Information
* Related to # (issue)
adding p-tuning Triton-FasterTransformer inference tutorial
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

adding p-tuning Triton-FasterTransformer inference tutorial

**Collection**: LLM

# Changelog 
- Add a new p-tuning deployment notebook

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Can save the base Diarizer model.
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I get the following error

```
AttributeError                            Traceback (most recent call last)

[<ipython-input-55-5354c1522db5>](https://localhost:8080/#) in <cell line: 1>()
----> 1 msdd_model.save_to(save_path="xyz.nemo")

1 frames

[/usr/local/lib/python3.10/dist-packages/nemo/collections/asr/models/msdd_models.py](https://localhost:8080/#) in save_to(self, save_path)
   1036             self.clus_diar._speaker_model.save_to(spkr_model)
   1037             self.msdd_model.save_to(neural_diar_model)
-> 1038             self.clus_diar.__make_nemo_file_from_folder(filename=save_path, source_dir=tmpdir)
   1039 
   1040     def extract_standalone_speaker_model(self, prefix: str = 'msdd._speaker_model.') -> Dict:

AttributeError: 'ClusteringDiarizer' object has no attribute '_NeuralDiarizer__make_nemo_file_from_folder'
```

**Steps/Code to reproduce bug**

This is literally all the code I am trying to run on google colab.
```
msdd_model = NeuralDiarizer(cfg=create_config())
msdd_model.save_to(save_path="xyz.nemo")
```
Everything else works the model works etc

**Expected behavior**

I expected the model to be saved.

**Environment overview (please complete the following information)**

 - Environment location: Collab
 - Method of NeMo install: !pip install nemo_toolkit[asr]

**Environment details**

- PyTorch version 1.13.1+cu116
Multi-lookahead cache-aware streaming models
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Adds support of multi-look ahead to cache-aware conformer models for streaming ASR. A model trained with multi-lookahead can support multiple context sizes during inference.


# Changelog 
- Added support of multi-lookahead for cache-aware Conformer model for streaming ASR

# Usage
User may specify a list of contexts in att_context_size in Conformer's config like:

`model.encoder.att_context_size=[[70,13],[70,6],[70,1],[70,0]]
`

During validation and inference, the default context would be the first one in the first. In order to change the current context for inference or validation, user may use the method set_default_att_context_size like:

```python
asr_model.encoder.set_default_att_context_size([70,1])
```
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Limit codeql scope
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Limits the scope of codeql to avoid memory errors

**Collection**: [Core]

# Changelog 
- Puts path limits on files that are scanned for codeql

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Fix k2 installation in Docker with CUDA 12
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Fix k2 installation in Docker with CUDA 12. 
Currently k2 is not installed in Docker (fails silently).
Relies on a fix in https://github.com/k2-fsa/k2/pull/1200

**Collection**: [ASR]

# Changelog 
- fix k2 installation source in `scripts/speech_recognition/k2/setup.sh`

# Usage

```python
import k2
# do anything you want with k2
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
check for first or last stage
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix bug when `inference.compute_logprob=True`. Only first and last pipeline stages have a response.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix k2 installation in Docker with CUDA 12
State: closed | Pull request: yes

# What does this PR do ?

Fix k2 installation in Docker with CUDA 12. 
Currently k2 is not installed in Docker (fails silently).
Relies on a fix in https://github.com/k2-fsa/k2/pull/1200

**Collection**: [ASR]

# Changelog 
- fix k2 installation source in `scripts/speech_recognition/k2/setup.sh`

# Usage

```python
import k2
# do anything you want with k2
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Wrong Confidence for Speaker Identification
State: closed | Pull request: no
Labels: name: stale

We made a finetune for the speaker identification for the Portuguese language, the best result we got was using these parameters:
Epochs 2
Batch size 32
Optimizer 'SGD'
Lr 0.011823477534453206
Scale 60
Margin 0.07095099595312822

But when testing with our test dataset, we found that some audios that are clearly different are giving a very high confidence rate. Here are some examples:
85% confidence rate
[85.zip](https://github.com/NVIDIA/NeMo/files/11545713/85.zip)

86% confidence rate
[86.zip](https://github.com/NVIDIA/NeMo/files/11545714/86.zip)

90% confidence rate
[90.zip](https://github.com/NVIDIA/NeMo/files/11545708/90.zip)

92% confidence rate
[92.zip](https://github.com/NVIDIA/NeMo/files/11545709/92.zip)

94% confidence rate
[94.zip](https://github.com/NVIDIA/NeMo/files/11545710/94.zip)

In another scenario, we validate whether they are in fact the same people. And in some cases it affected a low confidence rate, here are some examples:
59% confidence rate 
[59.zip](https://github.com/NVIDIA/NeMo/files/11545711/59.zip)

64% confidence rate 
[64.zip](https://github.com/NVIDIA/NeMo/files/11545712/64.zip)


Below is our model used for both scenarios:
["model"](https://drive.google.com/file/d/1CHyLBXmd_NoC3mg8Q8d_0gIsZdRWw8nd/view?usp=share_link)

Do these problems above make sense? Are there any parameter tweaks, or tweaks to the audios that can be done to fix this issue? 
Thanks in advance
karpnv/issues6690
State: closed | Pull request: yes

# What does this PR do ?

install ctc_decoders inside venv

**Collection**: ASR

# Changelog 
- Fix install_beamsearch_decoders.sh to work outside container 

# Usage
* Install NeMo ctc-decoders locally inside Conda venv

```bash
# make NeMo venv
conda create --name nemo python==3.8.10
conda activate nemo

# install NeMo
sudo apt-get update && apt-get install -y libsndfile1 ffmpeg
git clone https://github.com/NVIDIA/NeMo
cd NeMo
./reinstall.sh

# install ctc-decoders
./scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh $(pwd)
```
  
**PR Type**:
- [ X] Bugfix


## Who can review?
@titu1994 

# Additional Information
* Related to #6690
Fix validation with drop_last=False
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix validation with validation_drop_last=False flag.

**Collection**: NLP

# Changelog 
- fix validation with validation_drop_last=False flag.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix destructor for delayed mmap dataset case
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

When delaying dataset mmap, dataset destructor errored out if the mmap creation was not actually called, leading to misleading tracebacks.

**Collection**: NLP

# Changelog 
- Fix MMapIndexedDataset destructor for delayed case.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] corrected misleading deprecation warnings.
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

1. correct the misleading deprecation warnings;
2. organize the duplicate warning and replacing code snippet into a single function.
3. passed `pytest --nightly tests/collections/tts/models`.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] there are many unresolved reference needed to be fixed.
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

Please correct each item as shown below for `radtts`,
![Screenshot 2023-05-22 at 2 54 10 PM](https://github.com/NVIDIA/NeMo/assets/1646669/cb6bd1f2-23a3-433c-9e33-ecc522ec0453)
VP Fixes for converter + Config management
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix an issue with VP parameter loading. 
NOTE: this is not a complete fix for VP, bugs are still being fixed in later PRs.

**Collection**: [NLP]

# Changelog 
- force config to be correctly updated for megatron amp 02
- update VP logic when loading weights

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
[TTS] Fix adapter duration issue
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Fix TTS adapter duration issue.

**Collection**: [TTS]

# Changelog 
- `examples/tts/conf/fastpitch_align_44100_adapter.yaml`
- `examples/tts/fastpitch_finetune_adapters.py`
- `nemo/collections/tts/losses/aligner_loss.py`
- `nemo/collections/tts/models/fastpitch.py`
- `tutorials/tts/FastPitch_Adapter_Finetuning.ipynb`

# Usage
* You can potentially add a usage example below

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix Sentencepiece model not propagated for GPT init.
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

When using SentencePiece with custom model, `tokenizer.model` was not propagated to GPT Model initialization, hence resulted in error.

**Collection**: NLP

# Changelog 
- `nemo/collections/nlp/modules/common/tokenizer_utils.py`

# Usage
* When using a custom SentencePiece tokenizer model.

```python
model.tokenizer.model=...
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
NFA updates
State: closed | Pull request: yes

# What does this PR do ?

NFA updates: improve API, reduce memory consumption

**Collection**: [Note which collection this PR will affect]
Tools

# Changelog 
Numerous adjustments, including:
- Reduced memory requirements of viterbi decoding
- Removed need to specify model downsample factor
- Now ASS format files can also be created

# Usage
No more need to specify model downsample factor, so can use as follows:

```python
python <path_to_NeMo>/tools/nemo_forced_aligner/align.py \
        pretrained_name="stt_en_citrinet_1024_gamma_0_25" \
        manifest_filepath=<path to manifest of utterances you want to align> \
        output_dir=<path to where your ctm files will be saved> 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
WIP: NFA updates
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: Speaker Tasks

name: CI

name: common

# What does this PR do ?
Various NFA updates: improving APIs, reducing memory consumption, etc.

**Collection**: [Note which collection this PR will affect]
Tools

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)