renwei2024/nemo-github-issues huggingface dataset:
Tts fixed vocab
State: closed | Pull request: yes
Labels: name: TTS

name: common

# What does this PR do ?

Allows users to pass in a grapheme + phoneme list/set as an enforced symbol vocabulary when using the IPA classes. Other symbols, such as punctuation, OOV, etc. are still handled by other tokenizer arguments.

Once set, G2P dictionary entries will be filtered in the following way:
- Words with any illegal graphemes are removed entirely.
- Words with one unique pronunciation and any illegal phonemes are removed entirely.
- If a word has multiple pronunciations and all of them have at least one illegal phoneme, the word is removed entirely.
- If at least one pronunciation is valid, it is kept if `keep_alternate=True` (default), otherwise the word is removed entirely.

The `TTSDataset` will also filter out all manifest entries whose normalized forms have any illegal graphemes.

Note: Users should take care that the passed-in vocab is consistent with `grapheme_prefix` and `grapheme_case`, or else filtering may not work properly.

**Collection**: TTS

# Changelog 
- Added a `replace_symbols()` function to `IPAG2P` to enforce user-set vocab and filter existing G2P dict
- Modified `IPATokenizer` to support taking in a user-set vocab
- Modified `TTSDataset` to check for illegal graphemes if a fixed vocab is used
- Added unit tests for `replace_symbols()`
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Update container version in main readme
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update dataset preparation tool
State: closed | Pull request: yes

# What does this PR do ?

Refactor hyperparameter recommendation and add defaults for epochs, num virtual tokens and lr

**Collection**: nlp

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
metric update
State: closed | Pull request: yes

# What does this PR do ?

adds answer-field to metric.

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Graph RNNT: Grid- and Compose-Transducer. W-Transducer loss
State: closed | Pull request: yes
Labels: name: core

name: ASR

# What does this PR do?

Implement Grid- and Compose-Transducer along with W-Transducer loss according to the paper "Powerful and Extensible WFST Framework for Rnn-Transducer Losses" https://ieeexplore.ieee.org/document/10096679 (https://arxiv.org/abs/2303.10384)

**Collection**: [ASR]

# Changelog 
- `GraphTransducerLossBase` abstract class  with the interface for Graph-based loses
- RNN-T implementation in `GraphRnntLoss` with tests
- W-Transducer implementation in `GraphWTransducerLoss` with tests
- add `GraphRnntLoss` + `GraphWTransducerLoss` to RNN-T loss resolver

# Usage

```shell
# training some model with Graph-RNNT loss
python examples/asr/asr_transducer/speech_to_text_rnnt_bpe.py \
    <other params> \
    model.loss.loss_name=graph_rnnt \
    ++model.loss.graph_rnnt_kwargs.double_scores=true

# training some model with W-Transducer loss
python examples/asr/asr_transducer/speech_to_text_rnnt_bpe.py \
    <other params> \
    model.loss.loss_name=graph_w_transducer \
    ++model.loss.graph_w_transducer_kwargs.eps_weight=0

```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Added beam search to MegatronLMEncoderDecoderModel.decode
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

1. Added support in a custom token sampling function to `MegatronLMEncoderDecoderModel.decode`.
2. Added beam search sampling methods.

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to [merged PR](https://github.com/NVIDIA/NeMo/pull/6095)
[TTS] ClearML logging for validation samples
State: closed | Pull request: yes
Labels: name: TTS

name: stale

# What does this PR do ?

Add ClearML logging of samples produced in validation step

**Collection**: [TTS]

# Changelog 
- Add `tensor_to_wav` and `mel_to_audio` to `nemo.collections.tts.parts.utils.helpers`
- Add ClearML logging of samples in:
    - `nemo.collections.tts.models.univnet`
    - `nemo.collections.tts.models.hifigan`
    - `nemo.collections.tts.models.vits`
    - `nemo.collections.tts.models.fastpitch_ssl`
    - `nemo.collections.tts.models.fastpitch`
    - `nemo.collections.tts.models.mixer_tts`
    - `nemo.collections.tts.models.aligner`

# Usage
Auto-usage than ClearML logging enabled in cfg.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [x] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update docs for ExpManager and Exportable frameworks
State: closed | Pull request: yes
Labels: name: core

# What does this PR do ?

Adds documentation for Exportable and ExpManager.

**Collection**: [Core, Docs]

# Changelog 
- Fix exp manager subconfig docs
- Add exp manager loggers
- Add exp manager EMA docs
- Update Exportable docs and function docs

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation
Avoid unnecessarily accessing data loader with pipeline parallelism
State: closed | Pull request: yes
Labels: name: bug

name: NLP

# What does this PR do ?

I've been experiencing a hang in the first validation step when running GPT with interleaved pipeline parallelism.  This first shows up in https://github.com/NVIDIA/NeMo/pull/6049, so it seems that accessing the data loader in middle pipeline stages is the root cause.

**Collection**: NLP

# Changelog 
- Avoid unnecessarily accessing data loader with pipeline parallelism

# Usage
No change in usage

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to https://github.com/NVIDIA/NeMo/pull/6049
Upgrade setuptools
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Upgrade base container to PyTorch 23.02
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Adding basic preemption code
State: closed | Pull request: yes

Add preemption functionality in preemption_callback.py under utils 
Refactor the code to move NemoModelCheckpoint callback under callbacks

# What does this PR do ?

Adding basic functionality for cluster preemption.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Merge r1.16.0 release main
State: closed | Pull request: yes
Labels: name: TTS

name: NLP

# What does this PR do ?

Merges bugfixes and doc updates from the release branch into main.

(these should be done automatically by the github action but this PR will make sure the release branch can be rebased onto main)

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Add univnet scheduler
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Add scheduler for univnet model.

**Collection**: [TTS]

# Changelog 
- Add `get_warmup_steps` into `UnivNetModel`
- Change `configure_optimizers` in `UnivNetModel` for scheduler support

# Usage
Standart usage of scheduler

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Various fixes in docs and RNNT
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fixes for tests and docs

**Collection**: [Docs, ASR]

# Changelog 
- Fix dark mode in NeMo docs
- Fix issue with flaky RNNT test where the seq length can be off by one due to minute perturbation during beam search


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
[TTS] Make mel spectrogram norm configurable
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add optional configuration to AudioToMelSpectrogramPreprocessor to allow disabling area normalization in the librosa mel filter.

Applying area normalization to the mel spectrogram reduces its range of values from approximately [0, 100] to [0, 1], which is detrimental when you take the log of the [0, 1] values producing a poor distribution of negative features.

The intention is to experiment with disabling the area normalization and replacing the current mel spectrogram used in TTS which is:

> log(max(1E-5, area_norm(mel))

With a more accurate representation like:

> log(1.0 + mel)

This will also enable us to compute energy features using the mel spectrogram instead of the linear spectrogram.

**Collection**: [TTS], [ASR]

# Changelog 
- Add an optional parameter to AudioToMelSpectrogramPreprocessor to configure the mel filter normalization, defaulting to the librosa default of 'slaney'

# Usage
.yaml config:

> preprocessor:
>     _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
>     log: true
>     log_zero_guard_type: add
>     log_zero_guard_value: 1.0
>     mag_power: 1.0
>     mel_norm: null

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Sentence piece legacy false compatibility
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Makes prompt learning  compatible when sp tokenizer has legacy = False.

**Collection**: [NLP]

# Changelog 

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Bug] RoPE positional embedding doesn't work when sequence_parallel is enabled for GPT models
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When enabling `rope` position_embedding_type and  `sequence_parallel` in NeMo GPT pretraining, the training will give runtime error: 

```
  File "/workspace/NeMo/nemo/collections/nlp/modules/common/megatron/rotary_pos_embedding.py", line 59, in apply_rotary_pos_emb
    t = (t * freqs.cos()) + (_rotate_half(t) * freqs.sin())
RuntimeError: The size of tensor a (2048) must match the size of tensor b (1024) at non-singleton dimension 0
```

**Steps/Code to reproduce bug**

Follow the [tutorial](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/gpt/gpt_training.html) to train GPT model with `model.position_embedding_type=rope` and `model.sequence_parallel=True`

**Expected behavior**

Expect there will be no runtime error when both options are enabled. 

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model
check O2 for pre_process
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Rope based models require a modification to access the pre_process attribute.

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update branch for pynini install script in notebooks
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update apex commit
State: closed | Pull request: yes
Labels: name: cherry-pick

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update apex commit
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update dataset preparation tool to fix bug relating to non jsonl input file
State: closed | Pull request: yes

# What does this PR do ?

Update dataset preparation tool to fix bug relating to non jsonl input file

**Collection**: nlp

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
update speaker finetune configs
State: closed | Pull request: yes
Labels: name: Speaker Tasks

# What does this PR do ?

Update titanet finetune configs

**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Potential issue with exp_manager checkpoint loading
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

If need to resume checkpoints from `exp_manager.explicit_log_dir`, NeMo exp_manager will check the folder for checkpoints with `end.ckpt` or `last.ckpt` suffix. However, sometimes if one of the data_parallel_rank 0 failed to save the checkpoint or there are multiple `last.ckpt` in the same folder, the workers will fail to load the correct checkpoints. 

**Expected behavior**

In exp_manager, only the global rank 0 will check which last ckpt to load and then broadcast the ckpt path to other workers. 

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model
Davidm/cherrypick r1.16.0
State: closed | Pull request: yes
Labels: name: core

name: cherry-pick

name: NLP

name: CI

# What does this PR do ?

Adding important main PRs to 1.16 branch
 
**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Loading ar conformer
State: closed | Pull request: no
Labels: name: bug

name: stale

I'm trying to load a pertained conformer that I downloaded from the NGC model catalog using the following two lines 

```
import nemo.collections.asr as nemo_asr

conformer = nemo_asr.models.ASRModel.restore_from('/home/Conformer-CTC-L_spe128_ar-AR_2.0.nemo')
```

but an error arose with the message '  ReadError: not a gzip file '. 

you can find the model on 
https://catalog.ngc.nvidia.com/orgs/nvidia/teams/riva/models/speechtotext_ar_ar_conformer/files?version=trainable_v2.0
[Question] How to perform generation with the GPT model checkpoints trained with virtual staging
State: closed | Pull request: no
Labels: name: bug

name: stale

I have a question on how we can load the GPT model checkpoints trained with virtual staging and perform generation? With virtual staging in pipeline parallelism, each rank stores a list of modules. But I cannot find relevant code for dealing with such case when performing generation. It seems to me NeMo does not support inference when the pretrained model is trained with virtual staging.
Added ckpt to nemo for T5/T0 models
State: closed | Pull request: yes

# What does this PR do ?

Added ckpt to nemo for T5/T0 models

Can use ```model_type=[t5/gpt]``` to declare the model type in the config

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix character beam decoding algorithm with vocab index map
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fixes character decoding with CTC beam search framework

**Collection**: [ASR]

# Changelog 
- Add index vocab and vocab index maps for reverse decoding in character based models
- Update nemo and pyctcdecode inference


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

Fixes #6127
Adding Validation Metrics for GPT/T5 prompt learning
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Adds the following metrics for GPT/T5 prompt learning:
- Accuracy -> exact string match
- BLEU score
- ROUGE-1, 2, 3, L

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Spectrogram Enhancer: add option to zero out the initial tensor
State: closed | Pull request: yes
Labels: name: TTS

name: cherry-pick

# What does this PR do ?

Enhancer is based on StyleGAN 2, which features a randomly-initialized "initial" tensor. Zeroing this tensor out helps a bit in a downstream ASR task (reported by @artbataev)

This PR adds this option and makes it the default behavior.

**Collection**: TTS

# Changelog 
- Add a `start_from_zero` option to enhancer

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Spectrogram Enhancer: add option to zero out the initial tensor
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Enhancer is based on StyleGAN 2, which features a randomly-initialized "initial" tensor. Zeroing this tensor out helps a bit in a downstream ASR task (reported by @artbataev)

This PR adds this option and makes it the default behavior.

**Collection**: TTS

# Changelog 
- Add a `start_from_zero` option to enhancer

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
How use arrow data sets or LightningDataModule
State: closed | Pull request: no
Labels: name: bug

I try using NeMo to train a GPT-like model from scratch with large `.arrow` files for which I have a `LightningDataModule`. Are there any resources on how to integrate such data set into NeMo? I found the `model.setup_training_data` but am unsure how to properly use it. Any advice would be helpful.
[DIARIZATION] is clustering needed when number of speakers is known?
State: closed | Pull request: no

Hello,

I'm using the diarization model and already know the number of speakers. I thought that the clustering algorithm was only needed to determine the number of speakers, but it seems that it is still being run if when I use the following settings:

```
meta = {
    'audio_filepath': an4_audio,
    'offset': 0,
    'duration': None,
    'label': 'infer',
    'text': '-',
    'num_speakers': 2,
    'rttm_filepath': None,
    'uem_filepath': None
}
```
and

`config.diarizer.clustering.parameters.oracle_num_speakers = True`

Is that the expected behavior? My main issue is that while the other parts of the model can be run on the GPU, the  clustering is still run on the CPU so the overall inference is quite long.
Read amplification upon checkpoint read
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

NeMo currently executes load_checkpoint independently on all ranks, causing unnecessarily high filesystem reading. In contrast,
save_checkpoint functionality for nlp models is only executed on rank 0 / data_parallel_rank == 0: https://github.com/NVIDIA/NeMo/blob/ac50e5963ca4e23b7fd758f0732f6ed51200cc1d/nemo/collections/nlp/parts/nlp_overrides.py#L164-L171

**Steps/Code to reproduce bug**

See

https://github.com/NVIDIA/NeMo/blob/ac50e5963ca4e23b7fd758f0732f6ed51200cc1d/nemo/collections/nlp/parts/nlp_overrides.py#L195-L200

**Expected behavior**

Checkpoint is only read from filesystem on rank 0 / data_parallel_rank == 0 and broadcasted to all other ranks.
[TTS] remove old waveglow model that relies on torch_stft.
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

TTS collections currently have two waveglow models released in NGC model cards, however, only the lastest one (`tts_waveglow_88m`) works with the lastest code base without needs of `torch_stft`. The old one (`tts_waveglow_268m`) always pops up error below,
```
E           RuntimeError: Error(s) in loading state_dict for WaveGlowModel:
E           	Missing key(s) in state_dict: "audio_to_melspec_precessor.window".
E           	Unexpected key(s) in state_dict: "audio_to_melspec_precessor.stft.forward_basis", "audio_to_melspec_precessor.stft.inverse_basis".
```

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue) https://github.com/NVIDIA/NeMo/issues/4015
* Related to PR: https://github.com/NVIDIA/NeMo/pull/3993