renwei2024/nemo-github-issues huggingface dataset:
RoPE for GPT
State: closed | Pull request: yes
Labels: name: NLP

Signed-off-by: Hoo Chang Shin <hshin@nvidia.com>

# What does this PR do ?

Ported Megatron-LM "Rotary Positional Encoding" (RoPE) implementation for NeMo GPT.

**Collection**: GPT

# Changelog 
Add option to use 'rotary' positional encoding in the config - making it as default, instead of "learned_absolute" positional embedding.

# Usage

```python
`model.position_embedding_type: 'rotary'` 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Pin isort version
State: closed | Pull request: yes

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

# What does this PR do ?

Pin `isort` version. Use `>5.1.0`, `<6.0.0`, as suggested by the documentation https://pycqa.github.io/isort/docs/major_releases/release_policy.html

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
FilterbankFeaturesTA to match FilterbankFeatures
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fixes #5912 

**Collection**: asr

# Changelog 
- `FilterbankFeaturesTA` matches naming of `FilterbankFeatures`

# Usage
* When using `torchaudio` for preprocessing.


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #5912
FilterbankFeaturesTA imcompatible with FilterbankFeatures
State: closed | Pull request: no
Labels: name: bug

The new `FilterbankFeaturesTA` does not use the same naming convention as `FilterbankFeatures`.
This leads to errors when using the `FilterbankFeaturesTA` in a model. 

For instance here: https://github.com/NVIDIA/NeMo/blob/ddb92e319e9e3aa7b2db7e58b25810a71a4c2cc7/nemo/collections/asr/models/ctc_models.py#L160-L161

While `FilterbankFeatures` defines those attributes, `FilterbankFeaturesTA` uses different naming:
https://github.com/NVIDIA/NeMo/blob/ddb92e319e9e3aa7b2db7e58b25810a71a4c2cc7/nemo/collections/asr/parts/preprocessing/features.py#L540-L541
fixed missing long_description_content_type
State: closed | Pull request: yes

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Ensure tokens-in-batch is always an upper bound
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

This PR fixes an edge case where a batch may contain more tokens than allowed by `tokens-in-batch`

**Collection**: NLP

# Changelog 
- In case a batch that contains too many tokens has fewer than 8 sentences the last sentence is evicted to a new batch to ensure we don't break the `tokens-in-batch` bound
- Before it would not evict any sentence leading to batches that are too big.


**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Adithyare/tiny attn
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Correct doc for RNNT transcribe() function
State: closed | Pull request: yes
Labels: name: ASR

Signed-off-by: smajumdar <titu1994@gmail.com>

# What does this PR do ?

Corrects the documentation of RNNT and Hybrid CTC RNNT models

**Collection**: [ASR]

# Changelog 
- Fix docstring for two models

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Set providers for onnxruntime inference session
State: closed | Pull request: yes
Labels: name: ASR

Signed-off-by: athitten <abhishreetm@gmail.com>

# What does this PR do ?

Explicitly sets the providers parameter when instantiating InferenceSession with onnxruntime.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
update apex install instructions for 1.15
State: closed | Pull request: yes
Labels: name: cherry-pick

Signed-off-by: ericharper <complex451@gmail.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
update apex install instructions for 1.15
State: closed | Pull request: yes

Signed-off-by: ericharper <complex451@gmail.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
minimal test to outline the issue
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Can you kindly provide any evaluation file for megatron_bert just like the one for GPT - `megatron_gpt_eval.py`?
State: closed | Pull request: no

**Problem Description -> Trying to load the GatorTron model checkpoint into a pretrained MegatronBERT model**

Hi. I have the `.nemo` file for the Gatortron model which I downloaded from [here](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/gatortron_og/files) and I was looking to load this into a pretrained MegatronBert model, to perform some kind of language modeling just akin to the evaluation done in `megatron_gpt_eval.py` file. 
More specifically, I want to be able to input clinical text to the Gatortron weights loaded into a pretrained BERT and get relevant output, without having to perform any kind of finetuning, since I currently do not have any dataset to fine-tune upon.

**Describe the solution you'd like**

Could you kindly provide an example evaluation file - `megatron_bert_eval.py`, including all the dependent modules, just like the one for GPT? 

**Describe alternatives you've considered**
I am using the `nvcr.io/nvidia/nemo:22.09` docker image and the NeMo `v1.10.0` and I was trying to build my own evaluation file for MegatronBERT, taking reference from the file for GPT, but was running into a lot of errors. Hence I thought of asking over here whether it would be possible if anyone could kindly give some reference along this way.


**Additional context**

None
Block large files from being merged into NeMo main
State: closed | Pull request: yes

Signed-off-by: SeanNaren <snarenthiran@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
conformer transducer timestamp extraction
State: closed | Pull request: no

Good day! Thanks for your hard work!
I am trying to extract timestamps for a model stt_en_conformer_transducer_large
Following https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/asr/ASR_with_Transducers.ipynb#scrollTo=xkv_x8NAfpX3
But I'm having difficulty converting timestamps:
"Note that each timestep here is (roughly) timestep∗total_stride_of_model∗preprocessor.window_stride seconds timestamp" - I can extract "timestep" from hypotheses, but model.preprocessor doesn't have a window_stride field and I'm not sure which value to use here. Also, I couldn't figure out where the "total_stride_of_model" value comes from. Do I understand correctly, total_stride_of_model == len(audio) / 'window_stride'? By the way, the 'window_stride' field is also missing from model.preprocessor.
Thank you!
Update isort to the latest version
State: closed | Pull request: yes
Labels: name: core

name: ASR

name: NLP

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>

# What does this PR do ?

Update `isort` to the latest version

**Collection**: [Note which collection this PR will affect]

# Changelog 
- update `isort` to the latest version
- apply latest `isort` to the codebase

# Usage
```shell
isort .  # one can simply use `isort` command to fix all files or provided paths
python setup.py style --fix  # also will work for isort + black, as previously
```


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #5893
Use `mypy` for type checking
State: closed | Pull request: no
Labels: name: stale

name: discussion

**Is your feature request related to a problem? Please describe.**

In many places in NeMo, Python type annotations are used.
Since type annotations are not checked, there are some cases where annotations are incorrect or not precise (e.g., using one type when some union is really used).

**Describe the solution you'd like**

Add `mypy` type checking support. 

**Main principle:** do not require types, but if types are provided, check (require) correctness.

- firstly, add a minimal PR that doesn't affect existing code (except 2 lines, since duplicated type annotations are critical errors for mypy)
- provide PRs one-by-one to fix some of the errors to improve code quality, removing the error types from ignore list.

**Describe alternatives you've considered**

[Pyre](https://github.com/facebook/pyre-check) also can be considered as an alternative, but `mypy` is usually used as a default choice.

**Additional context**

...
Update `isort` to the latest version
State: closed | Pull request: no

**Is your feature request related to a problem? Please describe.**

NeMo currently requires a very old version of `isort` (<5). The latest version is 5.10.1

Benefits of supporting the latest `isort`:
- latest version => improved support, fewer bugs, etc
- no conflicts with new repos, more friendly for new developers
- explicit config instead of hard-coded options in setup.py
- it is possible to use the `isort` executable instead of a custom command from `setup.py`

**Describe the solution you'd like**

Update `isort` to the latest version.
Use updated config parameters (for `black` support).

**Cost of change:**
~35 files will be changed (the new version is a bit more restrictive)

**Describe alternatives you've considered**

...

**Additional context**

...
Pitch objective
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

name: common

some promising signs of improved spectrogram clarity:

![image](https://user-images.githubusercontent.com/6509132/215101429-c64267c2-e6d2-4cc5-9ad6-7392906daaf1.png)


summary of changes:
* model now has a first stage consisting of token and spectrogram encoders
* after the first stage an attention layer with a learned query vector is used to produce a "speaker encoding" from the spectrogram encodings
* The speaker encoding is added to the token encodings which are then used to predict token duration and pitch
* token encodings are then aligned with the spectrogram encodings, then run all together through the decoder
Streaming RNNT ASR with context 
State: closed | Pull request: no
Labels: name: stale

Im trying to implement streaming ASR from microphone with PyAudio. However, Im a bit stuck on which functions to use exactly.
Im now using pyaudio with stream_callback as provided by Online_ASR_Microphone_Demo.ipynb, but Im stuck in how to progress further.

I found the script of speech_to_text_cache_aware_streaming_infer.py, which uses conformer_stream_step() in combination with a model trained specifically for streaming, but there are also a lot of functions in streaming_utils.py as StreamingFeatureBufferer and FrameASR to provide buffers and transcribe from there, however in my understanding the transcription wouldnt use context.

Im a bit stuck on which functions I should for streaming ASR with the best possible result. Can I directly use the chunks from the audio in the conformer_stream_step() or do I need something like StreamingFeatureBufferer in between?

Could you provide an overview on which functions I should use to provide such a pipeline? And should I train a streaming model? Or would a full context model work fine.
Remove file history
State: closed | Pull request: yes

# What does this PR do ?

Removing a file from git history.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Cleanup docs on r1.13.0
State: closed | Pull request: yes
Labels: name: cherry-pick

name: stale

Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>

# What does this PR do ?

Some fixes and minor improvements to README.rst and RTD docs,

# Changelog 

- [x] Moves Requirements and Installation into single source of truth (on RTD)
- [x] Cleans up README.rst to make it less verbose and removing redundant and obsolete links
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation
No Convergence on SqueezeFormer character vocab based CTC Models
State: closed | Pull request: no
Labels: name: bug

name: stale

I have been trying to train the SqueezeFormer-based ASR model for the Hindi language. Model is only converging if I use BPE based vocabulary while if I used Non BPE based vocabulary on same training data model fails to converge.

Below are the experiment logs.
**BPE-BASED VOCABULARY**
![bpe](https://user-images.githubusercontent.com/18329235/215416244-fb4a23c3-cced-418d-b854-5179269f0911.png)

**NON BPE BASED VOCABULARY**
![non-bpe](https://user-images.githubusercontent.com/18329235/215416329-4b767c12-d4a7-49e5-aaa8-43205755fc46.png)

**MODEL DETAILS**
I am training SqueezeFormer Medium model with the following configurations as provided in the config file in the repo.
 
|Vocab type| Model        | d_model | n_layers | n_heads | time_masks | lr     | time_reduce_idx | GBS  |
|-------------|--------------|---------|----------|---------|------------|--------|-----------------|------|
|BPE         | Medium       | 324     | 20       | 4       | 7          | 1.5e-3 | 9               | 1024 |
|Non BPE| Medium       | 324     | 20       | 4       | 7          | 1.5e-3 | 9               | 1024 |

**Expected behavior**
Both model should be converged to nearly similar point.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal]
 - Method of NeMo install: [pip install or from source]
 `conda create --name nemo python==3.8`
 `conda activate nemo`
 `pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1 -f https://download.pytorch.org/whl/cu111/torch_stable.html`
 `apt-get update && apt-get install -y libsndfile1 ffmpeg`
 `pip install Cython`
 `pip install nemo_toolkit['all']`

**Environment details**
Otherwise, please provide:
- OS version: ubuntu 20.04
- PyTorch version : 1.10.1+cu111
- Python version: Python 3.8.0


Is there some configuration issue with bpe vs non bpe based model ??
fix(readme): fix typo
State: closed | Pull request: yes

Signed-off-by: Jean-Louis Queguiner <jean-louis.queguiner@gadz.org>

# What does this PR do ?

Fix typo in readme

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Support Transcribing a Manifest File in `transcribe()` method
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

This PR enables transcribing an already existing manifest file instead of providing a list of audio files in the variable `paths2audio_files`.

**Collection**: ASR

# Changelog 
- Modified the `transcribe` method in the `EncDecCTCModel` class to support transcribing from manifest file. I first check that one of the arguments  `paths2audio_files` or `manifest_filepath` exists. If `manifest_filepath` is provided, add it to the `config` dict. Otherwise, create a temp manifest file from `paths2audio_files` list.

# Usage
* It can be combined with other models' output manifests, e.g. Lang ID, and use as an input to the ASR model using the `transcribe` method.


# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
ONNX export for RadTTS
State: closed | Pull request: yes
Labels: name: TTS

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

# What does this PR do ?
ONNX export for RadTTS is now functional. 
Working around: https://github.com/pytorch/pytorch/pull/91526

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Dynamic freezing in Nemo
State: closed | Pull request: yes
Labels: name: core

Signed-off-by: Daniel Egert <degert@nvidia.com>

# What does this PR do ?

Allows you to selectively freeze modules in your model for the first N steps during training

**Collection**: Core

# Changelog 
- changed core/classes/modelPT.py to do dynamic freezing via PTL hooks
- updated core documentation at docs/source/core/core.rst

# Usage
You can add the following to your config.yaml:

```yaml
  model:
    ...
    freeze_updates:
      enabled: true  # set to false if you want to disable freezing
      
    modules:   # list all of the modules you want to have freezing logic for
      encoder: 200      # module will be frozen for the first 200 training steps
      decoder: [50, -1] # module will be frozen at step 50 and will remain frozen until training ends
      joint: [10, 100]  # module will be frozen between step 10 and step 100 (step >= 10 and step <= 100)
      transcoder: -1    # module will be frozen for the entire training run
```

Setting `enabled: false` will disable all of the freezing logic.

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS]Vits Available Model List needs to be updated
State: closed | Pull request: no
Labels: name: bug

name: TTS

Hello. I want to have a trial finetuning the newly updated Vits model in TTS module.
Trying to follow the instructions on:
https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/tts_en_lj_vits

But got the error of:
FileNotFoundError: Model tts_en_lj_vits was not found. Check cls.list_available_models() for the list of all available models.

Because currently print(VitsModel.list_available_models()) is an empty list.
This seems similar to the error in Tacotron2 I encountered last time here:
https://github.com/NVIDIA/NeMo/issues/5714

Thank you!
Set device specifically for core diarization clustering
State: closed | Pull request: yes
Labels: name: ASR

name: Speaker Tasks

# What does this PR do ?

This PR allows setting a different device (CPU) for the core clustering part of the diarization process so that it doesn't cause GPU OOM errors for long audio files.

**Collection**: ASR

# Changelog 
- Add `device` option to the `diarizer.clustering.parameters` configuration group so that [nemo.collections.asr.parts.utils.speaker_utils.perform_clustering()](https://github.com/NVIDIA/NeMo/blob/e1b3f5e2fe57174863110d3fa47b6c24238ab58e/nemo/collections/asr/parts/utils/speaker_utils.py#L402) (called from [ClusteringDiarizer.diarize()](https://github.com/NVIDIA/NeMo/blob/e1b3f5e2fe57174863110d3fa47b6c24238ab58e/nemo/collections/asr/models/clustering_diarizer.py#L440)) can be executed in the CPU (it's currently hard-coded to run on GPU).
  - This part of the code uses a huge amount of VRAM (over 10GB for an hour of audio) and barely uses the GPU at all, so the performance on CPU doesn't decrease dramatically (processing time increases from 150 seconds on an RTX3090 to 180 seconds on CPU for an hour of audio).
  - This PR is somewhat related to #5681, but instead of setting a device for all processing steps, it allows setting a different device specifically for the function mentioned above.
  - If the new device configuration is not set, it will use the globally set device.
  - The whole reason for this change is supporting audio files over an hour long. It seemed easier to me to run the clustering on CPU than splitting the input audio, diarizing the splits separately and combining the results later. It can surely be done, or there may be a better solution for this, but I don't know enough of Nemo's internals for that yet.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #5681
Reduce memory usage in getMultiScaleCosAffinityMatrix function
State: closed | Pull request: yes
Labels: name: ASR

Reduce memory usage in getMultiScaleCosAffinityMatrix function

Signed-off-by: gabitza-tech <gabriel.pirlogeanu@gmail.com>

# What does this PR do ?

Replace some tensor multiplication in the function for multi scale cos affinity matrix calculation with a single tensor in which we add the repeated embeddings multiplied with its scale weight.

Results show a decrease from 11GB to 2GB for 1 hour audio. It even managed to reduce from above 60GB memory usage for a [53000,53000] tensor to 28GB.
**Collection**: ASR (collections/asr/parts/utils/offline_clustering.py)

# Changelog 
`multiscale_weights = torch.squeeze(multiscale_weights,dim=0).to(device)`
The tensor was [1,6] shaped and had to squeeze the first dimension.
` fused_sim_d = torch.zeros(len(timestamps_in_scales[-1]),len(timestamps_in_scales[-1]))`
Initialized a tensor with the shape [s,s] with 0 values where s is the number of timestamps for the smallest scale. We will add the tensor for each scale to this tensor
` with torch.no_grad():  `
Gradients are not needed for inference.
`fused_sim_d = fused_sim_d + multiscale_weights[scale_idx] * repeated_tensor_1`
Instead of adding the tensor for each scale to a list, then stacking the tensor list and then matrix multiplying it with the weights, we simply add the repeated tensor to the previously initialized tensor before the for loop.

@tango4j is adding followings:
- Clear memory outside torch.jit.script functions since jit-script does not support torch.cuda.empty_cache()
- Fixed the bug where eigen value decomposition does not use GPU. Now GPU usage hits 100%
- Removed torch.jit.script decorators since it slows down if we run without jit.script export.
- Performance (without torch.jit.script) <!--td {border: 1px solid #cccccc;}br {mso-data-placement:same-cell;}-->

sess len in secs | Num of sessions | total time in secs | sec/Iter. | RTFx |  Peak Memory Usage
-- | -- | -- | -- | -- | --
1800 | 10 | 34 | 3.48 | 517.24138 | 3G
3600 | 10 | 70 | 7.02 | 512.82051 | 8G
7200 | 10 | 319 | 31.98 | 225.14071 | 20G

Now, 2hours of audio usually takes 30~40 seconds.


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. I would like to ask @titu1994, @redoctopus, @jbalam-nv, or @okuchaiev to review this pull request.
# Additional Information
* Related to https://github.com/NVIDIA/NeMo/issues/5637
Add pitch objective
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

name: common

some promising signs of improved spectrogram clarity:

![image](https://user-images.githubusercontent.com/6509132/215101429-c64267c2-e6d2-4cc5-9ad6-7392906daaf1.png)


summary of changes:
* model now has a first stage consisting of token and spectrogram encoders
* after the first stage an attention layer with a learned query vector is used to produce a "speaker encoding" from the spectrogram encodings
* The speaker encoding is added to the token encodings which are then used to predict token duration and pitch
* token encodings are then aligned with the spectrogram encodings, then run all together through the decoder
add ambernet to readme
State: closed | Pull request: yes
Labels: name: cherry-pick

Signed-off-by: fayejf <36722593+fayejf@users.noreply.github.com>

# What does this PR do ?

add ambernet to readme

**Collection**: [ASR]

# Changelog 
- add ambernet to readme



# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add ambernet to readme
State: closed | Pull request: yes

Signed-off-by: fayejf <36722593+fayejf@users.noreply.github.com>

# What does this PR do ?

add ambernet to readme

**Collection**: [ASR]

# Changelog 
- add ambernet to readme



# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix val loss computation in megatron
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

fix val loss computation
When drop last is set to false for validation, currently valid samples are computed assuming any token in a valid sample will not be masked. This case might not be true always. Also, last sample of the dataset could be incomplete as well. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix wrong label mapping in batch_inference for label_model
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

# What does this PR do ?

Fix groundtruth label mapping in `batch_inference` in `label_models.py` and add test. 

**Collection**: ASR

# Changelog 
-  The groundtruth label `batch_inference` was mapped wrongly. Fixed it
-  Add test for `batch_inference`

# Usage
* You can potentially add a usage example below

```python
from nemo.collections.asr.models import EncDecSpeakerLabelModel 
model = EncDecSpeakerLabelModel.from_pretrained('langid_ambernet')
embs, logits, gt_labels, mapped_labels = model.batch_inference(<path_to_manifest>)
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to https://github.com/NVIDIA/NeMo/pull/5278
removed WHATEVER(1)  ˌhwʌˈtɛvɚ from scripts/tts_dataset_files/ipa_cmudict-0.7b_nv22.10.txt
State: closed | Pull request: yes

# What does this PR do ?

changed WHATEVER  ˌwʌˈtɛvɚ to WHATEVER  ˌwʌtˈɛvɚ and deleted WHATEVER(1)
updated WHATEVER'S to ˌwʌtˈɛvɚz and delete WHATEVER'S(1).

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)