renwei2024/nemo-github-issues huggingface dataset:
Different outputs for a JIT operation
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I find that a JIT operation can have different outputs although the inputs are the same.

**Steps/Code to reproduce bug**

Suppose we have a checkpoint and load it to train a `MegatronGPTModel`. 

We print the inputs and outputs of this function: https://github.com/NVIDIA/NeMo/blob/v1.20.0/nemo/collections/nlp/modules/common/megatron/transformer.py#L556
```
layernorm_input = bias_dropout_add_func(attention_output, attention_bias, residual, self.hidden_dropout)
if torch.distributed.get_rank()==0:
    print(f"Layer: {self.layer_number} Attention checksum {layernorm_input.sum()}")
    print(f"Layer: {self.layer_number} Attention checksum {attention_output.sum()}")
    print(f"Layer: {self.layer_number} Attention checksum {attention_bias.sum()}")
    print(f"Layer: {self.layer_number} Attention checksum {residual.sum()}")
    print(f"Layer: {self.layer_number} Attention checksum {self.hidden_dropout}")
```
During the extra forward pass of the backward, we will get different outputs for this function despite the same inputs

For example, I get the following:
```
Layer: 1 Attention checksum -1.2981499433517456
Layer: 1 Attention checksum -0.10930676758289337
Layer: 1 Attention checksum 0.002560555934906006
Layer: 1 Attention checksum -1.1932671070098877
Layer: 1 Attention checksum 0
Layer: 2 Attention checksum -0.9067337512969971
Layer: 2 Attention checksum 0.4821169376373291
Layer: 2 Attention checksum 0.0022925958037376404
Layer: 2 Attention checksum -1.3912334442138672
Layer: 2 Attention checksum 0
Layer: 3 Attention checksum -1.014440894126892
Layer: 3 Attention checksum 0.02674156427383423
Layer: 3 Attention checksum -0.0017535686492919922
Layer: 3 Attention checksum -1.0347785949707031
Layer: 3 Attention checksum 0
Layer: 4 Attention checksum -1.6703314781188965
Layer: 4 Attention checksum -0.030592918395996094
Layer: 4 Attention checksum -0.0008977651596069336
Layer: 4 Attention checksum -1.6389871835708618
Layer: 4 Attention checksum 0
Layer: 1 Attention checksum -1.2990010976791382
Layer: 1 Attention checksum -0.10930676758289337
Layer: 1 Attention checksum 0.002560555934906006
Layer: 1 Attention checksum -1.1932671070098877
Layer: 1 Attention checksum 0
```
We can find that the outputs of `bias_dropout_add_func` in Layer 1 are different.

The key arguments I used are:
```
model.attention_dropout=0 
model.hidden_dropout=0 
model.transformer_engine=False
model.activations_checkpoint_granularity=full  
model.activations_checkpoint_num_layers=1
```

However, when setting `PYTORCH_JIT=0`, I will get the same outputs for `bias_dropout_add_func`.
**Expected behavior**

The function should always yield the same output.

**Environment overview (please complete the following information)**

I find this issue on NeMov1.20 docker
Fix lang id inference mengruw
State: closed | Pull request: yes
Labels: name: TTS

name: stale

name: ASR

name: NLP

name: Speaker Tasks

name: CI

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fixed lang id inference mengruw
State: closed | Pull request: yes
Labels: name: TTS

name: stale

name: ASR

name: NLP

name: Speaker Tasks

name: CI

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Fix] Save best NeMo model only when necessary
State: closed | Pull request: yes

# What does this PR do ?

Currently, when using `save_best_model`, the best model will be saved every time, even when it is not necessary to update it.
This fix ensures NeMo checkpoints are saved only when necessary.

**Collection**: Core

# Changelog 
- Fix member name when updating `previous_best_path`
- Added a debug log when saving is skipped when previous best is the same as the current best

# Usage
N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Codec] Update codec checkpoint config
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Update codec configs to always save NeMo checkpoint with best `val_loss`.

**Collection**: TTS

# Changelog 
- Updated `checkpoint_callback_params` to always save `NeMo` checkpoint with best `val_loss` and to save 5 best checkpoints
- Set `create_wandb_logger` to `true`, since `model.log_config.log_wandb` is set to `true` by default

# Usage
N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add support for finetuning with huggingface datasets
State: closed | Pull request: yes
Labels: name: ASR

name: CI

# What does this PR do ?

Add support for finetuning with HuggingFace datasets.


**Collection**: [ASR]

# Changelog 
- Add dataset wrappers that cast HuggingFace dataset to be compatible with NeMo ASR models.
- Add yaml for finetuning ASR models with HuggingFace datasets.

# Usage

```python
<NEMO_ROOT>/examples/asr/speech_to_text_finetune.py \
    --config-name='speech_to_text_hf_finetune'
```

Please refer to `speech_to_text_hf_finetune.yaml` for how to configure the dataset to use.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
GPU memory overhead when calling trainer.fit the second time
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

when calling `trainer.fit` the second time to train a NeMo model like:
```
trainer.fit(model)
trainer.fit_loop.epoch_loop.max_steps += 10
trainer.fit(model)
```
The second time would use more GPU memory than the first.

**Steps/Code to reproduce bug**
Create a `MegatronGPTModel` with `megatron_amp_O2=True` or with `distributed_fused_adam` and then use the above code for training.

When we print `torch.cuda.memory_allocated()` at the training step, we will find the GPU memory usage at the second `trainer.fit` is significantly higher.

**Expected behavior**

We expect the second `trainer.fit` should have the similar GPU memory usage to that in the first `trainer.fit`.

**Environment details**

Tried official Nemo v1.20 docker and it has this issue.
CTC segmentation tool does not support fastconformer hybrid model
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

It seems stt_en_fastconformer_hybrid_large_pc is supported by the force alignment tool, but not the CTC segmentation tool. CTC segmentation tool works on older models, but not the ctc rnnt hybrid model

./NeMo/tools/ctc_segmentation$ ./run_segmentation.sh  crash on latest stt_en_fastconformer_hybrid_large_pc model.


**Steps/Code to reproduce bug**
 
     NeMo/tools/ctc_segmentation$  ./run_segmentation.sh --MODEL_NAME_OR_PATH="stt_en_fastconformer_hybrid_large_pc" --DATA_DIR=./data/timit --OUTPUT_DIR=./res/timit_fastconformer_hybrid_large_pc


**Expected behavior**

 code crash with error message:    
    ctc_segmentation/scripts/prepare_data.py", line 357
    omegaconf.errors.ConfigAttributeError: Key 'vocabulary' is not in struct

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

- OS version    ubuntu "20.04.6 LTS (Focal Fossa)"
- PyTorch version     2.1.0           py3.10_cuda11.8_cudnn8.7.0_0
- Python version      3.10.12 

**Additional context**
Provides CPU fallback for `getKneighborsConnections` avoiding OOM
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

Provide a fallback to CPU for `getKneighborsConnections` preventing OOM when applying to large cluster of embeddings (i.e when applying diarization to a long audio with multiple speakers).

**Collection**: ASR

# Changelog 
- `getKneighborsConnections` now takes an optional argument `max_dim_on_gpu` that trigger a fallback to CPU when one of the dim is higher than `max_dim_on_gpu`.

# Usage

We were encountering OOMs when applying diarization to a longer audio (2h+) with multiple speakers. It appears that `getKneighborsConnections` double the VRAM needed to store the embedding when applying `binarized_affinity_mat = torch.zeros_like(affinity_mat).half()` as it use the same device as for `affinity_mat`.

This PR reduced the VRAM used when applying diarization on this same audio from 41Go (32 + 9Go) to 29Go by increase the inference time by only one seconde.

In order to allow user to use this param we either should add a kwargs to `getKneighborsConnections` (and all the function that called it) or use a more global param that's automatically propagated (i.e env var). As this choice need some thought on your side I prefer to only add `max_dim_on_gpu` to `getKneighborsConnections` for now and let you decide how to proceed (unless you already have some suggestion). 

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] ~~Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)~~
  - [x] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
multimodal/speech_cv collection, second pull request for Audio-Visual Speech Recognition
State: closed | Pull request: yes
Labels: name: stale

name: ASR

name: common

# What does this PR do ?

Update the multimodal/speech_cv branch to support audio-visual speech recognition (CTC, RNN-T and Hybrid)

**Collection**: [multimodal/speech_cv]

# Changelog 
- Update speech_cv branch
- bug fix
- Add audio-visual datasets
- Add audio-visual models: (AudioVisualEncDecCTCModelBPE, AudioVisualEncDecCTCModel, AudioVisualEncDecHybridRNNTCTCBPEModel, AudioVisualEncDecHybridRNNTCTCModel, AudioVisualEncDecRNNTBPEModel, AudioVisualEncDecRNNTModel)
- Add audio-visual modules (FusionModule, InterCTCMixin)

# Usage
* You can potentially add a usage example below

```python
import nemo.collections.multimodal.speech_cv as nemo_cv
from omegaconf import OmegaConf

config_path = "model_config.yaml"
params = OmegaConf.load(config_path)

# hybrid CTC/RNN-T BPE audio-visual speech recognition model
model = nemo_cv.models.AudioVisualEncDecHybridRNNTCTCBPEModel(cfg=params.model)
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
test jenkins pr - user is random
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
test jenkins pr - user is nvidian
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Test PR random user
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Test PR
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update README.rst 3
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
PR PR PR test
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Learning rate max step wrong when using batch sampler
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

when using the batch sampler such as [MegatronPretrainingBatchSampler](https://github.com/NVIDIA/NeMo/blob/177a67fa4a43ed6f7c4fb39a66c0a2998a27fef1/nemo/collections/nlp/data/language_modeling/megatron/megatron_batch_samplers.py#L141). The train_dataloader's drop_last argument is always set to False, which is accidentally picked up by this line of [code](https://github.com/NVIDIA/NeMo/blob/177a67fa4a43ed6f7c4fb39a66c0a2998a27fef1/nemo/core/optim/lr_scheduler.py#L877C47-L877C47) leading to our learning rate scheduler being off by 1step. 

Instead we should use drop last from the Batch sampler itself [here](https://github.com/NVIDIA/NeMo/blob/177a67fa4a43ed6f7c4fb39a66c0a2998a27fef1/nemo/core/optim/lr_scheduler.py#L870) when we detect it
Test jenkins
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
New PR @#2
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update README.rst
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update README.rst
State: closed | Pull request: yes

# What does this PR do ?

Test

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Remove undefined `trainer` in `get_lm_model_with_padded_embedding`
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Fixes errors using `BertDPRModel` and `BertJointIRModel`. Currently you get an exception due to an undefined variable `trainer` from method `get_lm_model_with_padded_embedding` in `BaseIRModel`.

**Collection**: nlp

# Changelog 
- Fixes errors using `BertDPRModel` and `BertJointIRModel` from undefined trainer name.

# Usage
* You can potentially add a usage example below

```python
from omegaconf import OmegaConf
from nemo.collections.nlp.models.information_retrieval import  BertJointIRModel

model_name_or_path = "intfloat/e5-small-v2"

cfg = OmegaConf.create({
    "language_model": {"sim_score_dropout": 0.1, "pretrained_model_name": model_name_or_path, "config_file": None, "config": None}, 
    "tokenizer": {"tokenizer_name": model_name_or_path, "vocab_file": None, "special_tokens": None, "tokenizer_model": None}
})
BertJointIRModel(cfg)
# => Raises NameError: name 'trainer' is not defined
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
First draft of mcore bert model in NeMo
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Adds the mcore bert model to the nemo codebase. 

**Collection**: nlp collection

# Usage
* Set mcore_bert config to True


# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add safetensors
State: closed | Pull request: yes
Labels: name: core

name: stale

name: ASR

name: NLP

# What does this PR do ?

By default, state_dicts are being saved as pytorch files which contain pickles. This is a security concern which this PR attempts to start the mitigation for while preserving backwards compatability.

**Collection**: all, but none specifically (core functionality)

# Changelog 
- add nemo.utils.secure.torch_load and nemo.utils.secure.torch_save
- torch.load and torch.save removed from save_restore_connector and alternates of it
- secure.torch_load and torch_save use safetensors when available, however if safe=False torch.load and torch.save are used to allow for backwards compatability
- safe is a named parameter and passed through from save_to and restore_from. The default is safe=False to preserve backwards compatability
- added unit tests to double check backwards compatability and secure functions work correctly.

# Usage
* When using nemo with untrusted .nemo files this will greatly reduce the potential for the user to be attacked.

```python
# Existing usage should be preserved. although the result will include a .safetensor inside the .nemo now as well
model.save_to(save_path=model_save_path)
model.save_to(save_path=model_save_path, safe=False)

# the .nemo will only include the .safetensor.
model.save_to(save_path=model_save_path, safe=True)

# Existing usage should be preserved. although if the .nemo contains a .safetensor, it will be used instead of the pytorch version
model.restore_from(model_save_path)
model.restore_from(save_path=model_save_path, safe=False)

# The restore will fail if a .safetensor isn't available to prevent an untrusted .nemo file from resulting in an exploit
model.restore_from(save_path=model_save_path, safe=False)
```

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [X] Did you write any new necessary tests?
- [X] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Duplex Model for Inverse Text Norm (ITN) does not work
State: closed | Pull request: no
Labels: name: bug

I follow the tutorial of inference the pretrained checkpoint as in [here](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/neural_text_normalization_t5) and [here](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/text_normalization/nn_text_normalization.html).

The TN option works just fine but when I use the mode of `ITN`, the model still output the Text Normalization instead.

The command I used:
```
cd NeMo/examples/nlp/duplex_text_normalization;

# run inference in interactive mode using pretrained tagger and decoder models
python duplex_text_normalization_infer.py \
    tagger_pretrained_model=neural_text_normalization_t5 \
    decoder_pretrained_model=neural_text_normalization_t5 \
    inference.from_file=False \
    lang=en \
    mode=itn
```

The output:
```
Entering interactive mode.
Type "STOP" to exit.
Input a test input:$123
Skipping post-processing of one hundred twenty three dollars for '$'
Prediction (ITN): one hundred twenty three dollars
Type "STOP" to exit.
Input a test input:one hundred twenty three
Prediction (ITN): one hundred twenty three
Type "STOP" to exit.
Input a test input:one two three
Prediction (ITN): one two three
Type "STOP" to exit.
Input a test input:one hundred twenty three dollars
Prediction (ITN): one hundred twenty three dollars
```

As you can see, the model still returns TN result.
Fix tn duplex
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
remove TN from ctc_segm tut
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
remove TN req from ctc_segm
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
ConfidenceMethodConfig DEPRECATED compatibility problem between r1.21.0 and r1.20.0
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

Loading models trained with Nemo version `r1.21.0` using Nemo `r1.20.0` codebase fails with the following error:
```
[NeMo E 2023-10-23 16:21:42 common:506] Model instantiation failed!
    Target class:       nemo.collections.asr.models.hybrid_rnnt_ctc_bpe_models.EncDecHybridRNNTCTCBPEModel
    Error(s):   'str' object has no attribute 'temperature'
```
This is due to changes in `nemo/collections/asr/parts/utils/asr_confidence_utils.py` - the newer [version ](https://github.com/NVIDIA/NeMo/blob/c0022aec9c8fedd59458935f0378a775b28d0444/nemo/collections/asr/parts/utils/asr_confidence_utils.py#L93-L120) supports the `str` value `DEPRECATED`: 

**Steps/Code to reproduce bug**

The bug occurs when using Nemo `r1.21.0` and finetuning the model [here](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_multilingual_fastconformer_hybrid_large_pc) using the script `examples/asr/speech_to_text_finetune.py` and the config file `examples/asr/conf/speech_to_text_finetune.yaml` with minor modifications.

**Expected behavior**

The `r1.20.0` codebase should be able to load models trained with `r1.21.0` unless the models differ in ways that simply can't be supported by the old code base.  In the later situation, a meaningful error message should indicate that code base is too old for the model.

**Environment overview (please complete the following information)**

 - Environment location: Bare Metal
 - Method of NeMo install: `python -m pip install 'git+https://github.com/NVIDIA/NeMo.git@r1.21.0#egg=nemo_toolkit[all]'`

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 20.04
- PyTorch version: pytorch-lightning 2.0.7
- Python version: Conda Python 3.10.13

**Additional context**

Add any other context about the problem here.
Example: GPU model: NVIDIA A100
Update Apex install command in Dockerfile
State: closed | Pull request: yes
Labels: name: cherry-pick

name: CI

# What does this PR do ?

Updates Dockerfile to install Apex correctly.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Error when running CTC Segmentation tutorial
State: closed | Pull request: no
Labels: name: bug

https://github.com/NVIDIA/NeMo/blob/main/tutorials/tools/CTC_Segmentation_Tutorial.ipynb
When running this colab we get error on the first few cells and fail to install requirements. We get the following error.
```site-packages (from requests->transformers->nemo_text_processing==0.1.6rc0->-r https://raw.githubusercontent.com/NVIDIA/NeMo/main/tools/ctc_segmentation/requirements.txt (line 3)) (2023.7.22)
Using cached num2words-0.5.13-py3-none-any.whl (143 kB)
Installing collected packages: setuptools, num2words, ctc_segmentation, nemo_text_processing
  Attempting uninstall: setuptools
    Found existing installation: setuptools 68.2.2
    Uninstalling setuptools-68.2.2:
      Successfully uninstalled setuptools-68.2.2
  Attempting uninstall: nemo_text_processing
    Found existing installation: nemo-text-processing 0.2.2rc0
    Uninstalling nemo-text-processing-0.2.2rc0:
      Successfully uninstalled nemo-text-processing-0.2.2rc0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
nemo-toolkit 1.21.0rc0 requires setuptools>=65.5.1, but you have setuptools 59.5.0 which is incompatible.
Successfully installed ctc_segmentation-1.7.1 nemo_text_processing-0.1.6rc0 num2words-0.5.13 setuptools-59.5.0
[GERMAN] Joint Intent and Slot Classification
State: closed | Pull request: no

Hello,

I have tried out the [Joint Intent and Slot Classification Notebook](https://github.com/NVIDIA/NeMo/blob/stable/tutorials/nlp/Joint_Intent_and_Slot_Classification.ipynb) in Google Collab and it works great.

In the notebook it is mentioned that "[their] Joint Intent and Slot classification model is comprised of the pretrained [BERT](https://arxiv.org/pdf/1810.04805.pdf) model with an Intent and Slot Classification layer on top of it."

I assume this model is inherited from [BERTModule](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/modules/common/bert_module.py).

Is there a way to use a German-based BERT and put the "Intent and Slot Classification layer on top of it."? For instance to plug-in some German [SentenceTransformer](https://huggingface.co/sentence-transformers) model?

Any help in the right direction is much appreciated, thanks!