renwei2024/nemo-github-issues huggingface dataset:
Update font for NeMo Website
State: closed | Pull request: yes

Signed-off-by: smajumdar <titu1994@gmail.com>

# What does this PR do ?

Update the font used in the NeMo website.

**Collection**: [Docs]

# Changelog 
- Use NVIDIA Sans font throughout

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation
[TTS][G2P] deprecate add_symbols
State: closed | Pull request: yes

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

# What does this PR do ?

```
remove this function since the use case is not fully clear right now.
```
**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Tn doc 16
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] deprecate AudioToCharWithPriorAndPitchDataset.
State: closed | Pull request: yes
Labels: name: TTS

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

# What does this PR do ?

```
AudioToCharWithPriorAndPitchDataset is deprecated. Please choose TTSDataset.
```
**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Speaker Diarization WinError: 'The filename or extension is too long'
State: closed | Pull request: no
Labels: name: bug

name: stale

Hi everyone,

when I run the following tutorial on my Windows 10 PC (Python 3.10.5): https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/ASR_with_SpeakerDiarization.ipynb

Then I get at the step "Run diarization with the extracted word timestamps":
`diar_hyp, diar_score = asr_diar_offline.run_diarization(cfg, word_ts_hyp)`
`print("Diarization hypothesis output: \n", diar_hyp['an4_diarize_test'])`

An error message stating that the name of the automatically created output folder is too long:
`vad: 100%|██████████| 1/1 [00:01<00:00,  1.42s/it]
[NeMo I 2023-02-07 21:01:01 clustering_diarizer:258] Converting frame level prediction to speech/no-speech segment in start and end times format.
FileNotFoundError: [WinError 206] The filename or extension is too long: 'C:\\Users\\XXXXX\\Desktop\\Audio_files\\data\\vad_outputs\\table_output_tmp_window_length_in_sec0.63_shift_length_in_sec0.01_smoothingFalse_overlap0.5_onset0.9_offset0.5_pad_onset0_pad_offset0_min_duration_on0_min_duration_off0.6_filter_speech_firstTrue_'`

Is there a way to alter the name of this output folder?

With Google Colab in conjunction with Google Drive, the long name of the output folder does not matter and everything is working fine. 

Thank you in advance for your help!
Fix hybridasr bug
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

# What does this PR do ?

Fixes a bug in the tokenizer/data loader of the Hybrid ASR models.

# Changelog 
The dataloader method was missing from the BPE version of the hybrid ASR model.


**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation
bugfix: file handlers are not closed.
State: closed | Pull request: yes
Labels: name: ASR

name: Speaker Tasks

name: common

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TN] bugfix file handler is not closed.
State: closed | Pull request: yes

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

# What does this PR do ?

```
files are not closed so that memory may leak.
```
**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Tn doc 16
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
GPT Eval changes
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

GPT eval changes


**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix new repo links
State: closed | Pull request: yes
Labels: name: NLP

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
TTS model documentation update
State: closed | Pull request: no
Labels: name: documentation

name: TTS

name: stale

TTS model documentation update

**A clear and concise description of what the bug is.**

The Readme's for the currently supported TTS models are outdated. E.g. there is no hint on that now VITS is supported, nor how SSL-Fastpitch is to be used and how it improves/deviates from the normal Fastpitch model. 

**Please list *minimal* steps or code snippet for us to be able to reproduce the bug**

Navigate to the TTS description: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/tts/intro.html. Search for VITS docuementation or SSL Fastpitch.


**Expected behavior**

Thorough documentation of the new language models or how to train and if possible current status and performance metrics.
Move settings to `pyproject.toml`. Remove deprecated `pytest-runner`
State: closed | Pull request: yes

# What does this PR do ?

- move settings from old-style `setup.cfg` to `pyproject.toml` (recommended by most of the tools)
- remove deprecated `pytest-runner` according to the [documentation](https://github.com/pytest-dev/pytest-runner#deprecation-notice)

**Collection**: [Note which collection this PR will affect]

# Changelog 
- create `pyproject.toml` with settings from `setup.cfg` for `isort` and `pytest`
- remove `setup.cfg`
- remove `pytest-runner` from `setup.py`

# Usage
It is not possible anymore to use `python setup.py test` (actually, not used in project). 

`pytest <options>` should be used for testing purpose.


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add file class based inference API for diarization
State: closed | Pull request: yes
Labels: name: ASR

name: Speaker Tasks

# What does this PR do?

Supports:

```python
from nemo.collections.asr.models import NeuralDiarizer

diarizer = NeuralDiarizer.from_pretrained(model_name='diar_msdd_telephonic')

annotation = diarizer("an4_diarize_test.wav")

print(annotation.to_rttm())
```

User can change the device:

```python
import torch
from nemo.collections.asr.models import NeuralDiarizer

diarizer = NeuralDiarizer.from_pretrained(model_name='diar_msdd_telephonic')
diarizer.to(torch.device('cuda'))
```

Supports passing the maximum number of speakers, or the actual number of speakers in a file (oracle):

```python
annotation = diarizer("an4_diarize_test.wav", num_speakers=2)

annotation = diarizer("an4_diarize_test.wav", max_speakers=4)
```

Relies on a temporary directory, which can be specified if preferred:

```python
annotation = diarizer("an4_diarize_test.wav", out_dir="my_temp/")
```

List of available models:

```python
from nemo.collections.asr.models.msdd_models import NeuralDiarizer

print(NeuralDiarizer.list_available_models())
"""
PretrainedModelInfo(
	pretrained_model_name=diar_msdd_telephonic,
	description=For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:diar_msdd_telephonic,
location=https://api.ngc.nvidia.com/v2/models/nvidia/nemo/diar_msdd_telephonic/versions/1.0.0/files/diar_msdd_telephonic.nemo
)
"""
```

### TODO
* Add tests, ensure that config options are passed correctly and updated correctly. The config logic is quite messy due to mutable objects, but this should be saved for a larger refactor.
* Hopefully in the future, `NeuralDiarizer` will be a NeMo PT model file too, save for a later refactor

**Collection**: ASR, Speaker Tasks

# Changelog 
- Add file class based inference API for Diarization.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation
Conformer : Training ASR for multiple language 
State: closed | Pull request: no
Labels: name: bug

name: stale

**What i did**

I took a multilingual SSL pretrained model and wanted to fine-tune it for ASR task on english and hindi together at once. I used a tokenizer that contains both English and Hindi manifest. 

**ERROR** 
After few steps the loss goes nan , detected inf or nan values in gradients! Setting gradients to zero.

**What i tried**
All of the hyper-parameters are kept as mentioned in GitHub repo.
i tried different combination of batch size and gradient accumulation. i tried lowering the learning rate.
____________________________________________________________
**AIM of my experiment**
let say the audio clip says : Aj mene apple khaya
want the transcription to be : आज मेने apple खाया
____________________________________________________________
how can this be achieved. 
Do i need to use the other method of tokenizing , where one make separate tokenizer for languages, and  use "agg" aggregate.
Can you suggest any method to achieve the above problem

An early response is always appreciated.
@titu1994 @meghmak13
Update NeMo Website
State: closed | Pull request: yes

# What does this PR do ?

Add Readme + other content to the NeMo website.

**Collection**: [Core]

# Changelog 
- Add detailed README.md
- Update links to NeMo Megatron
- Update blog posts in 2022


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation
[TTS][ZH] added new NGC model cards with polyphone disambiguation.
State: closed | Pull request: yes
Labels: name: TTS

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

# What does this PR do ?
```
add new chinese NGC model cards that improve with polyphone disambiguation.
```

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add new lannguages to doc
State: closed | Pull request: yes
Labels: name: NLP

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix code scanning alert - Potentially uninitialized local variable
State: closed | Pull request: no
Labels: name: bug

name: TTS

`context` is not assigned `if 'atn' not in self.include_modules and 'dec' not in self.include_modules:`
<!-- Warning: The suggested title contains the alert rule name. This can expose security information. -->

Tracking issue for:
- [x] https://github.com/NVIDIA/NeMo/security/code-scanning/871
Fix code scanning alert - Potentially uninitialized local variable
State: closed | Pull request: no
Labels: name: bug

name: TTS

`attn_logprob` is not guaranteed to be assigned `if 'atn' not in self.include_modules and 'dec' not in self.include_modules:`

<!-- Warning: The suggested title contains the alert rule name. This can expose security information. -->

Tracking issue for:
- [x] https://github.com/NVIDIA/NeMo/security/code-scanning/872
Fix code scanning alert - Potentially uninitialized local variable
State: closed | Pull request: no
Labels: name: bug

name: TTS

`spec_target_len` and `spec_target` will not be assigned if `audio is None`. 

<!-- Warning: The suggested title contains the alert rule name. This can expose security information. -->

Tracking issue for:
- [x] https://github.com/NVIDIA/NeMo/security/code-scanning/885
Fix code scanning alert - Potentially uninitialized local variable
State: closed | Pull request: no
Labels: name: bug

name: TTS

when `audio is None`, `z1, log_s_list, log_det_W_list` can not be assigned even `if self.mode == OperationMode.training or self.mode == OperationMode.validation:`.

<!-- Warning: The suggested title contains the alert rule name. This can expose security information. -->

Tracking issue for:
- [x] https://github.com/NVIDIA/NeMo/security/code-scanning/905
Fix code scanning alert - Potentially uninitialized local variable
State: closed | Pull request: no
Labels: name: bug

name: TTS

there are no assignments for `len_regulated` and `dec_lens` when `else:` is triggered.

<!-- Warning: The suggested title contains the alert rule name. This can expose security information. -->

Tracking issue for:
- [x] https://github.com/NVIDIA/NeMo/security/code-scanning/836
Fix code scanning alert - Potentially uninitialized local variable
State: closed | Pull request: no
Labels: name: bug

name: TTS

there is no assignment to `s` and `log_s` when `else:` is triggered.

<!-- Warning: The suggested title contains the alert rule name. This can expose security information. -->

Tracking issue for:
- [x] https://github.com/NVIDIA/NeMo/security/code-scanning/772
Fix code scanning alert - Potentially uninitialized local variable
State: closed | Pull request: no
Labels: name: bug

name: TTS

Please fix this bug: `affine_params` would be missing when `self.affine_model not in ["wavenet", "simple_conv"]`

<!-- Warning: The suggested title contains the alert rule name. This can expose security information. -->

Tracking issue for:
- [x] https://github.com/NVIDIA/NeMo/security/code-scanning/774
Fix Megatron Bert Broadcast when using Binary Head
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Fixes gpu hang caused by incompatible shapes during loss mean broadcast when binary head is on.  This may not be the cleanest solution but I am making a draft PR for awareness of the issue.

**Collection**: NLP

# Changelog 
- This adds an if statement to broadcast an shape [3] zeros tensor for loss mean reduction when the Binary Head is on.

```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [Bugfix]


# Additional Information
* cc. @shanmugamr1992
Speaker verification with low confidence
State: closed | Pull request: no
Labels: name: stale

Hello

I am using NeMo for my Portuguese language transcriber and I have great results, far superior to other engines, congratulations on the job.
Now I'm working with NeMo to use voice authentication, in my scenario I'm using phone call audio as a base, in 8K and with background noise. I tested all the pre-trained models that you suggest, the "best" being titanet_large and the result from the same people is giving between 50 to 65% confidence. To test I used the code:

`
speaker_model = nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained(model_name='titanet_large')
speaker_model.verify_speakers('/content/PERSON1-AUDIO1.wav', '/content/PERSON1-AUDIO2.wav')
`

And the sample audios (I can't provide more due to privacy concerns):
[NVIDIA.zip](https://github.com/NVIDIA/NeMo/files/10663690/NVIDIA.zip)

As you suggest, I did a finetunning using the same parameters suggested in the training https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb.
I only changed the number of Epochs, placing 250, this training ended after 1 day, the loss started at 13.7 and after 250 Epochs it was at 0.0000508

With this model in hand, I did the same tests, but the result got worse, it only gave high confidence when the person said the same thing said in the other audio. It gave about 90% if both audios had "Hello", for example, otherwise it gave 50%.
For this finetunning, I used approximately 60 hours of phone call audio as a basis, with approximately 1100 different speakers.

I opened an old issue with you about this same problem, but I still didn't see any improvements. I tested what was suggested from 6 seconds upwards and there were no improvements. I'm following titanet's own paper that comments that we should use at least 3 seconds
https://github.com/NVIDIA/NeMo/issues/2719

Now comes my questions:
Is NeMo suitable for my use case? Phone call audios that are in 8K (For training and testing I convert to 16K).
Do I need to test new parameters? What do you suggest for values ​​and what can I adjust.
Does spoken language matter for training? My test/training audios are in Portuguese.
Draft: support interface for p2p communication overlap and using batched p2p communication
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Add an interface to support P2P communication overlap with compute kernels. P2P communication overlap works only with interleaved pipelining.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [*] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
`tensorboard` not available in **tutorials/nlp/Entity_Linking_Medical.ipynb**
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I'm trying to run this notebook: https://github.com/NVIDIA/NeMo/blob/v1.0.2/tutorials/nlp/Entity_Linking_Medical.ipynb
A few pip dependency make problems but doesn't stop the executions, but then later one cell has a problem with the tensorboard module.
This cell:
```py
# Initialize the trainer and model
trainer = Trainer(**cfg.trainer)
exp_manager(trainer, cfg.get("exp_manager", None))
model = nemo_nlp.models.EntityLinkingModel(cfg=cfg.model, trainer=trainer)
```
produces ERROR:
```py
INFO:pytorch_lightning.utilities.rank_zero:Using bfloat16 Automatic Mixed Precision (AMP)
INFO:pytorch_lightning.utilities.rank_zero:GPU available: False, used: False
INFO:pytorch_lightning.utilities.rank_zero:TPU available: False, using: 0 TPU cores
INFO:pytorch_lightning.utilities.rank_zero:IPU available: False, using: 0 IPUs
INFO:pytorch_lightning.utilities.rank_zero:HPU available: False, using: 0 HPUs
[NeMo I 2023-02-06 02:31:00 exp_manager:362] Experiments will be logged at SelfAlignmentPretrainingTinyExample/2023-02-06_02-14-21
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
[<ipython-input-33-8586e3adaa65>](https://localhost:8080/#) in <module>
      1 # Initialize the trainer and model
      2 trainer = Trainer(**cfg.trainer)
----> 3 exp_manager(trainer, cfg.get("exp_manager", None))
      4 model = nemo_nlp.models.EntityLinkingModel(cfg=cfg.model, trainer=trainer)

3 frames
[/usr/local/lib/python3.8/dist-packages/lightning_fabric/loggers/tensorboard.py](https://localhost:8080/#) in __init__(self, root_dir, name, version, default_hp_metric, prefix, sub_dir, **kwargs)
     91     ):
     92         if not _TENSORBOARD_AVAILABLE and not _TENSORBOARDX_AVAILABLE:
---> 93             raise ModuleNotFoundError(
     94                 "Neither `tensorboard` nor `tensorboardX` is available. Try `pip install`ing either."
     95             )

ModuleNotFoundError: Neither `tensorboard` nor `tensorboardX` is available. Try `pip install`ing either.
```

**Steps/Code to reproduce bug**

run cells one by one: https://github.com/NVIDIA/NeMo/blob/v1.0.2/tutorials/nlp/Entity_Linking_Medical.ipynb


**Environment overview**

 - Environment location: Collab
 - Method of NeMo install: 
 ```py
BRANCH = 'main'
!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]
```
Support Alignment Extraction for all RNNT Beam decoding methods
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Adds alignment extraction for all rnnt beam algorithms

**Collection**: [ASR]

# Changelog 
- Adds support for TSD, ALSD and mAES alignment extraction 
- Adds unittest that downloads a small model and runs on AN4 test set to compute alignments from greedy and beam search algorithms

# Usage
You can now submit `preserve_alignments=true` to in the RNNTDecodingConfig for all beam search algos and extract alignments.

```python

import nemo.collections.asr as nemo_asr

filepath = "<audio filepath>"

model = nemo_asr.models.ASRModel.from_pretrained('stt_en_conformer_transducer_large')  # type: nemo_asr.models.EncDecRNNTBPEModel
# Create default decoding config
model.change_decoding_strategy(None)
decoding_cfg = model.cfg.decoding
decoding_cfg.strategy = "beam"  # can be beam, tsd, alsd or maes
decoding_cfg.preserve_alignments = True  # the important argument !
decoding_cfg.beam.beam_size = 8
decoding_cfg.beam.return_best_hypothesis = False

# Prepare beam search decoding strategy
model.change_decoding_strategy(decoding_cfg)

best_hypothesis, all_hypothesis = model.transcribe([filepath], return_hypotheses=True)
print("Num hypothesis per sample :", len(best_hypothesis[0]))
print("Result type :", best_hypothesis[0][0].__class__.__name__)
for candidate in range(len(best_hypothesis[0])):
    print(f"Beam Search {candidate + 1} :", best_hypothesis[0][candidate].text, best_hypothesis[0][candidate].score)

# Use the following commented print statements to check
# the alignment of other algorithms compared to the default
all_hyps = all_hypothesis
for idx, hyp_ in enumerate(all_hyps):  # type: (int, rnnt_utils.Hypothesis)
    print("Hyp index", idx + 1, "text :", hyp_.text)

    for t in range(len(hyp_.alignments)):
        t_u = []
        for u in range(len(hyp_.alignments[t])):
            logp, label = hyp_.alignments[t][u]
            t_u.append(int(label))
        print(f"Tokens at timestep {t} = {t_u}")
    print()

```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Getting acoustic model probabilities form ctc beam search decoder 
State: closed | Pull request: no

Hey, I've been using the beam BeamSearchDecoderWithLM module which returns a combined score of AM + bpe-ngram, and basically, I need acoustic model probabilities for each hyp to use downstream.
I *think* this would be a correct way to obtain them but just wanted to double check

-(kenlm.score(encoded_text) * alpha - score + encoded_length * beta)
which would be using the same ngram used for decoding to re-obtain seperate n-gram probs
gc not imported
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

https://github.com/NVIDIA/NeMo/tree/main/nemo/collections/asr/parts/utils)/speaker_utils.py

In this file gc is not imported, therefore while performing clustering without cuda, its showing error that gc is not imported

**Steps/Code to reproduce bug**

By adding import statement we can fix the bug


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model
remove conda pynini install
State: closed | Pull request: yes

Signed-off-by: ekmb <ebakhturina@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
GPT support for BF16 grad reductions
State: closed | Pull request: yes
Labels: name: core

name: NLP

name: CI

# What does this PR do ?

Adds GPT support for BF16/FP16 gradient reductions, with embedding grad reductions in FP32.

**Collection**: NLP

# Changelog 
- Wrapper for distributed Adam optimizer separately optimizes any parameters that require explicit FP32 gradients, e.g. embeddings.

# Usage

Set the optimizer to `distributed_fused_adam` in the config file:

https://github.com/NVIDIA/NeMo/blob/65c277bcff81023211b2aa4748ae5edb79326b2d/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml#L193

Configure the optimizer with `grad_sync_dtype: bf16`.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Depends on https://github.com/NVIDIA/apex/pull/1575
* Depends on https://github.com/NVIDIA/TransformerEngine/pull/75
Fix Windows bug with save_restore_connector
State: closed | Pull request: yes
Labels: name: core

Signed-off-by: Daniel Egert <degert@nvidia.com>

# What does this PR do ?

Fixes a bug with saving Nemo models using model.save_to()

**Collection**: core

# Changelog 
- Changed some ordering in nemo/core/connectors/save_restore_connector.py

# Usage
To trigger the bug, and check it's fixed:

```python
from nemo.collections.asr.models import ASRModel

model = ASRModel.restore_from(...)

model.save_to(...)    # crashes on Windows without this fix
```

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information

If you look at all of the uses of `with tempfile.TemporaryDirectory()` in save_restore_connector.py, you will see they all adhere to putting the tempfile context outer to the try-finally for the os.chdir call. However, this is not the case for `_handle_artifacts` which leads me to believe this is a bug where someone wrote `_handle_artifacts` without paying attention to the previous design pattern for tempfile context used throughout the same file. Unfortunately, deviating from that design pattern has catastrophic problems on Windows, but once we fix it to work like all the other entries in the same file, everything works correctly again. This is what I have done.
Use  spawn with GPT
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

This PR updates the GPT pretraining script to use 'spawn' multiprocessing.

This removes the CUDA init error that is seen sometimes on clusters but needs some testing.

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update container to 23.01
State: closed | Pull request: yes
Labels: name: CI

Signed-off-by: ericharper <complex451@gmail.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Core]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Merge release main
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)