renwei2024/nemo-github-issues huggingface dataset:
Add Bert HF checkpoint converter
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add BERT HF checkpoint converter and HF Bert Support in NeMo

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

jenkins
GPU memory overhead when calling trainer.fit the second time
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

Reopen the issue https://github.com/NVIDIA/NeMo/issues/7833 

**Steps/Code to reproduce bug**

when calling `trainer.fit` the second time to train a NeMo model like:
```
trainer.fit(model)
trainer.fit_loop.epoch_loop.max_steps += 10
trainer.fit(model)
```
The second time would use more GPU memory than the first.

**Steps/Code to reproduce bug**
Create a `MegatronGPTModel` with `megatron_amp_O2=True` or with `distributed_fused_adam` and then use the above code for training.

When we print `torch.cuda.memory_allocated()` at the training step, we will find the GPU memory usage at the second `trainer.fit` is significantly higher.

**Expected behavior**

We expect the second `trainer.fit` should have the similar GPU memory usage to that in the first `trainer.fit`.

**Environment details**

Check the official Nemo v1.20 docker and NeMo v1.22 branch, both have this issue.


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
fix device setting to allow using accelerator cpu
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

It is currently impossible to run wav2vec with `trainer.accelerator=cpu` if the machine has a gpu. 
The padding_mask is already on the device set in `trainer.accelerator` but if the machine has a GPU the DEVICE variable will contain cuda and this error message will be received:

```
File /mnt/NVM/oren/anaconda3_10/envs/nemo/lib/python3.10/site-packages/nemo/collections/asr/modules/wav2vec_modules.py:348, in Wav2VecTransformerEncoder.create_padding_mask(self, length)
    345 padding_mask = torch.arange(max_len, device=DEVICE)
    347 # Switch to binary for transformer, 1 for valid tokens, 0 for padding
--> 348 padding_mask = (padding_mask.expand(len(length), max_len) < length.unsqueeze(1)).type(torch.uint8)
    350 return padding_mask

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

```
  
**PR Type**:
- [x] Bugfix


# Additional Information
* Related to # (issue)
* 
@titu1994, @redoctopus, @jbalam-nv, or @okuchaiev

jenkins

Thanks @titu1994 , see also here:
https://github.com/NVIDIA/NeMo/issues/8085

jenkins 

@titu1994 i know you pushed but just for record I'm not seeing any issues so review is approved

Oh thanks for the update, I think I misremembered as this having your approval on slack somehow, turns out it was a different topic different pr. Sorry about that. 

do you @titu1994  or @tbartley94 have any idea about this one:
https://github.com/NVIDIA/NeMo/issues/8085

Hmm, lemme take a look. I'd wager it's some tech debt between PTL versions. (fervor for wav2vec dipped before PTL 2.0 was released, so the code is likely working off a dated namespace.)
How to improve ASR speed, Tested on Nvidia T4, L4, A100 give me same speed
State: closed | Pull request: no
Labels: name: stale

Hello, 

I want to know how to speed up NEMO.

Originally, I ran test with **[stt_zh_citrinet_1024_gamma_0_25](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_zh_citrinet_1024_gamma_0_25)** model with 30 seconds audio clip on Nvidia T4,
It give me around 1 - 2 seconds. It too slow for our requirements.

Then, I ran test on both Nvidia L4 and A100 which is more powerful than T4, 
But it still give me same speed (1-2 seconds), 

So, How to improve ASR speed?  Maybe I did something wrong. Please advise

PS. I'm sure CUDA is enabled, Using CPU give me more than 10 seconds

python = 3.10
nemo_toolkit = 1.21
cuda = 12.1
debian = 11

Can you check if your are loading the model every time ? 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Online Diarizer
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

The online diarizer is not working a excpected. when trying it on a file with two speakers it only outputs one as active.
**Steps/Code to reproduce bug**
creating a class instance of : https://github.com/NVIDIA/NeMo/blob/274a21b492f337702f83f7b82abca338d6c9c09e/nemo/collections/asr/models/online_diarizer.py 
and diarize().

There is no wraper code for the class so it might be the case.

**Environment overview (please complete the following information)**

 - Environment location: DOCKER
 - Method of NeMo install: source

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version : UBUNTU 20.04
- PyTorch version 1.12.1+cu116
- Python version 3.8.10

**Additional context**

Add any other context about the problem here.
Example: GPU model RTX A6000


Hi.

Please understand that speaker diarization is a type of pattern recognition system, 
so it does not have 100% accuracy on any dataset and should not be regarded as a bug. 
There is a chance of errors in terms of speaker labels or speaker counts.

However, you can try changing [self.min_frame_per_spk](https://github.com/NVIDIA/NeMo/blob/274a21b492f337702f83f7b82abca338d6c9c09e/nemo/collections/asr/parts/utils/online_clustering.py#L606C9-L606C26) variable which controls "minimum required number of segments to count a speaker" value.
Lowering this value (below default 15)  will increase the chance of counting more speakers.
Check out this `min()` operation at [This line](https://github.com/NVIDIA/NeMo/blob/274a21b492f337702f83f7b82abca338d6c9c09e/nemo/collections/asr/parts/utils/online_clustering.py#L724).

We implemented this threshold because it is very challenging to count the number of speakers in a short audio (less then 15 sec per speaker)

 
 

Hi, Thanks for the insights!
I tried it and still got one speaker only. is there an example config file for online clustering? 

Yoav

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
update broken links
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

name: NLP

# What does this PR do ?

Fix Broken Links

**Collection**: Docs

# Changelog 
- update rst files

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation
update broken links
State: closed | Pull request: yes
Labels: name: ASR

name: NLP

# What does this PR do ?

Fix Broken Links

**Collection**: Docs

# Changelog 
- update rst files

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

jenkins
run with non-dev option
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

# What does this PR do ?

Looks like pip installs run with -e option, don;t load well without restarting. 

**Collection**: [TUTORIALS]

# Changelog 
- Run without pip editable

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

jenkins
run with non-dev option
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Looks like pip installs run with -e option, don;t load well without restarting. 

**Collection**: [TUTORIALS]

# Changelog 
- Run without pip editable

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
RuntimeError: cusolver error: CUSOLVER_STATUS_NOT_INITIALIZED, when calling `cusolverDnCreate(handle)
State: closed | Pull request: no
Labels: name: bug

name: stale

I am facing 
`RuntimeError: cusolver error: CUSOLVER_STATUS_NOT_INITIALIZED, when calling cusolverDnCreate(handle)
`

I am getting this issue while running the msdd_model.diarize() function, I have checked my GPU and it has enough space to run the file.

I saw the same issue in https://github.com/PanQiWei/AutoGPTQ/issues/478, but in my case there is no AutoGPTQ. It is occuring when I run the line msdd_model.diarize().
My Cuda and Pytorch versions are these:
Cuda version 11.7 (Cuda compilation tools, release 11.7, V11.7.99)
Pytorch Version -2.0.1+cu117


A clear and concise description of what the bug is.

Description
I encountered a RuntimeError: cusolver error: CUSOLVER_STATUS_NOT_INITIALIZED when calling cusolverDnCreate(handle) while running the msdd_model.diarize() function.

Steps to Reproduce

from nemo.collections.asr.models import NeuralDiarizer
from nemo.collections.asr.models.msdd_models import create_config

msdd_model = NeuralDiarizer(cfg=create_config(temp_path)).to("cuda")

msdd_model.diarize()


**Expected behavior**

I expected the msdd_model.diarize() function to run without errors.


**Environment overview (please complete the following information)**

Environment Location: JupyterHub
Method of NeMo Install: pip install
Commands Used for Installation: `!pip install --no-build-isolation nemo_toolkit[asr]==1.21.0 --user`
OS Version: Linux 5.4.254-170.358.amzn2.x86_64
PyTorch Version: 2.0.1+cu117
Python Version: 3.10
GPU Model: NVIDIA A10G

**Environment details**
Ensure that the GPU has enough space.
CUDA Version: 11.7 (Cuda compilation tools, release 11.7, V11.7.99)

**Additional context**

Additional Context
I saw a similar issue reported in [GitHub Issue #478](https://github.com/PanQiWei/AutoGPTQ/issues/478), but my case does not involve AutoGPTQ. The error occurs specifically when running the msdd_model.diarize() function.


Using `torch.backends.cudnn.enabled = False` before model.diarize() solves the issue temporarily


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
update reqs
State: closed | Pull request: yes
Labels: name: cherry-pick

name: CI

# What does this PR do ?

Update requirements

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
update reqs
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Update requirements

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Update HF Transformers version
State: closed | Pull request: yes

# What does this PR do ?

Updates HF version

**Collection**: [Common]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation


jenkins
fix for missing mcore arg
State: closed | Pull request: yes

# What does this PR do ?

Mcore refactor introduced an error when trying to merge.
```
Loading distributed checkpoint with TensorStoreLoadShardedStrategy
[NeMo I 2023-12-13 12:13:12 nlp_overrides:752] Model MegatronGPTModel was successfully restored from /workspace/models/nemo_version/llama2-7b.nemo.
Error executing job with overrides: []
Traceback (most recent call last):
  File "/workspace/NeMo/scripts/nlp_language_modeling/merge_lora_weights/merge.py", line 198, in main
    lora_model_cfg = MegatronGPTLoRAModel.restore_from(
TypeError: NLPModel.restore_from() got an unexpected keyword argument 'mcore'
```
This PR is a patch for this error.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

Addressed in #8113 so closing this
Handle in HF AutoTokenizer with pad_token=None
State: closed | Pull request: yes
Labels: name: common

# What does this PR do ?

In the NML data preprocessing script `scripts/nlp_language_modeling/preprocess_data_for_megatron.py` 
the builder the pad_id thru:
`pad_id=tokenizer.pad_id if hasattr(tokenizer, "pad_id") else 0`

While HF wrapper AutoTokenizer class has `pad_id` as a property, but doesn't actually have pad_token, which will throw None error.
This pr checks if special token exists before encode to get ID of special token.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

Is it possible to reopen the pr?

jenkins

@yaoyu-33 Could you review the resolved issue and say jenkins again? Thanks!!

jenkins

jenkins

jenkins
[Fix] Increase length check tolerance to prevent test failing
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

This PR relaxes constrain on the length to prevent intermittent test failures.

**Collection**: ASR tests

# Changelog 
- Increased threshold for 1 to 2

# Original test failure
* You can potentially add a usage example below

```python
=========================== short test summary info ============================

FAILED tests/collections/asr/decoding/test_rnnt_decoding.py::TestRNNTDecoding::test_beam_decoding_preserve_alignments[beam_config4] - AssertionError: assert tensor(2) <= 1

 +  where tensor(2) = abs((31 - tensor(33)))
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Various fixes for typos and urls
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

name: NLP

# What does this PR do ?

**Collection**: [ASR, NLP, TTS]

# Changelog 
- See PR https://git

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Fix various issues with broken links and bugs
State: closed | Pull request: yes
Labels: name: TTS

name: cherry-pick

name: ASR

name: NLP

# What does this PR do ?

**Collection**: [ASR, NLP, TTS]

# Changelog 
Fixes bugs and typos in
- https://nvbugspro.nvidia.com/bug/4413845
- https://nvbugspro.nvidia.com/bug/4411322
- https://nvbugspro.nvidia.com/bug/4410625
- https://nvbugspro.nvidia.com/bug/4407526
- https://nvbugspro.nvidia.com/bug/4405781
- https://nvbugspro.nvidia.com/bug/4426768

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Fix various issues with broken links and bugs
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

name: NLP

# What does this PR do ?

**Collection**: [ASR, NLP, TTS]

# Changelog 
Fixes bugs and typos in
- https://nvbugspro.nvidia.com/bug/4413845
- https://nvbugspro.nvidia.com/bug/4411322
- https://nvbugspro.nvidia.com/bug/4410625
- https://nvbugspro.nvidia.com/bug/4407526
- https://nvbugspro.nvidia.com/bug/4405781
- https://nvbugspro.nvidia.com/bug/4426768

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Use NLPDDPStrategyNotebook in Multitask_Prompt_and_PTuning.ipynb
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Use NLPDDPStrategyNotebook in Multitask_Prompt_and_PTuning.ipynb
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Critical CVE in `transformers` in versions < `4.36.0`
State: closed | Pull request: no
Labels: name: stale

The `transformers` requirement is bounded here https://github.com/NVIDIA/NeMo/blob/39d883f16ce57f06c40be19a99c881324b93360d/requirements/requirements_lightning.txt up to version `4.33.3`

There is a Critical CVE affecting versions < `4.36.0`, related information
- https://www.cve.org/CVERecord?id=CVE-2023-6730
- https://github.com/huggingface/transformers/commit/1d63b0ec361e7a38f1339385e8a5a855085532ce
- https://huntr.com/bounties/423611ee-7a2a-442a-babb-3ed2f8385c16/

This also would solve https://github.com/NVIDIA/NeMo/issues/7765 and https://github.com/NVIDIA/NeMo/pull/7620#issuecomment-1745874618


Thanks for letting us know. @athitten could you take a look if we can safely unpin Transformers ? Fyi @ericharper 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

Any news related to this issue?

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
How to reproduce the throughput in the blog
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
Refer to this blog, training LLaMA 13B with NeMo 23.08 could achieve 2357 Tokens/s/GPU
![image](https://github.com/NVIDIA/NeMo/assets/73142299/da55a7eb-0992-440c-9c32-4aba21150d99)

Yet, I can only achieve under 1900 Tokens/s/GPU
![image](https://github.com/NVIDIA/NeMo/assets/73142299/4f5a5626-2e8b-42aa-a4f3-f14ba4973d20)


**Steps/Code to reproduce bug**
I'm running on nvcr.io/nvidia/nemo:23.08 with 8xA100
Here is my launching script
```
INDEX_MAPPING_DIR=""
DATA_PREFIX=""
TOKENIZER_DIR=""
TP_SIZE=4
PP_SIZE=1

torchrun --nproc_per_node=8 \
/workspace/nemo/examples/nlp/language_modeling/megatron_llama_pretraining.py \
   trainer.precision=bf16 \
   trainer.devices=8 \
   trainer.num_nodes=1 \
   model.num_layers=40 \
   model.hidden_size=5120 \
   model.ffn_hidden_size=13824 \
   model.num_attention_heads=40 \
   model.num_query_groups=40 \
   model.micro_batch_size=1 \
   model.global_batch_size=32 \
   model.tensor_model_parallel_size=${TP_SIZE} \
   model.pipeline_model_parallel_size=${PP_SIZE} \
   model.megatron_amp_O2=False \
   model.sequence_parallel=True \
   model.activations_checkpoint_granularity=selective \
   model.activations_checkpoint_method=uniform \
   model.optim.name=distributed_fused_adam \
   model.optim.lr=2e-4 \
   model.tokenizer.model=${TOKENIZER_DIR} \
   model.data.index_mapping_dir=${INDEX_MAPPING_DIR} \
   ++model.data.data_prefix=${DATA_PREFIX} \
   exp_manager.create_wandb_logger=False \
   exp_manager.explicit_log_dir=/results \
   exp_manager.resume_if_exists=False \
   exp_manager.resume_ignore_no_checkpoint=True \
   exp_manager.create_checkpoint_callback=True \
   exp_manager.checkpoint_callback_params.monitor=validation_loss \
   ++cluster_type=BCP
```
The `megatron_llama_pretraining.py`
```
import torch.multiprocessing as mp
from omegaconf.omegaconf import OmegaConf, open_dict

from nemo.collections.nlp.models.language_modeling.megatron_gpt_model import MegatronGPTModel
from nemo.collections.nlp.parts.megatron_trainer_builder import MegatronTrainerBuilder
from nemo.core.config import hydra_runner
from nemo.utils import logging
from nemo.utils.exp_manager import exp_manager

mp.set_start_method("spawn", force=True)


@hydra_runner(config_path="conf", config_name="megatron_llama_config")
def main(cfg) -> None:
    logging.info("\n\n************** Experiment configuration ***********")
    logging.info(f'\n{OmegaConf.to_yaml(cfg)}')

    trainer = MegatronTrainerBuilder(cfg).create_trainer()
    exp_manager(trainer, cfg.exp_manager)

    model = MegatronGPTModel(cfg.model, trainer)

    trainer.fit(model)


if __name__ == '__main__':
    main()
```


**Expected behavior**
Do I have to modify the configuration to get better performance? If so, could you show the config🥺?


I managed to reproduce🙃
Fixing wrapper and moving it to base class
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

1) Move wrapper to base class
2) Need to pass in args[module] = self.model to bert model 

**Collection**:  NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
[SE Tutorial] USe GPU for inference, when available
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Fixes a bug where inference on CPU was segfaulting.

**Collection**: Tutorials

# Changelog 
- Moved tensor to GPU when possible

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix crash when old checkpoint does not exist anymore
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

This prevents NeMo from crashing when trying to delete a checkpoint that has already been deleted.

This situation may happen for instance after manually fixing corrupted checkpoints.

Note that I believe PTL also has a similar safety guard: https://github.com/Lightning-AI/lightning/blob/119039b932c79f1e5daef67345628e675ab3c0bf/src/lightning/fabric/plugins/io/torch_io.py#L94

**Collection**: nlp

# Changelog 
- Fix crash when attempting to delete an old checkpoint that does not actually exist anymore

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

Re-opening, can someone please help review this minor change?

+1 on this issue

Closing as this issue has been addressed in #8130
Broken links in main README (leading to NVIDIA docs)
State: closed | Pull request: no
Labels: name: bug

Links in README lead to docs on NVIDIA website, but at least some of them seem to be broken. I checked the NLP section, specifically these two:

NLP
[NeMo Megatron pre-training of Large Language Models](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/intro.html)
[Neural Machine Translation (NMT)](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/machine_translation/machine_translation.html)

and the links send me to a page with this error:

An error occurred while processing your request.
Reference #30.4cf01202.1702990306.31b50b8c

Worth fixing the links

The docs should be up and running again, I'll inform the team about the broken links. 

Links are updated, pls let us know if you find any issues.
crash when run gpt with ub_tp_comm_overlap
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

crash when run gpt with ub_tp_comm_overlap
![image](https://github.com/NVIDIA/NeMo/assets/30709861/d15b8c2c-f266-4990-a71f-ebe1e1b9a6b0)


**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**
PyTorch Docker: nvcr.io/nvidia/pytorch 23.10-py3

**Additional context**

Add any other context about the problem here.
Example: GPU model


I found that enabling ub_tp_comm_overlap will create a new mpi process group, which will cause problems. I don't know if it has something to do with the mpi version.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
[SE Tutorial] USe GPU for inference, when available
State: closed | Pull request: yes

# What does this PR do ?

Fixes a bug where inference on CPU was segfaulting.

**Collection**: Tutorials

# Changelog 
- Moved tensor to GPU when possible

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Temporary pin Lightning-Utilities version due to broken NamedTuple
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Pin Lightning-Utitilies package version due to broken NamedTuple in `0.10.0`
https://github.com/Lightning-AI/utilities/pull/160 introduced the issue, 
https://github.com/Lightning-AI/utilities/blob/a00e1144dd300e3bc2fc8746a8e85510ffe35aa7/src/lightning_utilities/core/apply_func.py#L67 here NamedTuple force converted to tuple.

This fixes the bug https://nvbugspro.nvidia.com/bug/4405798 (broken `ASR_TTS_Tutorial`)

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

I think that this is not needed in main, since lightning-utilities has released fixed version https://github.com/Lightning-AI/utilities/releases/tag/v0.10.1
I'm closing this PR
cc: @titu1994, @ericharper
Nemo Documentation User Guide doesn't work 
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Nemo Documentation User Guide doesn't work 

Link:
https://docs.nvidia.com/deeplearning/nemo/user-guide/index.html

Output Message:
An error occurred while processing your request.
Reference #30.3ff01202.1702902642.32d3d058



Thanks for letting us know, we're looking into it. 

This should be fixee now. Thanks for letting us know !
fix gated_linear_unit bug
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix the issue when running Llama2 eval or LLama2 SFT.
```
megatron.core.dist_checkpointing.core.CheckpointingException: Global shape mismatch for loaded ((32, 22016, 4096)) and expected ((32, 11008, 4096)) tensor for key model.decoder.layers.mlp.linear_fc1.weight
```

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

merging this as this is critical to unblock any llama2 workflows
Unexpected Space Removal Before Punctuation in ASR Model Decoding
State: closed | Pull request: no
Labels: name: bug

name: stale

Hello,

I've come across an issue in the `nemo/collections/asr/metrics/wer.py` file, specifically within the `decode_hypothesis` function. There's a particular segment of code, marked for future removal (noted with a TODO comment) at line 546, which I believe should be eliminated.

This issue becomes apparent during the training of models that include punctuation marks as separate tokens, mirroring the ground-truth data which incorporates spaces preceding these punctuation marks. The problematic code collapses these spaces during decoding, resulting in significantly higher WER and loss calculations due to mismatches in space formatting. This code segment appears redundant when training models without punctuation marks, as it has no effect in such scenarios. It only impacts models trained with punctuation in the specific context I've outlined.

For instance, consider the ground-truth data: `This is a test , with some extra spaces .` and its prediction: `This is a test, with some extra spaces.` This alters the prediction by removing spaces before the punctuation marks.

Here's the code snippet:
`# TODO: remove`
` # collapse leading spaces before . , ? for PC models`
` hypothesis = re.sub(r'(\s+)([\.\,\?])', r'\2', hypothesis)`
`
Could you please confirm if you're aware of this and if there are plans to remove this section of code?

Thank you

We are aware of this, hence the Todo to remove it. But our condition for removal is that we update all the PC models we release such that they do not have spaces between special tokens. So the removal won't have an effect at that point since all the models will anyway emit no space between special tokens. 

The simplest solution to this problem is to collapse the space between special tokens in both the ground truth and the predicted models, rather than keep the spaces and anyway get incorrect metrics during inference. 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
The 'get_commonvoice_data.py' script is unable to download dataset from common voice
State: closed | Pull request: no
Labels: name: stale

The script is being used in the tutorial `Fine-tuning CTC Models on New Languages (https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/asr/ASR_CTC_Language_Finetuning.ipynb#scrollTo=Inwx4OE97guu)` to download the Japanese dataset from Mozilla's Common Voice. However, the download link to the script is no longer valid (I opened it on my browser) and it throws the error `ERROR 404: Not Found.` in colab. I also discovered that the download links generated for the common Voice datasets are not permanent links.
What can be done about this? 
![common_voice_error](https://github.com/NVIDIA/NeMo/assets/52848633/2539911b-aeca-4908-9327-d7a839b09379)


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.