renwei2024/nemo-github-issues huggingface dataset:
Add legacy_dataset flag to use legacy NeMo dataset path instead of MCore
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins
[pre-commit.ci] pre-commit suggestions
State: closed | Pull request: yes
Labels: name: stale

<!--pre-commit.ci start-->
updates:
- [github.com/pre-commit/pre-commit-hooks: v4.3.0 → v4.5.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.3.0...v4.5.0)
- [github.com/PyCQA/isort: 5.12.0 → 5.13.2](https://github.com/PyCQA/isort/compare/5.12.0...5.13.2)
- [github.com/psf/black: 19.10b0 → 24.3.0](https://github.com/psf/black/compare/19.10b0...24.3.0)
<!--pre-commit.ci end-->

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
chat tn paper update
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
PEFT keeps frozen model in train mode instead of eval mode
State: closed | Pull request: yes
Labels: name: bug

name: NLP

# What does this PR do ?

We have been experiencing poor convergence when running PEFT in FP8. The root cause is because the frozen model is set in evaluation mode, and so the FP8 scaling factors are never updated from their initial value of 1.0.

**Collection**: NLP

# Changelog 
- PEFT keeps frozen model in train mode instead of eval mode

# Usage

Run GPT finetuning, e.g. with the config at https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/tuning/megatron_gpt_finetuning.py.

Enable FP8 support with model.fp8=True.

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information


jenkins
add TE import guards
State: closed | Pull request: yes
Labels: name: core

name: stale

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
AttributeError: module 'nemo.core.classes' has no attribute 'dataset'
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

NeMo version: '1.23.0rc0'

Running on MAC OS <Sonoma 14.3.1> ,  Conda environment 

 import nemo.collections.asr as nemo_asr gives the following error:

---> 31 class ASRDatasetConfig(nemo.core.classes.dataset.DatasetConfig):
     32     manifest_filepath: Optional[Any] = None
     33     sample_rate: int = MISSING

AttributeError: module 'nemo.core.classes' has no attribute 'dataset'




Can you show steps on how you installed NeMo. Chances are you probably installed one NeMo version but using different version code.

NeMo version: '1.23.0rc0' 
I followed the instructions as under Conda :


# clone the repo and install in development mode
git clone https://github.com/NVIDIA/NeMo
cd NeMo
pip install 'nemo_toolkit[all]'

If you're cloning the repo you need to follow the installation from source part of the readme - pip install -e .[asr]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Tutorial BUG
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

1. `GPTDatasetConfig` got unexpected keyword `mmap_bin_files` (can be solved if I install main of Megatron LM instead of megatron-core-r0.5)
2. `GPTDatasetConfig` got unexpected keyword `is_build_on_rank`
3. merge file and vocab file seem to be useless, since `AutoTokenizer` downloads `gpt2` repo anyway.

**Steps/Code to reproduce bug**

follow https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/nemo_megatron/gpt/gpt_training.html

**Environment overview (please complete the following information)**

Cuda 12
Torch 2.1
Megatron-LM 9de386d08770d7296263a590171ace4ae45348ad
NeMo e64b2227b4759665187d061784927d2f9d0868b3

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Can I finetune from a sub-word based model to a character based model ?
State: closed | Pull request: no

Hi. First of all, thank you for your great work!
I 've followed the ASR language finetuning tutorial. In this tutorial, you finetuned character-based model and sub-word based model from English to corresponding model type in Japanese. I have a question, can I finetune from a sub-word base model in English to a character-based model in Japanese ?
 

Yes it's quite easily doable. What you can do is build a sentence piece "char" tokenizer instead of "bpe" or "unigram" - the char spe tokenizer provides a character encoding but with similar API to subword one so model can continue working with just change_vocabulary(). 

The only think to watch out for is if you're using CTC models, then you need to test that your audio length / text length ratio is always about 1.1 or 1.2 (we have a test for this in the eng to japanese tutorial). If not, your model will fail to train. 

Thank you ! Now I'm having a 70-hours Japanese dataset and I want to finetune from Character based ConformerTranducer pretrained on Mandarin. Can you give me some advice about hyper params selection (lr, batch size, warm up steps , ...) ?
fix: fix diarisation pickle error
State: closed | Pull request: yes
Labels: name: stale

name: common

# What does this PR do ?
- Applies suggested change in https://github.com/NVIDIA/NeMo/issues/3421 to fix pickle error when running speaker diarisation

**Collection**: [common]

# Changelog 
- Use defined SpeechLabelEntity class instead of `collections.namedtuple`

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [x] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to [[# (issue)](https://github.com/NVIDIA/NeMo/issues/3421)](https://github.com/NVIDIA/NeMo/issues/3421)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

@kengjichow is this ready for review? 

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Fastconformer loss reduction method for gradient accumulation
State: closed | Pull request: no
Labels: name: stale

Hi, I see that the fastconformer model is trained with mean_volume loss reduction (both for ctc and transducer loss). However, when using gradient accumulation (such as 8), the mean_volume loss is aggregated for each single batch rather for all of the 8 batches. I haven't successfully trained fastconformer transducer so I suspect that this is a possible issue. Do you have any suggestion on changing the reduction method or changing the learning rate when using gradient accumulation? 

In some tutorials I've used grad accumulation 8 without modifying any LR etc and the model is able to trim and converge fine. You need a decently large global batch size, otherwise model will not begin to train unless you start with a pretrained encoder 

yes, I used batch size of 256 with 8 accumulation steps for a total of 2,048. I got good result for the fastconformer CTC model but not yet for the fastconformer transducer. Perhaps there's some other issues. Thanks!

When you get batch size of 256, not necessary to have such large grad accumulation but it does not hurt, just maybe slow down convergence a little bit at the beginning. 
Have you tried hybrid models? 
What do you mean by not good results? You mean wer is worse than CTC?

I haven't tried the hybrid models. I couldn't get the fastconformer transducer's wer close to the Conformer-L Transducer's wer yet. But I'm working with Japanese, which have longer average acoustic length for each character (especially kanji), leading to higher loss when using mean_volume reduction than in other languages. I guess that I might need to tune the learning rate.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
Error in Tacotron2 Training: Reached max decoder steps 1000
State: closed | Pull request: no
Labels: name: bug

name: stale

**Description**:
While training with the default configuration in the NVIDIA NeMo Framework, I encountered the following warning message:

```
[NeMo W 2024-03-31 19:25:05 tacotron2:338 rank:2] Reached max decoder steps 1000.
```

This warning suggests that the decoder reached the maximum number of steps (1000) during training. It's important to investigate why this limit is being reached and whether it impacts the model's performance.

**Steps to Reproduce**:
1. Train Tacotron2 using the default configuration.
2. Observe the warning message in the logs.

**Expected Behavior**:
The decoder should not reach the maximum step limit unless there is a specific reason for doing so. It's crucial to identify the cause and address it appropriately.

**Additional Context**:
LJSpeech-1.1 dadaset

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Missing Support for MultiStepLR in NeMo Framework
State: closed | Pull request: no
Labels: name: bug

name: stale

The NVIDIA NeMo Framework, a powerful generative AI framework, currently lacks support for the MultiStepLR learning rate scheduler, which is available in PyTorch’s torch.optim.lr_scheduler module. This scheduler is commonly used for adjusting learning rates at specific milestones during training.

Expected Behavior: NeMo should include an implementation or provide guidance on how to use MultiStepLR effectively within its training pipelines.

Steps to Reproduce:

Attempt to use MultiStepLR with NeMo conf yaml.
Observe that it is not directly supported.
**Additional Context:** Support for MultiStepLR would enhance the flexibility and adaptability of NeMo’s training process, especially for researchers and developers working on large language models, multimodal models, automatic speech recognition, and text-to-speech synthesis.

how to fix:
add MultiStepLR to https://github.com/NVIDIA/NeMo/blob/main/nemo/core/config/schedulers.py

Please feel free to put a PR with this added lr. 
You can add it directly from pt scheduler similar to other schedulers as shown here: https://github.com/NVIDIA/NeMo/blob/6ca22351ea7dd231dfb94284d141de938cdb2822/nemo/core/optim/lr_scheduler.py#L963 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Huvu/mcore retro eval
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: Speaker Tasks

name: CI

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix import due to recent megatron change
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Fix `ModuleNotFoundError: No module named 'megatron.core.deploy'` error due to [a recent refractor of Megatron-LM](https://github.com/NVIDIA/Megatron-LM/commit/38644dd756c46142787b1bd5dc08ed82ca0e6de1).


# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [X] Did you write any new necessary tests?
- [X] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
update README fix frame_vad_infer_postprocess no have dateset Field 
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

There's no changes in this PR, can we close it?

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
FileNotFoundError: Model stt_zh_citrinet_1024_gamma_0_25 was not found. Check cls.list_available_models() for the list of all available models.
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I try offline_diar_with_asr_infer wish zh asr model 
!python ./clustering_diarizer/offline_diar_with_asr_infer.py \
    diarizer.manifest_filepath="/home/zhibo/Eval_Ali/Eval_Ali_far/audio_dir/temp_manifest_input.json"  \
    diarizer.out_dir='demo_asr_output' \
    diarizer.speaker_embeddings.parameters.save_embeddings=False \
    diarizer.asr.parameters.asr_based_vad=True \
    diarizer.asr.model_path="stt_zh_citrinet_1024_gamma_0_25" 


**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model


I try restory form local print error info :
ValueError: `cfg` must have `tokenizer` config to create a tokenizer !


@weiqingw4ng will submit a PR to fix this. Thank you Weiqing 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.

@weiqingw4ng is working on this, he is working to improve the code for other non-english languages as well. 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Speaker Diarization Inference: cuda=False, using CPU for eigen decomposition when creating model from NeuralDiarizer.from_pretrained()
State: closed | Pull request: no
Labels: name: stale

Hi, im having the same problem as above, 

I tried to create the model with:
```
        model = NeuralDiarizer.from_pretrained('diar_msdd_telephonic', map_location='cuda')
```
```
        model = NeuralDiarizer.from_pretrained('diar_msdd_telephonic', map_location='cuda:0')
```
```
        model = NeuralDiarizer.from_pretrained('diar_msdd_telephonic', map_location='cuda').to('cuda')
```
```
        model = NeuralDiarizer.from_pretrained('diar_msdd_telephonic', map_location='cuda:0').to('cuda:0')
```
and even
```
        model = NeuralDiarizer.from_pretrained('diar_msdd_telephonic').to('cuda:0')
```

On a certain machine (the jupyter notebook on AWS Sagemaker, running on Data Science 3.0, ml.g4dn.xlarge machine), all of these option worked with GPU flawlessly, this only takes 6 seconds for my audio file.

But when i deploy it on AWS Sagemaker Endpoint(with the container [PyTorch 2.0.0 with HuggingFace transformers](763104351884.dkr.ecr.ap-southeast-1.amazonaws.com/huggingface-pytorch-inference:2.0.0-transformers4.28.1-gpu-py310-cu118-ubuntu20.04), running on ml.g4dn.xlarge), Nemo keeps telling me:
```
... speaker_utils:464] cuda=False, using CPU for eigen decomposition. This might slow down the clustering process.
``` 
just as above. This took 112 seconds to complete with the same audio file.

i dont understand why with the same audio file, same code, it could result in two completely different outcomes?

I appreciate any help, thank you in advanced

I'm having this same issue!

The environment you are deploying the NeMo diarization seems like CUDA is not enabled.
If your CUDA setting is not properly set, it will run it on CPU then it will get very slow.
Check `torch.cuda.is_available()` is True in your container environment on the instance you are using.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
Update README.rst
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Added Nvidia Learning  to the readme file just before publications

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines] (https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Get rid of some docs warnings (for title underline and code-block spacing)
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

As per title.

**Collection**: Docs

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Mwawrzos/fix torch inductor pytorch to t
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Support for the latest PyTorch version, which upgraded to triton 3.0.0.

**Collection**: requirements

# Changelog 
- Update requirements,
- Update guards accordingly,
- Update outdated API used by the guards.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

Thanks so the fix, launched e2e testing. I'll let you know the results

jenkins

closing in favor of https://github.com/NVIDIA/NeMo/pull/8805
NeMo upgrade to ToT mcore & ToT TE
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
Convert Llama-7b to nemo: checkpoint ckpts are not saved as a model_weights.ckpt file
State: closed | Pull request: no
Labels: name: bug

name: stale

While I am trying to convert llama-hf model to Nemo via scripts/nlp_language_modeling/convert_llama_hf_to_nemo.py, the checkpoints are saved with each params type as a separate folder, as below
<img width="1284" alt="image" src="https://github.com/NVIDIA/NeMo/assets/157778011/cb83ef47-eb43-4c08-8093-c169c628d8f7">
However, when I am running Nemo-aligner, I was told that a model_weights.ckpt file is required, what is the problem here?

btw: I am using the latest version of Nemo


The converter `convert_llama_hf_to_nemo.py` should produce a `.nemo` file, not a dir. Can you try using NeMo docker image? https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
update README fix frame_vad_infer_postprocess no have dateset Field
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Changes to enable CUDA graph for LLM
State: closed | Pull request: yes
Labels: name: core

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins
Cannot run quantization example
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**
I tried to run the `basic_usage.py` file under `/examples/quantization` and reported an error.

**Steps/Code to reproduce bug**
`python basic_usage.py`

```shell
# error report info
  return self.fget.__get__(instance, owner)()
2024-03-26 12:05:10 INFO [auto_gptq.modeling._base] Start quantizing layer 1/12
2024-03-26 12:05:11 INFO [auto_gptq.modeling._base] Quantizing self_attn.k_proj in layer 1/12...
Traceback (most recent call last):
  File "/root/autoGPTQ/AutoGPTQ/examples/quantization/basic_usage_local.py", line 74, in <module>
    main()
  File "/root/autoGPTQ/AutoGPTQ/examples/quantization/basic_usage_local.py", line 29, in main
    model.quantize(examples)
  File "/usr/local/corex-3.2.1.20231222.40/lib64/python3/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/root/anaconda3/envs/auto_gptq/lib/python3.10/site-packages/auto_gptq/modeling/_base.py", line 368, in quantize
    scale, zero, g_idx = gptq[name].fasterquant(
  File "/root/anaconda3/envs/auto_gptq/lib/python3.10/site-packages/auto_gptq/quantization/gptq.py", line 110, in fasterquant
    H = torch.linalg.cholesky(H)
RuntimeError: cusolver error: Unknown cusolver error number, when calling `cusolverDnCreate(handle)`. If you keep seeing this error, you may use `torch.backends.cuda.preferred_linalg_library()` to try linear algebra operators with other supported backends. See https://pytorch.org/docs/stable/backends.html#torch.backends.cuda.preferred_linalg_library
```


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: `Docker`
 - Method of NeMo install: `BUILD_CUDA_EXT=0 pip install auto-gptq==0.5.1`
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: `Linux aarch64`
- PyTorch version: `2.1.0`
- Python version: `3.10`

**Additional context**

Add any other context about the problem here.
Example: GPU model


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Nemo ASR + NFA + TN + Seq2Seq
State: closed | Pull request: no
Labels: name: stale

Hello!
Has anyone already used technology "[NeMo Forced Aligner](https://nvidia.github.io/NeMo/blogs/2023/2023-08-forced-alignment/), [Text Normalization](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/text_normalization/wfst/wfst_text_normalization.html) and Seq2Seq" in one project?
Which direction should we go? In what order?
Most likely, when normalizing the text, the alignment will not be as accurate

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Test quantization.algorithm=int4_awq in Jenkins
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add **int4_awq** quantization test in Jenkinsfile. 

**Collection**: [NLP]

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.


jenkins

I'm closing this as there are CI issues here and **int4_awq** case will be tested elsewhere
missing EQUAL at '<EOF>', Unable to run sft.
State: closed | Pull request: no
Labels: name: bug

name: stale

Unable to run sft.

first covert  mixtral-8x7B-hf to test2.nemo:
python3 ./scripts/checkpoint_converters/convert_mixtral_hf_to_nemo.py --input_name_or_path mixtral-8x7B-hf --output_path test2.nemo

then 
python ./examples/nlp/language_modeling/tuning/megatron_gpt_finetuning.py \
   trainer.precision=bf16 \
   trainer.devices=8 \
   trainer.num_nodes=1 \
   trainer.val_check_interval=0.1 \
   trainer.max_steps=50 \
   model.restore_from_path= test2.nemo \
   model.micro_batch_size=1 \
   model.global_batch_size=8 \
   model.tensor_model_parallel_size=1\
   model.pipeline_model_parallel_size=8 \
   model.megatron_amp_O2=True \
   model.sequence_parallel=True \
   model.activations_checkpoint_granularity=selective \
   model.activations_checkpoint_method=uniform \
   model.optim.name=distributed_fused_adam \
   model.optim.lr=5e-6 \
   model.answer_only_loss=True \
   model.data.train_ds.file_names=./training.jsonl \
   model.data.validation_ds.file_names=./validation.jsonl \
   model.data.test_ds.file_names=./test.jsonl \
   model.data.train_ds.concat_sampling_probabilities=[1] \
   model.data.train_ds.max_seq_length=2048 \
   model.data.validation_ds.max_seq_length=2048 \
   model.data.train_ds.micro_batch_size=1 \
   model.data.train_ds.global_batch_size=8 \
   model.data.validation_ds.micro_batch_size=1 \
   model.data.validation_ds.global_batch_size=8 \
   model.data.test_ds.micro_batch_size=1 \
   model.data.test_ds.global_batch_size=256 \
   model.data.train_ds.num_workers=0 \
   model.data.validation_ds.num_workers=0 \
   model.data.test_ds.num_workers=0 \
   model.data.validation_ds.metric.name=loss \
   model.data.test_ds.metric.name=loss \
   exp_manager.create_wandb_logger=False \
   exp_manager.explicit_log_dir=/results \
   exp_manager.resume_if_exists=True \
   exp_manager.resume_ignore_no_checkpoint=True \
   exp_manager.create_checkpoint_callback=True \
   exp_manager.checkpoint_callback_params.monitor=validation_loss \
   exp_manager.checkpoint_callback_params.save_best_model=False \
   exp_manager.checkpoint_callback_params.save_nemo_on_train_end=True \
   exp_manager.checkpoint_callback_params.mode=min
Error parsing override 'test2.nemo'
missing EQUAL at '<EOF>'
See https://hydra.cc/docs/1.2/advanced/override_grammar/basic for details

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
ImportError: cannot import name 'CTCDecodingConfig' from 'nemo.collections.asr.parts.submodules' 
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

when I run speech_to_text_with_vad.py with some parameters below

!python /Users/whs/PycharmProjects/NeMo/examples/asr/asr_vad/speech_to_text_with_vad.py \
    manifest_filepath=/Users/whs/PycharmProjects/ava-speech-downloader/temp_manifest_input.json \
    vad_model=vad_multilingual_frame_marblenet \
    vad_config=../conf/vad/frame_vad_infer_postprocess.yaml

**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version  mac 14.2.1 
- PyTorch version 
- Python version 3.11

**Additional context**

Add any other context about the problem here.
Example: GPU model


Which version of NeMo is installed? If you're using NeMo 1.23, you need to change branch to r1.23.0 before selecting and using the script 

> Which version of NeMo is installed? If you're using NeMo 1.23, you need to change branch to r1.23.0 before selecting and using the script

I try install  pip3 install 'nemo_toolkit[asr]==1.23.0'  print errer info :

INFO: pip is looking at multiple versions of nemo-toolkit to determine which version is compatible with other requirements. This could take a while.
ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
ERROR: Could not find a version that satisfies the requirement triton (from nemo-toolkit) (from versions: none)
ERROR: No matching distribution found for triton

I am using an M series mac which does not support triron.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.