renwei2024/nemo-github-issues huggingface dataset:
Update docs: add "NeMo Fundamentals" page
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
conformer_ctc_bpe.yaml, I am getting high wer error rate above 90% , I am finetuning stt_en_conformer_small  model using common voice hindi 20 recorded hours dataset...
State: closed | Pull request: no
Labels: name: stale

i follow the , https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/examples/kinyarwanda_asr.html this example..

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
More Informative Performance Callback Data - NemoMegatron GPT
State: closed | Pull request: no
Labels: name: stale

Made some updates to the Docker image to support Pytorch NGC 23.10 so that i can consume the performance updates in NCCL 2.19.3. I am running some benchmarks with a prepared NemoMegatron 7B w/ Wikipedia and was wondering how i get a more representative performance callback data.
For example
```
Epoch 0: :   3%|▎         | 9000/301500 [1:42:42<55:37:46, v_num=0-56, reduced_train_loss=nan.0, global_step=9e+3, consumed_samples=6.3e+5, train_step_timing in s=0.615, val_loss=7.040]
Epoch 0, global step 9000: 'val_loss' reached 7.04238 (best 7.04238), saving model to '/data/nemo_megatron/nemo_experiments/megatron_gpt/2023-11-29_23-30-56/checkpoints/megatron_gpt--val_loss=7.04-step=9000-consumed_samples=630000.0.ckpt' as top 3
```
Wondering how i can get MFU or tokens/sec from this. In the legacy Megatron-LM implementation it had forward-compute/backward-compute and collective timings reported per step.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.

Hi!

Any update on this issue? I'm interested at least on logging `tokens_per_sec` to be able to compare performance between different configs.

Thanks,

Toni
multimodal/speech_cv collection update (Audio Visual Speech Recognition)
State: closed | Pull request: yes
Labels: name: stale

name: ASR

name: common

# What does this PR do ?

Update the multimodal/speech_cv branch to support audio-visual speech recognition (CTC, RNN-T and Hybrid)

**Collection**: [multimodal/speech_cv]

# Changelog 
- Update speech_cv branch
- bug fix
- Add audio-visual datasets
- Add audio-visual models: (AudioVisualEncDecCTCModelBPE, AudioVisualEncDecCTCModel, AudioVisualEncDecHybridRNNTCTCBPEModel, AudioVisualEncDecHybridRNNTCTCModel, AudioVisualEncDecRNNTBPEModel, AudioVisualEncDecRNNTModel)
- Add audio-visual modules (FusionModule, InterCTCMixin)
- Optional ConformerEncoder module 'pre_encode' by setting 'feat_in' to None

# Usage
* You can potentially add a usage example below

```python
import nemo.collections.multimodal.speech_cv as nemo_cv
from omegaconf import OmegaConf

config_path = "model_config.yaml"
params = OmegaConf.load(config_path)

# hybrid CTC/RNN-T BPE audio-visual speech recognition model
model = nemo_cv.models.AudioVisualEncDecHybridRNNTCTCBPEModel(cfg=params.model)
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
asr.transcribe() freezes inside multiprocessing class - rabbitmq queues
State: closed | Pull request: no
Labels: name: stale

Hi everybody, 

I have an application that communicates through rabbitmq queues. I am using a class that inherits multiprocessing.Process:
`class ThreadedConsumer(multiprocessing.Process)`

In this class, I would like to instantiate 3 processes, where each process transcribes an input file. 

Now, in order to use as little memory as possible, I would like to only create the 'asr_model' once, in the constructor of the ThreadedConsumer class, and each process to share the same model.

I have the following problem: when I create the self.asr_model inside the __init__ class, when i do asr.transcribe() inside the callback (when I receive a message) everything freezes. (probably a thread gets latched or something) When I create the self.asr_model each time I receive a new message in the queue, everything works fine. However, in this way I would have 3 models loaded into the memory.. I do not understand why declaring the model inside the constructor results in the pipeline getting stuck. 

Could you please help me with an idea for model sharing, is it possible for ASR models? I know that it would be a solution to use a model with batch_size=3, however once new message come into the queue, I would like another queue that's free to transcribe them and not continously wait for 3 messages.

If anybody has any insights or ideas on how to share the model/ use transcriber with multiprocessing.Process class, I would greatly appreciate any help!

Thank you in advance,
Gabi

@titu1994 have you worked something similar before? 

Is your model on Cuda ? 

Cuda does not work with multi threading, only with multi processing. It simply does not support instantiating two separate cuda contexts (that's why all data loaders are multi process) on the same process. That can be one reason why it's hanging. 

By default, ASR transcribe does not use worker processes. But it does use a PT data loader with 0 workers. 

I haven't implemented the queueing step but I have simply Fast API based inference pipeline that loads a single model and keeps it in memory calling transcribe one by one with async coroutine. But if you use multiple workers then your model gets replicated due to guvicorn. 

What about rearchitecing your solution a bit ? Have two async coroutines - one that queues your input audio and one that runs your model ? Don't put your model on a separate process, keep it on main process, but keep the queueing of input output only via rabbitmq under async coroutines 

Personally I've never seen a single process class used to launch 3 processes. 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Unfinished checkpoints handling
State: closed | Pull request: yes
Labels: name: core

# What does this PR do ?

Handle incomplete checkpoints.

With this fix, if last checkpointing operation was interrupted for some reason and left incomplete data, training should be resumed from the last available full checkpoint, instead of trying to read the incomplete checkpoint and crashing. This should simplify training auto-resume logic.

Implementation overview:
Create "unfinished" marker file for a checkpoint that is being written or removed. 
The marker file is deleted after the checkpoint IO operation completes.
When resuming, checkpoints marked as unfinished are ignored and subsequently removed.

**Collection**:
Core

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [X] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

## Who can review?

@ericharper, @titu1994, @blisc, @okuchaiev

jenkins

Hi @titu1994,

Thank you for the review! 
I added the docstrings and comments.

Regarding your question: If all checkpoints gets filtered out in Exp Manager, then depending on `resume_ignore_no_checkpoint` option  - training starts from scratch (resume_ignore_no_checkpoint=True) or raises NotFoundError (resume_ignore_no_checkpoint=False).

@jbieniusiewi thanks for the PR. PTL 2.1 creates a symlink for the last checkpoint so we wouldn't have an issue where saving last checkpoint is incomplete. Would this PR be still relevant with 2.1 ?
CC @titu1994 @ericharper 

Hi @athitten
I think some parts of this PR might be still relevant even with symlinks to the latest checkpoint introduced in TPL 2.1 E.g., incomplete checkpoints cleanup. Certainly this PR will need to be reviewed for PTL 2.1. 

 

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

jenkins

jenkins

@titu1994 is this good to be merged or do you have any comments to be resolved ?

This will need to be rechecked for PTL 2.1 release when we update
Enable pipelined tp comm overlap for mcore path
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Enable pipelined tp comm overlap for mcore path

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


MCore code path ignores config of enabling or disabling TE, so this LGTM. Thanks.

jenkins

jenkins

jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Draft: SFT Jinja Templating
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Adds support for Jinja2 templates for input prompt and chat prompts.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
The training time for SpeakerNet is too long
State: closed | Pull request: no
Labels: name: bug

Hi,
 I am training a SpeakerNet, and my process is as follows:
Step 1: In the `SpeakerNet_verification_3x2x256.yaml` file,
   - Set `train_ds manifest_filepath` to `path/to/my/trainset`.
   - Set `validation_ds manifest_filepath` to `path/to/my/trainset`.
   It's worth noting that `train_ds` and `validation_ds` have the same `manifest_filepath` because I found that not doing so results in errors.

Step 2: Execute `speaker_reco.py`.

I noticed that the program spends a considerable amount of time in the `Sanity Checking DataLoader` and also in the `Validation DataLoader` stage. The program estimates that it will take 66452 hours to complete one epoch. I believe there is some issue, and I'm wondering if anyone can provide me with advice on this matter.

Tool version
python 3.10.13
NeMo 1.20.0
pytorch-lightning 1.9.4
torch 2.1.0



I found that setting `num_sanity_val_steps: 0` and` limit_val_batches: 0.0` allows the training stage to run without the `Sanity Checking DataLoader"`and `Validation DataLoader`. However, I still want to know if there is a better way to handle this, as I believe a time requirement of 66452 hours is not a normal value.

Hi, its not a good idea to have same manifest for training and validation. The requirement is to have subset of speaker classes from training manifest. So prepare validation manifest accordingly. 

Training time depends on multiple factors: dataset size, batchsize, GPU, number of GPUs, utterance length of each sample etc. 

Hi, @nithinraok 

Thank you for letting me know about the validation requirement. 

Although the training time depends on many factors, my training stage takes about 1 day, while the validation stage takes 66,452 hours. 

I believe there is a problem here, but I don't know how to deal with it. 

Besides my current practice of skipping the validation stage, do you have any other suggestions?

My suggestion is to have 10% of training hrs of speakers in validation set. 

ok, thanks
Multiple last checkpoints
State: closed | Pull request: no
Labels: name: bug

Received the following error when I killed the job and try to resume it. It uses mcore gpt.
```
Traceback (most recent call last):
  File "/opt/NeMo/examples/nlp/language_modeling/megatron_gpt_pretraining.py", line 34, in main
    exp_manager(trainer, cfg.exp_manager)
  File "/opt/NeMo/nemo/utils/exp_manager.py", line 348, in exp_manager
    check_resume(
  File "/opt/NeMo/nemo/utils/exp_manager.py", line 622, in check_resume
    raise ValueError(f"Multiple checkpoints {last_checkpoints} that matches *last.ckpt.")
ValueError: Multiple checkpoints [PosixPath('/results/checkpoints/boson-consumed_samples=6224.0-last'), PosixPath('/results/checkpoints/boson-consumed_samples=4800.0-last')] that matches *last.ckpt.
```

This happens when the job is killed while saving a checkpoint. Currently you need to manually cleanup the checkpoints folder to be able to resume. Figure out which is the most recent *complete* "-last" checkpoint, and delete the other(s).

pls re-open if solution doesn;t work or if you face any other issue. 

@szhengac To allow PTL to exit gracefully, if your job has some max runtime, you could tell PTL to exit earlier, so that the program has buffer time to clean up the old checkpoint. For example, if your job can only run for max 4 hours, you could set `++trainer.max_time="00:03:50:00"`.

Instead of trainer.max_time, you should use exp_manager.max_time_per_run  

https://github.com/NVIDIA/NeMo/blob/main/nemo/utils/exp_manager.py#L175
[TTS] Scale sampler steps by number of devices
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

The weighted random sampler is configured to produce a fixed number of training samples during each epoch. This means if you use multiple GPUs, then the number of steps in each epoch is scaled down by the number of GPUs. This change multiplies the number of samples by the world size to keep the step count independent of number of GPUs used.

**Collection**: [TTS]

# Changelog 
- Scale number of samples per epoch by world size when using weighted random sampler.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

jenkins
only enable query key scaling during fp16
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Enable query key scaling only during fp16, since this is when TE enables query key scaling. https://github.com/NVIDIA/TransformerEngine/blob/666539f36275fa9c0fbc99f9ea50f2d6e29e336f/transformer_engine/pytorch/attention.py#L940 


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
Packed Sequence
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Support training with packed sequences for SFT and PEFT.
In this scenario, sequences in a batch are "concatenated" in the sequence length dimension, instead of "stacked" in a batch dimension. 
This is shown to significantly improve the training throughput.
Packed sequences will need to be prepared in a particular way (script in #8682 )

**Collection**: NLP

# Changelog 
- new GPTSFTPackedDataset class to handle the new dataset format
- pass `cu_seqlens` and `qkv_format` to the forward() function of mcore model

# Usage
Note: only train_ds is packed. Validation and test datasets are unchanged.
```python
python /opt/NeMo/examples/nlp/language_modeling/tuning/megatron_gpt_peft_tuning.py \
...
+model.data.train_ds.packed_sequence=True
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins
Instructions for running ci on pr template
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add checks for unit tests that are looking for data from CI machine
State: closed | Pull request: yes
Labels: name: ASR

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

tested and it looks good to me
Added a procedure for Windows users, README
State: closed | Pull request: yes
Labels: name: documentation

Added a brief guide and official links for troubleshooting WSL installation for Windows users

# What does this PR do ?
Adds instruction for Windows users


**Collection**: [Note which collection this PR will affect]
Documentation

# Changelog 
Added text


# Usage
Reading


# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [-] Did you write any new necessary tests?
- [+] Did you add or update any necessary documentation?
- [-] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [+] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


[celebrate]     Georgy Zelenfroind reacted to your message:
________________________________
From: Somshubra Majumdar ***@***.***>
Sent: Tuesday, December 12, 2023 3:54:39 PM
To: NVIDIA/NeMo ***@***.***>
Cc: Georgy Zelenfroind ***@***.***>; Assign ***@***.***>
Subject: Re: [NVIDIA/NeMo] Added a procedure for Windows users, README (PR #7942)


@titu1994 approved this pull request.

This is very nice ! Thanks !

—
Reply to this email directly, view it on GitHub<https://github.com/NVIDIA/NeMo/pull/7942#pullrequestreview-1777862998>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AI4QZ2DQQAQDV37562F3XX3YJB437AVCNFSM6AAAAAA74MRP2OVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTONZXHA3DEOJZHA>.
You are receiving this because you were assigned.Message ID: ***@***.***>


[celebrate]     Georgy Zelenfroind reacted to your message:
________________________________
From: Somshubra Majumdar ***@***.***>
Sent: Tuesday, December 12, 2023 3:54:52 PM
To: NVIDIA/NeMo ***@***.***>
Cc: Georgy Zelenfroind ***@***.***>; Assign ***@***.***>
Subject: Re: [NVIDIA/NeMo] Added a procedure for Windows users, README (PR #7942)


Merged #7942<https://github.com/NVIDIA/NeMo/pull/7942> into main.

—
Reply to this email directly, view it on GitHub<https://github.com/NVIDIA/NeMo/pull/7942#event-11225619307>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AI4QZ2AJ4XXAHQWGFSYHF7DYJB44ZAVCNFSM6AAAAAA74MRP2OVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJRGIZDKNRRHEZTANY>.
You are receiving this because you were assigned.Message ID: ***@***.***>
Fix HF dataset streaming conversion
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Hi 👋, tiny modif here to save the right item into the manifest.



This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
SDE2: GPU acceleration
State: closed | Pull request: yes
Labels: name: stale

GPU acceleration added, 
Data loading optimization with adding multiprocessing,
Code refactoring

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Nemotron-3-8b finetuning
State: closed | Pull request: no

I was looking into this [newly released model](https://developer.nvidia.com/blog/nvidia-ai-foundation-models-build-custom-enterprise-chatbots-and-co-pilots-with-production-ready-llms/) and wanted to experiment with either SFT or PEFT-finetuning but it seems documentation is lacking on it (the [only tutorial](https://huggingface.co/nvidia/nemotron-3-8b-base-4k) available seems to be on inference at the moment). Actually, I couldn't find any mention to the model in the code so I'm not even sure training is possible at all with code available.

In support to above, I've tried the code using the latest available image (`nvcr.io/nvidia/nemo:23.08`) with a small-sized model (gpt-2) and it works fine. However, when plugging in the nemotron weights it raises an exception at [L1221](https://github.com/NVIDIA/TransformerEngine/blame/db589510a69b5b5c655d439ad3521ea890c7b454/transformer_engine/pytorch/attention.py#L1221) due to an unsupported Q,K,V layout. Note that this code no longer exist in main after [this PR](https://github.com/NVIDIA/TransformerEngine/pull/403).

Could you please confirm that the code available in this public repo _supports nemotron finetuning_ **and** provide some guidance on how to do it (e.g. pointer to the correct docker image, example usage would also be great)?

Tagging @ericharper and @nithinraok hoping they can help, thank you in advance!

Is `nemofw-training:23.08.03` subset of `nvcr.io/nvidia/nemo:23.08`? 

The official guide [here](https://docs.nvidia.com/nemo-framework/user-guide/latest/playbooks/llama2sft.html) suggests to use `nemofw-training:23.08.03`, but I got the following error when I `docker pull nvcr.io/ea-bignlp/ga-participants/nemofw-training:23.08.03`:
`Error response from daemon: pull access denied for nvcr.io/ea-bignlp/ga-participants/nemofw-training, repository does not exist or may require 'docker login': denied: requested access to the resource is denied`

Same error even after I docker logged in. Do you guys have any idea what is the reason and which one is supposed for SFT?

Hi
Are you logging correctly?
You must register on the site below and use your created api-key to login.
ngc.nvidia.com

@alireza0r Thank you for the reminder. Yes, it was related to the api-key. `docker login` command gave me login success info, but it didn't work. I have to use command `docker login nvcr.io -p api-key` to be able to successfully pull the image. 

> @alireza0r Thank you for the reminder. Yes, it was related to the api-key. `docker login` command gave me login success info, but it didn't work. I have to use command `docker login nvcr.io -p api-key` to be able to successfully pull the image.

You actually need to have the proper access provided by NVIDIA.
How to apply early stopping criterion while finetuning conformer?
State: closed | Pull request: no
Labels: name: stale

name: ASR

I am trying out conformer-ctc and conformer-transducer finetuning with 5hrs of data using https://storage.googleapis.com/vakyansh-open-models/pretrained_models/vakyansh-conformer-ssl/ssl_conformer_large_e178.nemo pretrained model . I have seen that during training after certain number of epochs the wer is increasing during validation. So, I want to apply early stopping criterion for that. Can anybody kindly please help me out how to do that? 


@okuchaiev A gentle reminder for the same

@titu1994 can you kindly please answer this

@KunalDhawan could you answer this?

Hi @mukherjeesougata, we do provide support for early stopping based on [PTL's EarlyStopping callback](https://lightning.ai/docs/pytorch/stable/common/early_stopping.html). The config parameters **create_checkpoint_callback** and **checkpoint_callback_params** in [ExpManager](https://github.com/NVIDIA/NeMo/blob/main/nemo/utils/exp_manager.py) might be of interest to you - https://github.com/NVIDIA/NeMo/blob/main/nemo/utils/exp_manager.py#L156-L157

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
How to remove Punctuation and Capitalization from appearing in a P&C ASR model's output?
State: closed | Pull request: no

I am running `transcribe_speech.py` on the [STT En FastConformer Hybrid Transducer-CTC Large P&C](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_fastconformer_hybrid_large_pc) model, but I do not need the P&C feature. Is there a flag I can give to the script to stop the model from generating those tokens, or do I have to filter out the tokens as a postprocessing step myself?

Hi @abarcovschi, we don't support P&C stripping currently but would be adding this feature very soon! The easiest way would be to filter-out/ convert (capitalization) the tokens in post-processing. We would soon be adding text post-processors for ASR, initial implementation (not tested/merged) is available here - [TextPostProcessor](https://github.com/NVIDIA/NeMo/blob/asr_normalize/nemo/collections/asr/parts/postprocessing/asr_text_postprocessor.py) class, [language specific normalizations](https://github.com/NVIDIA/NeMo/tree/asr_normalize/nemo/collections/asr/parts/postprocessing/language_normalizations).
[NLP] HF -> NeMo conversion enhancements
State: closed | Pull request: yes
Labels: name: core

name: NLP

# Announcement📢📢📢

I'm closing this PR in favor of a series smaller PRs with all the corresponding changes applied.
* https://github.com/NVIDIA/NeMo/pull/7977
* https://github.com/NVIDIA/NeMo/pull/7978
* https://github.com/NVIDIA/NeMo/pull/7979
* https://github.com/NVIDIA/NeMo/pull/7980
* https://github.com/NVIDIA/NeMo/pull/7981
* https://github.com/NVIDIA/NeMo/pull/7982

Hopefully, this makes it easier to review and integrate the changes.

------------------------------------------------------------------------------------------

# What does this PR do ?

This PR improves community models conversion scripts: Llama2, MPT, and StarCoder.

**Collection**: NLP

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [x] Documentation


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

Closing to follow up elsewhere - see the description above for details.
add fim support
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Adds FIM support for GPTDataset
Increases max number of training data files

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

The FIM part looks good to me. Thank you!

jenkins

jenkins

jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Pass in rotary_base to mcore and from HF
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

This depends on the following mcore commit https://github.com/NVIDIA/Megatron-LM/commit/d931ba8a492801d9d74efba6be94f82387090e57, so if it's not yet integrated into our main branch, we'd need to wait before merging this PR.

**Collection**: NLP
# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

This PR incremented the version of Megatron Core from `e122536b7645edcb7ebf099b5c92a443f7dbf8e7` to `973330e9c3681604703bf1eb6b5a265d1b9b9b38`, that is from 0.3.0 to 0.4.0, but did not change the [requirements file](https://github.com/NVIDIA/NeMo/blob/main/requirements/requirements_nlp.txt#L13) accordingly
Added YAML parsing for UB/TP overlap configs
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Added a YAML parsing for UB/TP overlap configurations.

jenkins

This PR aims to reverse changes from this PR: https://github.com/NVIDIA/NeMo/pull/7003
"Added a YAML parsing for UB/TP overlap configurations." doesn't describe the problem that you want to solve. Please be more verbose.
The https://github.com/NVIDIA/NeMo/pull/7003 removed module loading, opening file and parsing arguments from the model class. All of this is what hydra is for.
Fix for gpt3 eval hang with PP (a dtype issue)
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix for gpt3 eval hang with PP (a dtype issue)

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins
Fix pinned triton version
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

* Remove pinned version in Dockerfile.
* Raise error if triton version is incorrect when users use Flash Attention Triton.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins
migration to PTL 2.0 for spellmapper model
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

migration to PTL 2.0 for spellmapper model, same as in #7597

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


@athitten this change is similar to yours, could you review?

@titu1994, can you review or assign somebody?

@athitten can you review this?

jenkins

@ericharper, why my jenkins hangs forever? I don't have rights to see on it

jenkins

@titu1994, jenkins passed


@athitten, @ericharper what about this pr? All checks passed
Add interface to set NCCL options of each process group
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add interface to set NCCL options of each process group. Can pass a path to yaml file that can specify the NCCL options (`min_ctas`, `max_ctas`, `cga_cluster_size`) for each process group.

Below shows an example yaml file that specifies config of data-parallel, pipeline-parallel, and tensor-parallel proc groups.

```
dp:                                                                                       
  min_ctas: 4                                                                             
  max_ctas: 4                                                                             
pp:                                                                                       
  min_ctas: 2                                                                             
  max_ctas: 2                                                                             
tp:                                                                                       
  min_ctas: 24                                                                            
  max_ctas: 24                                                                            
```


# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

@athitten Please review and merge