renwei2024/nemo-github-issues huggingface dataset:
Fix the automatic computation of `max_steps` when resuming an experiment
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

This fixes a bug in #7518 where `max_steps` wasn't computed properly when resuming an experiment in the middle of an epoch (my fault -- I hadn't realized the training dataloader had a smaller length on resume).

I have tested it locally to confirm both the bug and the fix.

I don't really like that we need to mess with the sampler but that's the best I could think of. Happy to take other suggestions as well.

NB: I also added an extra safety check for `limit_train_batches` but this is unrelated to the bug being fixed.

**Collection**: core

# Changelog 
- Fix learning rate schedule in Megatron models when max_steps is not set and an experiment is resumed


# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #7518


jenkins

Does this fix work with all 4 classes of Samplers inside the train dataloader? (MegatronPretrainingSampler, MegatronPretrainingRandomSampler, MegatronPretrainingBatchSampler, MegatronPretrainingRandomBatchSampler)

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Open source export and deploy modules
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Open sourcing the export and deploy module that exports a nemo checkpoint to TensorRT-LLM and serves it using PyTriton.

**Collection**: [None]

# Changelog 
- Added export and deploy folders under the main folder.

# Usage
* You can potentially add a usage example below

```python

# A simple example is as follows. 

# Run the following on the server side

from nemo.export import TensorRTLLM
from nemo.deploy import DeployPyTriton

# To export to TRT-LLM

trt_llm_exporter = TensorRTLLM(model_dir="/path/to/trt_llm_engine_folder")
trt_llm_exporter.export(
    nemo_checkpoint_path="/path/to/nemo/ckpt.nemo",
    model_type="model_type",
    n_gpus=1,
)

# To start serving on Triton server

nm = DeployPyTriton(model=trt_llm_exporter, triton_model_name="LLM")
nm.deploy()
nm.run()

# To query the model, please run the following on a the client side

from nemo.deploy import NemoQuery

nq = NemoQuery(url="localhost:8000", model_name="LLM")
output = nq.query_llm(
    prompts=["what is the color of a banana?"], , top_k=1, top_p=0.0, temperature=1.0,
)

```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests? (Test location: https://github.com/oyilmaz-nvidia/NeMo/blob/oss-export/tests/export/test_nemo_export.py) 
- [x] Did you add or update any necessary documentation? (Added documentation on the gitlab repo for the docs. Will need to make some updates though)
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.


@titu1994 Thanks for your review and all of the suggestions. Tried to address all of your comments. I agree with most of them and updated the code based on your suggestion. Some of them, I left comment. Can you please do another review? Would love to merge this asap because there are many tasks depend on this PR.

@ericharper Please let us know what you think as well.

@titu1994 Added the deploy and the export to exclude package list here https://github.com/oyilmaz-nvidia/NeMo/blob/oss-export-deploy/setup.py#L237 

Jenkins 

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
Fix "index 0 is out of bounds for dimension 1 with size 0"
State: closed | Pull request: yes
Labels: name: stale

name: ASR

In the code, the variable `audio` containing the original audio recording gets overwritten during the operation `audio[start: start + duration]`. Consequently, on subsequent iterations of the loop, a segment of a new length cannot be processed correctly because the timestamps of the segment exceed the boundaries of the overwritten `audio`.

This PR addresses this issue by proposing a fix to ensure that the overwritten `audio` variable does not cause incorrect processing of subsequent segments.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Reconfigure limit_train_batches in terms of micro batches
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Converts the limit_train_batches in terms of num of micro batches.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins
Restoring from pretrained checkpoint is broke for "stt_ua_fastconformer_hybrid_large_pc"
State: closed | Pull request: no
Labels: name: bug

name: stale

Hi folks! I'm trying to restore `EncDecHybridRNNTCTCModel` from checkpoint trained on Ukrainian

```
In [4]: from nemo.collections.asr.models import EncDecHybridRNNTCTCBPEModel
In [5]: model = EncDecHybridRNNTCTCBPEModel.from_pretrained("stt_ua_fastconformer_hybrid_large_pc")
```

model can't be restored because of keyword argument `conv_pointwise_type`

```
InstantiationException: Error in call to target 'nemo.collections.asr.modules.conformer_encoder.ConformerEncoder':
TypeError("ConformerEncoder.__init__() got an unexpected keyword argument 'conv_pointwise_type'")
```

Search through NeMo repository I only found `conv_pointwise_type` keyword  in one [PR](https://github.com/NVIDIA/NeMo/pull/6476), and nowhere in the code base, that looks suspicious. Is there any recommendation how one would make that checkpoint compatible?


Env:
- Nemo version: nemo_toolkit==1.23.0
- OS: Ubuntu 22.04

I figured how to load the parameters, though with the hack.

1. I obtained model's config by doing this:
```
ckpt_name = "stt_ua_fastconformer_hybrid_large_pc"
cfg = nemo_asr.models.EncDecHybridRNNTCTCBPEModel.from_pretrained(model_name=ckpt_name, return_config=True)
omegaconf.OmegaConf.save(cfg, f="cfg.yaml")
```

2. then I commented out the `conv_pointwise_type` attribute of encoder's parameters.

3. loaded model by providing a patched config:
```
model = nemo_asr.models.EncDecHybridRNNTCTCBPEModel.from_pretrained(
    model_name=ckpt_path,
    override_config_path="./cfg_modified.yaml",
```

The above hack works, though I wonder if the loaded model works correctly? Also it will be good to update the official checkpoints in the NGC registry.

Same issue happens to `stt_ru_fastconformer_hybrid_large_pc` russian checkpoint.


Thank you for reporting the issue and the fix ! We'll take a look soon. I think we updated the huggingface version but not NGC 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

Still relevant for https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_ru_fastconformer_hybrid_large_pc
docker: nvcr.io/nvidia/nemo:24.01.speech

And this model has a different tokenizer from the one in ngc.
https://huggingface.co/nvidia/stt_ru_fastconformer_hybrid_large_pc/tree/main

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Add SDXL support to NeMo
State: closed | Pull request: yes
Labels: name: NLP

name: Multi Modal

# What does this PR do ?

Add sdxl support 

**Collection**: NeMo multimodal


# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
Remove unnecessary attention mask
State: closed | Pull request: yes
Labels: name: core

name: NLP

name: Run CICD

# What does this PR do ?

Remove unnecessary attention masks. [Related MCore MR is here.](https://gitlab-master.nvidia.com/ADLR/megatron-lm/-/merge_requests/1259)

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

jenkins
AED transcribe bug fix[ASR]
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

* fixing AED transcribe failure case when input is manifest 
* Fixing pos encoding ids bug during inference. 

**Collection**: [ASR]

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.


**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Pass groupedgemm flag to spec maker; re-enable --transformer-engine spec.
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

Notice: with this PR merged the user must supply `transformer_engine: True` explicitly in their configuration to enable transformer engine in Mcore (now enabled by default). If Mcore is used, but no transformer engine option is not supplied, the user will receive a warning in their terminal/logs.
Model quantization manual
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Documentation for model quantization using AMMO.

**Collection**: NLP


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Support for multi-channel audio data
State: closed | Pull request: no
Labels: name: feature

**Describe the bug**

NeMo training and decoding scripts do not support multi-channel audio data.  

**Steps/Code to reproduce bug**

It does not support specifying which channel to use  for each audio file in each line in train.manifest.json or test.manifest.json file. 

I was able to run  ./examples/asr/speech_to_text_eval.py with "channel_selector=" to specify the channel for all the audio in a manifest.json file, but I can't find a way to specifying them for each audio file inside the manifest.json file.




**Expected behavior**
 
Can NeMo team add this useful feature to work with a diverse set of multi-channel training and testing audio data so data from different channel can be mixed within a manifest.json file?

**Environment overview (please complete the following information)**

NeMo was installed by pip in a conda environment. It works for single channel audio data. 
 
 


Thanks @yunbin, we'll take a look at adding this functionality over the next couple weeks. We'll post future updates here.

@anteju  Any update on getting the channel feature implemented in NeMo training scripts?

@yunbin, please check #9007.
The change there enables using channel selector from manifest for `nvidia/canary-1b` model.
fix for CP
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Setup CP Groups in GPTSFT model

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
[TTS] Add tutorial for training audio codecs
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Add tutorial about training audio codecs.

**Collection**: [TTS]

# Changelog 
- Add tutorial showing how to train or fine-tune audio codec models.
- Add config file for 22.05khz mel codec for documentation and easy fine-tuning.
- Make logging config in different config files consistent, disabling all logging by default except for audio samples. Tutorial shows how to update values to enable different types of logging.

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation


jenkins
Fix pickle issue for CLIP training
State: closed | Pull request: yes
Labels: name: Multi Modal

# What does this PR do ?

Fix an CLIP training issue found in nightly MM CI

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins
Fix transcribe for hybrid ctc rnnt model
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fix the transcribe signature for hybrid ctc rnnt model

**Collection**: [ASR]

# Changelog 
- Fix the transcribe signature for Hybrid CTC RNNT model
]
# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation


jenkins
Add guards for mcore imports [GPT FIM]
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Remove python 3.9 from setup.py
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Changes to run FP8 Debug in TE
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?
Changes required to run FP8 debug in TransformerEngine.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix the nsys profiling step condition
State: closed | Pull request: yes
Labels: name: bug

# What does this PR do ?

Fix the nsys profiling start and end step comparison condition to support different initial micro-batch start points.

# Changelog 
- Change the `current_step == target_step` to `current_step >= target_step`

# Usage
N/A

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Docs: PEFT and Packed Seq
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?
- Mistral/Mixtral PEFT support 
- Update converter path
- Add packed sequence docs

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
llm embeddings with position encoding models
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

bug fix for base models that use position encoding.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Tutorials broken due to Megatron-Core dependency
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

The NeMo [tutorials ](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/starthere/tutorials.html) on Google Colab do not run due to a dependency issue with Megatron-Core. Currently, NeMo is pinned to Megatron-Core 0.5.0, but references the `IndexedDataset` class. In 0.5.0 this class is called `MMIndexedDataset`. In Megatron-Core `main` branch is is now `IndexedDataset`.

**Steps/Code to reproduce bug**
1. Load the tutorial at https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/00_NeMo_Primer.ipynb
2. Run the first three cells.

On the third cell:
```
import nemo.collections.asr as nemo_asr
import nemo.collections.nlp as nemo_nlp
import nemo.collections.tts as nemo_tts
```

You get the error:
```
ImportError                               Traceback (most recent call last)
[<ipython-input-6-424875f6207b>](https://localhost:8080/#) in <cell line: 2>()
      1 import nemo.collections.asr as nemo_asr
----> 2 import nemo.collections.nlp as nemo_nlp
      3 import nemo.collections.tts as nemo_tts

5 frames
[/usr/local/lib/python3.10/dist-packages/nemo/collections/nlp/data/language_modeling/megatron/gpt_fim_dataset.py](https://localhost:8080/#) in <module>
     17 import numpy as np
     18 from megatron.core.datasets.gpt_dataset import GPTDataset, GPTDatasetConfig
---> 19 from megatron.core.datasets.indexed_dataset import IndexedDataset
     20 from megatron.core.datasets.utils import Split
     21 

ImportError: cannot import name 'IndexedDataset' from 'megatron.core.datasets.indexed_dataset' (/usr/local/lib/python3.10/dist-packages/megatron/core/datasets/indexed_dataset.py)
```

It looks like I see the same error on any of the NLP tutorials

**Expected behavior**

The tutorial should run to completion.

**Environment overview (please complete the following information)**

Google Collab
Tutorials loaded from https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/starthere/tutorials.html




There was a temporary error due to the `stable` tag pointing to NeMo main instead of NeMo r1.23.0. Please try again, it should work now. 

That worked. Thanks!
Fix expert model parallel rank issue with TP specified
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix the wrong expert model parallel rank when tensor_model_parallel_size >1.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
bug fix in convert_gpt_nemo_to_mcore.py
State: closed | Pull request: yes

# What does this PR do ?

minor fix...

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
minor fix in conversion script
State: closed | Pull request: yes

# What does this PR do ?

fixes precision attribute  during conversion
**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
CONF-TSASR
State: closed | Pull request: no
Labels: name: stale

The article ["Conformer-based Target-Speaker Automatic Speech Recognition for Single-Channel Audio"](https://arxiv.org/pdf/2308.05218v1.pdf) states that the model CONF-TSASR will be open-sourced through NVIDIA NeMo toolkit. Is it already open-sourced? Will pretrained weights be available? If yes, where can I found it?

Up!
I wish to try it as well to check if reported results are as good as they say.
Can we already find it and if no, when will it be available?

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

Still waiting for the answer, thank you.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

Up

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.

wait for it
No module named 'nemo'
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I run Speaker_Diarization_Inference.ipynb on my mac book pro(M2)

**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version.  14.2.1
- PyTorch version 2.0.1
- Python version 3.11

**Additional context**

Add any other context about the problem here.
Example: GPU model


Please install NeMo with pip install nemo_toolkit[all] 

> Please install NeMo with pip install nemo_toolkit[all]

Thank you ,I solved it. It's an environmental problem.
Switch NeMo to safetensors with Compat mode
State: closed | Pull request: yes
Labels: name: core

name: ASR

name: NLP

# What does this PR do ?

All checkpoints statedicts will be switched to safetensors following security recommendations against use of pkl (torch internally uses pickle format).

Additionaly, fix the model filter logic for HF based API search.

**Collection**: [Core]

# Changelog 
- Add the support of "ckpt_security_level" flag to SaveRestoreConnector
- Add compact and standard security modes
- In standard mode, only safetensor files can be read and written. If a legacy .ckpt checkpoint is attempted to be loaded into NeMos restoration engine, it will error out.
- In compat mode (the default), both pickle files and safetensors can be loaded for model restoration, however only safetensor files will be written to disk.
  - Compat mode is the default and will enable full backward compatibility with all older checkpoints.
  - It is the default mode (set as default inside the constructor). It must be explicitly updated to "standard" security mode that is stricter and breaks backward compatibility for .ckpt files.
- Fix a bug in an older test for Huggingface filters

# Usage
```python

from nemo.core.connectors.save_restore_connector import SaveRestoreConnector

connector = SaveRestoreConnector()
connector.ckpt_security_level = "standard"

# Will succeed if the checkpoint was saved with safetensors, fail if saved with .ckpt pickle file
model = ModelPT.restore_from("path/model.nemo", save_restore_connector=connector)
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation


jenkins

jenkins

jenkins

jenkins
Github io: add NeMo FW diagrams
State: closed | Pull request: yes

# What does this PR do ?

Add NeMo FW diagrams to the github io website and add mention of NVIDIA NIM.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix for get_all_params_for_weight_decay_optimization
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Fix multimodal nightly build issues 2403
State: closed | Pull request: yes
Labels: name: Multi Modal

# What does this PR do ?

Fix multimodal nightly build issues 2403 in CLIP and Neva

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkijns

jenkins

jenkins

jenkins
Quantization fixes
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Fixing quantization issues based on more extensive tests:

- model configured with `megatron_amp_O2: true`
- model w/o artifacts
- model using HF tokenizer
- remove broken "pileval" dataset

**Collection**: NLP

# Changelog 

See above.

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to https://github.com/NVIDIA/NeMo/pull/8444


jenkins

jenkins

jenkins