renwei2024/nemo-github-issues huggingface dataset:
Fix multimodal tutorial links in user guide
State: closed | Pull request: yes

# What does this PR do ?

Fix multimodal tutorial links in user guide

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Add implementation of slimIPL algorithm. Add support for using ais da…
State: closed | Pull request: yes
Labels: name: core

name: stale

name: ASR

name: common

…tasets using local manifest.

# What does this PR do ?

Adds implementation of slimIPL algorithm.

# Usage
This can be used for pseudo-labeling if in the config "ipl" parameters are given



# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

jenkins



jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
add alpha scaling to lora
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: CI

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update k2
State: closed | Pull request: yes

# What does this PR do ?

Update k2 version commit for r1.23.0 (cherry-pick from main, #8478 )

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
MoE parameter passing
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Add fp8 support for SD/Update notebook paths
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add FP8 support for SD training.

**Collection**: [multimodal/text_to_image/statble_diffusion]

# Changelog 
- Add config argument - use_te_fp8 to switch between fp16 and fp8 training
- Add transformer engine fp8 layers to Stable diffusion Unet
- Update a few legacy paths in notebook tutorial

# Usage

```yaml
unet_config:
    use_te_fp8=True
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins
Typo this needs to be replace
State: closed | Pull request: no

https://github.com/NVIDIA/NeMo/blob/88e332d31a9ed54d5e19c5250e876eac18ad8936/tools/nemo_forced_aligner/utils/data_prep.py#L155

@prathwishshetty 
I have raised a PR for this at https://github.com/NVIDIA/NeMo/pull/8498
Let's wait to get it merged soon. Thanks for raising PR

This can be closed as PR gets merged

Thank you for the quick PR!
Add audio codec models
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Add NeMo Audio Codec Models to the collection 

**Collection**: [TTS]

# Changelog 
- Add audio codec 16kHz config
- Add NGC model to pretrained list

# Usage

### Automatically load the model from NGC

```python
from  nemo.collections.tts.models  import  AudioCodecModel
nemo_codec_model  =  AudioCodecModel.from_pretrained('audio_codec_16khz_small')
```

### Getting discrete tokens from Audio

```python
import librosa
import torch
audio,sr = librosa.load("<path_to_audio>", sr=16000)
device = 'cuda' if torch.cuda.is_available() else 'cpu'
audio_tensor = torch.from_numpy(audio).unsqueeze(dim=0).to(device)
audio_len = torch.tensor([audio_tensor[0].shape[0]]).to(device)
nemo_codec_model.freeze()
encoded_tokens, encoded_len = nemo_codec_model.encode(audio=audio_tensor, audio_len=audio_len)
```

### Reconstructing audio from discrete tokens

```python
reconstructed_audio = nemo_codec_model.decode(tokens=encoded_tokens, tokens_len=encoded_len)
```

### Listen to audio

```python
import soundfile as sf
output_audio  =  reconstructed_audio.cpu().numpy()[0]
sf.write(<path_to_output_audio>,  output_audio,  samplerate=16000)
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation


jenkins

jenkins

jenkins

jenkins
add alpha scaling to lora
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: CI

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Add settings to suppress bf16 compile errors in CI on V100
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Add settings to suppress bf16 compile errors in CI on V100
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins
Fix PTL2.2 saving multiple `*-last.ckpt` checkpoints in resumed training
State: closed | Pull request: yes
Labels: name: core

# What does this PR do ?

Fix PTL2.2 saving multiple `*-last.ckpt` checkpoints in resumed training. 

In current PTL2.2 [model_checkpoint.py](https://github.com/Lightning-AI/pytorch-lightning/blob/master/src/lightning/pytorch/callbacks/model_checkpoint.py#L783), it doesn't delete the previous `*-last.ckpt` checkpoint that the model was resumed on when saving a new `*-last.ckpt` checkpoint, which results in multiple checkpoints. This will cause errors of "multiple *-last.ckpt checkpoints" when trying to resume from a previous job.

The fix is to check whether the checkpoint to be saved is also a `-last.ckpt` checkpoint. If it is, delete the previous `-last.ckpt` if they are under the same directory.


**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation




This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

@stevehuang52 this problem basically happens when we try to resume 2nd time right ? 

> @stevehuang52 this problem basically happens when we try to resume 2nd time right ?

@athitten  Yes, it happens when we resume for the 2nd time, the first resume is fine

jenkins

jenkins

jenkins

jenkins

@athitten seems that Jenkins passed without issues. Could you please review the PR again and see if there's other changes or tests needed? Thanks~
PEFT better error message and doc
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

PEFT better error message and doc

**Collection**: NLP

# Changelog 
- Better error message for setting `packed_sequence` to True but using an unpacked dataset
- Better error message for using a different TP value for PEFT evaluation from training
- Documentation for using `megatron_gpt_generate.py` on a pretrained base model

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins
Update k2 version
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

@GNroy FYI: I am able to build a container from `nvcr.io/nvidia/pytorch:23.09-py3` using this commit, but `nvcr.io/nvidia/pytorch:23.10-py3` and higher fails.
Locally I can install k2 with pytorch 2.2 (compiled with Cuda 12.1)
using onnx model for language identification inference
State: closed | Pull request: no
Labels: name: stale

hi,
I have export the language identification model to onnx and I want to use it for inference.
how should I use this model for inference on an audio file?
how should be the format of my audio input?
is there any example or tutorial?
after using the following code:


```
import nemo
import nemo.collections.asr as nemo_asr
vad_model = nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained(model_name="langid_ambernet")
vad_model.export('qn.onnx')
ort_session = onnxruntime.InferenceSession('qn.onnx')
ttt = torch.rand(1,80,8000)    
outputs = ort_session.run(None, {ort_session.get_inputs()[0].name:to_numpy(ttt)})
```



i get this error:


```
ValueError                               Traceback (most recent call last)
[<ipython-input-64-4dddcd66c01b>](https://localhost:8080/#) in <cell line: 1>()
----> 1 onnx_outputs = ort_session.run(None, {ort_session.get_inputs()[0].name:to_numpy(ttt)})
[usr/local/lib/python3.10/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py](https://localhost:8080/#) in _validate_input(self, feed_input_names)
    196                 missing_input_names.append(input.name)
    197         if missing_input_names:
--> 198             raise ValueError(
    199                 f"Required inputs ({missing_input_names}) are missing from input feed ({feed_input_names})."
    200             )

ValueError: Required inputs (['length']) are missing from input feed (['audio_signal']).
```

and how map the output embedding to labels?


any help is really appreciated.
thanks

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Add CTC-WS documentation
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a description of the CTC-WS context-biasing method to the documentation.

**Collection**: [ASR]

# Changelog 
- Rename docs/source/asr/asr_language_modeling.rst -> docs/source/asr/asr_language_modeling_and_customization.rst
- Add context-biasing description to docs/source/asr/asr_language_modeling_and_customization.rst
- Update docs/source/asr/intro.rst

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Must wait https://github.com/NVIDIA/NeMo/pull/8462


jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

jenkins

jenkins
Replace argsort to reduce memory footprint
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

Reduce memory footprint by replacing torch.argsort. With my local sample, diarization pipeline works on 12GB. Without the patch, it failed with an error `torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.60 GiB. GPU 0 has a total capacty of 23.64 GiB of which 3.21 GiB is free. Including non-PyTorch memory, this process has 20.35 GiB memory in use. Of the allocated memory 16.07 GiB is allocated by PyTorch, and 665.59 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF`

**Collection**: [asr]

# Changelog 
Replace torch.argsort to reduce memory usage

# Usage
Running MSDD on long audio sample

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


@dkurt 
Thank you for preparing this PR.
Let me test this and review it.
Need to verify on multiple samples. 

@tango4j, any updates?

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

@tango4j , friendly reminder. Maybe I can run some tests on my side?

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

@tango4j, reminder

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Add Gemma Support and Conversion Scripts
State: closed | Pull request: yes
Labels: name: NLP

name: common

# What does this PR do ?

Add Gemma Support and Conversion Scripts

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
Add Gemma Model Support and Conversion Scripts
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add Gemma Model Support and Conversion Scripts

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Hiddens modules bugfixes
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Hiddens modules such as `a_mim` and `vae` output a number of keys, including `{"z", "z_mean"}` for example. The current implementation supports specifying a single key to be used as the encoder output, and generally in configs this is set to `z` for those variational losses. This is not what you expect at inference time though. In that case you want the latent space to be a point estimate (the mean) rather than a noisy sampling. This PR accomplishes this by adding an optional second encoder output key to be used at inference time, and if it is not set then the current key is used for both. Inference/training is determined by checking the pytorch module property `self.training`. 

**Collection**: nlp

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [x] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

I need to get back to this.

On Wed, Mar 20, 2024 at 6:44 PM github-actions[bot] <
***@***.***> wrote:

> This PR is stale because it has been open for 14 days with no activity.
> Remove stale label or comment or update or this will be closed in 7 days.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/NVIDIA/NeMo/pull/8466#issuecomment-2011033150>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADQCBXYXVSTXGBJXW6C7DDYZI3QHAVCNFSM6AAAAABDSCNISGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJRGAZTGMJVGA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
pin to 0.5.0
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Support relative audio paths in Lhotse `LazyNeMoIterator`
State: closed | Pull request: yes
Labels: name: common

# What does this PR do ?

Support relative audio paths in Lhotse `LazyNeMoIterator`

**Collection**: ASR

# Changelog 
- Support relative audio paths in Lhotse `LazyNeMoIterator`

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

Closing this in favor of https://github.com/NVIDIA/NeMo/pull/8449
Add ASR context-biasing tutorial
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a context-biasing tutorial (based on CTC-WS method) for CTC and RNN-T models.

**Collection**: [ASR]

# Changelog 
- Add tutorials/asr/ASR_Context_Biasing.ipynb tutorial

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

@artbataev thank you so much for the review!

About your suggestions/comments:

1. Done
2. Done
3. I leaved installing graphviz optional. The installation takes 5-10 minutes during which the window may become glitchy.
4. I made a fix, but it works only on my remote desktop. I do not know why Google Colab does not display the right colors. I removed comments about colors.
5. Done
6. Done
7. Done 


jenkins
Video Pre-Training for Video NeVA
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Code to enable video pretraining for NeVa.

[W&B Project](https://wandb.ai/joc/video_neva)

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
FSDP support for GPT fine-tuning
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

1) FSDP bug fix.
2) FSDP support for GPT fine-tuning.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins
Which container to use for nemofw-inference.
State: closed | Pull request: no
Labels: name: stale

In https://github.com/NVIDIA/NeMo-Megatron-Launcher/pull/233, 

training container image update to `nvcr.io/nvidia/nemo:24.01.framework`

but nemo inference container image is still in private repository `nvcr.io/ea-bignlp/ga-participants/nemofw-inference`

Is there a plan to migrate the nemo inference container image to nvcr.io/nvidia as well?

Hi, can you add me to the org/team ea-bignlp/ga-participants. I can't get any info about private repository, so i can't pull that container. Thanks you so much!

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Prepare a dataset for speaker verification training
State: closed | Pull request: no
Labels: name: stale

Hello, how can I prepare a dataset to fine-tune or pretrain a model for speaker verification task, knowing that my current dataset has these fields: (audio, transcription, speaker-id)?
Thank you

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
How to discriminate valid phones and silence using nemo forced aligner
State: closed | Pull request: no
Labels: name: stale

**Is your feature request related to a problem? Please describe.**
I tried to use forced aligner to generate timestamps for my audio files, I got ctm results in tokens like this:
100001 1 0.00 0.44 \<b>
100001 1 0.44 0.04 WORD1
100001 1 0.48 0.20 \<b>
100001 1 0.68 0.04 WORD2
100001 1 0.72 0.20 \<b>
100001 1 0.92 0.04 WORD3
100001 1 0.96 0.16 \<b>
100001 1 1.12 0.04 WORD4

Every valid token(none blank ones) has the same duration 0.04, while \<b>'s duration changes. when I tried to use another model , the fixed value changed from 0.04 to 0.08, but still keeps the same for valid phones.
I believe the real duration for a valid phone varies, and some parts of the vowels were wrongly moved to \<b>, which makes it hard to discriminate valid phones and real silence part. It should be quite common that there are no noteworthy silence between words in some sentences.

**Describe the solution you'd like**

Is it possible to make the timestamp around \<b> more accurate, or add some other symbols for noteworthy silence like "sp"/"sil"? 


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
How to do fine-tuning for Nvidia - NEMO canary-1b models
State: closed | Pull request: no

**Is your feature request related to a problem? Please describe.**

- Can you explain, how to fine-tune Nvidia latest ASR models like Canary 1B and NeMo Parakeet ASR Models.

https://nvidia.github.io/NeMo/blogs/2024/2024-02-canary/
https://nvidia.github.io/NeMo/blogs/2024/2024-01-parakeet/

**Describe the solution you'd like**

- A notebook/code block explaining how to fine-tune Nvidia Canary models for low-resource Indic languages like Malayalam.


For Parakeet, we have several tutorials in the Nemo tutorials directory - here is one to get started - https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/ASR_CTC_Language_Finetuning.ipynb

Most of the concepts will apply to Parakeet CTC / RNNT/ TDT. 

For Canary, it is a very new model, we are still adding documentation and tutorials for finetuning and scratch training 

Also, if you want a finetune script, you can use this - https://github.com/NVIDIA/NeMo/blob/main/examples/asr/speech_to_text_finetune.py

Thank @titu1994 for sharing these details

> For Parakeet, we have several tutorials in the Nemo tutorials directory - here is one to get started - https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/ASR_CTC_Language_Finetuning.ipynb
> 
> Most of the concepts will apply to Parakeet CTC / RNNT/ TDT.
> 
> For Canary, it is a very new model, we are still adding documentation and tutorials for finetuning and scratch training

Hello, I would like to know about the progress on Canary finetuning. Thanks.
fix canary chunk infer bug
State: closed | Pull request: yes
Labels: name: ASR

name: common

# What does this PR do ?

- fix canary chunk infer rare bug, where the last segment might have mismatched shape with pre-allocated buffer.
- update audio_segment.py to allow duration is None
- add lhotse support for relative audio paths


jenkins

jenkins

jenkins

jenkins

could you add unit tests that cover these edge cases? btw I didn't see that PR earlier and added the relative audio path support independently; we can either merge my PR or port the test I wrote to yours and close mine (up to you) https://github.com/NVIDIA/NeMo/pull/8464

> could you add unit tests that cover these edge cases? btw I didn't see that PR earlier and added the relative audio path support independently; we can either merge my PR or port the test I wrote to yours and close mine (up to you) #8464

Sure I'll add the tests~I think it's better to use the `get_full_path` function to resolve the relative paths, since it's well tested.

jenkins

jenkins

jenkins

jenkins
Adding landing pages to collections
State: closed | Pull request: yes
Labels: name: TTS

name: stale

name: ASR

name: NLP

# What does this PR do ?

This PR adds landing pages for LLM, CV, and multimodal collections. Also, some suggestions on how to split the introduction into a few sections for better readability.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.