renwei2024/nemo-github-issues huggingface dataset:
Refactor SFT Dataset Class for Video Support [Video-NeVA]
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Adds video support for SFT dataset class in NeVA

**Collection**: Multimodal

# Changelog 
- Adds `video_processor` option in config for processing video data for SFT dataset class (defaults to `LanguageBindVideoProcessor`)
- Adds logic for image-only, video-only, or image and video data loading / preprocessing in `LazySupervisedDataset.__getitem__`

# Usage
* You can potentially add a usage example below

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
add alpha scaling to lora
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

step in line with the original lora and added alpha.

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins


jenkins

jenkins

jenkins
[canary] Refactor: `PromptedAudioToTextLhotseDataset` and `EncDecMultiTaskModel`
State: closed | Pull request: yes
Labels: name: ASR

This PR restores `transformer_bpe_models.py` and `audio_to_text_lhotse.py` to their previous shape and separates out two classes: `EncDecMultiTaskModel` and `PromptedAudioToTextLhotseDataset` that contain the modified code for Canary, generalized to support other model instances of this type.


Merging to `canary-2` for further development to unblock others -- we should address the rest of comments there.
Incrementally update black formatter
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Incremental black formatter update
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
No such file or directory: 'megatron/core/requirements.txt' when installing nemo 1.22
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

No such file or directory: 'megatron/core/requirements.txt' when installing nemo 1.22

**Steps/Code to reproduce bug**
```
File "/tmp/pip-build-env-93j56cnr/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 52, in <module>
    File "<string>", line 45, in req_file
  FileNotFoundError: [Errno 2] No such file or directory: 'megatron/core/requirements.txt'
```
A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model


using the docker container is highly recommended https://github.com/NVIDIA/NeMo?tab=readme-ov-file#docker-containers

You are using the wrong version of python, you need to use python 3.10.12.

For some reasons, I need to stick to python3.8.
Previous Nemo version installed and worked ok (I do not use anything related to megatron). Is the reported error (missing file) related to Python version?

If you need just ASR, you can safely install nemo_toolkit[ASR]. However it may be time to update. 3.8 has been deprecated for over a year now, and ASR has also started using syntax from 3.10 for Lhotse datasets. So in the coming month or two even pip install nemo_toolkit[ASR] will no longer safely work on python ,3.8

As to docker, it's actually not the primary path we recommend for local 1 GPU development. 

NeMo ASR at least is designed to be a pip installable package. Ofc, docker is very convenient when you just want to start working on clusters, and we too only use docker containers in those cases 

This should be fixed with this PR: https://github.com/NVIDIA/NeMo/pull/8465

Please try again and close the bug if your issue is solved.

thanks
Huvu/mcore retro
State: closed | Pull request: yes
Labels: name: core

name: NLP

# What does this PR do ?
Implementing RETRO's mcore model, dataloader's wrapper, training script.

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Attention encoder-decoder models for multiple speech-to-text tasks 
State: closed | Pull request: yes
Labels: name: core

name: ASR

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
Karpnv/beamsearch
State: closed | Pull request: yes
Labels: name: ASR

# Preserve one Flashlight beamsearch with Ngram LM 

Support Flashlight and Pyctcdecode decoding with pure KenLM and NeMo KenLM 
Standardize API of CLI inference scripts

**Collection**: ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

```python
python3 scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py \
nemo_model_file=am_model.nemo  \
input_manifest=manifest.json  \
preds_output_folder=/tmp   \
ctc_decoding.strategy=flashlight \
ctc_decoding.beam.kenlm_path=am_model.kenlm \
ctc_decoding.beam.beam_size=[4]   \
ctc_decoding.beam.beam_alpha=[0.5]   \
ctc_decoding.beam.beam_beta=[0.5] \
ctc_decoding.beam.flashlight_cfg.lexicon_path=am_model.flashlight_lexicon \ # DEFAULT_TOKEN_OFFSET
batch_size=32  \
beam_batch_size=1 \
device=cuda:1

python3 examples/asr/speech_to_text_eval.py  \
nemo_model_file=am_model.nemo \ 
input_manifest=manifest.json \
decoder_type=ctc  
ctc_decoding.strategy=flashlight \  
ctc_decoding.beam.kenlm_path=am_model.bin \
ctc_decoding.beam.beam_size=4   \
ctc_decoding.beam.beam_alpha=0.5   \
ctc_decoding.beam.beam_beta=0.5 \
ctc_decoding.beam.flashlight_cfg.lexicon_path=am_model.flashlight_lexicon \ # DEFAULT_TOKEN_OFFSET
ctc_decoding.beam.return_best_hypothesis=true \
batch_size=32  \
output_manifest=/tmp/manifest_out.json 
```
**PR Type**:
- [ V] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Correct padding for SFT input data to account for sequence parallel + TE's fp8 op dimension requirements
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Alters GPTSFTDataset and GPTSFTPackedDataset to pad to 8*TP when sequence parallel is enabled.

**Collection**: nlp/language_modeling

# Changelog 
- Adds an argument to GPTSFTDataset and GPTSFTPackedDataset which allows setting what value sequence lengths will be padded to a multiple of (previously always 16)
- Sets this argument from MegatronGPTSFTModel to ensure that sequences will be padded to a multiple of 8*TP when sequence parallel is enabled so that the sequence dimension will remain divisible by 8 after being split (required by TE for the first dimension)
- Sequences will still be padded to a multiple of 16 when sequence parallel is not used (see [PR-7904](https://github.com/NVIDIA/NeMo/pull/7904))

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


@cuichenx / @ericharper can you review?

jenkins
Allow MegatronPretrainingRandomSampler to do multi-epoch training
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Allows the MegatronPretrainingRandomSampler class to correctly handle multi-epoch training by calculating length correctly when consumed_samples > total_samples

**Collection**: NLP

# Changelog 
- nemo/collections/nlp/data/language_modeling/megatron/data_samplers.py: modified MegatronPretrainingRandomSampler to correctly return length for multi-epoch training when consumed_samples > total_samples
- nemo/collections/nlp/data/language_modeling/megatron/megatron_batch_samplers.py: fixed len calculation to be correct when drop_last=False, and also cleaned up constructor logic caused by these fixes: https://github.com/NVIDIA/NeMo/pull/7728/files#diff-26575f60355f81991aef1f635e947a368b086afe220c5a81817e35b1c00b2e38

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins
[TTS] Add modules for mel spectrogram codec
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Add code and example for how to train audio codec while encoding mel spectrogram.

Note there is an extra "FullBandMelEncoder" definition which is not in the example config, but which I am adding to try to and keep inference working with some existing checkpoints.

**Collection**: [TTS]

# Changelog 
- Add codec modules for encoding mel spectrogram.
- Add example config file.

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

jenkins

jenkins

jenkins

> What about model class file? Would it be added later?

This is trained using the same `audio_codec.py` model class.

jenkins
TransformerLMModel loads tokenizer from the local file instead of .nemo trained file
State: closed | Pull request: no
Labels: name: bug

name: stale

**TransformerLMModel loads tokenizer from the local file instead of .nemo trained file**

While using `TransformerLMModel.restore_from(<.nemo file>)`, the tokenizer is loaded from the local tokenizers file instead of the tokenizer found in the `.nemo` model file.

As shown in the snippet below from [nemo/collections/nlp/models/language_modeling/transformer_lm_model.py](https://github.com/NVIDIA/NeMo/blob/0b7467ebd158ef6876dd147189517b25ef626982/nemo/collections/nlp/models/language_modeling/transformer_lm_model.py#L57), the models uses `tokenizer_model` instead of `cfg.tokenizer.tokenizer_model` to load the tokenizer, which refers to the path to the local tokenizer.
This implementation is causing an issue when I try to load the model in another location other than where it's trained from.

```python
self.setup_tokenizer(
            tokenizer_name=cfg.tokenizer.get("tokenizer_name", "yttm"),
            tokenizer_model=cfg.tokenizer.get("tokenizer_model", None),
            vocab_file=cfg.tokenizer.get("vocab_file", None),
            bpe_dropout=cfg.tokenizer.get("bpe_dropout", 0.0),
            special_tokens=OmegaConf.to_container(cfg.tokenizer.special_tokens)
            if cfg.tokenizer.get("special_tokens", None)
            else None,
        )
```


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
add fsdp support for fine-tuning
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Adds FSDP support for NeMo fine-tuning.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.
Typo in data preparation script
State: closed | Pull request: no
Labels: name: stale

https://github.com/NVIDIA/NeMo/blob/d275d68048049e3a231ebf1a9a31ef810a4adcc7/tools/nemo_forced_aligner/utils/data_prep.py#L155

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
use_flash_attention does not control  NVTE_FLASH_ATTN flag for transformer engine
State: closed | Pull request: no
Labels: name: bug

name: stale

use_flash_attention=False should set NVTE_FLASH_ATTN=0

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
PEFT training of an LLM as an embedding model
State: closed | Pull request: yes
Labels: name: core

name: ASR

name: NLP

name: CI

# What does this PR do ?

train a Lora model to perform sentence embedding tasks which can be used for IR.

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins



jenkins

jenkins

closed in favor of #8622
Syncing random seed between ranks in generate
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

While random_seed was added to generate, it was missing from the sync between ranks and so was not possible to use with MegatronServer. This PR fixes that.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Fix preprocessing of data for Megatron when loading data files from a directory
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Fix preprocess data for megatron: loading files from a directory.

**Collection**: nlp

# Changelog 
- Open a JSON file from the specified directory.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

NLP collection: @MaximumEntropy, @ericharper

# Additional Information
* Related to # (issue)


Anyone willing to review? It's really a small change.

@MaximumEntropy, @ericharper, @ekmb, @yzhang123, @VahidooX, @vladgets, @okuchaiev

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Can't download vad_multilingual_frame_marlblenet
State: closed | Pull request: no
Labels: name: bug

name: stale

Hi,

For the last couple of days I have tried to download the VAD frame-multilingual marblenet model and I receive an error:

```
 wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/nemo/vad_multilingual_frame_marblenet/versions/1.20.0/zip -O vad_multilingual_frame_marblenet_1.20.0.zip
--2024-01-23 16:14:29--  https://api.ngc.nvidia.com/v2/models/nvidia/nemo/vad_multilingual_frame_marblenet/versions/1.20.0/zip
Resolving api.ngc.nvidia.com (api.ngc.nvidia.com)... 54.188.231.142, 34.223.159.108
Connecting to api.ngc.nvidia.com (api.ngc.nvidia.com)|54.188.231.142|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-01-23 16:14:30 ERROR 404: Not Found.
```

Could you please help? Also tried direct download.

Best regards,
Gabriel

Hi, any updates?

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
[docs] Add --force_codec to tarred dataset creation examples
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add --force_codec to tarred dataset creation examples

**Collection**: All speech collections

# Changelog 
- [docs] Add --force_codec to tarred dataset creation examples

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Temporarily use the previous RNN-T decoding algorithm as default
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Use previous RNN-T decoding algorithm as default (`loop_labels=False`)

**Collection**: [ASR]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

@GNroy, @andrusenkoau FYI
Is there a method to add an N-gram to streaming ASR with cache-aware model?
State: closed | Pull request: no
Labels: name: stale

Hi,

I would like to use a cache-aware streaming model with CTC decoder + n-gram. Is this feature supported or could you give me some ideas on how I could implement this?

Thank you in advance! Any help is greatly appreaciated!

Best regards,
Gabi

Streaming ASR at the moment does not support ngram

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Context-biasing by CTC-based Word Spotter (CTC-WS)
State: closed | Pull request: yes
Labels: name: feature

name: ASR

# What does this PR do ?

A new context-biasing method for CTC and Transducer (RNNT) models (only Hybrid Transducer-CTC in case of Transducer) by CTC-based Word Spotter (CTC-WS). The idea is to use CTC logprobs for fast decoding with context-biasing graph (tree). The context-biasing graph is built according to the context-biasing list of words/phrases and CTC transition topology. Obtained results (words with start/end timestamps) are merged with greedy CTC/Transducer word-level predictions. 

CTC-WS allows context-biasing in greedy decoding mode which is much faster than approaches based on shallow fusion (beam search decoding with external lm or context-biasing graph).

A detailed description of the CTC-WS algorithm and the comparison results will be published later.

**Collection**: [ASR]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage

```python
python scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py \
            nemo_model_file=<path to the .nemo file of the model> \
            input_manifest=<path to the evaluation JSON manifest file \
            preds_output_folder=<optional folder to store the predictions> \
            decoder_type=<type of model decoder [ctc or rnnt]> \
            acoustic_batch_size=<batch size to calculate log probabilities> \
            apply_context_biasing=<True or False to apply context biasing> \
            context_file=<path to the context biasing file with key words/phrases> \
            beam_threshold=[<list of the beam thresholds, separated with commas>] \
            context_score=[<list of the context scores, separated with commas>] \
            ctc_ali_token_weight=[<list of the ctc alignment token weights, separated with commas>] \
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
Pre-compute max_seqlen and cu_seqlens_argmin in all model-parallel cases
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Pre-compute max_seqlen and cu_seqlens_argmin in all model-parallel cases.

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Jenkins
Improve communication overlapping in FP8 distributed optimizer
State: closed | Pull request: yes
Labels: name: feature

name: core

name: NLP

name: CI

# What does this PR do ?

When training GPT, the Apex distributed Adam optimizer overlaps its first parameter all-gather with the optimizer step. This optimization has been applied to both FP8 and non-FP8 models.

**Collection**: NLP

# Changelog 
- Wait until all distopt buckets finish optimizer step before updating FP8 scaling factors
- Support dist opt buckets with multiple dtype configs
- Put GPT "leftover params" (i.e. embeddings, layer norm params, biases) in same distopt bucket as first layer params

# Usage

Run GPT, e.g. with the config at https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml.

Enable FP8 support with `model.fp8=True`, FP8 parameters with `model.fp8_params=True`, the distributed optimizer with `model.optim.name=distributed_fused_adam`, and overlapped param all-gathers with `model.optim.overlap_param_sync=True`

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* This PR was implemented together with https://github.com/NVIDIA/apex/pull/1771. Each of the two PRs should be able to run without the other, but there may be performance degradations when training GPT with FP8 params. With the Apex PR and without the NeMo PR, there will be redundant amax reductions. Without the Apex PR and with the NeMo PR, the first bucket all-gather will not be overlapped with the optimizer step.
* This communication optimization used to be present for non-FP8 models, but was removed in https://github.com/NVIDIA/NeMo/pull/7359
* Distributed optimizer support for FP8 models was added in https://github.com/NVIDIA/NeMo/pull/7909

jenkins

jenkins

jenkins

jenkins
Add video support to NeVA LLM conversation data [VideoNeVA]
State: closed | Pull request: yes
Labels: name: stale

name: Multi Modal

# What does this PR do ?

Refactors `preprocess_multimodal` to support video tokens in LLM conversation data

**Collection**: Multimodal - NeVA

# Changelog 
- Adds `videoneva_dataset.py` and `conversation.py` to `/nemo/collections/multimodal/data/`
- `videoneva_dataset.py` implements `TarOrFolderVisualLoader` (modified from `neva_dataset.py`), `tokenize`, and `preprocess_multimodal`
    - `TarOrFolderVisualLoader` is a modification of `TarOrFolderImageLoader` (from `neva_dataset.py`) which extends support to video files with the `open_video` function
    - `tokenize` is identical to `neva_dataset.py` implementation
    - `preprocess_multimodal` extends `preprocess_multimodal` from `neva_dataset.py` to include support for video tokens 
- `videoneva/conversation.py` is a modification of `neva/conversation.py` to include additional video-related constants

# Usage
* You can potentially add a usage example below

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


@yaoyu-33

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Add support for HF style Bert
State: closed | Pull request: no

**Is your feature request related to a problem? Please describe.**

Post-LN is not supported in MCore transformer at the moment (everything is Pre-LN). In order to convert HF checkpoints, we want to add post-LN style to MCore as well. Post-LN requires an additional LN in the beginning of the network as well.

**Describe the solution you'd like**

Please refer to NeMo code update here for HF Bert support: https://github.com/NVIDIA/NeMo/pull/8088
NeVA Tutorial Notebook
State: closed | Pull request: yes

# What does this PR do ?

Adds NeVA tutorial for training / inference

**Collection**: Multimodal

# Changelog 
- New Jupyter notebook under `NeMo/tutorials/multimodal` outlining steps for pretraining and instruction fine-tuning NeVA

# Usage
* Documentation purposes

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


@yaoyu-33 

Verified `torchrun` can be executed in NeMo Container 1.23.10 within the notebook:

**Pretrain**: 
![torchrun-pretrain](https://github.com/NVIDIA/NeMo/assets/30813477/d4323996-795e-4d38-b5ab-a69e5acbecc8)

**Finetune**:
![neva-finetune](https://github.com/NVIDIA/NeMo/assets/30813477/bf9a2580-5a9f-4f90-96c6-2cf24db7d4e6)

**Inference**:
![neva-inference](https://github.com/NVIDIA/NeMo/assets/30813477/0662f743-6fc4-4173-9786-682545b1be01)


jenkins

jenkins
Fix to peft & virtual pipeline parallel unsupported check
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Corrects the check added in [PR-7964](https://github.com/NVIDIA/NeMo/pull/7964) to raise a ValueError when attempting to run virtual pipeline parallel with peft.

**Collection**: nlp/language_modeling

# Changelog 
- Moves this safety check from `MegatronGPTSFTModel.__init__` (where self.use_peft had not yet been set to true, so the check did not function) to `NLPAdapterModelMixin.add_adapter`

# Usage
Running with the unsupported combination of peft and virtual pipeline parallel now yields a more useful error message:

```
python examples/nlp/language_modeling/tuning/megatron_gpt_peft_tuning.py \
    ++model.virtual_pipeline_model_parallel_size=10 \
    ++model.peft.peft_scheme=lora \
    ...
```
gives
```
  File "/opt/NeMo/examples/nlp/language_modeling/tuning/megatron_gpt_peft_tuning.py", line 73, in main
    model.add_adapter(peft_cfg_cls(model_cfg))
  File "/opt/NeMo/nemo/collections/nlp/parts/mixins/nlp_adapter_mixins.py", line 178, in add_adapter
    raise ValueError('Virtual pipeline model parallel is not supported when using PEFT')
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


@cuichenx / @ericharper can you review?  Thanks!

jenkins