renwei2024/nemo-github-issues huggingface dataset:
Mingyuanm/sdxl quantization notebook
State: closed | Pull request: yes
Labels: name: Multi Modal

name: Run CICD

# What does this PR do ?

Update quantization script and adding tutorial/documentation

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
GPU-based vectorized `SpecAug`
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Context: together with @galv we found feature normalization and specaug to take approx 30% of the total forward step time in Canary training, due to CPU-bottlenecked implementations. Feature normalization is addressed in #8964. 

This PR adds a GPU based SpecAugment. The original implementation loops over every example and every mask, then waits on CPU RNG to sample the numbers. The new fast implementation still applies masks sequentially, but is vectorized on batch size and uses GPU's RNG. We found approx. 5x speedup (70ms -> 17ms in profiling, but both numbers include profiler overhead). I also added a flag to be able to revert to the old implementation in case anybody encounters a compatibility issue.

I validated visually that the new impl behavior is as expected. 

Old:
<img width="1696" alt="image" src="https://github.com/NVIDIA/NeMo/assets/15930688/78c5b421-e2ba-4bc7-a121-20eab75f9c29">

New:
<img width="1696" alt="image" src="https://github.com/NVIDIA/NeMo/assets/15930688/31df5106-326b-4679-9ca0-3c9446c0a49e">

**Collection**: ASR

# Changelog 
- 5x faster SpecAugment to reduce typical forward step time by ~10%.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Closing in favor of https://github.com/NVIDIA/NeMo/pull/9155
"greedy_batched" methods should support "partial_hypotheses" option
State: closed | Pull request: no
Labels: name: stale

**Is your feature request related to a problem? Please describe.**

I've been experimenting with `examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py`. One of the things I've noticed is that the "greedy_batched" strategy does not support partial hypotheses. We should add support for this. Right now, streaming of RNN-T models is horrendously slow because we are running the decoder at batch size 1, because we must use the "greedy" strategy when doing streaming. The encoder basically isn't meaningfully contributing to the runtime. The decoder is the main slowdown.

FYI @artbataev .



This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Memory Allocation Error during alignement (tools/nemo_forced_aligner/align.py)
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I am trying to run the `tools/nemo_forced_aligner/align.py` on a very small toy dataset but get the following error:

```
return torch._C._nn.pad(input, pad, mode, value)
RuntimeError: [enforce fail at alloc_cpu.cpp:114] data. DefaultCPUAllocator: not enough memory: you tried to allocate 109086779069136 bytes.
```
The same code works on relatively large dataset but with short audio-clips while in the current dataset I have audios of length
[14, 2, 20, , 2, 16, 19, 2, 14, 2, 7] minutes (`batch_size=2`)


**Steps/Code to reproduce bug**

python tools/nemo_forced_aligner/align.py

Params:
model_path=speech_to_text_ctc_bpe__checkpoint.nemo
manifest_filepath=metadata_small.json
output_dir=save_dir

**Expected behavior**
 
 The  alignements stored in save_dir as follows:
 
 ***ass*** - directory with corresponding *tokens* & *words* folders containing the .ass files
  ***ctm*** - directory with corresponding *tokens*, *segments* & *words* folders containing the .ctm files
 
**Environment overview**

 - Environment location: Local
 - Method of NeMo install: descibed in the NeMo docs

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Windows 10 Pro (22H2) | OS build - 19045.4291
- PyTorch version: torch2.2.1+cu121
- Python version: 3.10.12

**Additional context**

GPU model: GeForce RTX 3090
CUDA: 12.2

I have modified some torch and pytorch_lightning source codes to make the `nemo` work on my Windows. But those changes were mostly related to multiprocessing strategy and other parralelization things which should have nothing to do with the given error, as I managed to run the aligner over a big dataset (but many short audios: up to 15 seconds).


Hi, could you please show more of the error message/where exactly in the NFA code the error happens?
Does the error also happen with `batch_size=1`?

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
TypeError: EncDecRNNTBPEModel.change_vocabulary() got an unexpected keyword argument 'new_vocabulary'
State: closed | Pull request: no
Labels: name: bug

Dear Team,

I am getting this error message, while I am trying to train EncDecRNNTBPEModel model.
Meanwhile the code is Ok with other models, for example with  stt_en_quartznet15x5.  

`TypeError: EncDecRNNTBPEModel.change_vocabulary() got an unexpected keyword argument 'new_vocabulary'
`

**Describe the bug**

```
char_model = nemo_asr.models.EncDecRNNTBPEModel.from_pretrained(model_name="stt_en_conformer_transducer_xlarge", map_location='cpu')
char_model.change_vocabulary(new_vocabulary=list(train_set))

```

Thank you in advance!

**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model


Actually no one of the new models has new_vocabulary argument.
You need to use "change_vocabulary" method

```
char_model.change_vocabulary(
        new_tokenizer_dir="your_tokenized_folder",
        new_tokenizer_type="bpe"
    )

```

You may receive "your_tokenized_folder" whit the help of script/process_asr_text_tokenizer.py

`python process_asr_text_tokenizer.py --data_file=your_plain_text --data_root="your_tokenized_folder" --vocab_size=42 --tokenizer="wpe" --log
`
Enhance Distributed Adam
State: closed | Pull request: yes
Labels: name: core

name: NLP

# What does this PR do ?

This PR enhances distributed adam by:

* Fix deprecation warning of `NCCL_ASYNC_ERROR_HANDLING`;
* Do not overwrite user desired configs for distributed adam;
* Preserve memory format when creating contiguous param buffers;
* Expose one API to let user lock SHArP tree;

This PR works together with https://github.com/NVIDIA/apex/pull/1794.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
New mcore transformer block spec
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

name: Run CICD

# What does this PR do ?

This PR leverages new mcore gpt spec handling. (not yet merged to mcore)

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Add Mcore DistributedDataParallel and distributed optimizer into Nemo
State: closed | Pull request: yes
Labels: name: core

name: NLP

name: Run CICD

# What does this PR do ?

Add Mcore DistributedDataParallel and distributed optimizer into Nemo `examples/nlp/language_modeling/megatron_gpt_pretraining.py`

# Changelog 
- Port DistributedDataParallel to Nemo
- Add a wrapper McoreDistributedOptimizer to bypass torch/PTL assertion check
- Add the optim name `mcore_distributed_optim` to turn on mcore distributed optimizer and a few other mcore related flags (details in next section).
- Verified memory and accuracy between mcore and apex optimizer.

# Usage
* Mcore distributed optimizer usage example is as below: 

```python
optim:
    name: mcore_distributed_optim
    overlap_grad_sync: false 
    overlap_param_sync: false 
    grad_sync_dtype: fp32
    delay_param_gather: false
    delay_grad_reduce: True
    ddp_bucket_size: null
    check_for_nan_in_grad: false
    lr: 0.00012
    weight_decay: 0.1
    betas:
    - 0.9
    - 0.95
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


- EP=2 NGPU=8 
- EP=1 TP=1 NGPU=8 
These also work correctly

@ericharper Can you trigger test?

the tests are triggered, only need to add the "Run CICD" label 

@ericharper will the tests be re-triggered automatically after I made new changes with the CICD label? It looks like it still needs to be manually re-triggered.
[Video-NeVA] Convergence Run Implementation
State: closed | Pull request: yes
Labels: name: stale

name: Multi Modal

# What does this PR do ?

Multi-frame implementation for convergence runs. See `neva_dataset.py` for modified video dataloader and how video frames are merged into T_img dimension.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Fix import in lora merge script
State: closed | Pull request: yes

# What does this PR do ?

Fix import from a deleted script. Add the function directly in merge.py

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update mm dataprep notebook based on feedback
State: closed | Pull request: yes
Labels: name: Multi Modal

# What does this PR do ?

Clarify installation instructions

**Collection**: multimodal

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Implement async distributed checkpoint save
State: closed | Pull request: yes
Labels: name: core

name: NLP

name: Run CICD

# What does this PR do ?

Adds async distribtued checkpoint save implementation.

The base of this PR is set to `mblaz/async-dist-ckpt-minimal-base`, since this PR requires #9016 and #9015 to be merged first and the base branch contains combined changes from those branches.

**Collection**: NLP

# Changelog 
- Adds AsyncFinalizableCheckpointIO async saving wrapper (along with AsyncFinalizerCallback)
- Adds temporary PyT Dist async save strategy (to be replaced with MCore v0.7 version)

# Usage
* set config `exp_manager.checkpoint_callback_params.async_save=True`

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Added tests and docs, undrafting
Support required for fine tuning cache aware streaming model
State: closed | Pull request: no
Labels: name: stale

Hi
I want to fine tune "stt_en_fastconformer_hybrid_large_streaming_multi" model on my custom data.
I want to know some best practices that we can follow to fine tune tune cache aware streaming model.

1. I am using audio of fixed length (2s). Is this good? Can I have audios' of different lengths? Total duration of audio required to finetune on a dataset of different domain (Medical Data)?
2. Which tokenizer to use? Should we finetune using custom tokenizer which will be created with new data or should we fine tune using the default tokenizer and just with new audio? 
3. How can I make this model work with a different language? Can I fine tune this model directly on audio of different language for eg Spansh audio? Or how can we use this on different language?
4. How to resume training for this model because I cannot train in one go? If I finetune using NeMo/examples/asr/speech_to_text_finetune.py? 
5. Should I use speech_to_text_finetune.py or speech_to_text_hybrid_rnnt_ctc_bpe.py, I want to try out with old vocabulary as well as new one and I want to stop and continue training multiple times.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Developer Documents for mcore RETRO
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Saving and reloading the pretrained model's vocab breaks the tokenizer.
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

So I picked [nvidia/parakeet-ctc-0.6b](https://huggingface.co/nvidia/parakeet-ctc-0.6b) and I untarred the the `.nemo`  file.
After that, I then loaded the model and changed the the vocab this way:


**Steps/Code to reproduce bug**

```python
model.change_vocabulary(
            new_tokenizer_dir=vocab_extension_path, new_tokenizer_type="bpe"
        )
```
where `vocab_extension_path` = the path of the pretrained model.
 



**Expected behavior**
The model's tokenizer is supposed to remain intact and not start generating gibberish because I am just reloading the extact tokenizer that was used to pretrain the model.



**Why I need this**
I need to replace some tokens in the model's vocab while keeping the order tokens intact. If I cant keep other parts of the tokenizer intact then my replacement of tokens cannot work.

It should be path to a tokenizer directory not model. 

the directory should contain:
* `tokenizer.model`
* `tokenizer.vocab`
* `vocab.txt`

Yes, that's what I'm doing.
Infact, I've been able to edit the pretrained model's tokenizer and changed the tokens inside of it. 
What I found out is that just reloading the pretrained tokenizer with the change_vocab method scatters the whole decoding process.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Fix mixtral_nemo_to_hf conversion
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Fix convert_mixtral_nemo_to_hf.py

**Collection**: scripts/checkpoint_converters

# Changelog 
 * Fix two typos
 * Allow for custom tokenizer to be passed in

**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

## Who can review?

@cuichenx 

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.


Is it fine & really required to introduce extra param `--input_tokenizer`?

If yes, should we update relevant docs? https://github.com/NVIDIA/NeMo/blob/main/docs/source/ckpt_converters/dev_guide.rst

CC @yaoyu-33 - please have a look, maybe you will have some comments

> Is it fine & really required to introduce extra param --input_tokenizer?

I'm modeling this off #8000. In both cases, the problem is that the converter tries to use the default HuggingFace tokenizer, while the custom model that's being converted may well be using a different tokenizer.

Perhaps the correct solution in both cases is to unpack the tokenizer out of the .nemo file during the conversion script, instead of adding a `--input_tokenizer` argument.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Adding unit test for mcore RETRO model
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

…kenizer

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Alit/griffin
State: closed | Pull request: yes
Labels: name: NLP

name: Run CICD

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Support for Specifying Start and End Time when Reading WAV File
State: closed | Pull request: no

I would like to inquire whether the  toolkit currently supports specifying the start and end time when reading a WAV file. This feature would be beneficial for tasks such as audio segmentation or processing specific sections of longer audio files. 

Like we have a long audio file of `L` length, and which is labeled only from `x1` to `x2` seconds. How can one read the audio file and extract features only for this interval `x1` to `x2`.

In lhotse this is possible using
```python
cut_set = cut_set.trim_to_supervisions(keep_overlapping=False)
```
I am wondering we can do something like that in Nemo.

Thank You

I found it.
this can be done using `offset`.  Supplying this key in your json manifest file allows the dataloader to know your desired start time in an audio file. 
Example: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/speech_classification/datasets.html#voxlingua107
Update black test3 [NOT FOR MERGING]
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Reformatting works as expected, closing
Restore PTQ tests for Llama2
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Restoring PTQ tests for Llama2 model. The tests are moved here from recently disabled [Jenkinsfile](https://github.com/NVIDIA/NeMo/blob/main/Jenkinsfile).

Testing all methods: FP8, INT8 SQ, INT4 AWQ and special case of NULL (just weights export step).

**Collection**: NLP

# Changelog 
- Setting up data and models
- Testing PTQ script (four variants)

# Usage

If you want to reproduce tests locally, scripts included in `tests.setup` module along with `convert_llama_hf_to_nemo.py ` can be used to create tiny test model and data with the following steps (set `TEST_DATA` variable accordingly):

```python
python tests/setup/data/create_sample_jsonl.py \
  --output_file ${TEST_DATA}/test_quantization/test.json

python tests/setup/models/create_hf_model.py \
  --model_name_or_path meta-llama/Llama-2-7b \
  --output_dir ${TEST_DATA}/llama-ci-hf-tiny \
  --config_updates '{"hidden_size": 256, "num_attention_heads": 4, "num_hidden_layers": 2, "num_key_value_heads": 4}'

python scripts/checkpoint_converters/convert_llama_hf_to_nemo.py \
  --input_name_or_path ${TEST_DATA}/llama-ci-hf-tiny \
  --output_path ${TEST_DATA}/llama_ci.nemo
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.


Closing: tests were added in https://github.com/NVIDIA/NeMo/pull/9064
test for black 2 [NOT FOR MERGING]
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Everything works as expected, thanks!
Implement DistributedCheckpointIO
State: closed | Pull request: yes
Labels: name: core

name: NLP

name: Run CICD

# What does this PR do ?

This PR sets the ground for more advanced distributed checkpointing features, like fully parallel save/load or async save which will be configured and implemented as part of DistributedCheckpointIO plugin. It doesn't introduce any new features, just refactors existing behavior.

**Collection**: NLP

# Changelog 
- implement DistributedCheckpointIO plugin for save/load of distributed checkpoints

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Prevent duplicated checkpoints
State: closed | Pull request: yes
Labels: name: core

name: NLP

name: Run CICD

# What does this PR do ?

This PR fixes the behavior of encountering an existing distributed checkpoint. Previously the save was skipped based on logic in `NLPDDPStrategy.save_checkpoint`, which is not a good place for such logic (strategy should just save the checkpoint). This PR reuses the PTL versioning mechanism to adjust the checkpoint name accordingly (add `-v1` suffix to ckpt name)

**Collection**: NLP

# Changelog 
- fix the behavior of skipping duplicate checkpoint by reusing PTL versioning mechanism

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Simplify sharded_state_dict saving
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Simplify NLPDDPStrategy.save_checkpoint methods by using native PTL mechanism for state dict construction.

**Collection**: NLP

# Changelog 
- Create optimizer state dict with `optimizer_state` method
- Create model state entirely in `on_save_checkpoint` method

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

There's no need to comment `jenkins` on the PR to trigger Jenkins CI.
The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.