renwei2024/nemo-github-issues huggingface dataset:
Token count and sequence length logging for MegatronGPTSFTModel
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Adds an option to enable progress bar logging for sequence length (including padding tokens) and real token count for each train step with MegatronGPTSFTModel to enable accurate calculation of tokens/sec and related metrics.

**Collection**: nlp/language_modeling

# Changelog 
- Alters GPTSFTDataset and GPTSFTPackedDataset to pass out a list of token counts as part of the batch
- Alters MegatronGPTSFTModel to expect, average, and log that list of token counts at the beginning of the step if enabled via config

# Usage

```
python megatron_gpt_peft_tuning.py \
    ... \
    ++model.log_token_counts=true
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.


@cuichenx / @ericharper can you review?

jenkins

jenkins

jenkins
langid bug fix 
State: closed | Pull request: yes
Labels: name: cherry-pick

name: stale

name: ASR

# What does this PR do ?

Bug fix for overwritten audio for num_segments>1

**Collection**: ASR

# Usage
* Can be used with num_segments>1

```python
lang_id_model.get_label(path2audio_file, segment_duration=20.0, num_segments = 3, random_seed: int = None)
```

**PR Type**:
- [ ] New Feature
- [ V] Bugfix
- [ ] Documentation




jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
langid bug fix 
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Bug fix for overwritten audio for num_segments>1

**Collection**: ASR

# Usage
* Can be used with num_segments>1

```python
lang_id_model.get_label(path2audio_file, segment_duration=20.0, num_segments = 3, random_seed: int = None)
```

**PR Type**:
- [ ] New Feature
- [ V] Bugfix
- [ ] Documentation




jenkins
Improve PEFT UX
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Two improvements of PEFT UX
1. Correct layer_selection logging messages in the case of more than one adapters.
2. Make output (label, ground truth) field optional for test data, for use cases where user only wants to generate text with gpt_peft_eval and not calculate any metrics. 
   - Note that using the default config file, `data.test_ds.label_key` and `data.test_ds.prompt_template` still reference an "output" field. These do not need to be changed. This PR simply ignores a key error if `data.test_ds.label_key` does not exist in the test data file.


**Collection**: NLP

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation
- [x] User Experience Improvement

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Use latest apex internal API
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Apex _fast_layer_norm added mandatory argument `memory_efficient`. This PR fixes call to this function to accommodate to the change.

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x ] Bugfix
- [ ] Documentation

## Who can review?

@mikolajblaz can you review this one? I can see that you introduced this line 17 months ago. Can we move to the APEX implementation entirely as suggested in the TODO? 

# Additional Information
* Related to # (issue)
Final multimodal PR with our recent developments on MM side
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

multimodal PR with our recent developments on MM side after we start the open sourcing process. Mainly update in Neva/Llava and stable diffusion

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
Fixes NVIDIA/apex installation to not erroneously install the `install` pkg
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

>  Fixes NVIDIA/apex installation to not erroneously install the `install` pkg

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update post
State: closed | Pull request: yes

# What does this PR do ?

Reorder content

**Collection**: [ASR]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation
Fix Adapter for MCore models
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Adapter for MCore models was broken due to upstream changes in Megatron-LM. This PR updates the adapter code paths in NeMo to accommodate the upstream changes.

Tested on Llama2-7b on Squad dataset and accuracy matches before

**Collection**: NLP

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
README edit to change Apple Silicon install instructions (to fix a break introduced by pytorch 2)
State: closed | Pull request: yes

# What does this PR do ?

This PR aims to correct a small error in the Apple Silcon installation instructions in README, where a triton dependency is problematic due to pytorch 2. The use of `/reinstall.sh` doesn't work, so using `pip install` instead. Also added a note about MacBook, to clarify that only asr toolkit is supported.

# Additional Information
* Related to #8116



git clone and cd'ing into NeMo became obsolete when installing from pypi. Imo those commands can be removed?
```shell
git clone https://github.com/NVIDIA/NeMo
cd NeMo
```
In my case, I only get v1.21 installed from pypi. With `pip install 'nemo-toolkit[asr]==1.22.0'` I again run into an triton dependency issue:
`ERROR: Could not find a version that satisfies the requirement triton (from nemo-toolkit) (from versions: none)`

According to [this comment](https://github.com/openai/triton/issues/1057#issuecomment-1500691831), macOS is not supported by triton at all.
What eventually _did_ work for me was removing `triton` from the requirements altogether and install from the local checkout.
`$ pip install '.[asr]'`
Draft:add fp8 weight caching to mcore_gpt
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


@erhoo82 can you please review?

Can you add the case handling the validation step as well? At validation, we need to cache the fp8 w/t at the first micro-batch in the first global batch, and no need to cache in the following global batches. I think you can use `model.training` to tell if the current micro-batch belongs to the training or validation routine. Ned to tell if the current global-batch is the first global-batch after a training step routine.

This is the reference. 
https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/modules/common/megatron/transformer.py#L1543-L1544

I tested validation and it seems ok; the global batch size is the same between validation and training, so the is_first_microbatch logic is unaffected, and validation and training both share `get_forward_output_and_loss_func`
The case where the global batch size is changed should be handled by this [code](https://github.com/NVIDIA/NeMo/blob/99fa120c637bafd7ec364ff2159661ec0784a4e0/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py#L890): 

In validation, we only need to cache the fp8 w/t at `the first global-batch` + `the first micro-batch`. Did you check if fp8 w/t is not cached in the other first micro-batch in the later global-batch?

Good point thanks, I added a fix for validation 

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
fix gated_linear_unit config
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins
Add missing audio_filepath validation for Canary
State: closed | Pull request: yes
Labels: name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Remove Pydantic Version Constraint Due to Inflect Update
State: closed | Pull request: no

Currently, pydantic is pinned to version 1, which leads to conflicts with common frameworks such as fastapi.

https://github.com/NVIDIA/NeMo/blob/d3c589646b453f0395f19c2eef1ed958d048c6c7/requirements/requirements_common.txt#L4

It seems that inflect now supports pydantic 2, see https://github.com/jaraco/inflect/issues/187. Their requirements now list `pydantic >= 1.9.1`. Given this update, is it possible to remove the pydantic version constraint without further changes?

I'll unpin it tomorrow, seems it should be fine.
Added marathi ITN support in documentation
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Added marathi ITN support to the documentation in wfst_text_normalization.rst. This corresponds to the pr https://github.com/NVIDIA/NeMo-text-processing/pull/134 in nemo_text_processing which adds ITN support for Marathi language.

# Changelog 
- Added marathi ITN support to the given file NeMo/docs/source/nlp/text_normalization/wfst/wfst_text_normalization.rst .

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you add or update any necessary documentation?
  
**PR Type**:
- [x] New Feature
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?
Anyone in the community is free to review the PR once the checks have passed. 

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

Commenting to remove stale tag

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Installation error on M1 Macbook - No matching distribution found for triton
State: closed | Pull request: no
Labels: name: bug

I am installing NeMo on M1 Macbook, following the installation guide for M1:
https://github.com/NVIDIA/NeMo?tab=readme-ov-file#mac-computers-with-apple-silicon

All works fine in Conda environment, up to running ./reinstall.sh. After a few successful packages installed I get this error:

```
INFO: pip is looking at multiple versions of nemo-toolkit to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement triton (from nemo-toolkit) (from versions: none)
ERROR: No matching distribution found for triton
```

The triton compatibility statement on the GitHub readme says Linux only:
https://github.com/openai/triton#compatibility

Is there a way to work around this in the M1 Mac installation?

- OS version: 14.1.2 (23B92)
- PyTorch version: 2.1.2 
- Python version: 3.11.0


Triton dependency comes from pytorch 2.0 - you can try to install pytorch CPU first and then install NeMo instead of installing everything at the same time 

Thank you! I used conda to install, like this:

![image](https://github.com/NVIDIA/NeMo/assets/16166531/fbe278fb-a853-46a7-94cb-0acf637784f1)

But still getting this from ./reinstall.sh

![image](https://github.com/NVIDIA/NeMo/assets/16166531/fb67636a-27fe-4f4b-8fa3-c093fba46f1c)

I'll dig into it a bit more.

Btw only the asr toolkit can be guaranteed installed on MacBook. Don't use reinstall.sh, use pip install nemo_toolkit[asr] 

Those instructions were fine for pytorch 1.x but triton complicates things 

Thank you so much for your help - that worked with `pip install 'nemo_toolkit[asr]'`
Finetuning ASR models and change the vocabulary error！！！
State: closed | Pull request: no
Labels: name: bug

name: stale

My code:
....
model = nemo_asr.models.ASRModel.from_pretrained("stt_en_citrinet_512")
model.change_vocabulary(
    new_tokenizer_dir="./tokenizers/an4/tokenizer_spe_unigram_v32/",
    new_tokenizer_type="wpe"
)

**error:
You tried to register an artifact under config key=tokenizer.vocab_path but an artifact for it has already been registered**.
![1704261547987](https://github.com/NVIDIA/NeMo/assets/38098690/61c3d694-a2fe-487c-9f23-9cf34e8cf0c9)


That's more of a warning than an error. The actual crash is because of the OSErrorv- network is unreachable. HF wpe requires connecting to the HF servers even for local models, you need to use the environment flag TRANSFORMERS_OFFLINE=1 (or set that in os.environ before importing HF). 

Otherwise you can create and use a sentence piece tokenizer instead (recommended) 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Add NeMo Parakeet post
State: closed | Pull request: yes

# What does this PR do ?

Adds the content for Parakeet post.

**Collection**: [ASR]

# Changelog 
- Add parakeet post
- Update template to refer to 2024

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Fix lora merge script
State: closed | Pull request: yes

# What does this PR do ?

- Fix typo in "mcore" argument location
- Remove dependency on MegatronGPTLoRAModel, which has been deprecated since #7308

**Collection**: NLP


# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins


jenkins
Multi codec support (nemo audio codec), CTC loss and context patterns
State: closed | Pull request: yes
Labels: name: TTS

name: stale

name: ASR

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
transcribe fn for Canary models
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Implementation of Disfluencies in TextToSpeech
State: closed | Pull request: no
Labels: name: bug

name: TTS

name: stale

I am working on a project that involves creating dialogues that sound spontaneous and natural. A key feature I would like to implement is the use of disfluencies (such as “Mm-hmm”) during moments when calculations are being made or there are pauses in the dialogue.

So far, I have experimented with using phonemes to create these disfluencies, but the results have not been satisfactory. It's possible that I haven't found the correct IPA pronunciation for these expressions. (If you have any IPA pronunciation suggestions to try, they would be greatly appreciated).

My questions are as follows:

- Is the integration of disfluencies in dialogue currently supported?
- If possible, could you recommend a strategy or approach to implement these disfluencies more naturally?


Any update on this?

Hi @rodrigoGA, thanks for your interest in NeMo TTS toolkit. If sticking to our current phoneme-based TTS models, such as FastPitch, you have to add new IPA dictionary entries for those filler words, and better to have paired filler speech as well.  

So far as I know, we haven't added support for disfluencies synthesis yet, unless the training corpus has filler speech/text pairs and corresponding phonemes. 

If you have filler speech/text pairs, and can't figure out canonical phonemes for filler words, you may try to use grapheme-based tokenizer for FastPitch. 

Hi, thanks for the detailed response. It's really interesting and useful to consider support for disfluency synthesis. I hope this feature will be considered for future training versions.


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Pre-generate cu_seqlens argmin and max_seqlen to remove host-to-device sync
State: closed | Pull request: yes
Labels: name: NLP

…e data copies

# What does this PR do ?

At SFT, pre-generate `cu_seqlens_argmin` and `max_seqlen` to skip CPU-side computation on the main process.
This removes host-to-device sync, which is performance-critical when each kernel execution is short.

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins


jenkins

@ericharper
Please review this PR.

Just waiting on internal CI tests. They look good so far. We'll most likely merge first thing tomorrow.

jenkins

jenkins


jenkins

jenkins

jenkins

jenkins
[pre-commit.ci] pre-commit suggestions
State: closed | Pull request: yes
Labels: name: stale

<!--pre-commit.ci start-->
updates:
- [github.com/pre-commit/pre-commit-hooks: v4.3.0 → v4.5.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.3.0...v4.5.0)
- [github.com/PyCQA/isort: 5.12.0 → 5.13.2](https://github.com/PyCQA/isort/compare/5.12.0...5.13.2)
- [github.com/psf/black: 19.10b0 → 23.12.1](https://github.com/psf/black/compare/19.10b0...23.12.1)
<!--pre-commit.ci end-->

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Add a progress bar when loading model from pretrained
State: closed | Pull request: no
Labels: name: feature

name: stale

I'm attempting to download a pre-trained model from the NGC source. Unfortunately, my network connection is not performing optimally. The absence of a download progress bar makes it difficult for me to estimate the remaining time for completion, resulting in frustration and uncertainty during the download.

I suggest incorporating a progress bar into the pre-training model download process. This feature should display both the download progress and speed, providing users with real-time feedback and enhancing the overall downloading experience.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Speechllm tts paarth - Alignment CTC loss, custom number of codebooks (dac) and other changes
State: closed | Pull request: yes
Labels: name: TTS

name: stale

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Support transcoding audio formats when saving tarred datasets (FLAC, OPUS)
State: closed | Pull request: yes

# What does this PR do ?

Supports saving audio into compressed formats when creating a tarred dataset.

**Collection**: Any speech collection

# Changelog 
- Add option `--force_codec [name]` to `scripts/speech_recognition/convert_to_tarred_audio_dataset.py` 
- Acceptable values: `wav`, `flac`, `opus`, possibly others supported by libsndfile
- When `--force_codec` is provided, and the audio file extension is different than codec's name, we'll read the audio and transcode it in memory, then save to tar; otherwise the existing code path is selected
- Change the default to save files as `opus`. Soundfile uses 28kbps for OPUS which is adequate for podcasts/audiobooks according to https://wiki.xiph.org/Opus_Recommended_Settings -- the result is ~17x smaller than FLAC and 26x smaller than WAVE (current default). 

# Usage

```bash
python scripts/speech_recognition/convert_to_tarred_audio_dataset.py --manifest_path nemo_manifest.json --num_shards 4 --force_codec opus --workers 4 --max_duration 30
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins
Fix for #8100: Audio with filename containing spaces generate invalid rttm files
State: closed | Pull request: yes
Labels: name: stale

name: ASR

name: common

# What does this PR do ?

Resolves issue #8100. Ensure that the unique_name saved and expected to be found in rttm is uri-compliant  and does not contain spaces even if the original audio file contained spaces.
**Collection**: [Note which collection this PR will affect]

# Changelog 
- RTTM filenames are now uri encoded

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # 8100


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Reconfigure limit_val_batches only for int
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins
Read files BytesIO format
State: closed | Pull request: no
Labels: name: stale

**Is your feature request related to a problem? Please describe.**

I have audio files in BytesIO, how can I feed them into transcribe if this function expects file_path?

**Describe the solution you'd like**

The transcribe method expects not only file_path but also BytesIO

**Describe alternatives you've considered**

I made a custom EncDecCTCModelBPE and read librosa files, but the accuracy of the transcription is reduced in contrast to the basic EncDecCTCModelBPE.

**Additional context**

-


We currently do not support any input other than file path. However, subclassing and overriding the transcribe function should be fine, usually does not cause wer degradation. 

> We currently do not support any input other than file path. However, subclassing and overriding the transcribe function should be fine, usually does not cause wer degradation.


How Nemo reads sound files by default? DALI, librosa?


This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Canary: option to limit the number of open streams
State: closed | Pull request: yes
Labels: name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Privately reporting a Vulnerability to NVIDIA/NeMo 
State: closed | Pull request: no

Hey , 

I have identified a vulnerability in the latest codebase NVIDIA/NeMo. 
Could you please enable the GitHub feature for [private vulnerability reporting](https://docs.github.com/en/code-security/security-advisories/working-with-repository-security-advisories/configuring-private-vulnerability-reporting-for-a-repository), or in case NVIDIA Corporation has a bug bounty program in HackerOne platform for instance, I would like to report that.

Thank you  
@sim4n6

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

The report has been submitted in private : https://github.com/NVIDIA/NeMo/security/advisories/GHSA-x392-p65g-4rxx

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

Well, it is not in a stale state. But rather I've been waiting for a CVE to be assigned and the report to be published.
Transformer decoder with conditioning for canary
State: closed | Pull request: yes
Labels: name: ASR

name: NLP

name: common

# What does this PR do ?

Transformer decoder with conditioning for canary

**Collection**: ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
EncDecTransfModelBPE model has bugs in transcribe() method 
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

1. `setting return_hypotheses=True` in EncDecTransfModelBPE.transcribe() will crash because of undefined local variables ([here](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/models/transformer_bpe_models.py#L261))
2.  EncDecTransfModelBPE.transcribe() has different APIs from other ASR models, causing `transcripbe_speech.py` to crash. (to be fixed in #8087 )



This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
official training logs with loss/wer
State: closed | Pull request: no
Labels: name: stale

**Is your feature request related to a problem? - Describe the solution you'd like**

Firstly, I want to express my gratitude for the comprehensive documentation provided, which I find immensely valuable.

When initiating training for a new model, distinguishing between a potential bug and normal behavior during the training process can be quite challenging. Having access to continues logging metrics loss/WER (Word Error Rate) from your training sessions would significantly enhance our debugging capabilities and the security of our training procedures. Would it be feasible for you to release or publish these logs?

**Additional context**
Something of that kind would be amaizng:
https://wandb.ai/patrickvonplaten/wav2vec2-pretrained-demo/reports/Wav2Vec2-PreTraining-Demo-Run--VmlldzoxMDk3MjAw?accessToken=oa05s1y57lizo2ocxy3k01g6db1u4pt8m6ur2n8nl4cb0ug02ms2cw313kb8ruch
But even just screenshot of tensorboard will be great

Probably won't be possible to share wandb

thanks @titu1994 , wandb would be a best case scenario, but even a tensorboard screenshot will be amazingly valuable 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.