renwei2024/nemo-github-issues huggingface dataset:
Added Vietnamese language
State: closed | Pull request: yes
Labels: name: TTS

name: common

# What does this PR do ?
Add Vietnamese into tokenizer package
Add a one line overview of what this PR aims to accomplish.
Define and set all of this (Vietnamese's alphabet, symbols, punctuations, tildes, phonemes) properties

# Changelog 
tts_tokenizers.py
ipa_lexicon.py
tokenizer_utils.py

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Did you add or update any necessary documentation?

  
**PR Type**:
- [x] New Feature
- [x] Documentation
Pipeline Parallel Evaluation for SFT/PEFT
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

This PR enables the inference of pipeline parallel fine-tuned language models using the `megatron_gpt_generate.py` script. Middle stages during pipeline parallel are expected to have no output during the response generation. We handle this case in the PR to enable evaluation of SFT and PEFT models trained with pipeline parallelism. 

**Collection**: NLP

# Changelog 
- Reorganized logic for generation by checking if generation has any output (line 430).
- Middle stages of pipeline parallel have no output, so we set empty lists on line 439.

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.


jenkins
bug fix in long-form transcription for canary [ASR]
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

current code for canary long-form transcription adds "none" before every segment. 

**Collection**: [ASR]

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
[SD] CUDA Graphs update
State: closed | Pull request: yes
Labels: name: bug

name: core

name: NLP

name: CI

name: Multi Modal

# What does this PR do ?

Restore CUDA Graphs support for SD.

**Collection**: multimodal

# Changelog 
- Update in the CUDA Graph callback to align changes in PL API.
- `MegatronLatentDiffusion.training_step` accepts `batch` input instead of `dataloader_iter` so CUDA Graphs are supported.
- Remove circular dependencies in `nemo/core/optim/megatron_fused_adam.py`.
- Add missing resblock_gn_groups parameter.
- Exception is raised when gradient reduction is misconfigured.
- Add synthetic data-loader for SD.
- Tensors are allocated directly on GPU to avoid synchronizations with host in few places.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins
PP Evaluation for SFT/PEFT
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: CI

name: common

name: Multi Modal

# What does this PR do ?

This PR enables the inference of pipeline parallel fine-tuned language models using the `megatron_gpt_generate.py` script. Middle stages during pipeline parallel are expected to have no output during the response generation. We handle this case in the PR to enable evaluation of SFT and PEFT models trained with pipeline parallelism. 

**Collection**: NLP

# Changelog 
- Reorganized logic for generation by checking if generation has any output (line 430).
- Middle stages of pipeline parallel have no output, so we set empty lists on line 439.

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.


jenkins
bug fix in transcribe_speech.py [ASR]
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Bug fix in transcribe_speech.py: temp manifest file was deleted before calling writing predictions.

**Collection**: [ASR]

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Woops, i fast merged, @pzelasko if you need any changes let me know I'll revert 

looks good :)
[DRAFT] Add timeout parameter
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Based on https://github.com/NVIDIA/Megatron-LM/pull/723, this PR aims to make user-set timeout applicable to all ProcessGroups instead of just default ProcessGroups

**Collection**: [Note which collection this PR will affect]

# Changelog 
During model parallel initialization, it will pass a timeout parameter (which is the existing attribute of `NLPDDPStrategy`) to new ProcessGroup creation (`new_group`)

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
Need to wait https://github.com/NVIDIA/Megatron-LM/pull/723 to be merged first


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Add data_prefix check for float limit_val_batches
State: closed | Pull request: yes
Labels: name: stale

name: NLP

name: CI

# What does this PR do ?

Set num of eval samples as 1 for float `limit_val_batches` only when the data_prefix contains single prefix, since otherwise it can lead to no validation.

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Key 'num_classes' is not in struct
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I am trying to run a train process of fastconformer_hybrid_transducer_ctc_bpe_streaming, but obtaining an error

```
omegaconf.errors.ConfigKeyError: Key 'num_classes' is not in struct
    full_key: decoder.num_classes
    object_type=dict

```
**Steps/Code to reproduce bug**

`OC_CAUSE=1 HYDRA_FULL_ERROR=1 python ${NEMO_ROOT}/examples/asr/asr_ctc/speech_to_text_ctc_bpe.py   --config-path=${PWD}/conf/conformer/   --config-name=fastconformer_hybrid_transducer_ctc_bpe_streaming_kazakh   exp_manager.name="KAZ_MCV_ISSAI_V1"    exp_manager.exp_dir=results/   ++model.encoder.conv_norm_type=layer_norm`

**Expected behavior**

to run a train

**Environment overview (please complete the following information)**

 - Environment location: GCP
 - Method of NeMo install: pip install['all']

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- Ubuntu 22.04
- PyTorch version: 2.2.1
- Python version: 3.11.2

**Additional context**

The part of the config where num_classes is:
```
  # The section which would contain the decoder and decoding configs of the auxiliary CTC decoder
  aux_ctc:
    ctc_loss_weight: 0.3 # the weight used to combine the CTC loss with the RNNT loss
    use_cer: false
    ctc_reduction: 'mean_batch'
    decoder:
      _target_: nemo.collections.asr.modules.ConvASRDecoder
      feat_in: null
      num_classes: -1
      vocabulary: []
    decoding:
      strategy: "greedy"
```

Your config seems to be for a hybrid model, not CTC. You'll need to use this instead - https://github.com/NVIDIA/NeMo/blob/main/examples/asr/asr_hybrid_transducer_ctc/speech_to_text_hybrid_rnnt_ctc_bpe.py
multimodal retriever -- clip score fusion model
State: closed | Pull request: yes
Labels: name: stale

name: Multi Modal

# What does this PR do ?

Multimodal retriever (CLIP score fusion model & training)

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
fix race condition in exp manager
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

in exp manager we move files on rank0, but this move is not always completed by the time the other ranks go and do other things such as try to run `ls` on the file system. So this adds a sync to make sure


jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Use mcore adapter hooks
State: closed | Pull request: yes
Labels: name: stale

name: NLP

name: CI

# What does this PR do ?

Use hook functions in mcore for PEFT logic instead of copy-pasting mcore code.
This prevents any mcore change from propogating to nemo
Also add mcore PEFT CI tests for all models.

Keeping this PR ready to run CI tests. Do not merge until mcore PR is finalized

**Collection**: NLP

# Usage
* You can potentially add a usage example below

no usage change

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation
- [x] Better dev experience

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Prevent Redundant Gather for LoRA Sequence Parallel
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Get gathered output from TE layernorm to prevent the redundant gather in parallel adapter. This improves the performance of LoRA SP.
Corresponding TE PR: https://github.com/NVIDIA/TransformerEngine/pull/697

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Fix bug in training Frame-VAD documentation
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Fix ia3 mlp infused adapter
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add back lines mistakenly deleted in #7988 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins
Update docs for NeMo Framework
State: closed | Pull request: yes

# What does this PR do ?

Updates docs for NeMo framework 

**Collection**: [All]

# Changelog 
- Move LLM to top, ASR next, Multimodal 3rd
- Move introduction to index.rst Som Note: Injected entire intro.rst into index.rst for now - it looks cleaner and if user explicitly visits the intro page they get same content as just visiting the index site. Moved the links to the very bottom (cannot hide them, its used as the ordering of the items on the left). 
- Speech Processing to Speech AI - try not to break links
- Move migration guide to bottom of list called Upgrade guide
- Rename “Foundation Vision Models in NeMo” -> Vision Models 
- Move NeMo Core to the top
- Move Text processing to inside of NeMo Core -- Som Note: I didn’t merge into nemo core, instead I renamed the collection to “Common collection” - that way it remains at the bottom (low visibility is good for common collection) and doesn’t break links.
- Make all links - black with blue hyperblink 

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation


FYI @titu1994 : regarding 

> Moved the links to the very bottom (cannot hide them, its used as the ordering of the items on the left).

You can hide the links while preserving their ordering for the sidebar by including the `:hidden:` option like this:

```rst
.. toctree::
  :hidden:
```

See [sphinx docs](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#:~:text=You%20can%20also%20give%20a%20%E2%80%9Chidden%E2%80%9D%20option%20to%20the%20directive%2C%20like%20this%3A).
Update text for new version 
State: closed | Pull request: yes

# What does this PR do ?

Updates text for domains on website

**Collection**: [Website]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.
fix: start_time type of int instead of float
State: closed | Pull request: yes

# What does this PR do ?

Fix : You can now completely use the `minimum_timestamp_duration` option, which bugged if the value was too large.

**Collection**: ASR

# Changelog 
As we could see in the traceback, the problem appeared during the use of the `get_ctm_line` function because `start_time` was a type of **int** at [this line](https://github.com/NVIDIA/NeMo/blob/main/tools/nemo_forced_aligner/utils/make_ctm_files.py#L98) :
At the beginning, `boundary_info_.t_start` is definitely a **float** with the value 0.0 and `start_time` is also a float after [copying](https://github.com/NVIDIA/NeMo/blob/main/tools/nemo_forced_aligner/utils/make_ctm_files.py#L88). Everything is fine for now.
The source of the problem is at [this line](https://github.com/NVIDIA/NeMo/blob/main/tools/nemo_forced_aligner/utils/make_ctm_files.py#L98) : When using the `max` function, if we have set a `minimum_timestamp_duration` value that is too high. `start_time` will become 0 BUT type of **int** and **no more** type of **float**.
So when we get to the use of the `get_ctm_line` function, this one checks the type of `start_time` and since it's not the right type, it raises a type exception [here](https://github.com/NVIDIA/NeMo/blob/935b9c78f4e75e873f8511121bd846e7dabca4f3/nemo/collections/asr/parts/utils/manifest_utils.py#L97).

Consequently, the **max function argument** should be changed to **0.0** to be always a type of **float**.


# Usage
- You can now really use the option **ctm_file_config.minimum_timestamp_duration=0.5** to change the value of `minimum_timestamp_duration` no matter how long the tokens/words/segments of your transcription is.
Example : 
```
python <path to NeMo>/tools/nemo_forced_aligner/align.py pretrained_name="stt_en_fastconformer_hybrid_large_pc" manifest_filepath=nemo/manifest.json output_dir=nemo/output ctm_file_config.minimum_timestamp_duration=0.5
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #8592
ValueError when higher minimum_timestamp_duration given : "start_time must be a float or str containing float, but got int"
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
I wanted to put a `minimum_timestamp_duration` to visualize the result given by nemo forced aligner with [Gecko](https://github.com/gong-io/gecko) as the documentation recommends. 
When I change the value of `minimum_timestamp_duration` to 0.4, it's all good.
If I try to change the `minimum_timestamp_duration` for a higher value like 0.5, I get a `ValueError` : 
![image](https://github.com/NVIDIA/NeMo/assets/66870443/ca90e8df-a4f2-41d4-a3a2-1867e0b37cae)
**Steps/Code to reproduce bug**

- Notice the option **ctm_file_config.minimum_timestamp_duration=0.5** to change the value of `minimum_timestamp_duration`. To be sure to have the error, you can put an extremely high value like 50.
```
python <path to NeMo>/tools/nemo_forced_aligner/align.py pretrained_name="stt_en_fastconformer_hybrid_large_pc" manifest_filepath=nemo/manifest.json output_dir=nemo/output ctm_file_config.minimum_timestamp_duration=0.5
```
The value that is too high and causes the error **depends on** the **respective duration of your tokens/words/segments** that are calculated in this [loop](https://github.com/NVIDIA/NeMo/blob/1882d0d0025832a530063e5b699347253b2ab768/tools/nemo_forced_ligner/utils/make_ctm_files.py#L83). The loop iterates over **each token/word/segment** of the transcription and finally produces the `ctm` file).


**Expected behavior**

- No error should be raised.

**Environment overview**

 - Method of NeMo install: source

**Additional context**
**_This is a minor error that I've already fixed locally, so I'll make a pull request in a few minutes_**





PR with fix has been merged, so closing this. Thank you for the fix.
qkv format error in GPT Eval when bs=1 and using fused rope kernel
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

Opening a github issue for public tracking purposes. (internal #4489257)

megatron_gpt_eval.py throws the following error:
```
...
    core_attn_out = super().forward(
  File "/usr/local/lib/python3.10/dist-packages/transformer_engine/pytorch/attention.py", line 2099, in forward
    qkv_layout, query_layer, key_layer, value_layer = _get_qkv_layout(
  File "/usr/local/lib/python3.10/dist-packages/transformer_engine/pytorch/attention.py", line 1209, in _get_qkv_layout
    raise Exception("The provided qkv memory layout is not supported!")
Exception: The provided qkv memory layout is not supported!
This only happens when all of the following are true:
```

micro batch size = 1 (mbs = 2 works)
apply_rope_fusion = True
rope fusion is available (i.e. container has latest Apex)
running an inference workload – megatron_gpt_eval.py, or megatron_gpt_generate.py, or validation loop during training. (training with mbs = 1 and mbs = 2 both work)

**Steps/Code to reproduce bug**

run inference with any gpt-style model (e.g. llama2)
```
torchrun --nproc_per_node=1 /opt/NeMo/examples/nlp/language_modeling/megatron_gpt_eval.py \
    gpt_model_file=<path to model> \
    inference.greedy=True \
    inference.add_BOS=True \
    trainer.devices=1 \
    trainer.num_nodes=1 \
    tensor_model_parallel_size=1 \
    pipeline_model_parallel_size=1 \
    prompts=["deep learning is"]
```


**Environment overview (please complete the following information)**

nvcr.io/nvidia/nemo:24.01.framework



This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

fixed in nemo 24.03 release / megatron core 0.6.0 release
fix to RNNT bug
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Safer fix for https://github.com/NVIDIA/NeMo/pull/8587#event-12018166291

Metrics should accumulate between calls. This little fix allows that function for `WER`.

**Collection**: [ASR]

# Changelog 
- Added accumulation to wer metrics

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [Y] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [N] Did you write any new necessary tests?
- [N] Did you add or update any necessary documentation?
- [N] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [Y] New Feature
- [Y] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix bug in RNNT Joint WER calculation for fused batch
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fix metric calculation in the case fused batch size > 1 during training / validation calculation.

Previously, the RNNT metric was stateful while the CTC one was not ([r1.22.0](https://github.com/NVIDIA/NeMo/blob/r1.22.0/nemo/collections/asr/metrics/rnnt_wer_bpe.py#L419-L420), [r1.23.0](https://github.com/NVIDIA/NeMo/blob/r1.23.0/nemo/collections/asr/metrics/wer.py#L333))

Therefore this calculation in the RNNT joint for fused operation worked properly. However with the unification of metrics in r1.23.0, a bug was introduced where only the last sub-batch of metrics calculates the scores and does not accumulate.

**Collection**: [ASR]

# Changelog 
- Preserve the list of metric outputs from each sub-batch
- take average of batch wer, compute running sum of numerator and denominator


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation


jenkins
validation consumed samples is always set to 0 in megatron gpt pretraining
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

validation consumed samples is always set to 0 in megatron gpt pretraining ([link](https://github.com/NVIDIA/NeMo/blob/a06835f50dd18a67f5b4fedfb56985f9b3573985/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py#L1342)). On resuming training from a checkpoint, consumed samples for validation dataset should be set from the checkpoint instead of getting initialized to 0.

**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model


+@athitten

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
WER.update doesn't work 
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
As shown in
(https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/metrics/wer.py#L349-L350), the new scores and words will be assigned to the object and the pervious scores and words are dropped. 
1) we might rename this function to WER.set
or 
2) we might update the code as
> self.scores += torch.tensor(scores, device=self.scores.device, dtype=self.scores.dtype)
> self.words += torch.tensor(words, device=self.words.device, dtype=self.words.dtype)

The current code could lead to WER report inconsistent during training and inference if we use fuse_loss_wer in the Transducer model training, i.e., [model.joint.fuse_loss_wer=True and model.joint. fused_batch_size > 1](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/conformer/conformer_transducer_bpe.yaml#L194).  In this setting, only the last sub-mini-batch WER is accumulated during validation stage. 




Thank you very much for raising this! We have fixed it in this PR - https://github.com/NVIDIA/NeMo/pull/8587 
It occurred due to a large refactor and unification of metrics in ASR to make it simpler to extend in the long run. 

The patch will be there in the next NeMo release, and we have added a release note in the 1.23 release page https://github.com/NVIDIA/NeMo/releases/tag/v1.23.0 so that users are aware and can utilize correct metrics during evaluation by using the speech to text eval script (or disabling fused batch explicitly)

Fixed via #8587
Add config key for dropout position in LoRA adapter
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do?

Added dropout_position key in the Lora config to decide the dropout position in Lora adapter. Both [official](https://github.com/microsoft/LoRA/blob/main/loralib/layers.py#L149) Lora implementation and [PEFT](https://github.com/huggingface/peft/blob/9119b780ebac7859db5753ebad50d94ba803c99c/src/peft/tuners/lora/layer.py#L437) one are using dropout as a first operation while Nemo allows it only as a last operation. The dropout_position key allows to set position to pre to use it in the same way as the official implementation or post to leave it as it is right now.

**Collection**: nlp

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


jenkins

jenkins

jenkins

jenkins

jenkins
Multiprocessing fork, spawn issue when attempting to run Nemo ASR on a Sanic server
State: closed | Pull request: no
Labels: name: bug

name: stale

I'm trying to create a Sanic server for Nemo ASR. It seems Nemo uses some fork method, while Sanic requires spawn.

**Code:**
```
import nemo.collections.asr as nemo_asr
from sanic import Sanic
app = Sanic(__name__)  
if __name__ == '__main__':
    app.run(debug=True)
```

**Error Message:**
`RuntimeError: Start method 'spawn' was requested, but 'fork' was already set.`

**Things I already tried, but nothing works:**

```
import multiprocessing as mp
mp.set_start_method('spawn', force=True)
```
```
torch.multiprocessing.set_start_method('spawn', force=True)  
```
```
try:
         mp.set_start_method('spawn', force=True)
         print("spawned")
except RuntimeError:
         pass 
```

I request experts to help me or share some code snippets if they are publicly available.

Thanks 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Fixed missing copy import in rnnt_decoder.py
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

When using compute_timestamps with RNNT models, an error is encountered because of a missing `copy` import (where `copy` is Python's built-in [copy](https://docs.python.org/3/library/copy.html) module).

**Collection**: ASR

# Changelog 
- One-line change: `import copy` at top of [rnnt_decoding.py](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/parts/submodules/rnnt_decoding.py) file.

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
    - Not applicable, tiny bugfix.
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Speech Synthesizer cannot produce more than 11 seconds of audio.
State: closed | Pull request: no
Labels: name: stale

Is there some parameter needed to increase the length of audio that the speech synthesizers supported by NeMo can produce?
This code here does not produce audio that covers the Text I provide in the code below, I only get 11 seconds of Audio. Also what is the maximum length of audio that can be produced at a time?
```
import torch
import IPython.display as ipd
from matplotlib.pyplot import imshow
from matplotlib import pyplot as plt
from nemo.collections.tts.models.base import SpectrogramGenerator, Vocoder
import soundfile as sf
import sys

# List pretrained models available in NeMo
print("Spectrogram Models")
print(SpectrogramGenerator.list_available_models())

print()
print("Vocoders")
print(Vocoder.list_available_models())


if torch.cuda.is_available():
  device = "cuda"
else:
  device = "cpu"

# Load our spectrogram and vocoder models onto our CPU
spectrogram_model = SpectrogramGenerator.from_pretrained("tts_en_tacotron2").eval().to(device)
vocoder = Vocoder.from_pretrained("tts_en_hifigan").eval().to(device)

text = "Within the chasmic depths of my mind, vague shapes, sounds and visions fight to be witnessed. A scattering of incoherent memories..... gateways to the blessed palace...... Darkness......Phase equations made flesh.....Darkness.....Waking dreams of a passionless mortal life long though buried....Darkness....Archs of impossible colors claw desperately across the back of my electronic eyes, my machine souls last attempt to ignite conciousness....Darkness....Light, waking neurons fired into life like a birthing galaxy aroused by the sound of plaintiff cries, not human or A.I, not daemon, distant. Ethereal to my spirit."

# Normalize the text and convert it into individual phonemes/tokens.
tokens = spectrogram_model.parse(text, normalize=True)

# Generate spectrogram from text
spectrogram = spectrogram_model.generate_spectrogram(tokens=tokens)

# Invert the spectrogram into audio samples
audio = vocoder.convert_spectrogram_to_audio(spec=spectrogram)

# Convert output from pytorch tensor to numpy array
spectrogram = spectrogram.cpu().detach().numpy()[0]
audio = audio.cpu().detach().numpy()[0]
audio_file = "output_audio.wav"
sf.write(audio_file, audio, samplerate=22050)
print(f"Audio saved to {audio_file}")

print(f'"{text}"\n')

ipd.Audio(audio, rate=22050)
```

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.
Fixing data types for cached states
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

Some dimensions of the cached states should be exported as static, not dynamic.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Jenkins CI
To run Jenkins, a NeMo User with write access must comment `jenkins` on the PR.

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.
Cannot disable progress bar
State: closed | Pull request: no
Labels: name: bug

When setting `+trainer.enable_progress_bar=False` in the config, the application crashes since `CustomProgressBar` is always passed as callback to PyTorch lightning trainer.

```
    raise MisconfigurationException(
lightning_fabric.utilities.exceptions.MisconfigurationException: Trainer was configured with `enable_progress_bar=False` but found `CustomProgressBar` in callbacks list.
```

@athitten, can you take a look at this when you have time?

@ksivaman this issue should be fixed in this PR: [PR 8663](https://github.com/NVIDIA/NeMo/pull/8663). Can you please help review it ? Thank you.