renwei2024/nemo-github-issues huggingface dataset:
RuntimeError: shape '[16, -1, 128]' is invalid for input of size 113920
State: closed | Pull request: no
Labels: name: bug

name: stale

@titu1994 @meghmak13
Dear Team,
I am training conformer based SSL model where I have configured the parameters
min_duration : 3.2
patch_size: 16
mask_patches: 0.5
num_negatives: 40
sample_from_same_utterance_only: true
sample_from_non_masked: false
codebook_size: 300
num_groups: 2
num_classes: 90000

but I am getting following error
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1129, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 987, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1129, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 82, in forward
    output = self.module.training_step(*inputs, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/utils/model_utils.py", line 364, in wrap_training_step
    output_dict = wrapped(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/ssl_models.py", line 468, in training_step
    loss_value, loss_val_dict = self.decoder_loss_step(
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/ssl_models.py", line 450, in decoder_loss_step
    current_loss_value = current_loss(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1129, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/core/classes/common.py", line 963, in __call__
    outputs = wrapped(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/losses/ssl_losses/contrastive.py", line 187, in forward
    out_masked_only = out_masked_only.reshape(bs, -1, out_masked_only.shape[-1])
RuntimeError: shape '[16, -1, 128]' is invalid for input of size 113920

I am using Nemo-22.05 images(ngc) and 48, A100-SXM4-40GB GPU i.e 6 nodes for the training. Kindly suggest me what is need to be done ? Thanks in advance
fix-causal-fa-infer
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Previously, we check `is_causal` for FA if we have `self.attn_mask_type == AttnMaskType.causal` and `s_q == s_k`. However, we may need causal self-attention for `s_q != s_k` if we have context parallelism proposed by @ksivaman . I change the condition `s_q == s_k` to `inference_mode` in order to prevent using causal when we have `s_q==1` during inference.
 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
megatron gpt training fix 
State: closed | Pull request: yes
Labels: name: NLP

Case where sequence parallel=True, transformer engine=False, and bias=False

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Uncomment removal of exp_dir in JenkinsFile
State: closed | Pull request: yes
Labels: name: CI

The removal of result dir after the test 'Megatron GPT with KERPLE Pretraining and Resume Training TP=2' was commented out by mistake that was leading to the next test using the checkpoint from this test. Hence rolling it back.

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Add license to ported EnCodec code
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Move code ported from EnCodec github into encodec_modules and add MIT Meta license to it.

https://github.com/facebookresearch/encodec/blob/main/LICENSE

**Collection**: [TTS]

# Changelog 
- Copy/paste code from audio_codec_modules.py and vector_quantization.py into encodec_modules.py
- Add MIT license

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation
Fix T5 using FA
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

* Fix T5 mask when using FA
* Add additional configs

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
minor fix for conformer subsampling docstring.
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
deprecation warning
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

adds a warning for multiple peft scripts.

**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
List index out of range when transcribing with RNNT conformer large
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

A clear and concise description of what the bug is.

Hello, we are running a Dutch trained version of the conformer RNNT large model.
However, in a small amount of cases we receive the following list index out of range error.

![image](https://github.com/NVIDIA/NeMo/assets/10922796/5341394d-a158-43d5-a724-52261052f0bb)

I saw that in [https://github.com/NVIDIA/NeMo/issues/4981 ](url) the issue seemed solved for the word timings, but maybe this issue still exists for the confidence scores?

We use the following to enable the confidence scores:

`
decoding_config = copy.deepcopy(conformer.cfg.decoding)
`
`
confidence_cfg = ConfidenceConfig(preserve_word_confidence=True, preserve_token_confidence=False)
`
`
with open_dict(decoding_config):
`
`
decoding_config.preserve_alignments = True,
`
`
    decoding_config.confidence_cfg = confidence_cfg
`
`
conformer.change_decoding_strategy(decoding_config)
`

**Expected behavior**

Transcription succeeding without errors

**Environment overview (please complete the following information)**

Docker nvcr.io/nvidia/pytorch:23.04-py3
NeMo installed from main with git clone and ./reinstall.sh ( cloned today )


**Additional context**

Ran on a 4090 GPU
Fix ZeroDivisionError in DistributedBucketSampler
State: closed | Pull request: yes
Labels: name: TTS

name: stale

In the `_create_buckets` method of the `DistributedBucketSampler` class, a ZeroDivisionError was being raised when `len_bucket` was zero. This fix adds a conditional check to skip the problematic division if `len_bucket` is zero, thus avoiding the error and ensuring that the rest of the code functions as expected.

# What does this PR do ?

Fixes a ZeroDivisionError in the DistributedBucketSampler class to ensure proper functionality.

**Collection**: TTS Collection

# Changelog 
- Added a conditional check in the `_create_buckets` method to prevent division by zero when `len_bucket` is zero.

# Usage

```python
from nemo.collections.tts.data.dataset import DistributedBucketSampler

# Initialize the sampler with your dataset, batch_size, and boundaries
sampler = DistributedBucketSampler(dataset, batch_size, boundaries)

# Use the sampler in your training loop
for batch in sampler:
    # Process the batch
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [x] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
[Fix] AudioCodecModel training fails with PTL 2.0
State: closed | Pull request: yes
Labels: name: TTS

Currently, `AudioCodecModel` training fails on `main` due to changes in PTL 2.0.

# What does this PR do ?

This PR is fixing two issues when using PTL 2.0.

**Collection**: TTS

# Changelog 
- Support for `training_epoch_end` has been removed in v2.0.0, and we use `on_train_epoch_end`
  - Reference: https://github.com/Lightning-AI/lightning/pull/16520
- Enable the detection of unused parameters in DDP, either by setting the string value `strategy='ddp_find_unused_parameters_true'`
  - Reference: https://github.com/Lightning-AI/lightning/pull/16520

# Usage

N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #6433
Make NFA subtitles stay until end of video
State: closed | Pull request: yes

# What does this PR do ?

Write an extra line at the end of the ASS files which will ensure the subtitles of the final segment are visible until the end of the video. Without this, it means that if we create an MP4 file from an audio file by writing the subtitles on a plain background, at the end of the video we are always left with a blank screen, which is not a nice UX as the text feels to disappear very suddenly.

**Collection**: `tools/nemo_forced_aligner.py`

# Changelog 
- extra code to get the duration of the audio file and write the extra lines at the end of the ASS files.

# Usage
No change in usage.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ N/A ] Did you write any new necessary tests?
- [ N/A ] Did you add or update any necessary documentation?
- [ N/A ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Fix TTS recipes with PTL 2.0
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Update TTS configs so they run with PTL 2.0

**Collection**: [TTS]

# Changelog 
- Replace _ddp_ with _ddp_find_unused_parameters_true_ for all vocoder recipes, otherwise it fails because there are unused parameters during the alternating generator and discriminator training steps.
- Update training_epoch_end() for AudioCodec to be compatible with new PTL interface.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Concerns about the implementation of `end_of_generation_condition()`
State: closed | Pull request: no
Labels: name: bug

The code of `end_of_generation_condition()` https://github.com/NVIDIA/NeMo/blob/13791d27ec8b322da11e8324ff07124aa4e6c55f/nemo/collections/nlp/modules/common/text_generation_strategy.py#L138 seems weird to me => opening an issue to clarify whether it is working as intended (cc @gshennvm @yidong72)

I have two concerns:

1. I find it weird to see `<extra_id_1>` hardcoded in this function -- I thought that the goal of providing `end_strings` as argument would be to avoid having to hardcode special strings
2. I am confused by something that is best explained through an example. Let's say I want to use `end_strings = ["abc"]`, which is tokenized into [1, 2, 3] (tokens for "a", "b", "c"), while `<extra_id_1>` is tokenized into [0]. It seems to me that this code will lead to `ids_1 = tokenizer.text_to_ids(f'<extra_id_1>abc') = [0, 1, 2, 3]`, `ids_2 = [0]`, and `token_id = ids_1[len(ids_2) :][0] = 1`. As a result, as soon as `p` (the last token generated) is equal to 1 (i.e., as soon as an `"a"` is generated), we consider that the generation has ended. Am I missing something?
Speaker emb -> speaker encoder
State: closed | Pull request: yes
Labels: name: TTS

name: stale

# What does this PR do ?

Rename spk_emb api to speaker_encoder
**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix token_ids list truncation to specified max_len
State: closed | Pull request: yes
Labels: name: stale

name: NLP

name: common

# What does this PR do?

Fixed samples processing:
- removed multiple conversions from text_to_ids and ids_to_tokens which causes the difference in the token amounts between these operations
- changed the prompt template processing: tokens inside the prompt template (eg "<extra_token_1") are converted to token_ids and after merging with "input" and "output" token_ids lists 
- due to some tokenizers have no specified space tokens (like " " and "\n") the prompt_template is mandatory (or it will be set by default as "{input} <extra_id_1> {output}"

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [+] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
allreduce layernorm params when enable sequence_parallel
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Refer to the [comments](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py#L677), Layernorm parameters need to be all-reduced across model parallel nodes when sequence parallelism is used.
NeMo find layernorm parameters through '`sequence_parallel` attribute. Code is [here](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py#L665).
But layernorm set the attribute with `sequence_parallel_enabled`  in apex. Code is [here](https://github.com/NVIDIA/apex/blob/master/apex/transformer/layers/layer_norm.py#L30).

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
use the MixedFusedLayerNorm which will set sequence_parallel_enabled
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

NeMo uses `apex.normalization.fused_layer_norm.MixedFusedLayerNorm`  when the `persistent_layernorm` is false.  
But the  `apex.normalization.fused_layer_norm.MixedFusedLayerNorm` doesn't set sequence_parallel.
And the  `apex.transformer.layers.layer_norm.MixedFusedLayerNorm`  will set sequence_parallel.
This PR use the `apex.transformer.layers.layer_norm.MixedFusedLayerNorm` instead of `apex.normalization.fused_layer_norm.MixedFusedLayerNorm`. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix t5 trainer test m
State: closed | Pull request: yes
Labels: name: NLP

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
MSDD model share memory.
State: closed | Pull request: no
Labels: name: stale

Hi.
I tried to run my MSDD model (using TITANET-large as speaker embedding model) in serving mode, meaning loading the model once and inference it in several processes, and in order to do so I need to use torch share_memory method.
when I use share_memory method I get the following error when I start the process:
AttributeError: Can't pickle local object 'FilterbankFeatures.__init__.<locals>.<lambda>'

Is there any guidelines how to share memory for all MSDD modules or any utils script to do so?
add missing import
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: stale

name: ASR

# What does this PR do ?

Not sure why missing lib import did not raise any errors.

* added missing import for `Trainer` and `NemoConfig`;
* unify `'Trainer'` into `Trainer` as a variable type hint.


**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Error] On running this tutorial notebook on Colab
State: closed | Pull request: no
Labels: name: stale

## ASR Tutorial - Online_Offline_Speech_Commands_Demo.ipynb


I tried to execute the notebook in google colab (`https://github.com/NVIDIA/NeMo/blob/stable/tutorials/asr/Online_Offline_Speech_Commands_Demo.ipynb`) 

While running the code block:


```python
wave_file = demo_wave

CHANNELS = 1
audio, sample_rate = librosa.load(wave_file, sr=SAMPLE_RATE)
dur = librosa.get_duration(audio)
print(dur)
``` 

The below error is produced:
```bash
TypeError                                 Traceback (most recent call last)
[<ipython-input-20-1dea7463f5dd>](https://al43ym753q8-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230804-060128-RC00_553764177#) in <cell line: 5>()
      3 CHANNELS = 1
      4 audio, sample_rate = librosa.load(wave_file, sr=SAMPLE_RATE)
----> 5 dur = librosa.get_duration(audio)
      6 print(dur)

TypeError: get_duration() takes 0 positional arguments but 1 was given
```
fine-tuning hifi-gan with libritts
State: closed | Pull request: no
Labels: name: TTS

name: stale

I try finetune hifi-gan, which was pre-trained as 'tts_en_hifigan', with libritts, but it failed. I guess I need to modify the libritts data, I have resampled it to 22050kHz, and it did not work. Could you please give me some help, thanks!
Fix logging for Megatron
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

The PR fixes the progress bar stdout in megatron models to remove the `s/it` or  `it/s` and instead display the actual cuda synchronized train step timing.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fixes for lightning 2.0 upgrade
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Fix the following issues with the previous lightning 2.0 upgrade PR:

1) Decrease `val_check_interval` to 1 while resuming from a ckpt in some CI tests where max_steps is same as that of training and lightning 2.0 errors: [fit_loop.py#L259](https://github.com/Lightning-AI/lightning/blob/2.0.6/src/lightning/pytorch/loops/fit_loop.py#L259). Lightning 2.0 calls `setup_data` right at the beginning of fit_loop.run unlike 1.9 that did it only when `self.skip` for fit was False, hence the error din't show up with 1.9
2) Remove the usage of `trainer._checkpoint_connector` as it's redundant in lightning 2.0 and use `trainer.ckpt_path` directly. Also having `trainer._checkpoint_connector = _CheckpointConnector(trainer)` overrode ckpt_path with None leading to starting training from scratch during resuming from a ckpt.
3) Remove `optimizer_idx` arg in `optimizer_step` of `MegatronHalfPrecisionPlugin` class as the parent function in lightning does not have it anymore.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update README.rst
State: closed | Pull request: yes

# What does this PR do ?

update readme regarding container version and python version

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Allow applications to control the encoder mask type
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

When TensorRT optimizes Nemo GPT models it uses a different mask shape in the Context phase (triangular mask, i.e. causal) and in the Generation phase (1D mask, i.e. padding).
To support this TensorRT needs to control GPT's encoder mask type (which defaults to causal) via the model configuration.

**Collection**: NLP

# Changelog
- `MegatronGPTModel::model_provider_func` reads an optional `encoder_attn_mask_type` argument from the config file and passes it to the GPTModel constructor. The default value is a causal mask.
- Add an optional `encoder_attn_mask_type` argument to GPTModel constructor. The argument defaults to `AttnMaskType.causal`.
- Pass this argument to `get_language_model`  (previously hard-coded: `encoder_attn_mask_type=AttnMaskType.causal`)

**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
1D conv block in citrinet
State: closed | Pull request: no

I am looking at the citrinet paper (https://arxiv.org/pdf/2104.01721.pdf) and I am trying to understand the architecture. Simultaneously I downloaded citrinet model (stt_en_citrinet_1024_gamma_0_24) and printed the model. Part of it looks like follows

---------------------
     (1): JasperBlock(
        (mconv): ModuleList(
          (0): MaskedConv1d(
            (conv): Conv1d(1024, 1024, kernel_size=(3,), stride=(1,), padding=(1,), groups=1024, bias=False)
          )
          (1): MaskedConv1d(
            (conv): Conv1d(1024, 1024, kernel_size=(1,), stride=(1,), bias=False)
          )
          (2): BatchNorm1d(1024, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
          (3): ReLU(inplace=True)
          (4): Dropout(p=0.1, inplace=False)
          (5): MaskedConv1d(
            (conv): Conv1d(1024, 1024, kernel_size=(3,), stride=(1,), padding=(1,), groups=1024, bias=False)
          )
          (6): MaskedConv1d(
            (conv): Conv1d(1024, 1024, kernel_size=(1,), stride=(1,), bias=False)
          )
........................
---------------------

There are 2 conv1d blocks before batchnorm where as there is only 1 conv1d block as per the paper. 
Is this observation correct ? 

I saw few lines of the code , it looks like there is residual module for every sub block in the megablock . Am I correct ?  But as per paper, there is only residual connecting from beginning to end of the the megablock.

Kindly clarify
Upgrading `hydra-core` past 1.3
State: closed | Pull request: no

**Is your feature request related to a problem? Please describe.**

Current `hydra-core` version (`hydra-core>=1.2.0,<1.3` from `requirements/requirements_lightning.txt`) doesn't work with Python 3.11: see #7166

**Describe the solution you'd like**

Now, after [Upgrade to pytorch lightning 2.0](https://github.com/NVIDIA/NeMo/commit/67de251e980ef8d8aec3b8339e70e96f6c952122) commit, would it be safe to upgrade `hydra-core` past `1.3.0`? `hydra-core==1.3.2` at least imports fine without `1.2.0` `dataclasses` related issues.

**Describe alternatives you've considered**

The only alternative I can think of is creating a fork of `1.2.0` version and patching it, but I think it's counter-productive since it has to be maintained separately.

Thanks :smiley:
context parallelism
State: closed | Pull request: yes
Labels: name: core

name: NLP

name: CI

# What does this PR do ?

GPT training with long-context input (e.g., sequence length of 16, 32K, 64K) can easily overflow GPU memory with huge activations. Context parallelism splits long-context input along the dimension of sequence length, parallelizes partitioned sequence segments among multiple GPUs. In this way, each GPU only needs to store activations of a part of sequence length, so that we can avoid memory overflow.

# Usage
Add a new config in yaml file, e.g., "context_parallel_size: 2".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
initial scripts
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: CI

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Merge release r1.20.0 to main
State: closed | Pull request: yes
Labels: name: ASR

name: NLP

name: CI

# What does this PR do ?

Make sure all bugfixes and doc updates were properly cherry picked to main.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
NFA subtitle file config - specify colors and vertical alignment
State: closed | Pull request: yes

# What does this PR do ?

Modify the `ASSFileConfig` so we can specify the colors of the text and its vertical alignment.

**Collection**: `tools/nemo_forced_aligner`

# Changelog 
For specifying vertical alignment, removed `ass_file_config.marginv` parameter, which was being used as a rough way to modify the vertical placement of the text. Now, there is a parameter `ass_file_config.vertical_alignment`, which can take the values `"center"` (default), `"top"`, or `"bottom"`

For specifying the colors of the text, added 3 new parameters:
- `ass_file_config.text_already_spoken_rgb`
- `ass_file_config.text_being_spoken_rgb`
- `ass_file_config.text_not_yet_spoken_rgb`

# Usage
Can specify new parameters in `ASSFileConfig` e.g.:

```bash 
python <path to NeMo>/tools/nemo_forced_aligner/align.py \
  pretrained_name="stt_en_fastconformer_ctc_large" \
  manifest_filepath="manifest.json" \
  output_dir=nfa_output/ \
  additional_segment_grouping_separator="|" \
  ass_file_config.text_already_spoken_rgb=[252,186,3] \ # bright yellow/orange
  ass_file_config.text_being_spoken_rgb=[240,26,240] \ # bright pink
  ass_file_config.text_not_yet_spoken_rgb=[62,26,240] \ # bright blue
  ass_file_config.vertical_alignment="top"
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ N/A ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ N/A ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)