renwei2024/nemo-github-issues huggingface dataset:
PipelineMixedPrecisionPlugin: dtype is `None`
State: closed | Pull request: no
Labels: name: stale

In [this logic](https://github.com/NVIDIA/NeMo/blob/c56119623a3ad1a0b2e2767f88b65ddcac0cbd1b/nemo/collections/nlp/parts/nlp_overrides.py#L549), dtype might be `None` and causes errors like:

```
Traceback (most recent call last):
  File "/tmp/nvflare/nemo/peft_fedavg_345M/simulate_job/app_site-1/custom/megatron_gpt_peft_tuning.py", line 203, in main
    plugins.append(PipelineMixedPrecisionPlugin(precision=plugin_precision, device='cuda', scaler=scaler))
  File "/usr/local/lib/python3.10/dist-packages/nemo/collections/nlp/parts/nlp_overrides.py", line 525, in __init__
    torch.set_autocast_gpu_dtype(dtype)
TypeError: dtype must be a torch.dtype (got NoneType)
```

There should be a informative error message in case dtype is not set correctly.
[TN][Docs] update language coverage matrix and refs
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

update language coverage matrix and refs for TN/ITN

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Tracking Artifacts as well along with metrics and parameters in Mlflow
State: closed | Pull request: no
Labels: name: stale

**Is your feature request related to a problem? Please describe.**

While we are training ASR model on nemo framework, we are able to track parameters and metrics with minimal inputs in the config.sh. But we are not able to track artifacts i.e. Nothing getting stored here.

**Describe the solution you'd like**
`exp_manager:
    ...
    create_checkpoint_callback: True
    create_mlflow_logger: True
    mlflow_logger_kwargs:
        experiment_name: ${name}
        tags:
            <Any key:value pairs>
        save_dir: './mlruns'
        prefix: ''
        artifact_location: None
        # provide run_id if resuming a previously started run
        run_id: Optional[str] = None`

Tracking artifacts would be really helpful along with metrics and parameters. Thanks in advance!
Ctc bpe conformer overfit
State: closed | Pull request: no

Hi, I'm trying to train a bpe-level conformer and am facing overfitting.

In this regard, I have a few questions and I would be very grateful for the answers to them:
1) Have you encountered overfitting of the BPE conformer and BPE models in general?
2) Have you tried to experiment with char level vs bpe level models? If yes, what are you facing? I mean over/underfitting, stability, approximate increase in metrics.
3) I saw [configs ](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/conformer/conformer_ctc_bpe.yaml) for training conformer from nemo, thanks for posting them, but I couldn't figure out what the global batch size was. [2048](https://github.com/NVIDIA/NeMo/issues/4183#issuecomment-1128444967)?
4) What are some tips for tuning regularization for a conformer?
5) How trainning stability depends on the size of the dictionary?

Before that, I used the char-level conformer, and did not encounter overfitting. I don't have a lot of data, 370 hours for one domain and 130 for another. Therefore, I first train the base model, and then finetune it on domain data. I did some experiments:
1) I finetuned stt_ru_conformer_ctc_large using my domain data and got pretty good quality (better my char level model)
2) I trained my base model, just like I did before with char-level models, and then tried to finetuned it on domain data

The training data configurations are completely the same, except for the weights of models and tokens of tokenizers (the sizes of models and tokenizers are the same).
The length of my audios is from 0.1 to 10 seconds, the size of the bpe dictionary is 128, the average transcription length is 53 characters or 31 tokens.
I use the same (as in nemo models) spec augs, except mask value (mean instead of 0).

I used AdamW, a constant scheduler with lr = 0.0001, global batch size = 512 and faced with overfitting (see plots), then I reduced lr by 10 times and increased global batchs size by 3 times (1536 in total), but this did not solve the problem cardinally. The metrics I get (before the start overfitting) are worse than not only the experiment on finetuning the base model from NeMo, but also worse than the experiments on finetuning my char-level base model.

Plots for char based model:
![newplot (18)](https://github.com/NVIDIA/NeMo/assets/49618087/d32b972b-d27f-4332-9af1-0fc3bb810bc4)
![newplot (15)](https://github.com/NVIDIA/NeMo/assets/49618087/3818469d-2a1c-4b50-a989-75b867915842)

Plots for bpe based model:
![newplot (20)](https://github.com/NVIDIA/NeMo/assets/49618087/6c35247c-fa0b-457e-9675-1222838749e9)
![newplot (19)](https://github.com/NVIDIA/NeMo/assets/49618087/788d546d-b82f-44e4-b015-600df36437df)

@titu1994 

Thank you in advance
Upgrade hydra and omegaconf
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
T5 PEFT
State: closed | Pull request: yes
Labels: name: stale

name: NLP

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Hidden transforms model parallel config + CI with Perceiver
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

This PR addresses:
1. Compatibility with added ModelParallelConfig configuration.
2. Adding to CI test for hidden transformations and Perceiver bottleneck models.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Are sriLM models compatible with nemo decoders?
State: closed | Pull request: no

I have an ARPA 3-gram model trained with sriLM. I know nemo decoders use kenLM models, but I thought it might be compatible with the current decoders as it should also be word level trained. When I run eval_beam_search_ngram_transducer.py and supply kenlm_model with the sriLM, I don't receive any error. However, after performing grid search over a big set parameters.. there is absolutely no improvement, only degraded results. When I run beam_width=1 and beam_alpha=0, I receive the same results as greedy decoding.

```
[NeMo I 2023-08-14 21:14:11 eval_beamsearch_ngram_transducer:384] ==============================Starting the maes decoding===============================
[NeMo I 2023-08-14 21:14:11 eval_beamsearch_ngram_transducer:387] Grid search size: 1
[NeMo I 2023-08-14 21:14:11 eval_beamsearch_ngram_transducer:388] It may take some time...
[NeMo I 2023-08-14 21:14:11 eval_beamsearch_ngram_transducer:389] ==============================================================================================
Loading the LM will be faster if you build a binary file.
Reading /homes/gpirlo/git_repos/nemoasr/examples/asr/train_dir/models/lm_2020_3g_large_200k_cn.3.GramLM
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
The ARPA file is missing <unk>.  Substituting log10 probability -100.
```

Thank you in advance!
remove deprecated scripts from ci
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Removing deprecated script from the CI.


**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
import FastPitchModel fails with unrelated error from prompth_encoder.py
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Running `tutorials/tts/FastPitch_Finetuning.ipynb` on Colab get error on `import FastPitchModel`

**Steps/Code to reproduce bug**
```
[/usr/local/lib/python3.10/dist-packages/nemo/collections/nlp/modules/common/prompt_encoder.py](https://localhost:8080/#) in TPMLP()
    138     def __init__(
    139         self,
--> 140         config: ModelParallelConfig,
    141         total_virtual_tokens: int,
    142         hidden_size: int,

NameError: name 'ModelParallelConfig' is not defined
```
Add mcore installation to Dockerfile
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Temporary fix until pip package is released

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
can't run sft with gpt-2b-001 using FP8, model initialization parameters mismatch error
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**
Hi,

I want to compare the SFT training convergence between bf16 and fp8.

So I am trying to do SFT training in NeMo using fp8 on a H100 GPU.

However, it reports an error during model initialization.

The error log file is in the attachment.
[sft_fp8.log](https://github.com/NVIDIA/NeMo/files/12340953/sft_fp8.log)


It seems that the NeMo fp8 layer names are different from bf16 model.


**Steps/Code to reproduce bug**

```shell
EXE=./nemo/sft/nemo_sft.fp8.py
NEMO_PATH=./GPT-2B-001_bf16_tp1.nemo
FP8=True
   python ${EXE} --config-path=${CONFIG_DIR}/conf --config-name=megatron_gpt_sft.yaml \
        trainer.num_nodes=1 \
        trainer.devices=[${GPU}] \
        trainer.max_steps=8 \
        exp_manager.create_checkpoint_callback=false \
        exp_manager.checkpoint_callback_params.every_n_train_steps=128 \
        exp_manager.exp_dir=${EXP_DIR} \
        exp_manager.name=nemo-fp8-2b \
        exp_manager.create_wandb_logger=${USE_WANDB} \
        exp_manager.wandb_logger_kwargs.project=SFT-FP8 \
        exp_manager.wandb_logger_kwargs.name=gpt-2b \
	model.transformer_engine=${FP8} \
	model.fp8=${FP8} \
        model.restore_from_path=${NEMO_PATH} \
        model.tensor_model_parallel_size=1 \
        model.optim.name=fused_adam \
        model.optim.lr=1e-5 \
        model.data.train_ds.max_seq_length=1024 \
        model.data.train_ds.micro_batch_size=2 \
        model.data.train_ds.global_batch_size=32 \
        model.data.train_ds.file_names=[${DATAFILE}]"
```



**Additional context**
Model: https://huggingface.co/nvidia/GPT-2B-001
Image: nvcr.io/ea-bignlp/nemofw-training:23.07-py3
Fix GPTSFTDataset
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?
* Change middle tokenization logic from tokenization/detokenization to separate prompt template 
* Add multiple contexts and truncation fields
* Add truncation augmentation

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
_D.R suggest we document the changes here, e.g._ 

**Fixes** 
- Remove the old logic prompt_template -> tokenize -> compute total tokens-> truncate -> detokenize -> prompt_template -> tokenize
- Add new logic prompt_template -> separate prompt_template -> tokenize -> compute total tokens -> truncate
- In separate prompt_template, we need to carefully adding spaces to the next token
- In separate prompt_template,  we need to separate from left to right
-
**New features**
Remove query_key and make context_key  have multiple keys by a ,  for example context_key="context,query"
Make truncation_field can have multiple keys, logic the same as context_key
Add multiple truncation if we have multiple truncation_field . Current logic is to equally truncate each field
Add truncation augmentation (random truncate)

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add new script for finetuning asr models
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add single script for finetuning ASR models. 

**Collection**: ASR

# Changelog 
- Add script and config yaml file for finetuning
- TODO: Add tests and docs

# Usage

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add precision and megatron_amp_O2 configs to adapters
State: closed | Pull request: yes
Labels: name: stale

name: NLP

name: CI

- Add precision and megatron_amp_O2 parameters to adapters
- Make adapter modules follow the config of base model for precision and megatron_amp_O2

# What does this PR do ?

Make adapter modules follow the config of base model for precision and megatron_amp_O2

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Can timestamps and confidence be extracted from a conformer model when used with beam search and LM? 
State: closed | Pull request: no

I have finetuned an english conformer transducer bpe model on romanian and I have a couple of questions after following the tutorials on "Offline ASR" and "ASR confidence estimation":

1. I want to add a kenLM trained model in order to solve some small erros related to missing characters/phonetically transcribed foreign words ("meeting" -> "miting", "business" -> "bisnis"). This is expected because this is how they are pronounced in romanian and the word "meeting" might not appear often in the training set, but I thought that a language model might easily solve this problem. 

However, we would also need a timestamp and confidence estimate per word and from what I understood this is possible only for "greedy decoding without LM". Is it actually possible to extract timestamps/confidence for words for beam search+LM? If it is not yet implemented, how complicated would it be and could you suggest a starting point?

2. From what I understood, transducer decoding is more complicated than CTC, so is there any implementation of confidence+timestamp estimation for CTC models with beam search+LM?  

Thanks in advance!!
Add only trainable parameters to optimizer group in PEFT
State: closed | Pull request: yes
Labels: name: NLP

* Add only parameters with requires_grad=True to the optimizer group

# What does this PR do ?

- Add only the parameters with requires_grad=True to the optimizer group and skip frozen parameters. This change can reduce memory usage.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Install mcore in nlp import github action
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Install mcore commit in the Github Action. Can revert this when the next pip package for mcore is released. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Remove frozen parameters from PEFT optimizer group
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: NLP

name: CI

# What does this PR do ?

Remove frozen parameters from the optimizer group of MegatronGPTPEFTModel

# Changelog 
- Check the requires_grad attribute of the parameter and only add it to the optimizer group if `requires_grad=True`

# Before your PR is "Ready for review"
**Pre checks**:
- [x ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add bf16-mixed and 16-mixed in module.py
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
StatsPoolLayer is inconsistent
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

There are 2 problems with StatsPoolLayer

1. `biased` argument actually means unbiased (unbiased estimator uses N-1, instead of N)

https://github.com/NVIDIA/NeMo/blob/ab749e4401a3dcdfa5ea969347aaee20b7947c7c/nemo/collections/asr/parts/submodules/tdnn_attention.py#L35-L36

See PyTorch doc: https://pytorch.org/docs/stable/generated/torch.std.html

2. `biased` argument has no effect when `length` is not provided -> discrepancy between when `length` is provided and not provided. Unbiased estimator (PyTorch's default) is always used.

https://github.com/NVIDIA/NeMo/blob/ab749e4401a3dcdfa5ea969347aaee20b7947c7c/nemo/collections/asr/parts/submodules/tdnn_attention.py#L59-L65

**Expected behavior**

I believe `biased` argument should be `unbiased` instead. This argument should be passed to `torch.std()` when `length` is not provided.
main_param not initialized when param.type() not in ['torch.cuda.HalfTensor', 'torch.cuda.BFloat16Tensor']
State: closed | Pull request: no
Labels: name: stale

main_param is not initialized when param.type() not in ['torch.cuda.HalfTensor', 'torch.cuda.BFloat16Tensor']
but it is used in the following line:

https://github.com/NVIDIA/NeMo/blob/f4afdf07ad742bdb2f45135f05d5d7dd8270eb51/nemo/core/optim/optimizer_with_main_params.py#L305
Unable to resume multi-epoch experiments with a `MegatronGPTModel`
State: closed | Pull request: no
Labels: name: bug

name: stale

_NB: I am experiencing this issue with a custom subclass of `MegatronGPTModel` but I expect the same problem to occur with the base class_

**Describe the bug**

I am trying to:
1. Run two epochs:
```yaml
trainer:
  max_epochs: 2
```
2. *but* stopping the experiment in the middle, then resuming it (due to cluster availability constraints)

Now let's see what happens when stopping after the first epoch, and trying to resume afterwards.
The first experiment runs fine, but resuming doesn't work because the following happens when running the second one:
1. `MegatronGPTModel.setup()` is called at the start, which triggers a call to `build_pretraining_data_loader()` which creates a `MegatronPretrainingSampler` on [this line](https://github.com/NVIDIA/NeMo/blob/ab749e4401a3dcdfa5ea969347aaee20b7947c7c/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py#L993), with `consumed_samples` set to the number of consumed samples during the first run, i.e. `consumed_samples == len(dataset)` (at least with the fix from #7212)
2. As a result, the `MegatronPretrainingSampler` crashes on [this line](https://github.com/NVIDIA/NeMo/blob/ab749e4401a3dcdfa5ea969347aaee20b7947c7c/nemo/collections/nlp/data/language_modeling/megatron/data_samplers.py#L43) because we have already consumed the whole dataset

If you stop before the end of the first epoch (or if you don't have the fix from #7212), the behavior will be a bit different, but it won't work either, at best you'll be able to finish the first epoch but it won't do the second one properly.

**Expected behavior**

The second run should be able to pick up where the first one stopped, and continue until the desired number of epochs have been completed.

**Environment overview**

 - Environment location: Docker
 - Method of NeMo install: Docker

**Discussion**

I have some potential ideas on how to fix this, but I thought I'd ask here first in case I would be missing something obvious (I find it hard to believe I'd be the first one to try and run more than one epoch on a time-limited cluster...)
Make image centering in NFA README actually work
State: closed | Pull request: yes

# What does this PR do ?

Make image centering in NFA README actually work

**Collection**: `tools/nemo_forced_aligner`

# Changelog 
- fix align="center" in README

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ N/A ] Did you write any new necessary tests?
- [ N/A ] Did you add or update any necessary documentation?
- [ N/A ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
NFA docs: update READMEs and links, add docs page
State: closed | Pull request: yes

# What does this PR do ?

Update documentation for NFA.

**Collection**: `tools/nemo_forced_aligner`

# Changelog 
- Update NeMo README
- Update NFA README - add links to new materials, move most info to new NFA docs page
- Update links to NFA tutorials

# Usage
N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ N/A ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ N/A ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
NFA blog posts
State: closed | Pull request: yes

# What does this PR do ?

Adding 2 blog posts: "Introducing NFA" and "How forced alignment works".

**Collection**: Blogs

# Changelog 
- add blog posts
- add a new 'Install dependencies' stage containing `pip install mkdocs-video` to the `.github/workflows/gh-docs.yml` file

# Usage
N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ N/A ] Did you write any new necessary tests?
- [ N/A ] Did you add or update any necessary documentation?
- [ N/A ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix val_check_interval on Resume tests
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

For tests that run pretraining and then resume (by re-running pretraining) and don't train for more steps after resuming, `val_check_interval` should be 1.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- set `trainer.val_check_interval=1` for several tests

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Resume tests validation
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix attention mask inference
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix `inference.compute_attention_mask=True` if not using FA

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix `consumed_samples` which was off by one batch
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

It fixes `consumed_samples`, which was off by one batch

**Collection**: nlp

# Changelog 
- Fix `consumed_samples`, which was off by one batch

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

# Additional Information

This could be an issue for instance when resuming an experiment and using the number of consumed samples to know where to start from (we would end up re-processing a batch that should have been skipped).
NFA tutorial notebook
State: closed | Pull request: yes

# What does this PR do ?

Add notebook tutorial for how to use NFA.

**Collection**: `tutorials/tools`

# Changelog 
- Add new notebook.

# Usage
N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [N/A] Did you write any new necessary tests?
- [will add links later] Did you add or update any necessary documentation?
- [ N/A ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Fix] Discriminator update in AudioCodecModel
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

This change removes random selection when deciding to update the discriminator in the `AudioCodecModel`.

**Collection**: TTS

# Changelog 
- Instead of using a random number, update the discriminator `disc_updates_per_period` times for every `disc_update_period` batches
  - This fixes an issue when training with multiple GPUs
- Added logging in `audio_codec.py` training script

# Usage
N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
NFA: replace ellipses in text with 3 periods
State: closed | Pull request: yes

# What does this PR do ?
Currently NFA breaks if there is a 'horizontal ellipses' character (`…`) in the text - specifically the 'restore punctuation' code breaks because it doesn't reconcile the single ellipses character with the 3 periods generated by the tokenizer. A quick for this is to just replace the ellipses with the 3 periods when we read the data (because the text will always end up being 3 periods even if we don't try to restore the punctuation.

**Collection**: `tools/nemo_forced_aligner/`

# Changelog 
- add a line which replaces horizontal ellipses character with 3 periods when reading the text in the original manifest.

# Usage
No usage change

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ N/A ] Did you write any new necessary tests?
- [ N/A ] Did you add or update any necessary documentation?
- [ N/A ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
support SFT/distOpt mcore
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Hongbin supported SFT/distOpt for Llama on nemo-mcore

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
eternal import nemo.collections.asr in the Offline_ASR.ipynb
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

After the first run of the "Installation", when trying to run it again, colab endlessly executes the line "import nemo.collections.asr as nemo_asr".

**Expected behavior**

Import all required modules in a normal period of time

**Environment details**

Free colab (Nvidia Tesla T4)



![image](https://github.com/NVIDIA/NeMo/assets/75932124/29339e76-e4e6-4575-84e2-8c470e765cdb)