renwei2024/nemo-github-issues huggingface dataset:
NeMo Megatron GPT Unit Tests
State: closed | Pull request: yes

# What does this PR do ?

Adding unit tests for GPT

_Need to add merge and vocab files to the test tarfile_

**Collection**: NLP

# Changelog 
- Test tokenizer
- Test constructor
- Test forward

# Usage

```bash
TRANSFORMERS_OFFLINE=1 pytest tests/collections/nlp/test_gpt_model.py
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Allow relative paths in manifest
State: closed | Pull request: yes

# What does this PR do ?

 Allow to use relative paths in manifest file.

**Collection**: ASR or any other collection which use manifest file with field "audio_file"  

# Changelog 
- Allow to use relative paths in the ASR manifest file 

# Usage
* Allow to use relative paths in the manifest file. Field audio_file may be relative to the folder of manifest file


# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Bug fix for Issue #4059, word time stamps for single word cases
State: closed | Pull request: yes

Signed-off-by: Taejin Park <tango4j@gmail.com>

# What does this PR do ?

This PR fixes the bug reported in Issue #4059. 
Before this fix, QuartzNet ASR Model could not generate word timestamps for an utterance with a single word.

**Collection**: [Note which collection this PR will affect]

ASR

# Changelog 

- Function named get_word_ts_from_spaces() has been changed to handle the cases where there is only one word in the decoded output.
- Few changes in docstrings.


# Usage

Please refer to [README](https://github.com/NVIDIA/NeMo/blob/d4bf4ed37d9465baa5a83dbf824864b0081c6098/examples/speaker_tasks/diarization/README.md)

```python
python offline_diarization_with_asr.py \
    diarizer.manifest_filepath=<path to manifest file> \
    diarizer.out_dir='demo_asr_output' \
    diarizer.speaker_embeddings.model_path=<pretrained modelname or path to .nemo> \
    diarizer.asr.model_path=<pretrained modelname or path to .nemo> \
    diarizer.asr.parameters.asr_based_vad=True
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [] Did you write any new necessary tests?
- [] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Fix link to tutorial in dialogue docs
State: closed | Pull request: yes

# What does this PR do ?

Fix link to tutorial in dialogue docs

**Collection**: [nlp]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Tarred fix label models
State: closed | Pull request: yes

# What does this PR do ?

Fix for an issue where during tarred dataset, max_steps computation fails in label models

**Collection**: ASR

# Changelog 
- Adds it during setting up of training data
- Deleted duplication code for mapping tarred audio paths
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Generalize adapter merge strategy for future adapters
State: closed | Pull request: yes

Signed-off-by: smajumdar <titu1994@gmail.com>

# What does this PR do ?

Generalize the way adapters are merged with their input, so as to allow more complicated adapters for the future, such as SimAdapter or MetaAdapter

**Collection**: [Core, ASR]

# Changelog 
- Add `AbstractAdapterStrategy` as a base strategy that is used for other adapter strategies
- Add `ResidualAddAdapterStrategy` for simple residual addition between input and adapter output
- Add tests

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Cherry-pick fix for megatron ckpt conversion script when using BCP
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: NLP

# Changelog 
- Update script to use plugin necessary for BCP

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Punctuation and Capitalization model is not run on multiple GPUs when `devices>1` or `gpus>1`.
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Punctuation and capitalization model is run on 1 GPU even if `strategy=ddp`, `accelerator=gpu`, `devices=2`. For running on several GPUs a user has to set parameter `num_processes`.

**Steps/Code to reproduce bug**

Run script `examples/nlp/token_classification/punctuation_capitalization_train_evaluate.py` with `trainer.accelerator=gpu`, `trainer.devices > 1`, `trainer.strategy=ddp` and check GPU utilization.

**Expected behavior**

Several GPUs used.

**Environment overview (please complete the following information)**

 - Environment location: bare-metal
 - Method of NeMo install: from source

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 20.04.2 LTS
- PyTorch version: 1.10.1
- Python version: 3.8

**Additional context**

The reason for the problem is that in this [line](https://github.com/NVIDIA/NeMo/blob/f6987f4fb9e6c45afe0559e82d5d09e7d28a365d/examples/nlp/token_classification/punctuation_capitalization_train_evaluate.py#L101) default values of trainer config from this [file](https://github.com/NVIDIA/NeMo/blob/main/nemo/core/config/pytorch_lightning.py) are added. Default value of `num_processes=1` blocks the model from running on multiple GPUs.
Upgrade container to NGC PyTorch 22.04
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: Core
# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add async grad allreduce and chunk optimization
State: closed | Pull request: yes

# What does this PR do ?

Add async grad allreduce, work for T5 and GPT-3.
Increase allreduce granularity by grouping params into multiple chunks. Every time a chunk is finished, we start an allreduce for the chunk (sending large chunk of data can better utilize bandwidth).
Current implementation works for BF16 O2 and PP=1.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Update train_step of GPT-3 and T5 for async grad allreduce
- Add chunk optimization in MainParamsOptimizerWrapper

# Usage
add a knob "grad_allreduce_chunk_size_mb", default config is 125MB. You can change it for different models and/or model sizes.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
ASR finetuning/training 7x slower on v1.8.2 
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I recently updated from NeMo 1.7.2 to 1.8.2
When running a training session, the speed is now `4.04s/it` whereas it used to be `2.97it/s`.
Note it's taking ~4 **seconds per iteration** whereas it used to be ~3 **iterations per second**!

Here's my trainer config since it's the only things that's different from the NeMo examples config for CitriNet-1024 with SPE tokenizer:

```yaml
trainer:
  devices: 8
  accelerator: gpu
  max_epochs: 100
  max_steps: -1
  num_nodes: 1
  strategy: ddp
  accumulate_grad_batches: 4
  enable_checkpointing: false  # Provided by exp_manager
  logger: false                # Provided by exp_manager
  log_every_n_steps: 100
  val_check_interval: 1.0
  precision: 16
```

**Steps/Code to reproduce bug**

Run any training example script.

**Expected behavior**

Speed should not be significantly affected.

**Environment overview (please complete the following information)**

 - Environment location: Docker (tried `nvcr.io/nvidia/pytorch:22.03-py3` and `nvcr.io/nvidia/pytorch:22.02-py3`)
 - Method of NeMo install: `pip install nemo_toolkit[all]==1.8.2`
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used
 ```
docker run --rm -it --gpus all --ipc=host --env-file .env train
```

**Additional context**

Running this on an AWS p3dn.24xlarge instance (8xV100)
TTS Deprecation
State: closed | Pull request: yes

# What does this PR do ?

Remove TTS models that were deprecated and marked for deletetion.

**Collection**: TTS, ASR

# Changelog 
- Removes the following tts models and associated modules
  - DegliModel
  - EDMel2SpecModel
  - FastPitchHifiGanE2EModel
  - FastSpeech2Model
  - FastSpeech2HifiGanE2EModel
  - UniGlowModel
  - TalkNet models: TalkNetDursModel, TalkNetPitchModel, TalkNetSpectModel\
  - SqueezeWaveModel
  - MelGanModel
  - GlowTTSModel
 - Removed `torch_stft` dependency from ASR and TTS
   - Removed `stft_exact_pad`, and `stft_conv` parameters
 - Removed the following datasets from the ASR and TTS collections
   - ASR: AudioToCharWithDursF0Dataset, AudioToCharWithPriorDataset, AudioToCharWithPriorAndPitchDataset, FastPitchDataset
   - TTS: AudioDataset, MelAudioDataset, SplicedAudioDataset, NoisySpecsDataset, FastSpeech2Dataset

# TODO
- [ ] Update documentation to remove old models
- [ ] Test tutorials
- [ ] Update NGC model cards to mark them as deprecated
Fix link to prompt tuning page
State: closed | Pull request: yes

# What does this PR do ?

After #4046 fix the link in the README to point to the correct page!

**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Update LJSpeech whitelist file path
State: closed | Pull request: yes

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

# What does this PR do ?

Updates various files with the new path for the LJSpeech whitelist file

**Collection**: TTS
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

# Additional Information
* Related to #4076
Added exception handling for audio player in SDE
State: closed | Pull request: yes

# What does this PR do ?

The PR adds logger message for exceptions during audio processing in SDE player.

**Collection**: Tools/SDE

# Changelog 
- Added exception handling for audio player in SDE

# Usage

N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #4067

Signed-off-by: Vitaly Lavrukhin <vlavrukhin@nvidia.com>
`bias_dropout_add_fused_train` drops `grad_fn` in the output tensor
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

when running p-tuning of T5 model in branch1.8.0, freezing the LM model weights, it raises 
```
Exception has occurred: RuntimeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
element 0 of tensors does not require grad and does not have a grad_fn
```

It turns out to be a problem with `bias_dropout_add_fused_train` method.
The `attention_output` has `grad_fun` as arguments for the following code in line717 of `transformer.py`:
```python
            layernorm_input = bias_dropout_add_func(
                attention_output, attention_bias.expand_as(residual), residual, self.hidden_dropout
            )
```           
![image](https://user-images.githubusercontent.com/43824965/165551818-f0e12553-a3c0-48d9-80a1-f869fe51fdee.png)
However, the result doesn't have the `grad_fn` and `requires_grad=False`
![image](https://user-images.githubusercontent.com/43824965/165553101-bad41e8a-bf4e-427b-8744-60cddb0d5410.png)

If use the none-fused version, it works fine.
![image](https://user-images.githubusercontent.com/43824965/165552804-8409ae6d-b865-4666-a5cb-26d1579b87bc.png)

It looks like to be a `pytorch.script` problem. Following is how it is implemented
```python
def bias_dropout_add(x, bias, residual, prob, training):
    # type: (Tensor, Tensor, Tensor, float, bool) -> Tensor
    out = torch.nn.functional.dropout(x + bias, p=prob, training=training)
    out = residual + out
    return out


@torch.jit.script
def bias_dropout_add_fused_train_(
    x: torch.Tensor, bias: torch.Tensor, residual: torch.Tensor, prob: float
) -> torch.Tensor:
    # type: (Tensor, Tensor, Tensor, float) -> Tensor
    return bias_dropout_add(x, bias, residual, prob, True)
```
The work around is to set `bias_dropout_fusion=False`
Cherry pick tts fix
State: closed | Pull request: yes

# What does this PR do ?

Cherry picks fastpitch bugfix.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Draft] CI test debug
State: closed | Pull request: yes

Signed-off-by: ekmb <ebakhturina@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Make saving prompt encoder embeddings non-configurable
State: closed | Pull request: yes

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Made saving prompt encoder embeddings to prompt table non-configurable

Add a one line overview of what this PR aims to accomplish.

# Changelog 
- updated MegatronGPTPromptLearning class
- removed line from example config

# Usage
removed `save_tuned_prompts_to_prompt_table`
```python
p_tuning:
  dropout: 0.0
  num_layers: 2
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation? (Updated in separate PR)
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Replace sampled tokens with EOD after EOD has been sampled once
State: closed | Pull request: yes

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Replace sampled tokens with EOD after EOD has been sampled once

**Collection**: BigNLP

# Changelog 
- Updated 

# Usage
Same as before

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Added answer only loss for prompt learning
State: closed | Pull request: yes

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Adds ability to calculate loss on answer portion of the prompt only for p-tuning/prompt-tuning.


**Collection**: BigNLP

Updated 
- GPTPromptLearningDataset class
- MegatronGPTPromptLearningModel class
- megatron_gpt_prompt_learning_config

# Usage
Same as before, but task templates in the config now have ``answer_only_loss`` and ``answer_field`` added to them.
They are added to the config like:

```python
- taskname: 'rte'
  prompt_template: '
{text}{answer}'
  total_virtual_tokens: 100 
  virtual_token_splits: [100]
  truncate_field: null
  answer_only_loss: True
  answer_field: 'answer'
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation? **(In separate PR)**
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Text memmap dataset
State: closed | Pull request: yes

Signed-off-by: Micha Livne <mlivne@nvidia.com>

# What does this PR do ?

Has mechanism to retire older ind files by updating internal idx version.

Indexing speed of 1443990774 samples in 147 files using 6 workers
```
```

Loading speed
```
[NeMo I 2022-04-29 00:23:22 text_memmap_dataset:85] Time loading 147 mem-mapped files: 0:00:04.395558
```

```
In [9]: len(ds)
Out[9]: 1443990774
# Timing without tokenizer
In [10]: %timeit -n 1000  ds[np.random.randint(len(ds))]
555 µs ± 16.9 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
# TIming with 'byte-level' tokenizer
In [20]: %timeit -n 1000  ds[np.random.randint(len(ds))]
724 µs ± 19.8 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
```

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Added `TextMemMapDataset`
- Added `CSVMemMapDataset`
- Retired `MegatronDataset`
- Added `nemo/collections/nlp/data/language_modeling/text_memmap_dataset.py` to preprocess indices (else happs on the fly at first run)
- Added `nemo/collections/nlp/data/machine_translation/sequence_to_sequence_dataset.py`
-  Added `scripts/nlp_language_modeling/build_index_memmap_data.py`

# Usage

Example for caching index files:

```bash
NeMo/scripts/nlp_language_modeling/build_index_memmap_data.py *.txt
```

Index files will be created when instantiating a memory mapped dataset if missing.

```python
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
not able to play audio in speech data explorer
State: closed | Pull request: no
Labels: name: bug

I am trying to explore ASR training data through the speech explorer tool and prepared the required manifest file in json format. It has three fields, audio_filepath, duration, and text. 

**Steps/Code to reproduce bug**

python tools/speech_data_explorer/data_explorer.py manifest.json

**Expected behavior**
It is loading the data in an expected manner. I am able to see text, duration, and file name. However, I am not able to play the audio. The audio player does not seem to be working.  

**Environment overview (please complete the following information)**

pip install -r tools/speech_data_explorer/requirements.txt

 - Using on an M1 mac machine

**Environment details**

- OS version - macOS Monterry M1
- Python version - 3.8.11
conformer fine-tuning with conformer_ctc_char.yaml
State: closed | Pull request: no

Hi thanks for your great tutorial, 
I have an error `Interpolation key 'model.labels' not found` when I set up the training params model. I do the following steps:
1) I used `stt_en_conformer_ctc_small` as pre-trained model.
2) I load `conformer_ctc_char.yaml ` file.
3) I change the model.labels in this file manually to new language characters. 
4) I change the train test monifest path and lr.
5) fit the model.
I attach my code. so what is the problem? How can I fine-tune the conformer for another language in CTC char-based?

```
import nemo
import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.EncDecCTCModelBPE.from_pretrained(model_name="stt_en_conformer_ctc_small")

from omegaconf import OmegaConf, open_dict
params = OmegaConf.load("conformer_ctc_char.yaml")
print(OmegaConf.to_yaml(params))

train_manifest = "./train.json"
test_manifest = "./test.json"
params.model.train_ds.manifest_filepath = train_manifest
params.model.validation_ds.manifest_filepath = test_manifest

params.model.spec_augment.rect_masks = 0

import copy
new_opt = copy.deepcopy(params.model.optim)
new_opt.lr = 0.001

print(params.model.optim)

print(asr_model.decoder.vocabulary)

asr_model.setup_optimization(optim_config=new_opt)

asr_model.setup_training_data(train_data_config=params['model']['train_ds'])

asr_model.setup_validation_data(val_data_config=params['model']['validation_ds'])

asr_model.encoder.freeze()

import pytorch_lightning as pl
from pytorch_lightning.callbacks.early_stopping import EarlyStopping

early_stop_callback = EarlyStopping(
   monitor= 'val_wer',
   min_delta=0.05,
   patience=3,
   verbose=True,
   mode='min'
)

trainer = pl.Trainer(max_epochs=2,
                     accumulate_grad_batches=1,
                     checkpoint_callback=False,
                     logger=False,
                     log_every_n_steps=5,
                     check_val_every_n_epoch=1,               
                     callbacks=[early_stop_callback],
                     gpus=0, 
                     # accelerator='ddp',
                     # plugins='ddp_sharded'
                    )

asr_model.set_trainer(trainer)

trainer.fit(asr_model)
```

**Environment overview**

Environment location: Google Colab
Method of NeMo install: !pip install nemo_toolkit[asr]
NeMo branch version: main
Learning Rate: 1e-3

**Environment details**

OS version : "Ubuntu20.04.3 LTS"
PyTorch version : "1.10.0"
[Core] Support pre-extracted nemo checkpoint for restoration
State: closed | Pull request: yes

Signed-off-by: smajumdar <titu1994@gmail.com>

# What does this PR do ?

Adds support to SaveRestoreConnector to employ a pre-extracted nemo directory path as a restoration directory.
This is required for extremely large language model restoration, which cannot be extracted partially during multinode inference.

**Collection**: [Core, Megatron]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Note
- During restore_from, the path of the nemo file is ignored in its entirety since the actual pre-extracted directory is passed.
- The filepath passed to restore_from in such a case is not preserved, it is overwritten by the value of `connector.model_extracted_dir` inside AppState.

# Usage

```python
# Extract a nemo file apriori to restoration
# Here we will use the SaveRestoreConnector's method to extract the checkpoint to a given directory 

connector = MySaveRestoreConnector()
MySaveRestoreConnector._unpack_nemo_file(nemo_filepath, extracted_directory)

# Then update SaveRestoreConnector with path of the extracted directory
connector.model_extracted_dir = extracted_directory

# Call restore as always
# note, we pass in the "old" nemo_filepath, stored somewhere other than the extracted directory
restored_model = ModelPT.restore_from(nemo_filepath, save_restore_connector=connector)
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation


## Who can review?
@okuchaiev @ericharper
placeholder PR for diarization decoder
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Error in ASR_with_SpeakerDiarization script when transcribing only one word
State: closed | Pull request: no
Labels: name: bug

Hi 
I'm using the script https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/ASR_with_SpeakerDiarization.ipynb in Colab and I came across a problem. When trying to transcribe an audio that has the transcription of only one word, it gives the following error in the "Run ASR and get word timestamps" block:

![image](https://user-images.githubusercontent.com/94137522/165177730-a7b39b2e-99a1-4489-af75-d53cb8c58e32.png)

The audio only contains transcription of the word "Eleven". In case you need it, here's the audio:
[audio.zip](https://github.com/NVIDIA/NeMo/files/8558127/audio.zip)
Update num worker calculation due to PTL flag changes
State: closed | Pull request: yes

Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

# What does this PR do ?

Removes conditionals in favor of simple calculation of num workers due to changes in PTL flags, as per conversation with @nithinraok and prior discussion in #3589 

# Changelog
- Update calculation of num workers in model class (optimizer setup)
- Update calculation of num workers in TTS helpers
- Bump PTL version requirement from 1.6.0 -> 1.6.1

**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Tn electronic
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Cherry pick HF integration and bug fixes from 1.8.1
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Problem with speaker diarization using long audios
State: closed | Pull request: no
Labels: name: bug

Good morning, 

Thank you very much for your excellent repository. 

I am having problems when I want to use Nemo for speaker diarization with "long" audios, could you please help me to make it work correctly?.

The error is the following:


```
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "/usr/local/lib/python3.7/dist-packages/nemo/collections/asr/data/audio_to_label.py", line 420, in vad_frame_seq_collate_fn
    return _vad_frame_seq_collate_fn(self, batch)
  File "/usr/local/lib/python3.7/dist-packages/nemo/collections/asr/data/audio_to_label.py", line 194, in _vad_frame_seq_collate_fn
    audio_signal = torch.stack(audio_signal)
RuntimeError: stack expects a non-empty TensorList
```


I leave an example of the code to reproduce the error:

https://colab.research.google.com/drive/1DnrIBrqaBiIZX-nPcoEMg3jrEGFR1zWF?usp=sharing

And one of the audios I am testing with:

https://drive.google.com/file/d/1eyzfzLh7u_xo3pzkGJ_GqYx38nukolrR/view?usp=sharing

Thank you very much for your time
Fix Punctuation and Capitalization model batching. An issue with shuffling.
State: closed | Pull request: yes
Labels: name: bug

# What does this PR do ?

1. If in DDP mode number of batches is not evenly divisible by number of processes, some batches may be processed several times. This lead to distorted metrics. This PR adjusts number of batches to number of processes.
2. In DDP mode batches in different processes are shuffled independently. This PR sets random seed for this shuffling. Before this fix, shuffling lead to some batches being processed several times and some batches not processed at all. 

**Collection**: nlp

# Changelog 
- [adjusting number of batches](https://github.com/NVIDIA/NeMo/blob/45e9deca8c29b6f3142294352c7350f385ba6b28/nemo/collections/nlp/data/token_classification/punctuation_capitalization_dataset.py#L1354)
- [setting random seed](https://github.com/NVIDIA/NeMo/blob/45e9deca8c29b6f3142294352c7350f385ba6b28/nemo/collections/nlp/models/token_classification/punctuation_capitalization_model.py#L787)

# Usage

To notice changes support numbers you can use this [stage](https://github.com/NVIDIA/NeMo/blob/91e6c2552a9ca8d00bf8ce758adea68d9c49f803/Jenkinsfile#L1219) in Jenkins.
  
**PR Type**:
- [x] Bugfix

Closes bug https://nvbugswb.nvidia.com/NvBugs5/SWBug.aspx?bugid=3610560&cmtNo=
WFST grammars export and test error
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I am looking for text normalization tools and select nemo text normalization tools.

In order to use it in c++, I follow the tutorial in [https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/text_normalization/wfst/wfst_text_processing_deployment.html](url)

and run test tn grammar code based on sparrowhawk and get result like:

<img width="828" alt="image" src="https://user-images.githubusercontent.com/29497466/164977748-1d21a641-7398-4bde-b8d0-11eeba3c1255.png">

It seems that numbers in time pattern is removed. Is this right?

Besides, many "Unknown field" error occurred like:

<img width="1269" alt="image" src="https://user-images.githubusercontent.com/29497466/164977824-fac1188c-c0ca-46bb-9ce9-ffc13695f5e8.png">

Really need your help about WFST grammar export and sparrowhawk errors.

Thanks a lot !
RMSNorm, Normformer and fixes from merging 1.8.0 into main
State: closed | Pull request: yes

# What does this PR do ?

Implements the following

1. RMSNorm normalization, used in T5 when not using bias terms.
2. Normformer that adds extra normalization layers after self-attention and between the MLP feedforward layers. Also adds the option of configuring the additional MHA scale parameter.
3. Fixed some issues with merging code from 1.8.0 to main that removed code relevant to removing bias terms.

**Collection**: NLP

# Changelog 
- Mostly changes to `transformer.py`

# Usage
* You can potentially add a usage example below

Configure the following args in your T5/BART yaml file

encoder_arch: "normformer"
decoder_arch: "normformer"
headscale: True
normalization: rmsnorm
bias: False

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Prompt Learning Docs
State: closed | Pull request: yes

# What does this PR do ?

Adds a prompt learning page to the NeMo docs. 

**Collection**: BigNLP

# Changelog 
- Added prompt_learning.rst
- Edited megatron.rst

# Usage
N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [X] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [X] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Fastpitch Tutorial fix
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)