renwei2024/nemo-github-issues huggingface dataset:
[TTS] Using a "reference" audio clip to control duration when synthesizing audio?
State: closed | Pull request: no
Labels: name: bug

name: stale

If I have an audio clip, for example the "Tears in rain" monologue from Blade Runner, is there any way to replicate the pacing/duration of that audio when generating a spectrogram for TTS? 

The "Inference_DurationPitchControl" tutorial covers changing the pace of the entire clip by some fixed percentage, but what I actually want is to change the duration of the individual words/phonemes (and the gaps between them) to match their durations in the reference audio.

If this isn't currently available, is it feasible as a future feature?
Greedy decoding confidence for CTC and RNNT
State: closed | Pull request: yes

# What does this PR do ?

Confidence estimation based on the maximum probability and entropy methods for CTC and RNNT models.

**Collection**: ASR

# Changelog 
- Hypothesis now has frame_confidence, token_confidence, and word_confidence attributes

# Usage
* You can potentially add a usage example below

```python
python scripts/speech_recognition/benchmark_asr_confidence.py pretrained_name=stt_en_conformer_transducer_large_ls dataset_manifest=<librispeech>/test_other.json output_dir=<your_output_dir> 'grid_params="{\"aggregation\": [\"mean\", \"min\"], \"temperature\": [0.33, 0.5]}"'
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?
@vsl9 @hainan-xv @titu1994
Tutorials and Docs for Multi-scale Diarization Decoder
State: closed | Pull request: yes

# What does this PR do ?

This PR adds tutorials and documentations for the newly published model "Multi-scale diarization decoder".

Not only tutorial and docs, we need to ha

**Collection**: ASR 

# Changelog 

Major updated files:
`tutorials/speaker_tasks/Speaker_Diarization_Inference.ipynb`
`tutorials/speaker_tasks/Speaker_Diarization_Training.ipynb`
`docs/source/asr/speaker_diarization/*.rst` files

Minor updates: 
Some changes are made to use a model published on NGC.
`nemo/collections/asr/models/msdd_models.py`
`nemo/collections/asr/models/clustering_diarizer.py`


# Usage
* You can potentially add a usage example below

These are tutorials and documents.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

NeMo ASR contributors
Upgrade to NGC PyTorch 22.08 Container
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: Core

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Sparrowhawk deployment fix
State: closed | Pull request: yes

Signed-off-by: ekmb <ebakhturina@nvidia.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[ASR][Tools] RIR corpus generator
State: closed | Pull request: yes

# What does this PR do ?

This PR aims to add a tool for generating a set of multi-channel RIRs.

**Collection**: ASR

# Changelog 
- Added `RoomCorpusGenerator` in `asr/data/data_simulation.py`
- Added unit tests in `tests/collections/asr/test_asr_data_simulation.py`
- Added an example config and a readme file in `tools/rir_corpus_generator`

# Usage
The tool can be used as

```bash
python rir_corpus_generator.py output_dir=OUTPUT_DIR
```

where `OUTPUT_DIR` is a path to the output directory. This will use the default configuration in `conf/rir_corpus_v1.yaml`.

The output will be structured as

```bash
OUTPUT_DIR
+--{train, dev, test}
|	+--*.h5
+--config.yaml
+--{train, dev, test}_manifest.yaml
+--{train, dev, test}_info.png
```

Each directoy, e.g, `{train, dev, test}`, corresponds to a subset of data and contain the `*.h5` files with RIRs. Corresponding `*_manifest.yaml` files contain metadata for each subset.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* n/a
Text Processing Export Grammar is raising error while creating Docker image
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

While exporting grammar we are facing error while creating the docker image.

**Steps/Code to reproduce bug**

``` bash
cd tools/text_processing_deployment
bash export_grammars.sh --GRAMMARS=tn_grammars --LANGUAGE=en --INPUT_CASE=cased --FORCE_REBUILD=True --OVERWRITE_CACHE=True
```

Above command is stopping with the following error:
![docker image build error](https://user-images.githubusercontent.com/1061389/189801110-4b014c4c-f161-456b-bdd5-c538e287122e.png)

**Expected behavior**

To export the grammar the sparrowhawk image has to be built completely.

**Environment details**

``` text
Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:02:57 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:03 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 20.04
- Python version: Python 3.8.10 (on the host machine)
[WIP] G2P integration with TTS
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Evaluation script for VAD models
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**

I'd like to be able to evaluate the performance of speech/audio classification models (in particular, VAD/SAD models) with different collars, using metrics such as are available in `md-eval.pl` in SCTK (used by Kaldi's SAD recipes) or PyAnnote metrics (see https://github.com/pyannote/pyannote-metrics/blob/develop/pyannote/metrics/detection.py).

**Describe the solution you'd like**

I'd like to have a script that takes a manifest (formatted the same way as for train/validation in speech classification; not necessarily preprocessed to have only short sections) as input and produces error/accuracy metrics. As PyAnnote Metrics is already used elsewhere in NeMo, it could probably easily used for this purpose. Or perhaps there already exist internal/unpublished scripts for that purpose that could be added to NeMo.

**Describe alternatives you've considered**

Creating RTTM output from VAD predictions and evaluating them using SCTK/md-eval.pl.

Also tagging @fayejf

Thanks!
Add support for skipping validation on resume + extend saving last ckpt test
State: closed | Pull request: yes
Labels: name: bug

# What does this PR do ?

When restarting training from a saved checkpoint during validation, validation is re-run as a first step (as the checkpoint was saved just before validation). This PR introduces a new loop injected via the `exp_manager` that skips validation if we're restarting.

There was also a mistake with the test definition from #4905 and I extended the test further to check @yaoyu-33s' case.

cc @MaximumEntropy @ericharper @titu1994 

# Changelog 
- Skip initial validation by default when resuming from checkpoint which saved after a validation pass has occurred.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Nemo gets installed in legacy version because it couldn't build a wheel
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I am using google colab for using nemo, nemo supports python 3.8 but colab has python 3.7, I can succesfully install python 3.8 in colab and pip but when I try to install nemo, it gives me an error in the end that nemo got installed in legacy version becaue pip failed to build a wheel. After using the `!pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[all] --verbose --log ./pip.log` and reading the logs, I find this 

> Traceback (most recent call last):
> File "<string>", line 2, in <module>
> File "<pip-setuptools-caller>", line 34, in <module>
> File "/tmp/pip-install-0anzil8f/nemo-toolkit_8b9af28d0a0a4f5d861548d9508e4c38/setup.py", line 209, in <module>
> setuptools.setup(
> File "/usr/local/lib/python3.8/dist-packages/setuptools/__init__.py", line 153, in setup
> return distutils.core.setup(**attrs)
> File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
> dist.run_commands()
> File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
> self.run_command(cmd)
> File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "/usr/lib/python3/dist-packages/wheel/bdist_wheel.py", line 256, in run
> self.egg2dist(self.egginfo_dir,
> File "/usr/lib/python3/dist-packages/wheel/bdist_wheel.py", line 426, in egg2dist
> pymeta = pkginfo_to_dict(metadata_path,
> File "/usr/lib/python3/dist-packages/wheel/metadata.py", line 177, in pkginfo_to_dict
> new_requirements = sorted(convert_requirements(requirements))
> File "/usr/lib/python3/dist-packages/wheel/metadata.py", line 234, in convert_requirements
> parsed_requirement = pkg_resources.Requirement.parse(req)
> File "/usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py", line 3154, in parse
> req, = parse_requirements(s)
> ValueError: too many values to unpack (expected 1)

**Steps/Code to reproduce bug**

Install python 3.8 and then run this


**Expected behavior**
Nemo should be installed correctly and its logs should show any problem.

**Environment overview (please complete the following information)**

 - Environment location: Colab
 - Method of NeMo install: [pip install or from source]. `!pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[all] --verbose --log ./pip.log`

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- Ubuntu (colab)
- pytorch 1.12+cu116
- Python 3.8
merge r1.11 to main
State: closed | Pull request: yes

# What does this PR do ?

This PR is to merge `r1.11.0` branch to the `main` branch. There were merge conflicts according to several updates in all domains. I fixed all conflicts. For some conflicts, I chose the `main` changes, while for other conflicts, I made a decision after reading detailed changes in both `r1.11.0` and `main`. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [] Did you write any new necessary tests?
- [] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?
Owners of PRs in every domain, please help to review the changes. Thanks.

# Additional Information
* Related to # (issue)
Incorrect WER reported by nemo wer function
State: closed | Pull request: no
Labels: name: bug

On my test set when I run  trainer.test() I am getting  a wer of 0.081. 

 global_step        │            0.0            │
│         test_loss         │     7.94022274017334      │
│         test_wer          │    0.08155332505702972  


but when I try running 
jiwer.wer or 
from nemo.collections.asr.metrics.wer import word_error_rate 

I am getting wer == 0.139 why is there such a big difference. 


have also tried : 

WER = WERBPE(decoding=ctc_bpe_decoding, use_cer = False,log_prediction=False, fold_consecutive=False,dist_sync_on_step=True)

Even this gives me a wer of 0.139

I am running this function for every sample and then taking mean 


```
from nemo.collections.asr.metrics.wer_bpe import CTCBPEDecoding,  WERBPE
from nemo.collections.asr.metrics.wer_bpe import CTCDecodingConfig

ctc_bpe_decoding  = CTCBPEDecoding(decoding_cfg = CTCDecodingConfig(), tokenizer = asr_model.tokenizer)

asr_model._wer.log_predictions  = False
asr_model._wer.use_cer  = False

WER = WERBPE(decoding=ctc_bpe_decoding, use_cer = False,log_prediction=False, fold_consecutive=False,dist_sync_on_step=True)

@torch.no_grad()
def cal_wer(gt, pred):
    global WER
    # print(asr_model.tokenizer.text_to_tokens(gt))

    gt = asr_model.tokenizer.text_to_ids(gt)
    pred = asr_model.tokenizer.text_to_ids(pred)

    len_gt = torch.tensor([len(gt)])
    len_pred = torch.tensor([len(pred)])
    
    gt = torch.tensor([gt])
    pred = torch.tensor([pred])

    WER.update(targets = gt, predictions = pred, target_lengths = len_gt, predictions_lengths=len_pred)

    return WER.compute()[0].item()
```
[TTS] Ssl tts final
State: closed | Pull request: yes
Labels: name: core

name: TTS

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Draft] Diffs comparing R1.11.0 to main
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS][German][Single Speaker][Fastpitch + HifiGAN] Bad Interference RTF on CPU
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

The RTF for interference is greather than 2 on Intel CPUs. 

**Steps/Code to reproduce bug**

The below code produces a 7 second audio (speech2.wav) after properly loading models and takes more than 14 seconds. For me its unclear how many cores are utilized and where to configure this.

git clone https://github.com/NVIDIA/NeMo/
cd NeMo
python3.8 -m venv venv
source venv/bin/activate
pip3 install a-lot-of-missing-requirements when executing
python3 tts_german.py where inference code is as follows tts_german.py

```
import soundfile as sf
from nemo.collections.tts.models.base import SpectrogramGenerator, Vocoder
import time

# Download and load the pretrained fastpitch model
spec_generator = SpectrogramGenerator.from_pretrained(model_name="tts_de_fastpitch_singlespeaker")#.cuda()
# Download and load the pretrained hifigan model
vocoder = Vocoder.from_pretrained(model_name="tts_de_slr_hifigan_ft_fastpitch_singlespeaker")#.cuda()

# All spectrogram generators start by parsing raw strings to a tokenized version of the string
parsed = spec_generator.parse("Was schreibst Du da gerade Papa?")
# They then take the tokenized string and produce a spectrogram
spectrogram = spec_generator.generate_spectrogram(tokens=parsed)
# Finally, a vocoder converts the spectrogram to audio
audio = vocoder.convert_spectrogram_to_audio(spec=spectrogram)

# Save the audio to disk in a file called speech.wav
# Note vocoder return a batch of audio. In this example, we just take the first and only sample.
sf.write("speech.wav", audio.to('cpu').detach().numpy()[0], 22050)

starttime=time.time()
parsed = spec_generator.parse("In den alten Zeiten, wo das Wünschen noch geholfen hat, lebte ein König, dessen Töchter waren alle schön.")
spectrogram = spec_generator.generate_spectrogram(tokens=parsed)
audio = vocoder.convert_spectrogram_to_audio(spec=spectrogram)
sf.write("speech2.wav", audio.to('cpu').detach().numpy()[0], 22050)
endtime=time.time()
print(endtime-starttime)
```


**Expected behavior**

Faster interference time, RTF < 1. 

**Environment overview (please complete the following information)**

 See above

**Environment details**

See above

**Additional context**

--
Adoptive lookahead for streaming Conformer encoder
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.
- Adds support to train and test models with adoptive lookahead

**Collection**: [ASR]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below
Need to add/modify following flags to train streaming conformer. `ATT_CONTEXT_STYLE` can be set to `regular`/`chunked_limited`. If `CONV_DUAL_MODE` is set to `true`, then `CONV_CONTEXT_SIZE` should be set to `null`, otherwise `CONV_CONTEXT_SIZE` can be set to `causal` to train streaming models. `ATT_CONTEXT_SIZE` can be set something like `[102,[16,33,50,0],full]` to train chunk-based streaming (middle four value) and non-streaming (last) modes together. Probability or fraction of updates for non-streaming mode can be controlled by setting `NON_STREAMING_PROB` in [0, 1] range. If both modes are trained together, separate layer norm can be instantiated by setting `STREAMING_LAYER_NORM` to `true`.

```python
model.preprocessor.normalize=NA \
model.encoder.att_context_size=$ATT_CONTEXT_SIZE \
model.encoder.att_context_style=$ATT_CONTEXT_STYLE \
model.encoder.causal_downsampling=true \
model.encoder.conv_context_size=$CONV_CONTEXT_SIZE \
model.encoder.conv_norm_type=layer_norm \
++model.encoder.non_streaming_prob=$NON_STREAMING_PROB \
++model.encoder.conv_dual_mode=$CONV_DUAL_MODE \
++model.streaming_layer_norm=$STREAMING_LAYER_NORM
```

To compute wer for different modes, use the following command
```
python ./examples/asr/transcribe_speech.py model_path=$MODEL_PATH \
       +val_random_params_enc.att_context_size_right=$ATT_CONTEXT_SIZE_RIGHT
```
where `ATT_CONTEXT_SIZE_RIGHT` can be set to any desired value from`{0, 16, 33, 50, 'full'}` if the model was trained with `ATT_CONTEXT_SIZE=[102,[16,33,50,0],full]`
```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix ptl_load_state not providing cls
State: closed | Pull request: yes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

# What does this PR do ?

Fixes `ptl_load_state` calls not providing the class attribute.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix and refactor label models
State: closed | Pull request: yes

# What does this PR do ?

Fix and refactor label models

**Collection**: ASR

# Changelog 
- fix hanging issue happens in test after train in` speech_to_label.py `, `speaker_reco.py` and `speaker_reco_finetune`.py
- fix weighted cross entropy by disabling cal_label_occurences for val and test
- refactor loss instantiation part in label_models 
- updated yamls file for above refactory.

# Usage
Specify loss as below in config files for label_models
```python
loss:
    _target_: nemo.collections.common.losses.cross_entropy.CrossEntropyLoss
    weight: 'auto' # could be 'auto' or 1D tensor or null
# or
loss:
    _target_: nemo.collections.asr.losses.angularloss.AngularSoftmaxLoss # you could also use cross-entrophy loss
    scale: 30
    margin: 0.2
    
```
# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Merge r1.11 main
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Error while restarting training. 
State: closed | Pull request: no
Labels: name: bug

name: stale

Getting the following error : Early stopping conditioned on metric `val_loss` which is not available. Pass in or modify your `EarlyStopping` callback to use any of the following: ``

using nemo-toolkit['asr'] == 1.10.0

pytorch-lightning       1.7.3
torch                            1.12.1
torchmetrics               0.9.3


looks like this is a Pytorch-lightning help any work around this?


    trainer = ptl.Trainer(
                          val_check_interval= FLAGS.val_check_interval,
                          devices=-1, 
                          max_epochs=EPOCHS, 
                          accumulate_grad_batches=FLAGS.accumulate_grad_batches,
                          enable_checkpointing=FLAGS.enable_checkpointing,
                          logger=FLAGS.logger,
                          log_every_n_steps=FLAGS.log_every_n_steps, ##value should be high to save memory.
                         callbacks = [early_stop],
                          sync_batchnorm = FLAGS.sync_batchnorm,
                          gradient_clip_val=FLAGS.gradient_clip_val,
                          num_sanity_val_steps=0,
                          )



@titu1994
IA3 support for GPT and T5
State: closed | Pull request: yes

Signed-off-by: arendu <adithya.r@gmail.com>

# What does this PR do ?

- Adds adapter tuning support for t5 models
- Adds Infused Adapter (IA3) support for gpt and t5 models.
- Ensures that no test example is skipped due to max-length issues (a wrong answer is better than skipping)

**Collection**: [nlp]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```
python NeMo/examples/nlp/language_modeling/tuning/megatron_t5_adapter_tuning.py
python NeMo/examples/nlp/language_modeling/tuning/megatron_t5_ia3_tuning.py 
python NeMo/examples/nlp/language_modeling/tuning/megatron_gpt_ia3_tuning.py 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update the prompt learning to handle large lanague model
State: closed | Pull request: yes

# What does this PR do ?

Added the following features:
1.  Allow the model to load from the unpacked nemo file
2.  Remove LM model weights from optimizer
3.  It can apply various activation checkpoint optimizations to the large LM
4.  When Pytorch Lightning restore the model, the weights are in fp32 format for some reason even the checkpoint file are saved as bf16. I fix this issue by casting the floating number type to the correct precision. This saves the half of the initial GPU memory consumption. 
5.  Implement to enable the sequence parallel for p-tuning. It saves a lot of GPU activation memory by using it. Also, it enables selective activation, so it can save extra activation memory without sacrificing much performance. 
6.  Implement the Tensor Parallel version of the MLP prompt encoder, so it saves GPU memory and distributed computation in TP workgroup.
7.  Added a script to conver PTL p-tuning checkpoint files to nemo file for evaluation.
8.  Added cache capability to the prompt dataset to save the data processing time.
Fix saving the last checkpoint when using val check interval
State: closed | Pull request: yes
Labels: name: bug

# What does this PR do ?

When saving a checkpoint and the validation check interval isn't divisible by the number of batches, we can miss saving the final checkpoint. This ensures we save the final checkpoint.

I've noticed that the epoch saved in the checkpoint is incremented. Not sure if this will be an issue, but considering this is the final step I assume it should be fine.

cc @ericharper @okuchaiev

# Changelog 
- Fix saving the last checkpoint when `val_check_interval`.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Add mixed-representation config to PhonemizerTokenizer
State: closed | Pull request: yes

Signed-off-by: Ryan <rlangman@nvidia.com>

# What does this PR do ?

Extends the IPATokenizer and PhonemizerTokenizer classes so we can do mixed-representation grapheme/phoneme training for non-English languages.

**Collection**: [TTS]

# Changelog 
- Add phoneme_probability argument to PhonemizerTokenizer, with code changes to add the grapheme set for each language. The grapheme set was obtained by taking all graphemes in the words of each language's pronunciation dictionary.
- Removed "use_chars" logic in IPATokenizer, which currently adds hardcoded list of English ASCII chars, in favor of parsing them from the user-provided phoneme dictionary in IPAG2P (similar to how it infers its phoneme set from the dictionary).
- In IPAG2P I added accented characters & locale-agnostic digit to the word/token checker. In an earlier commit, I accidentally added this to the EnglishG2p instead (when refactoring the PR around the tokenizer file directory change). Though I don't see any reason to remove it from EnglishG2p.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

# Additional Information
[TTS] fix wrong g2p path.
State: closed | Pull request: yes

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

# What does this PR do ?

fixed wrong g2p path. Previously in https://github.com/NVIDIA/NeMo/pull/4690, g2p modules were moved to `nemo_text_processing.g2p.modules`. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
removed unused imports for all domains.
State: closed | Pull request: yes

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

# What does this PR do ?
LGTM.com detected 100 ish unused imports. Clean up to remove those warnings.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Support distributed Adam with T5 and support overlapped grad reductions with pipeline parallelism
State: closed | Pull request: yes

# What does this PR do ?

Generalize distributed Adam support for GPT-3 to T5 and other Megatron-LM models. It also implements several performance optimizations.

**Collection**: NLP
# Changelog 
- When params are BF16, distributed Adam will store 16-bit param remainders instead of FP32 main params
- Decouples distributed Adam support from Megatron O2-level optizations
- Add support for Apex distributed Adam optimizer with other Megatron-LM models, namely T5
- Add support for overlapped grad reductions with pipeline or sequence parallelism

# Usage

Set optimizer to `distributed_fused_adam` in config file:

https://github.com/NVIDIA/NeMo/blob/265f7b17876aef7937cf0fedf7a76885bdc63288/examples/nlp/language_modeling/conf/megatron_t5_config.yaml#L137

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Closes https://github.com/NVIDIA/NeMo/pull/4799
* Depends on https://github.com/NVIDIA/apex/pull/1475
Fix question answering docs r1p11
State: closed | Pull request: yes

# What does this PR do ?

Fix bug in documentation

**Collection**: nlp
# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue 4775)
Addition of word in Citrinet Model
State: closed | Pull request: no

stt_en_citrinet_512 and stt_en_citrinet_1024 both are trained on large English speech data, is there any way we can add few more words such as name(Example Hanuman or Ganesha), pronunciation of these words might be different depends upon the language , if we have enough speech data to add these words for improvement its transcription accuracy. 

In short, I want to add few more words in dictionary of this model.
Update tutorials.rst for question answering
State: closed | Pull request: yes

# What does this PR do ?

Documentation bug fix.

**Collection**: nlp

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue 4775)