renwei2024/nemo-github-issues huggingface dataset:
[ASR] Fix for old models in change_attention_model
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fixes some issues:
-change_attention_model should work for old models which don't have some of the relevant parameters in config
-also it shouldn't run when nothing to change was submitted

**Collection**: ASR

  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation
Revert data iter list
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Create dummy iters to satisy iter type len checks in core + update core commit
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

name: CI

# What does this PR do ?

Create dummy iters to satisy iter type len checks in core + update core commit

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Raise value error when it falls back to native pytorch softmax and attn_mask is None
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

When softmax fused kernel is [not available](https://github.com/NVIDIA/apex/blob/85e9eddece9d4ac72b48c2407f8162f2173e1bf4/apex/transformer/functional/fused_softmax.py#L222), it will fall back to pytorch softmax forward in NeMo.

In apex, the mask is override based on your `attn_layer_type`. So it will always be set to either [causal or padding type of masking](https://github.com/NVIDIA/apex/blob/85e9eddece9d4ac72b48c2407f8162f2173e1bf4/apex/transformer/functional/fused_softmax.py#L205). While when it falls back to NeMo, it will actually do None masking fwd. This will cause in issues in training and inference when your tensor shape doesn't fit kernel or you are using 32 precision. 

We propose few improvement:
1. In apex, the user should be warned, no matter what they set in mask, it will be override based on `attn_layer_type`.
2. In every NeMo models, the attn mask should always be set when it's trying to use `MatchedScaleMaskSoftmax`.

**Collection**: Megatron models use fused softmax from apex, but attn_mask is None.

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Patch transcribe_util for steaming mode and add wer calculation back to inference scripts
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Patch transcribe_util for steaming mode and add wer calculation back to the inference scripts as requested

**Collection**: ASR

# Changelog 
- Patch `decoding` in `write_transcription`
- Temporarily set `decoding.beam.return_best_hypothesis = True` to true to be written to manifest to avoid`assert`, see in [discussion](https://github.com/NVIDIA/NeMo/pull/6063/files#r1178635367) @karpnv. May need to come up a better way for ngram/beam
- Add wer calculation back to
 `transcribe_speech.py`,
 `speech_to_text_buffered_infer_ctc`,
 `speech_to_text_buffered_infer_rnnt` as requested, but I still recommand to use asr evaluator and simply those inference script if possible. 
- Reflect changed in ASR evaluator 
# Usage
```python
python transcribe_speech.py \
    model_path=xxxx \
    dataset_manifest=xxx\
    clean_groundtruth_text=True \
    langid='zh' \
    use_cer=True \
    calculate_wer=True

[NeMo I 2023-05-08 15:43:25 transcribe_speech:360] Writing prediction and error rate of each sample to xxxx
[NeMo I 2023-05-08 15:43:25 transcribe_speech:361] {'samples': 2, 'tokens': 164, 'wer': 0.04878048780487805, 'ins_rate': 0.0, 'del_rate': 0.04878048780487805, 'sub_rate': 0.0}

```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to[PR#6559](https://github.com/NVIDIA/NeMo/pull/6559)
Create dummy iters to satisy iter type len checks in core + update core commit
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Create dummy iters to satisy iter type len checks in core + update core commit

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
A few small typo fixes
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: asr

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
merge lora weights into base model
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Piggy backing on eval script to merge lora weights into a base GPT model.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update SDP docs
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Add more detail to SDP section in NeMo docs, and add link to that page in the NeMo README.rst.

**Collection**: docs

# Changelog 
- update SDP section in NeMo docs
- update NeMo README

# Usage
N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [N/A] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ N/A ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Documentation for ASR-TTS models
State: closed | Pull request: yes
Labels: name: TTS

name: cherry-pick

name: ASR

# What does this PR do ?

Add documentation for hybrid ASR-TTS models.

**Collection**: [ASR]

# Changelog 

- add docs to ASR models
- add docs for text-only dataset construction
- add description for configs and training scripts
- add note about extracting final ASR model
- add datasets and hybrid ASR-TTS model to ASR collection API
- fix imports: optional library, partially initialized module
 

# Usage
* You can potentially add a usage example below

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Documentation for ASR-TTS models
State: closed | Pull request: yes
Labels: name: TTS

name: ASR

# What does this PR do ?

Add documentation for hybrid ASR-TTS models.

**Collection**: [ASR]

# Changelog 

- add docs to ASR models
- add docs for text-only dataset construction
- add description for configs and training scripts
- add note about extracting final ASR model
- add datasets and hybrid ASR-TTS model to ASR collection API
- fix imports: optional library, partially initialized module
 

# Usage
* You can potentially add a usage example below

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Make tensor split contiguous for qkv and kv in attention
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Make tensor split contiguous for qkv and kv in attention

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Patches from main to r1.18.0 for Virtual Parallel
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Cherry pick patches for VP

**Collection**: [NLP]

# Changelog 
- Adds VP support

# Usage

```python

# Megatron GPT + Virtual Pipeline parallelism
python megatron_change_num_partitions.py \
    --model_extracted_dir="<Directory of Pytorch Lightning ckpt folders>" \
    --target_file="<Name of the target NeMo file>" \
    --ckpt_name="<Name of a single Pytorch Lightning ckpt file inside the extracted dir>" \
    --tensor_model_parallel_size=<TP SIZE> \
    --target_tensor_model_parallel_size=<TARGET TP SIZE> \
    --pipeline_model_parallel_size=<PP SIZE> \
    --target_pipeline_model_parallel_size=<TARGET PP SIZE> \
    --virtual_pipeline_model_parallel_size=<VP SIZE> \
    --hparams_file="<Path to HPARAMS.yaml file>" \
    --precision=bf16


```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Error 'pitch_state.json' expecting ID when finetuning for spanish multispeaker model
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I'm trying to fine tune the tts_es_fastpitch_multispeaker.nemo model and have this issue with the pitch_state.json file. I just copy the format in "scripts/tts_dataset_files/openslr_es/pitch_stats.json" and I have this error.

`mismatched input 'pitch_state.json' expecting ID`

**Steps/Code to reproduce bug**

I only add my new speaker with a new number to the pitch_stats.json with his respective mean and std pitch. I use the config file in "NeMo/examples/tts/conf/es/fastpitch_align_44100_ipa_multi.yaml" and try modifing the sup_data_type to match the number of my new speaker but it doesn't work either.

`sup_data_types: ["align_prior_matrix", "pitch", 174]`

The code use for fine tuning:

`
!(python fastpitch_finetune.py --config-name=fastpitch_align_44100_ipa_multi.yaml \
  train_dataset=./transcript_train_spk.json \
  validation_datasets=./transcript_dev_spk.json \
  sup_data_path=./fastpitch_sup_data \
  pitch_stats_path= tts_dataset_diles/pitch_state.json \
  phoneme_dict_path=tts_dataset_files/es_ES_nv230301.dict \
  exp_manager.exp_dir=./ljspeech_to_9017_no_mixing_5_mins \
  +init_from_nemo_model=./tts_es_fastpitch_multispeaker.nemo \
  trainer.check_val_every_n_epoch=25 \
  model.train_ds.dataloader_params.batch_size=12 model.validation_ds.dataloader_params.batch_size=12 \
  model.n_speakers=175 model.pitch_mean=137.2 model.pitch_std=44.2 \
  model.pitch_fmin=30 model.pitch_fmax=512 model.optim.lr=2e-4 \
  ~model.optim.sched model.optim.name=adam trainer.devices=1 trainer.strategy=null \
  +model.text_tokenizer.add_blank_at=true \
)
`

**Environment details**

Use Google Collab following this [example](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_Finetuning.ipynb)
Add patches for Virtual Parallel conversion
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Fix an issue with virtual parallel conversion

**Collection**: [NLP]

# Changelog 
- Fixes issue where the resultant model loaded in VP does not correctly load subset of model parameters due to incorrect VP state set in megatron core

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.
Fix batch size reconf for T5 FT for multi-validation
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Current T5 FT batch reconf for drop_last = False didn't account for the case of multivalidation. This lead to incorrect setting of micro-batch calculator.
The PR fixes the issue

**Collection**:  NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
ASR inference stops without any error
State: closed | Pull request: no

I am following [this tutorial](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/examples/kinyarwanda_asr.html) for training and evaluating a conformer-BPE model. 
The training has finished properly and I have a `.nemo` file. I have tried following the inference script [here](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/examples/kinyarwanda_asr.html#inference-and-evaluation) on `colab pro` to no avail. The code just stops when it has to iterate over the batches. *There are no errors*. The final line is given below: 
```bash
Transcribing:   0% 0/235 [00:00<?, ?it/s]^C
``` 

This is the script I am using : 
```bash 
! HYDRA_FULL_ERROR=1 python3 ./NeMo/examples/asr/transcribe_speech.py \
  model_path=./assets/Conformer-CTC-BPE.nemo \
  dataset_manifest=./speech_test_dataset_transcript.json \
  output_filename=test_with_predictions.json \
  batch_size=32 \
  overwrite_transcripts=false
```

Can anyone please guide me here? I have also tried tinkering with the class file at  `NeMo/nemo/collections/asr/models/ctc_models.py`
transcribe multilanguage in multilingual model
State: closed | Pull request: no
Labels: name: bug

name: stale

I took a multilingual pretrained english model and wanted to fine-tune it for ASR task on english and arabic together at once. I used a tokenizer that contains both English and arabic manifest.

Can this https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/conformer/multilang/conformer_ctc_bpe_multilang.yaml
Setting give results like this if yes then how. I was not able to find one.
I want the transcription to be:
مرحبًا ، أريد apple لذيذة

I've checked the tutorial file https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/Multilang_ASR.ipynb but in this tutorial in the inference, it is a sample just for english and the other one is just for spanish. In fact it doesn't have an inference that transcribe a multi-languange like I said.

How one can do code switching? I wanna transcribe an audio file that a person speak in english and arabic simultaneously?
TTS Aligner Inference Example outputs attn_soft_tensor with nan values 
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I simply followed the [TTS Aligner Inference Example](https://github.com/NVIDIA/NeMo/blob/stable/tutorials/tts/Aligner_Inference_Examples.ipynb) but the visualized `attn_soft` is empty and indeed `attn_soft_tensor` has all `nan` values.

<img width="1082" alt="Screen Shot 2023-05-06 at 10 43 29 PM" src="https://user-images.githubusercontent.com/621009/236655018-9c65c525-684c-42f6-ba1a-45e67024113f.png">

**Steps/Code to reproduce bug**

You can simply proceed through [this notebook]((https://github.com/NVIDIA/NeMo/blob/stable/tutorials/tts/Aligner_Inference_Examples.ipynb)

**Expected behavior**

Display a graph with values described in the instruction.

**Environment overview (please complete the following information)**

Google Collab
Fix batch size reconf for T5 FT for multi-validation
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Current T5 FT batch reconf for drop_last = False didn't account for the case of multivalidation. This lead to incorrect setting of micro-batch calculator.
The PR fixes the issue

**Collection**:  NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Safe batch_norm
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?
adds an option for a safe batchnorm that does not accept nan values, converting them to zeroes

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Make tensor split contiguous for qkv and kv in attention
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Make tensor split contiguous for qkv and kv in attention

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
More streaming conformer export fixes
State: closed | Pull request: yes
Labels: name: core

name: cherry-pick

name: ASR

# What does this PR do ?

Bug fixes for cach-aware conformer export. 
replaces #6562 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

@VahidooX

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Turn autocast off when precision is fp32
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Fix for autocast when precision is fp32

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Separate punctuation by whitespace
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fix separate_punctuation=True option 

**Collection**: ASR

# Changelog 
Do 
```
'some text.' -> 'some text .'
'some text .' -> some text .'
```

instead of 
```
'some text.' -> 'some text .'
'some text .' -> some text.'
```

**PR Type**:
- [V ] Bugfix


## Who can review?

@Kipok
megatron-lm version issue  while installing nemo-toolkit
State: closed | Pull request: no
Labels: name: bug

Hi,

I'm facing the following issue while pip install nemo-toolkit.

ERROR: Could not find a version that satisfies the requirement megatron-lm==2.2.0; extra == "all" (from nemo-toolkit[all]) (from versions: 2.2.1, 2.2.2)
ERROR: No matching distribution found for megatron-lm==2.2.0; extra == "all"



Can someone help me solve this issue?
ONNX for VAD inference time improvement small
State: closed | Pull request: no

Hello, 

I am using the offline speaker diarization module and I managed to export vad_multilingual_marblenet.nemo model to onnx. 

Running voice activity detection with ONNX results in an improvement of about 20% less inference time. From what I have read ONNX usually results in a greater imrovement. 

Is this result normal? Did I implement it wrong? Did anybody try this before and has some profiling results? Are there any ways to improve  inference speed even more?

My code in /nemo/collections/asr/models/clustering_diarizer:

```
    def _run_vad(self, manifest_file):
        """
        Run voice activity detection. 
        Get log probability of voice activity detection and smoothes using the post processing parameters. 
        Using generated frame level predictions generated manifest file for later speaker embedding extraction.
        input:
        manifest_file (str) : Manifest file containing path to audio file and label as infer

        """

        shutil.rmtree(self._vad_dir, ignore_errors=True)
        os.makedirs(self._vad_dir)

        self._vad_model.eval()

        self._vad_model.export('vad.onnx')
        ort_session = onnxruntime.InferenceSession('vad.onnx')
        time_unit = int(self._vad_window_length_in_sec / self._vad_shift_length_in_sec)
        trunc = int(time_unit / 2)
        trunc_l = time_unit - trunc
        all_len = 0
        data = []
        for line in open(manifest_file, 'r', encoding='utf-8'):
            file = json.loads(line)['audio_filepath']
            data.append(get_uniqname_from_filepath(file))

        status = get_vad_stream_status(data)
        for i, test_batch in enumerate(
            tqdm(self._vad_model.test_dataloader(), desc='vad', leave=True, disable=not self.verbose)
        ):
            test_batch = [x.to(self._vad_model.device) for x in test_batch]
            with autocast():
                # ONNX inference
                audio_signal, audio_signal_len = test_batch[0], test_batch[1]
                audio_signal, audio_signal_len = audio_signal.to(self._vad_model.device), audio_signal_len.to(self._vad_model.device)
                processed_signal, processed_signal_len = self._vad_model.preprocessor(input_signal=audio_signal, length=audio_signal_len,)

                ort_inputs = {ort_session.get_inputs()[0].name: self.to_numpy(processed_signal), }
                ologits = ort_session.run(None, ort_inputs)
                alogits = np.asarray(ologits)
                logits = torch.from_numpy(alogits[0])
                probs = torch.softmax(torch.as_tensor(logits), dim=-1)
                pred = probs[:, 1]

```
Offline and streaming inference support for hybrid model
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Offline and streaming inference support for hybrid model

**Collection**: ASR

# Changelog 
- Add check to examine whether model and decoding type match
- Update`speech_to_text_buffered_infer_ctc.py` and `speech_to_text_buffered_infer_rnnt.py` to support hybrid + ctc and hybrid rnnt 
- Make default `model_stride` in eval.yaml to be 8 for `fastconformer`
- Update asr_evaluator accordingly
- Add relative path + manifest support (get_full_path) to buffered inference 

# Usage
```python
#  model / decode_type
# # buffered inference
# [x] ctc  + null
# [x] ctc + ctc 
# [x] ctc + rnnt - should fail
# [x] rnnt + null
# [x] rnnt + rnnt 
# [x] rnnt + ctc - should fail 
# [x] hybrid ctc  
# [x] hybrid null same as below
# [x] hybrid rnnt

# # offline
# [x] rnnt null 
# [x] rnnt rnnt 
# [x] rnnt ctc - should fail 
# [x] ctc null 
# [x] ctc ctc  
# [x] ctc rnnt - should fail
# [x] hybrid + ctc
# [x] hybrid + null same as rnnt
# [x] hybrid + rnnt
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Limit urllib3 version to patch issue with RTD
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Docs build fails currently due to older version of OpenSSL on readthedocs. Limit urllib3 version to <2

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Limit urllib3 version to patch issue with RTD
State: closed | Pull request: yes

# What does this PR do ?

Docs build fails currently due to older version of OpenSSL on readthedocs. Limit urllib3 version to <2

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
More streaming conformer export fixes
State: closed | Pull request: yes
Labels: name: core

name: ASR

# What does this PR do ?

Bug fixes for cach-aware conformer export. 
replaces #6562 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

@VahidooX

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Is NeMo nvidia's recommend way to run llm?
State: closed | Pull request: no
Labels: name: stale

Is NeMo the best way to run LLM on your hardware for conversation?

My second experience was that on 4090 the https://huggingface.co/nvidia/GPT-2B-001 did not work #6564
I want to play around with FP8 and it looked like this was fastest way but it was just not working, and when did it to run was 4x slower than regular hf transformers.

So simplest thing is have a really damn simple demo showing how to run models from hugging face.

- lmsys/fastchat-t5-3b-v1.0
- nvidia/GPT-2B-001
- google/flan-ul2

Using 4090 and A100.

Like I know it's possible but the demo you have for using your own model does not work, and it should be a big friendly button notebook that I run end to end in the official docker container to get those models running.
GPT-2B-001_bf16_tp1 gibberish on 4090
State: closed | Pull request: no
Labels: name: bug

name: stale

using the image `nvcr.io/nvidia/nemo:23.02`
4090
start server with
`python /workspace/nemo/examples/nlp/language_modeling/megatron_gpt_eval.py gpt_model_file=GPT-2B-001_bf16_tp1.nemo trainer.precision=bf16 server=True tensor_model_parallel_size=1 trainer.devices=1`
run unmodified sample code
```python
import json
import requests

port_num = 5555
headers = {"Content-Type": "application/json"}

def request_data(data):
    resp = requests.put('http://localhost:{}/generate'.format(port_num),
                        data=json.dumps(data),
                        headers=headers)
    sentences = resp.json()['sentences']
    return sentences


data = {
    "sentences": ["Tell me an interesting fact about space travel."]*1,
    "tokens_to_generate": 50,
    "temperature": 1.0,
    "add_BOS": True,
    "top_k": 0,
    "top_p": 0.9,
    "greedy": False,
    "all_probs": False,
    "repetition_penalty": 1.2,
    "min_tokens_to_generate": 2,
}

sentences = request_data(data)
print(sentences[0])
```
output (gibberish) 
> Tell me an interesting fact about space travel. My version of health. A method of therapy for gonorrhoea by a man. A man presented with this complaint of "despair for me for a man for a man with gonorrhea. I presented to a man with gonoragia. My man
Not doing CastToFloat by default
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Removing casts around norms from the set of default pre-export replacements and moving them to a separate function.
Latest TRT should not need that anymore - also the casts may prevent some fusion.
That function may still be used from Nemo2Riva explicitly should it be needed on case-by-case basis. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
More streaming conformer export fixes
State: closed | Pull request: yes
Labels: name: core

name: ASR

# What does this PR do ?

Bug fixes for cache aware / conformer export

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

@borisfom
@VahidooX 

# Additional Information
* Related to # (issue)