renwei2024/nemo-github-issues huggingface dataset:
[TTS] fixed wrong pronunciations.
State: closed | Pull request: yes

Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Add] SLURP models and examples
State: closed | Pull request: yes

The PR adds the new SOTA model we have for intent classification and slot filling for spoken language understanding.

- [x] Copy transformer modules from `nemo.collections.nlp.modules.common` to `nemo.collections.asr.modules`
- [x] Add models, loss and utils to nemo.collection.asr
- [x] Add example scripts for data downloading and processing 
- [x] Add example scripts for training and evaluation 
- [x] Refactor model to be independent of EncDecCTCBPE
- [x] Refactor to use built-in SmoothedCrossEntropyLoss
- [x] Write README for example 
- [x] Upload model checkpoints to NGC and create model cards
- [x] Update Docs and list_available_models()
- [ ] Create tutorial. // Plan to add in later PR

The NGC model card is [here](https://ngc.nvidia.com/models/nvstaging:nemo:slu_conformer_transformer_large_slurp)
The HF model card is [here](https://huggingface.co/nvidia/slu_conformer_transformer_large_slurp)
update readme
State: closed | Pull request: yes

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

# What does this PR do ?

Update README docs

**Collection**: [Speaker Tasks]

# Changelog 
- update docs on launching fine-tune script

# Usage
-

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation
Fix apex for r1.11
State: closed | Pull request: yes

Added installation of correct apex version.

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add support for RNNT Char/Word Timestamp Calculation
State: closed | Pull request: yes

# What does this PR do ?

Adds support for RNNT models to compute character level and word level timestamps using similar steps to CTC models.

**Collection**: [ASR]

# Changelog 
- Adds char and word level timestamp calculation for character and subword models

# Usage

```python

# Copyright (c) 2022, NVIDIA CORPORATION.  All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import pytorch_lightning as pl
from omegaconf import OmegaConf, open_dict

from nemo.collections.asr.models import ASRModel


def main():

    trainer = pl.Trainer(gpus=[0], precision=32, logger=False)

    # Char model
    # path = "Path to char model"

    # Subword model
    path = "Path to subword model"

    asr_model = ASRModel.restore_from(path, trainer=trainer, map_location='cpu')  # type: EncDecRNNTModel

    asr_model.eval()
    asr_model.freeze()

    asr_model.preprocessor.featurizer.dither = 0.0
    asr_model.preprocessor.featurizer.pad_to = 0

    decoding_cfg = OmegaConf.create({})

    with open_dict(decoding_cfg):
        decoding_cfg.strategy = "greedy_batch"
        decoding_cfg.preserve_alignments = True
        decoding_cfg.compute_timestamps = True
        decoding_cfg.rnnt_timestamp_type = "all"

    asr_model.change_decoding_strategy(decoding_cfg)

    hyp, _ = asr_model.transcribe(['/test-other-processed/367-130732-0003.wav'],
                   batch_size=1, return_hypotheses=True)

    print("Evaluating ---")
    print()
    print("Transcription: ", hyp[0].text)
    print()
    print("Total number of timesteps :", hyp[0].length)
    print()
    print("Character/Subword Time-Stamps :", hyp[0].timestep['char'])
    print()
    print("Word Time-Stamps :", hyp[0].timestep['word'])
    print()


if __name__ == '__main__':
    main()

```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [x] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Import Error 
State: closed | Pull request: no
Labels: name: bug

**Unable to import  ASR model from Nemo.Collections **


**import nemo.collections.asr as nemo_asr**



**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

** Error Message 
Error

ModuleNotFoundError Traceback (most recent call last)
in ()
3 # NeMo's ASR collection - this collections contains complete ASR models and
4 # building blocks (modules) for ASR
----> 5 import nemo.collections.asr as nemo_asr

17 frames
[/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/apply_func.py](https://localhost:8080/#) in ()
27 if _TORCHTEXT_AVAILABLE:
28 if _compare_version("torchtext", operator.ge, "0.9.0"):
---> 29 from torchtext.legacy.data import Batch
30 else:
31 from torchtext.data import Batch

ModuleNotFoundError: No module named 'torchtext.legacy'
**
 - Environment location: Collab
 - Method of NeMo install: pip install 
 
**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- PyTorch version - 1.12
- Python version - 3.7

**Additional context**

Add any other context about the problem here.
Example: GPU model
Megatron Encoder Decoder models with RPE and PP > 2
State: closed | Pull request: yes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

# What does this PR do ?

Adds support for Enc-Dec models with RPE and PP > 2

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* model.encoder.position_embedding_type="relative"
* model.decoder.position_embedding_type="relative"
* model.pipeline_model_parallel_size=4
* model.pipeline_model_parallel_split_rank=2

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Improve mAES algorithm with patches
State: closed | Pull request: yes

# What does this PR do ?

Adds the fixes from https://github.com/espnet/espnet/pull/4312 for mAES RNNT beam search.

**Collection**: [ASR]

# Changelog 
- Add deduplication checks
- Improve speed of mAES beam search

# Usage

```python

# Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import pytorch_lightning as pl
from omegaconf import OmegaConf, open_dict

from nemo.collections.asr.models import EncDecRNNTBPEModel

def main():

    trainer = pl.Trainer(gpus=[1], precision=16, logger=False)

    # Model
    # path = 'some model path'

    asr_model = EncDecRNNTBPEModel.restore_from(path, trainer=trainer, map_location='cpu')

    asr_model.eval()
    asr_model.freeze()

    asr_model.preprocessor.featurizer.dither = 0.0
    asr_model.preprocessor.featurizer.pad_to = 0

    cfg = asr_model.cfg

    test_cfg = cfg.test_ds
    test_cfg.manifest_filepath = "manifests/test_other.json"
    test_cfg.batch_size = 256

    asr_model.setup_multiple_test_data(test_cfg)

    decoding_cfg = cfg.decoding

    with open_dict(decoding_cfg):
        decoding_cfg.strategy = "maes"
        decoding_cfg.fused_batch_size = -1
        decoding_cfg.beam.beam_size = 4
        decoding_cfg.beam.tsd_max_sym_exp = 10
        decoding_cfg.beam.alsd_max_target_len = 2.5
        decoding_cfg.beam.maes_num_steps = 2
        decoding_cfg.beam.maes_prefix_alpha = 1
        decoding_cfg.beam.maes_expansion_beta = 2
        decoding_cfg.beam.maes_expansion_gamma = 2.3

    asr_model.change_decoding_strategy(decoding_cfg)

    if asr_model.prepare_test(trainer):
        trainer.test(asr_model)


if __name__ == '__main__':
    main()

```
 
# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [x] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Confidence score for each word in asr model
State: closed | Pull request: no
Labels: name: feature

name: stale

Hi! I am trying `nvidia/stt_en_conformer_transducer_xlarge` for my ASR system and the performance is impressive. I noticed that there is an option in transcribe function that can support lm_score in output. I wonder if it is possible to generate confidence score for each word or where should I modify the code so that it can show the confidence score for each word. Thanks in advance,
Pin lightning version to be < 1.7.0
State: closed | Pull request: yes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

# What does this PR do ?

Pins PTL version to be less than 1.7.0

**Collection**: All

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add sentencepiece legacy arg to megatron tokenizer configs
State: closed | Pull request: yes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

# What does this PR do ?

Makes the sentencepiece legacy arg configurable. This fixes GPT training with sentencepiece.

**Collection**: NLP

# Changelog 
- Makes the sentencepiece legacy arg configurable.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Testing a pre-trained model
State: closed | Pull request: no

Hello,

I am trying to test a pre-trained conformer transducer, but the reported WER is around 100 %, whereas it is much lower when I use the `transcribe()` method of the model and compute the WER myself. What am I doing wrong?

Script:

```python
import yaml
from nemo.collections.asr.models.rnnt_bpe_models import EncDecRNNTBPEModel
import pytorch_lightning as pl

trainer = pl.Trainer(devices=1, accelerator="gpu", max_epochs=50)
config = yaml.load(open("./config/config.yaml"), Loader=yaml.FullLoader)
ct = EncDecRNNTBPEModel.from_pretrained("stt_de_conformer_transducer_large")

ct.setup_test_data(config["data"]["test_ds"])
trainer.test(ct, dataloaders=ct.test_dataloader())
```

Config file:
```yaml
data:
    test_ds:
    manifest_filepath: data/train_split/debug.txt
    sample_rate: 48000
    batch_size: 1
    shuffle: false
    num_workers: 0
    pin_memory: true
    use_start_end_token: false
```
Sentence Piece tokenizer from script "process_asr_text_tokenizer"
State: closed | Pull request: no
Labels: name: bug

name: stale

I was finetuning  pretrained citrinet 512 model (Encoder Freeze) for filipino language where i trained a custom tokenizer with vocab size - 1024 and replaced the pretrained citrinet tokenizer section with my custom tokenizer.

`data_dir = "."
!python3 ./scripts/process_asr_text_tokenizer.py  
  --manifest="{data_dir}/Filipino/train_manifest.json"
  --data_root="{data_dir}/complete_tg/tokenizers/tokenizer_1/" 
  --vocab_size=1024 
  --tokenizer="spe" 
  --spe_type="bpe" 
  --spe_character_coverage=1.0 
  --log` 

Sample vocab -: 

▁
##a
##i
##o
##n
##t
##c
##e
##l
##r
##s
##p
##f
##g
##k
##m
##v


This script gives tokenizer.model and vocabulary, In vocab a special character "▁" is present. This is not a normal underscore which is "_".

Can someone explain me why i am getting this special token as due to this special character i am unable to deploy my asr in riva speech with flashlight decoder.
Error got during deployment with flashlight decoder -:

`I0712 13:32:15.565105 94 ctc-decoder-library.cc:23] TRITONBACKEND_ModelInstanceInitialize: nemo_asr_riva_pipeline_filipino-ctc-decoder-cpu-streaming_0 (device 0)
terminate called after throwing an instance of 'std::runtime_error'
  what():  [LoadWords] Invalid line: ▁
  > Riva waiting for Triton server to load all models...retrying in 1 second
/opt/riva/bin/start-riva: line 4:    94 Aborted                 (core dumped) tritonserver --log-verbose=0 --strict-model-config=true $model_repos --cuda-memory-pool-byte-size=0:1000000000`
Fix installation issues in virtual environment
State: closed | Pull request: yes

Fixes installation issues in a normal python virtual environment when conda is also present on the same system. Also remove double negation.

Signed-off-by: Anand Joseph <anajoseph@nvidia.com>

# What does this PR do ?
Fixes installation issues in a normal python virtual environment when conda is also present on the same system.

# Changelog 
- Check if "$CONDA_PREFIX" is set rather than check if "conda" is present.
- Replace double negation "! -z" with "-n" when checking for NVIDIA_PYTORCH_VERSION
Fixes installation issues in a normal python virtual environment when conda is also present on the same system.
State: closed | Pull request: yes

# What does this PR do ?

Fixes installation issues in a normal python virtual environment when conda is also present on the same system.

# Changelog 
- Check if "$CONDA_PREFIX" is set rather than check if "conda" is present. 
- Replace double negation "! -z" with "-n" when checking for NVIDIA_PYTORCH_VERSION
Mutiscale Diarization Decoder (MSDD) model and module files
State: closed | Pull request: yes

Signed-off-by: Taejin Park <tango4j@gmail.com>

# What does this PR do ?

Add multiscale diarization decoder model module, and yaml files. 
Additionally, yaml files are also uploaded.

- msdd_models.py

Three classes are included:
(1) ClusterEmbedding
This class is for taking care of cluster-average embeddings by launching clustering diarizer class.
The methods in this class read embedding vectors and clustering output that are written from clustering diarizer class.

(2) EncDecDiarLabelModel
MSDD model class.

(3) OverlapAwareDiarizer
A class that launches Clustering Diarizer and run MSDD infer then evaluate it.
It involves integration of pairwise inference from MSDD and split-infer that breaks down the input to
short sequences.

- msdd_diarizer.py
MSDD Neural model class. 

- speaker_utils.py
audio_rttm_map() function is modified to deal with the uniq_id with duration information.

-audio_to_diar_label.py, collections.py

These are couple of very small points where it needs to be changed like: rttm_files -> audio_files 
(We need to extract uniq_id when rtttm is not provided for evaluation)

- 4 x yaml files
msdd_5scl_15_05_50Povl_infer.yaml -> for inference
msdd_6scl_30_05_50Povl_infer.yaml -> for inference
msdd_5scl_15_05_50Povl_256x3x32x2.yaml  -> for training 
msdd_6scl_30_05_50Povl_256x3x32x2.yaml  -> for training 

The name of these files are up for debate.

* Tutorial and script for training data generation is excluded since it makes PR too big.


**Collection**: [Note which collection this PR will affect]
ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* Use the following command to run MSDD.
```python
HYDRA_FULL_ERROR=1 python $BASEPATH/multiscale_diar_decoder_infer.py
 --config-path='conf' --config-name='msdd_5scl_15_05_50Povl_infer.yaml' \
    diarizer.manifest_filepath=<TEST_MANIFEST> \
    diarizer.out_dir=<TEST_EMB_DIR> \
    diarizer.msdd_model.model_path=<MSDD_MODEL_PATH>\
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests? - Added unitest for msdd_model. Will add unit-test for audio_to_diar_label once micro diarization dataset is ready.
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
adding lang id to shard
State: closed | Pull request: yes

# What does this PR do ?

Adds / propagates language id when creating a tarred [multilingual] dataset from non-tarred 

**Collection**: [Note which collection this PR will affect]
ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.
One line change in the convert_to_tarred_audio_dataset.py script

# Usage
* You can potentially add a usage example below
No change to user interface. 

```python
# Add a code snippet demonstrating how to use this 
```
python convert_to_tarred_audio_dataset.py <params>

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add Github Action for auto webpage build
State: closed | Pull request: yes

# What does this PR do ?

Adds a github action to build the website for nvidia.github.io/NeMo/ on every push.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation
Update NeMo github.io website
State: closed | Pull request: yes

# What does this PR do ?

Modernize and update NeMo Github.io website.

**Collection**: [Docs]

# Changelog 
- Replace previous redirect with mkdocs for material based website

# Usage
- To build the website locally to see the contents, run `bash build_docs.sh` from the `gh-pages` branch.
- In order to add posts, follow the steps in the template inside `docs/blogs/template.md`

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation
Add MuTransfer Capablity to RETRO model pretraining
State: closed | Pull request: yes

# What does this PR do ?

It adds the muTransfer capablity to do RETRO model pretraining.
1. Included the mutransfer codes from MS github.
2. Add shape calculation script.
3. Add the mutransfer pretraining script.
4. Make sure it uses fused_softmax padding mode because of the customized masks
5. Include a CI test to make sure the model training performance matches. 

It also fix a bug that PTL grad sync is not called.

To use muTransfer for other models (GPT, T5, BERT etc), you can follow the same pattern.
1. register muadam or muadamw optimizer.
2. swap to use MuReadout to to convert model_d to logistics
3. Use the similar parameters init method in the mutransfer_pretrain file.
Mixer-TTS-X Colab Training example
State: closed | Pull request: no
Labels: name: feature

Hello.

Is any script for training Mixer-TTS-X?

I've used it in the model select inference example, and I really like how it sounds. Better than FastPitch and MixerTTS, in my opinion. Also in the paper shows a higher MOS.

I think it would be extremely useful.

Thanks,
Update pynini library to fix MacOS build
State: closed | Pull request: yes

# What does this PR do ?

Update pynini from `2.1.4` to `2.1.5` to fix MacOS build

**Collection**: [Text Processing]

# Changelog 
- Update pynini from `2.1.4` to `2.1.5`

# Reason

Pynini 2.1.4 can't be installed on MacOS (Intel processor, Conda, python 3.8). Using version `2.1.5` fixes the issue.
```
Building wheels for collected packages: pynini
  Building wheel for pynini (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [55 lines of output]
...
      building '_pywrapfst' extension
      creating build/temp.macosx-10.9-x86_64-3.8
      creating build/temp.macosx-10.9-x86_64-3.8/extensions
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /<user_dir>/anaconda3/envs/ml38/include -fPIC -O2 -isystem /<user_dir>/anaconda3/envs/ml38/include -I/<user_dir>/anaconda3/envs/ml38/include/python3.8 -c extensions/_pywrapfst.cpp -o build/temp.macosx-10.9-x86_64-3.8/extensions/_pywrapfst.o -std=c++17 -Wno-register -Wno-deprecated-declarations -Wno-unused-function -Wno-unused-local-typedefs -funsigned-char -stdlib=libc++ -mmacosx-version-min=10.7
      extensions/_pywrapfst.cpp:753:10: fatal error: 'fst/types.h' file not found
      #include <fst/types.h>
               ^~~~~~~~~~~~~
      1 error generated.
      error: command 'clang' failed with exit status 1
      [end of output]
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?
@yzhang123 

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix/need different cache dirs for different datasets
State: closed | Pull request: yes
Labels: name: bug

# What does this PR do ?

If several `ds_item`s are used for validation or testing and `cache_dir` is set, then pickled features for different features overwrite each other.

**Collection**: nlp

# Changelog 
- if `cache_dir` is set, then an absolute path to dataset is appended to `cache_dir` to avoid collisions
- Both text and labels file names are incorporated into pickled features file name. This is done to avoid collisions in rare cases when there are different labels files for 1 input text file.
- I also changed token classification pickled features name

# Additional Information
* Related to # (issue)
[chinese text normalization] PR for chinese TN part in text_normalization 
State: closed | Pull request: yes

# What does this PR do ?

PR for chinese TN part in text_normalization 

**Collection**: NeMo/norm_text_processing/text_normalization

Chinese Text Normalization
1. How To Use
python normalize.py --lang "zh" --text "text to be normalized"
2. TN Pipeline
There are 3 components in TN pipeline:

pre-processing (before tagger)
non-standard word normalization
post-processing (after verbalizer)
2.1 Pre-Processing
Char Width Conversion (全角 -> 半角)
苹果ＣＥＯ宣布发布新ＩＰＨＯＮＥ -> 苹果CEO宣布发布新IPHONE
他说：“我们已经吃过了！”。 -> 他说:"我们已经吃过了!".
covers English letters, digits, punctuations and some symbols
the complete mapping table data/char/fullwidth_to_halfwidth.tsv
Blacklist (Removal)
Sometime you may want to remove certain things like interjections/fillers "啊", "呃" etc

呃这个呃啊我不知道 -> 这个我不知道
customizable via data/blacklist/interjections.tsv
2.2 Non-Standard-Words(NSW) normalization
Numbers
  共465篇，约315万字 -> 共四百六十五篇，约三百一十五万字
  共计6.42万人 -> 共计六点四二万人
  同比升高0.6个百分点 -> 同比升高零点六个百分点
Fraction
  总量的1/5以上 -> 总量的五分之一以上
  相当于头发丝的1/16 -> 相当于头发丝的十六分之一
  3/2是一个假分数 -> 二分之三是一个假分数
Percentage
  同比增长6.3% -> 同比增长百分之六点三
  增幅0.4% -> 增幅百分之零点四
Date
  2002/01/28 -> 二零零二年一月二十八日
  2002-01-28 -> 二零零二年一月二十八日
  2002.01.28 -> 二零零二年一月二十八日
  2002/01 -> 二零零二年一月
Time
  8月16号12:00之前 -> 八月十六号十二点之前
  我是5:02开始的 -> 我是五点零二分开始的
  于5:35:36发射 -> 于五点三十五分三十六秒发射
  8:00 a.m. 准时开会 -> 八点 a m 准时开会
Math
  比分定格在78:96 -> 比分定格在七十八比九十六
  计算-2的绝对值是2 -> 计算负二的绝对值是二
  ±2的平方都是4 -> 正负二的平方都是四
Money
  价格是￥13.5 -> 价格是十三点五元
  价格是$13.5 -> 价格是十三点五美元
  价格是A$13.5 -> 价格是十三点五澳元
  价格是HKD13.5 -> 价格是十三点五港元
Measure
  重达25kg -> 二十五千克
  最高气温38°C -> 最高气温三十八摄氏度
  实际面积120m² -> 实际面积一百二十平方米
  渲染速度10ms一帧 -> 渲染速度十毫秒一帧
Number series (phone, mobile numbers)
  可以打我手机13501234567 -> 可以打我手机一三五零一二三四五六七
  可以拨打12306来咨询 -> 可以拨打一二三零六来咨询
Erhua(儿化音) Removal
  这儿有只鸟儿 -> 这有只鸟
  这事儿好办 -> 这事好办
  我儿子喜欢这地儿 -> 我儿子喜欢这地
erhua whitelist is customizable via data/erhua/whitelist.tsv
Whitelist(Replacement)
a set of user-defined hard mapping, i.e. exact-string matching & replacement

CEO -> C E O
GPU -> G P U
O2O -> O to O
B2B -> B to B
customizable via data/whitelist/default.tsv
2.3 Post-Processing
Punctuation Removal
If enabled, punctuations are removed.

Uppercase or Lowercase Conversion
If enabled, English letters are converted to uppercases / lowercases

Out-Of-Vocabulary(OOV) Tagger
If enabled, OOV chars are tagged with <oov> and </oov>, e.g.:

我们안녕 -> 我们<oov>안</oov><oov>녕</oov>
雪の花 -> 雪<oov>の</oov>花
default charset (national standard) [通用规范汉字表](https://zh.wikipedia.org/wiki/%E9%80%9A%E7%94%A8%E8%A7%84%E8%8C%83%E6%B1%89%E5%AD%97%E8%A1%A8)
you can extend charset via data/char/charset_extension.tsv
3. Credits
Author: Zhenxiang MA @ Tsinghua University

Advisors: [SpeechColab](https://github.com/SpeechColab) organization

The authors of this work would like to thank:

The authors of foundational libraries like OpenFst & Pynini
NeMo team and NeMo open-source community
[WIP] Multilingual punctuation restoration, true casing, and sentence boundary detection
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

This is a work-in-progress for a model and data set that performs multilingual punctuation restoration, true casing, and sentence boundary detection. See `Usage` below for a demo of what this PR does.

Some key features:
* Resolves most of the issues mentioned in #3819
* Emphasis on multi-lingual; data set handles the nuances of each language.
* Language-agnostic inference. Inputs do not need language labels, and batches can contain multiple languages. Text is processed in its native script (e.g., Chinese is processed without spaces, for Spanish we can predict inverted punctuation, etc.).
* User can train with plain-text files as input; all "art work" is done by the pre-processor at training time

**Collection**: NLP

# Changelog 
No changes to existing NeMo code, only additions.

# Usage
See current example model on the [HF hub](https://huggingface.co/1-800-BAD-CODE/punct_cap_seg_22lang1pass_xlmr). 

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to discussion #3819
Can't redefine method: __filter_banks_getter on class: __torch__.nemo.collections.asr.parts.preprocessing.features.FilterbankFeatures
State: closed | Pull request: no
Labels: name: bug

name: stale

**Steps/Code to reproduce bug**

I'm trying to repeat the model deployment [instruction ](https://github.com/pytorch/android-demo-app/blob/master/SpeechRecognition/create_wav2vec2.py) on Android.

```python
from torch.utils.mobile_optimizer import optimize_for_mobile

pretrained_model = nemo_asr.models.EncDecCTCModel.load_from_checkpoint(
    "my_model.ckpt",
    map_location="cpu"
)

pretrained_model.eval()
pretrained_model.encoder.freeze()
pretrained_model.decoder.freeze()

quantized_model = torch.quantization.quantize_dynamic(
        pretrained_model, qconfig_spec={torch.nn.Linear}, dtype=torch.qint8, inplace=True
)

scripted_model = torch.jit.script(quantized_model)
# optimized_model = optimize_for_mobile(scripted_model)
```
But torch.jit.script returns an error:
> RuntimeError: Can't redefine method: __filter_banks_getter on class: __torch__.nemo.collections.asr.parts.preprocessing.features.FilterbankFeatures (of Python compilation unit at: ...)

Can you advise how to fix this?
NeMo Megatron Doc updates1
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
WordTokenizer `build_vocab` constructs character-based vocab instead of words
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

`WordTokenizer` should construct a vocab out of the words in a string, e.g., ['A word here'] -> ['A','word','here']. Instead, in constructs a character-based vocabulary due to its re-use of the `CharTokenizer` code. Additionally, if a properly constructed vocab is attempted to be loaded, it fails due to the error checking that is again inherited from `CharTokenizer`.

**Steps/Code to reproduce bug**

For constructing vocab:
```python
>>> from nemo.collections.common.tokenizers import WordTokenizer
>>> WordTokenizer.build_vocab('/tmp/vocab-georgea0729.txt', 'A word goes here')
>>> with open('/tmp/vocab-georgea0729.txt') as fh:
...     print(fh.readlines())
... 
['{}\n', "' '\n", "'e'\n", "'o'\n", "'r'\n", "'A'\n", "'w'\n", "'d'\n", "'g'\n", "'s'\n", "'h'\n"]
>>> tokenizer = WordTokenizer('/tmp/vocab-georgea0729.txt')
>>> tokenizer.text_to_tokens('A word goes here')
['A', None, None, None]
>>> tokenizer.text_to_tokens('A e w o h')
['A', 'e', 'w', 'o', 'h']
```

For Loading vocab:
```python
>>> with open('/tmp/georgea-manual-vocab0729.txt', 'w') as fh:
...     fh.write("{}\n'A'\n'word'\n'goes'\n'here'\n")
... 
28
>>> tokenizer = WordTokenizer('/tmp/georgea-manual-vocab0729.txt')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/common/tokenizers/word_tokenizer.py", line 49, in __init__
    super().__init__(
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/common/tokenizers/char_tokenizer.py", line 134, in __init__
    self.check_token_from_file(token, vocab_file, i)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/common/tokenizers/char_tokenizer.py", line 241, in check_token_from_file
    raise ValueError(
ValueError: Each line in vocabulary have to be a Python string literal containing 1 character. Encountered 'word' on line 1 in file /tmp/georgea-manual-vocab0729.txt.
>>> 
```


**Expected behavior**

```python
>>> from nemo.collections.common.tokenizers import WordTokenizer
>>> WordTokenizer.build_vocab('/tmp/vocab-georgea0729.txt', 'A word goes here')
>>> with open('/tmp/vocab-georgea0729.txt') as fh:
...     print(fh.readlines())
... 
["{}\n", "'A'\n", "'word'\n", "'goes'\n", "'here'\n"]
>>> tokenizer = WordTokenizer('/tmp/vocab-georgea0729.txt')
>>> tokenizer.text_to_tokens('A word goes here')
['A', 'word', 'goes', 'here']
>>> tokenizer.text_to_tokens('A e w o h')
['A', None, None, None, None]
```
FIX special tokens in vocab to arguments of constructor
State: closed | Pull request: yes

# What does this PR do ?

This PR fixes #4627 by setting the initial vocab of `RegExTokenizer` to the supplied special tokens.

**Collection**: Common

# Changelog 
- Sets the initial vocab of `RegExTokenier` to the supplied special tokens.


# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?
@michalivne @titu1994 

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix itn pt time
State: closed | Pull request: yes

# What does this PR do ?

Fix ITN rules for Portuguese (time).

Fix the case when the text is "às três horas" (at three o'clock) to force to have "horas" (like o'clock) instead of "às três" (at three).

Collection: [Text Processing]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Fix HF check for model card info
State: closed | Pull request: yes

Signed-off-by: smajumdar <smajumdar@nvidia.com>

Fix HF test where model card might not contain info

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Core]


# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
RegexTokenizer: Default token arguments are not mapped to vocabulary index
State: closed | Pull request: no

The ID's of the default special tokens in RegexTokenizer  are set here:

https://github.com/NVIDIA/NeMo/blob/72d78d832c6332ccc7f44811fb777eb7044df1d8/nemo/collections/common/tokenizers/regex_tokenizer.py#L75-L82

However, the constructor, allows custom special tokens:

https://github.com/NVIDIA/NeMo/blob/72d78d832c6332ccc7f44811fb777eb7044df1d8/nemo/collections/common/tokenizers/regex_tokenizer.py#L42-L51


So this could cause issues (specifically extra unknowns) when text is converted to tokens, then to ids, then back.
Update megatron t5 interface to dialogue
State: closed | Pull request: yes

# What does this PR do ?

Add inference support for Megatron T5 in Dialogue module

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Tiny VAD refactoring for postprocessing
State: closed | Pull request: yes

# What does this PR do ?

Small fix for ignoring overlap median filtering and use shift > 0.01 in generating speech segments efficiently.

**Collection**: ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)