renwei2024/nemo-github-issues huggingface dataset:
[Text Processing] G2P for OOV and heteronyms
State: closed | Pull request: yes

Signed-off-by: ekmb <ebakhturina@nvidia.com>

# What does this PR do ?

Adds G2P models for OOV and heteronyms.

**Collection**: [Text Processing]

# Changelog 
G2P models for OOV and heteronyms.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix ITN pt
State: closed | Pull request: yes

# What does this PR do ?

Fix ITN rules for Portuguese (date and time).
In addiction, tests for these rules were added.

Collection: [Text Processing]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
[NLP/PC] add support for capitalization classes lower (L), upper (U), capitalize (C)
State: closed | Pull request: yes
Labels: name: stale

Signed-off-by: Iztok Lebar Bajec <ilb@fri.uni-lj.si>

# What does this PR do ?

While waiting for #3819 to be finished, upgrade of current punctuation capitalization model with support for lowercasing, uppercasing, and capitalisation. 

**Collection**: NLP/PC

# Changelog 
Modified the capitalisation decision from pure capitalisation, as soon as `capit_label` differs from noop (`O`), to one where the operation is based on three classes: lowercase (`L`), uppercase (`U`) and capitalize (`C`).

Warning: Due to the `class_label` previously used for capitalisation (`U`) and the way the decision was implemented prior to this PR, this change becomes a breaking change. Models trained prior to this PR will result in returning `all caps` instead of `capitalising` selected words. A retrain will, however, provide additional functionality.

# Usage
* You can potentially add a usage example below

**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

## Who can review?
@PeganovAnton
Support listing Hugging Face model info
State: closed | Pull request: yes

Signed-off-by: smajumdar <smajumdar@nvidia.com>

# What does this PR do ?

Add new apis to filter and obtain NeMo hugging face models via code.

**Collection**: [Core]

# Changelog 
- Adds two methods `get_hf_model_filter()` and `search_huggingface_models(model_filter=None)` to ModelPT.
- First method returns a basic ModelFilter object, can be overridden by the user. Used as default to next method if nothing is provided.
- Add support for metadata attributes to the filter to modify the result list of ModelInfo.
- `list_available_models_on_hf()` lists the nemo models that satisfy the filter.

# Usage

```python
from nemo.core import ModelPT

# You can replace <DomainSubclass> with any subclass of ModelPT.
# Get default ModelFilter
filt = <DomainSubclass>.get_hf_model_filter()

# Make any modifications to the filter as necessary
filt.language = [...]
filt.task = ...
filt.tags = [...]

# Add any metadata to the filter as needed
filt.limit_results = 5

# Obtain model info
model_infos = <DomainSubclass>.search_huggingface_models(model_filter=filt)

# Browse through cards and select an appropriate one
card = model_infos[0]

# Restore model using `modelId` of the card.
model = ModelPT.from_pretrained(card.modelId)

```


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Cuda Out of memory 
State: closed | Pull request: no
Labels: name: bug

name: stale

Hi,

I am trying to train a conformer-transducer model on LibriSpeech but I am getting out of memory error. I am using the `conformer_transducer_bpe.yaml` config to initialize a `EncDecRNNTBPEModel` class (also tried the `char` version) with `batch_size=8` on a `NVIDIA K80 GPU` with 12GB of memory and this is the error I am getting: 


 `...

  File "/home/ec2-user/anaconda3/envs/nemo/lib/python3.8/site-packages/nemo/collections/asr/modules/rnnt.py", line 992, in joint   
res = self.joint_net(inp)  # [B, T, U, V + 1]
  File "/home/ec2-user/anaconda3/envs/nemo/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/nemo/lib/python3.8/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/home/ec2-user/anaconda3/envs/nemo/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/nemo/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA out of memory. Tried to allocate 2.16 GiB (GPU 0; 11.17 GiB total capacity; 8.88 GiB already allocated; 1.76 GiB free; 9.01 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF`

I managed to train a `EncDecCTCModelBPE` model with much larger number of parameters on the same dataset with a larger batch size but I  am not sure why I am getting memory error for a smaller model.
[TTS] Fix off-by-1 bug in Beta Binomial Prior
State: closed | Pull request: yes

# What does this PR do ?

Fix a bug in the prior calculation used in the TTS aligner

**Collection**: [TTS]

# Changelog 
- Fix off-by-1 bug in prior calculation

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

# Additional Information

The prior here is supposed to help the TTS model align by teaching it to place more weight on phonemes along the diagonal, with it always starting on the first phoneme and ending on the last phoneme.

The current logic produces a prior that looks like this, which awkwardly ends earlier than expected with maximum weight on the last phoneme before the end of the sequence:

![image](https://user-images.githubusercontent.com/9942053/181093773-48065280-478d-4d26-9fdd-4ea1d87b8652.png)

After this change, the prior looks like this:

![image](https://user-images.githubusercontent.com/9942053/181093845-bfff79b2-23d6-4cce-8fae-3d0f39f423c5.png)

I think the intuition is that we model the attention as a binomial distribution (eg. coin flips with head being success, tail failure) where each success results in the attention moving on to the next phoneme. To reach the end of a sequence of N phonemes, you only need (N - 1) successes.
LossMisconfigurationError
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Add a LossMisconfigurationException that provides more context on how to fix certain issues that can occur during training.

**Collection**: ASR

# Changelog 

- Add a `LossMisconfigurationException` for `ContrastiveLoss`

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X] New Feature
- [ ] Bugfix
- [X] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to #4587
Improve normalize_batch ValueError message
State: closed | Pull request: yes

# What does this PR do ?

Improve `normalize_batch` error message with suggestions on how to fix.
Sometimes audio files with zero/small duration sneak by as files with a non-zero duration field, causing this error.

**Collection**: ASR

# Changelog 
* Improve `normalize_batch` `ValueError` message with suggestions on how to fix

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Add option to save VAD artifacts
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

* Adds utility option to save input and output tensors for VAD model inference useful for testing and comparison in isolation from pre/post processing
* Adds support to disable median filtering for VAD post processing, (`None`, `False`, `''`)

**Collection**: ASR

# Changelog 
- Add VAD options to save artifacts and disable median filtering

# Usage
examples/asr/conf/vad/vad_inference_postprocessing.yaml

```yaml
vad:
    parameters:
      smoothing: None
save_artifacts: False
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TN] bug fix "hundred" in Audio-based, added method so split text in sentences
State: closed | Pull request: yes

# What does this PR do ?

Bug fix for Audio-based TN when "hundred" is optional, new method to split text into sentences before TN

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ X] New Feature
- [ X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[ASR] Adding Speech separation models to NeMo
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?
Adds speech separation model to NeMo.

**Collection**: ASR

# Changelog 
- High level script for speech separation `examples/speech_tasks/separation/speech_separation.py`
- Sample config file `examples/speech_tasks/separation/conf/sep_transformer.yaml`
- Separation model `nemo/collections/asr/models/ss_model.py`
- Pre-processor `AudioToFeaturesConvPreprocessor` in `nemo/collections/asr/modules/audio_preprocessing.py`
- Decoder `nemo/collections/asr/modules/ss_decoder.py`
- Loss `nemo/collections/asr/losses/ss_losses/si_snr.py`
- Dataset `nemo/collections/asr/data/audio_to_audio.py`
- Inference `EncDecSpeechSeparationModel.extract_sources()`  in `nemo/collections/asr/models/ss_model.py`


# Usage

```python
# load model
cfg = {'init_from_nemo_model': '<path_to_nemo_ckpt>'}
cfg = OmegaConf.create(cfg)
model_cfg = ASRModel.restore_from(restore_path=cfg.init_from_nemo_model, return_config=True)
sep_model = EncDecSpeechSeparationModel(cfg=model_cfg)
sep_model.maybe_init_from_pretrained_checkpoint(cfg, map_location='cpu')

# run inference
paths2audio_files=[<list_of_audio_mixtures_to_separate>]
sep_model.extract_sources(
    paths2audio_files = paths2audio_files,
    save_dir='<path_to_save_separated_sources>',
    orig_sr=16000,
    num_sources=2,
    batch_size=1,
    num_workers=1,
)
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Update Offline ASR with CTC Decoding
State: closed | Pull request: yes

# What does this PR do ?

Add support for subword time stamping using CTC decoding strategy.

**Collection**: [ASR]

# Changelog 
- Fix mistake in selection of word boundaries for subword models
- Update tutorial with subword time stamp computation.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Updating the default parameters in the example adapters config file
State: closed | Pull request: yes

# What does this PR do ?
Updates the default values of the warmup and the layer norm in the example adapter config file. It also adds the option to specify tags as well as offline logging of wandb.

**Collection**: asr

# Changelog
File modified: `examples/asr/conf/asr_adapters/asr_adaptation.yaml`
- Adapters now use pre-norm by default
- The default number of warmup steps is set to 10% of the training set
- Added two parameters: `offline` and `tags` within the `wandb_logger_kwargs` config to allow specifying offline logging and tags for each run respectively.

# Before your PR is “Ready for review”
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?

**PR Type**:
- [x] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven’t finished some of the above items you can still open “Draft” PR.

## Who can review?
Anyone in NeMo ASR.
Updating the default parameters in the example adapters config file
State: closed | Pull request: yes

# What does this PR do ?

Updates the default values of the warmup and the layer norm in the example adapter config file. It also adds the option to specify tags as well as offline logging of wandb.

**Collection**: asr

# Changelog 
File modified: `examples/asr/conf/asr_adapters/asr_adaptation.yaml`

- Adapters now use pre-norm by default
- The default number of warmup steps is set to 10% of the training set
- Added two parameters: `offline` and `tags` within the `wandb_logger_kwargs` config to allow specifying offline logging and tags for each run respectively.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in NeMo ASR.
G2P Aligner
State: closed | Pull request: yes

# What does this PR do ?

Adds a script and a tutorial for using a trained RAD-TTS Aligner as a G2P disambiguation tool.

**Collection**: TTS

# Changelog 
- Created the `aligner_g2p.py` script for disambiguating heteronym pronunciations
- Added a new TTS Aligner example tutorial, which includes visualizations and demonstration of how heteronym disambiguation works

# Usage
For the disambiguation script:
```
python aligner_g2p.py \
    --model=<model_path> \
    --manifest=<manifest_path> \
    --out=<output_json_path> \
    --confidence=0.02 \
    --verbose
```
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation
r1.10.0 MegaMolBART Compatibility
State: closed | Pull request: yes

# What does this PR do ?

1. Fixes RegExTokenizer missing attributes.
2. Fixes broken Megatron encoder-decoder LM model ability to receive directly encoder hiddens.

```
    MegatronLMEncoderDecoderModel.forward(
        self,
        encoder_input_ids,
        decoder_input_ids,
        encoder_attn_mask,
        decoder_attn_mask,
        token_type_ids=None,
        lm_labels=None,
        # NEW API
        enc_output=None,
        enc_output_attn_mask=None,
        # NEW API
        output_enc_hidden_only=False,
        enc_input=None,
    ):
```

3. Added MegatronLMEncoderDecoderModel.encode method.
4. Improved decoding speed.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Label Smoothing in VocabParallelCrossEntropy
State: closed | Pull request: yes

# What does this PR do ?

Adds label smoothing for VocabParallelCrossEntropy

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Adds the label smoothing arg to VocabParallelCrossEntropy

# Usage
* `model.label_smoothing=0.1`

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Key Error when using conformer-transducer-bpe
State: closed | Pull request: no

Hi,
I am trying to train a conformer-transducer ASR model with BERT tokenizer. I am using  `conformer_transducer_bpe.yaml` as the base of my config with few changes to set the tokenizer. When running it, I get the following error:
```
    raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ConfigKeyError: Key 'num_classes' is not in struct
    full_key: decoder.num_classes
    object_type=dict
```
so I decided to set the decode.num_classes to -1 but then I get this error:
```
    raise InstantiationException(msg) from e
hydra.errors.InstantiationException: Error in call to target 'nemo.collections.asr.modules.rnnt.RNNTDecoder':
TypeError("__init__() got an unexpected keyword argument 'num_classes'")
```
Dividing generative question-answering CI tests
State: closed | Pull request: yes

Signed-off-by: Ameya Mahabaleshwarkar <ameyasm1154@gmail.com>

# What does this PR do ?

Split CI tests for generative question-answering into multiple stages to reduce memory usage

**Collection**: nlp

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Testing lengths output from decoder for ragged batching.
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Trying to return model output with lengths is not exporting correctly.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
T0 model and  dataset
State: closed | Pull request: yes

# What does this PR do ?

Adds the T0 Dataset and Model classes.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Adds the T0 dataset and model class files
- Repurposes the generic fine-tuning run script for T0

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Does non-English TTS training work properly now?
State: closed | Pull request: no

Hi all!

I've come across this [issue](https://github.com/NVIDIA/NeMo/issues/234) from 2020, which clearly shows that non-English TTS had some issues at that time. Has the state of things changed since then? 

I need to train Tacotron2 for Russian, but I face so many troubles in preprocessing stage. Mainly with text preprocessing. Unfortunately, I couldn't find any decent tutorials on how to train a model for a non-English dataset.

Can anyone help?
add kw asr models, add itn ru checkpoint (tagger-based)
State: closed | Pull request: yes

Signed-off-by: Alexandra Antonova <aleksandraa@nvidia.com>

# What does this PR do ?

Add two ASR models for Kinyarwanda (ctc and transducer large), add tagger-based ITN model for Russian

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
A new fused softmax kernel function
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?
Implemented the fused softmax GPU kernel that can be a drop-in replacement of the `apex`'s FusedScaleMaskSoftmax`. It addresses the following issues:
1. Removed the sequence length limitation. It works with tensor of any length.
2. Fix the `grad_input` modification issue https://github.com/NVIDIA/apex/issues/1430
3. Fix the problem when all keys are masked https://github.com/NVIDIA/apex/issues/1390

It uses the torch.cppextension to build the kernel dynamically like MegatronLM repo does. The kernels will be saved in the `/tmp` directory so it can be used in slurm with enroot clusters too.
It includes a unit test to cover a few cases.
The performance is comparable to the apex one. Here is a training step time graph when it is tested with training a RETRO model.
![image](https://user-images.githubusercontent.com/43824965/180662857-712f48e4-dc9b-40f7-aad5-13e82004a30b.png)
The blue curve is pytorch softmax. The bottom red curve is apex fused kernel and the middle red curve the fused kernel in this PR.
ASR for long audio files (streaming) with Beam Search/LM
State: closed | Pull request: no
Labels: name: feature

Hi
Following older answers for closed issues: https://github.com/NVIDIA/NeMo/issues/2307 and the tutorial: https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/Streaming_ASR.ipynb. 
Any updates on Nemo supporting LM with long audio (streaming)?
Add DALI char dataset support to SSL model
State: closed | Pull request: yes

# What does this PR do ?

Use DALI pipelines as part of `ssl_models`

**Collection**: ASR

# Changelog 

- Add DALI char dataset support to SSL model
  
**PR Type**:
- New Feature

Tested manually by running 1 validation step on a local dataset.

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information

* Related to #4591
SpeechEncDecSelfSupervisedModel does not support Dali datasets
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**

I noticed `SpeechEncDecSelfSupervisedModel`'s don't check if `use_dali` is set in their `_setup_dataloader_from_config()` method.

Is that intended? I noticed some performance improvements using Dali and the error traces it returns are much more helpful than the typical audio dataset ones.

**Describe the solution you'd like**

Just add 

```py
device = 'gpu' if torch.cuda.is_available() else 'cpu'
if config.get('use_dali', False):
    device_id = self.local_rank if device == 'gpu' else None
    dataset = audio_to_text_dataset.get_dali_bpe_dataset(
        config=config,
        tokenizer=self.tokenizer,
        shuffle=shuffle,
        device_id=device_id,
        global_rank=self.global_rank,
        world_size=self.world_size,
        preprocessor_cfg=self._cfg.preprocessor,
    )
    return dataset
```

to `_setup_dataloader_from_config` in `SpeechEncDecSelfSupervisedModel`.

I can open a PR if this if fine, but wanted to see if there was an explicit reason not to include it.
Contrastive Loss: Runtime error for certain audio files when reshaping out_masked_only
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

After a few steps when pretraining a `SpeechEncDecSelfSupervisedModel`, training fails with the following error

```sh
File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/ssl_models.py", line 468, in training_step
  loss_value, loss_val_dict = self.decoder_loss_step(
File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/ssl_models.py", line 450, in decoder_loss_step
  current_loss_value = current_loss(
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1129, in _call_impl
  return forward_call(*input, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/nemo/core/classes/common.py", line 963, in __call__
  outputs = wrapped(*args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/losses/ssl_losses/contrastive.py", line 187, in forward
  out_masked_only = out_masked_only.reshape(bs, -1, out_masked_only.shape[-1])
RuntimeError: shape '[16, -1, 128]' is invalid for input of size 547712
```

**Steps/Code to reproduce bug**

I used the [speech_pre_training.py](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/speech_pretraining/speech_pre_training.py) script with the default configuration for the conformer [here](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/ssl/conformer/conformer_ssl.yaml).

I only modified `train_ds.max_duration: 25.0`.
I also tried the config in the [Self_Supervised_Pre_Training.ipynb](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/Self_Supervised_Pre_Training.ipynb) notebook and I got the same error.

**Expected behavior**

Training should process normally and/or a more detailed explanation why the reshape failed and what parameters to change.

**Environment overview (please complete the following information)**

 - Environment location: Docker (`nvcr.io/nvidia/pytorch:22.05-py3`)
 - Method of NeMo install: `pip install 'nemo_toolkit[all]==1.10.0'`
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used
 
```sh
docker run --rm -it --gpus all --ipc=host --env-file .env train
```
Fixed in_features dim if bidirectional is True
State: closed | Pull request: yes

Signed-off-by: Faris Abdullah Alasmary <farisalasmary@gmail.com>

# What does this PR do ?

Fix a bug in LSTM decoder while using bidirectional LSTM

**Collection**: ASR

# Changelog 
- Adding one line of  code to check if the argument `bidirectional` is True and multiply `lstm_hidden_size` by 2 since the output vector of both directions have the same size and they are concatenated to form the final output of the LSTM model.

# Usage
N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
text normalization documentation should mention the 500 word limit
State: closed | Pull request: no
Labels: name: feature

name: TN/ITN

**Is your feature request related to a problem? Please describe.**

normalize() has a `500` word limit per here: https://github.com/NVIDIA/NeMo/blob/main/nemo_text_processing/text_normalization/normalize.py#L255

This is not documented in the documentation pages as far as I can see.

**Describe the solution you'd like**

Documentation should make this requirement clear up front, so when developing a pipeline which uses nemo text normalization one is not surprised by crashes when are large input comes through. Instead developers could split into sentences knowing it is a requirement of this method.

Bonus: explain why there is a 500 word limit.

**Describe alternatives you've considered**

Seems like a text normalizer model would know where sentence bounds are to be able to operate, so the normalizer could split into sentences itself and process the sentences, removing this limitation.
Fix metric setup for finetuning without a test set
State: closed | Pull request: yes

Signed-off-by: MaximumEntropy <sandeep.subramanian.1@umontreal.ca>

# What does this PR do ?

Fixes a bug in setup_metric() when fine-tuning without a specified test set.

**Collection**: NLP

# Changelog 
- Use the provided data_cfg to instantiate a metric rather than test_ds.

# Usage
* N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Pytorch lightning required but not installed for text_normalization
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Installing `nemo_toolkit[nemo_text_processing]` does not bring in `pytorch_lightning` but then 
```python
from nemo_text_processing.text_normalization import normalize
```
errors as pytorch_lightning is required to import

**Steps/Code to reproduce bug**

1. on a fresh venv, install `nemo_toolkit[nemo_text_processing]` 
2. run `from nemo_text_processing.text_normalization import normalize`
3. See error

**Expected behavior**

If text processing requires pytorch_lightning it should be installed as a requirement
Loss is getting zero when training Titanet with Adam
State: closed | Pull request: no

Hi

I am trying to train TitaNet with angular loss. When the optimizer is SGD everything is fine (loss decreases and TopK accuracy  increases). Switching optimizer from SGD to Adam or AdamW results zero loss and zero accuracy after some steps.  Do you have any clue about zero-loss?
Btw, number of speakers in my case is equal to ~25000.
[TTS][ASR] customize arguments for trimming the leading/trailing silence
State: closed | Pull request: yes

# What does this PR do ?

It supports overriding arguments of `librosa.effects.trim` when trimming silence. By default, we assign `trim: false` in YAML config. But when enabling it by `trim: true`, you have options to override original arguments of `librosa.effects.trim`, for example,

     trim: true
      trim_top_db: 50,
      trim_frame_length: ${model.n_window_size},
      trim_hop_length: ${model.n_window_stride},

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Question] Exporting HifiGanModel and Tacotron2Model for TTS on Android 
State: closed | Pull request: no
Labels: name: TTS

name: stale

I am using TTS using the below code. I want to run TTS offline on an Android device as a next step using the same pretrained models.
```python
import soundfile as sf
from nemo.collections.tts.models import HifiGanModel
from nemo.collections.tts.models import Tacotron2Model

spec_generator = Tacotron2Model.restore_from("tts_hi_female_tacotron2.nemo")
spec_generator.eval()

vocoder = HifiGanModel.from_pretrained(model_name="tts_hifigan")

parsed = spec_generator.parse("न्यू यॉर्क में सड़क धंसने से एक विशाल सिंकहोल बनने का वीडियो वायरल हो गया है। इसमें भारी बारिश के बाद इलाके में एक वैन को सिंकहोल में गिरते हुए देखा जा सकता है।")

spectrogram = spec_generator.generate_spectrogram(tokens=parsed)
audio = vocoder.convert_spectrogram_to_audio(spec=spectrogram)

sf.write("speech.wav", audio.to('cpu').detach().numpy()[0], 22050)
```
I went through the issue [How to use/port quartzNet model on mobile?](https://github.com/NVIDIA/NeMo/issues/1394) which suggests it is possible to export the model.

Pytorch documentation for exporting model to run on android shows below code
```python
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
traced_script_module_optimized = optimize_for_mobile(traced_script_module)
traced_script_module_optimized._save_for_lite_interpreter("app/src/main/assets/model.ptl")
```

Where I am stuck now ->
What to pass in example as second paramter to  torch.jit.trace for tacotron2 and hifigan?
I would need to export both tacotron2 and hifigan separately?
How to pass output from one model to input to another on android? (This only if I am able to achieve first two steps)
Wav2vec2.0 Script add
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Adds a script to load Wav2Vec2.0 weights from Fair to NeMo implementation. Also adjusts NeMo implementation to be similar to Fair's.

**Collection**: [ASR]

# Changelog 
- Added script to download and load state_dict from Fairseq Wav2Vec models to NeMo equivalents
- Adjusted order of normalization operations for wav2vec preprocessing
- Changed Wav2Vec module config so that projection operation of features is done in transformer encoder
- Altered contrastive loss so config can provide option to not combine groups. (Just added an optional boolean)

# Before your PR is "Ready for review"
**Pre checks**:
- [y ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ n] Did you write any new necessary tests?
- [ n] Did you add or update any necessary documentation?
- [ n] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [y ] New Feature
- [ y] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)