renwei2024/nemo-github-issues huggingface dataset:
Apply garbage collection interval to validation steps
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Apply garbage collection interval to validation steps


# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
Auto-applied when using `gc_interval > 0`

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fine-tuning  ASR (stt_en_conformer) on another language.
State: closed | Pull request: no

Hello,

I would like to use the pre-trained stt_en_conformer_ctc or stt_en_conformer_transducer models for transfer learning on another language ( I only have around 1000 hours of audio, and I would like to jump-start the training/try to see if it may bring better results). I am new to ASR, so I am not very familiar if this is possible. From my understanding, I could use the pre-trained weights from the encoder, and only change the tokens of the decoder ( but the tokenizer should have the same size as the pretrained model, even if the tokens are different). 

I would greatly appreciate some clarifications/corrections to what I mentioned above and if someone could guide me towards a tutorial/example that tackles something similar?

Thank you in advance!
Apply garbage collection interval to validation steps
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Apply garbage collection interval to validation steps


# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
Auto-applied when using `gc_interval > 0`

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Order of scheduler/optimizer calls
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

PyTorch is warning about the order at which is the scheduler is being called.

**Steps/Code to reproduce bug**

Any training script will generate this warning:
```
[...]/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:139: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
      warnings.warn("Detected call of `lr_scheduler.step()` before `optimizer.step()`. "
```



**Expected behavior**

Reorder the calls.

**Environment overview (please complete the following information)**

 - Environment location: Bare metal
 - Method of NeMo install: `pip isntall nemo_toolkit[asr]`

**Environment details**

- NeMo: 1.18.1
- OS version: Ubuntu 22.04
- PyTorch version: '2.0.1+cu117'
- Python version: Python 3.10.6

**Additional context**


GPUs: A100 40G (8x)
fix ptuning residuals bug
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

- fixes peft ptuning bug which was causing ptuning to use a residual connection
- simplifies mixin use for all NLP PEFT models
- ptuning includes a inference only table which is needed for FT 
- 
**Collection**: [NLP]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add API docs for NeMo Megatron
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Adds API documentation.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Bugfix] Move nemo file config logic of GPT eval script
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Move the cfg.gpt_model_file logic into the "if" block to avoid error during ckpt eval

**Collection**: [Note which collection this PR will affect] 
NLP
examples/nlp/language_modelling

# Changelog 
- Add specific line by line info of high level changes in this PR.
Move https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/megatron_gpt_eval.py#L172-L189
to after https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/megatron_gpt_eval.py#L191

# Usage
* You can potentially add a usage example below
Otherwise, if following this
```python
        python megatron_gpt_eval.py \
            checkpoint_dir=PATH_TO_CHECKPOINT_FILE \
            checkpoint_name=CHECKPOINT_FILE_NAME \
            hparams_file=HPARAMS_FILE \
            inference.greedy=True \
            inference.add_BOS=True \
            trainer.devices=1 \
            trainer.num_nodes=1 \
            tensor_model_parallel_size=-1 \
            pipeline_model_parallel_size=-1 \
            prompts=[prompt1,prompt2]
```
Since in this case gpt_model_file=null, the current script will error out due to restore path unknown
```
Traceback (most recent call last):
  File "megatron_gpt_eval.py", line 176, in main
    model_config = MegatronGPTModel.restore_from(
  File "/workspace/NeMo-1.18.0/nemo/core/classes/modelPT.py", line 425, in restore_from
    restore_path = os.path.abspath(os.path.expanduser(restore_path))
  File "/usr/lib/python3.8/posixpath.py", line 231, in expanduser
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

```


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests? No
- [ ] Did you add or update any necessary documentation? No
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc) No
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fade on long TTS inputs
State: closed | Pull request: no
Labels: name: bug

name: stale

GPU T4
Riva 2.10
TTS es multispeaker fastpitch-hifigan

I increased the default input when using TTS SSML tags because I need to use 1000 characters, when using it in offline mode there is a big audio quality degradation or fade at the end of the generated audio, to avoid this I want to use online/streaming mode but the size of the chunks is very small (less than 1s), is it possible to increase the size of the chunks to 10s? Is there another way to solve this problem? I couldn't find Riva documentation on how to fix it.

This is the configuration that I'm using but in online mode it stays inferring and does not synthesize any audio:

riva-build speech_synthesis -f \
             /servicemaker-dev/tts_rmir:tlt_encode \
             /servicemaker-dev/tts_es_fastpitch.riva:None \
             /servicemaker-dev/tts_es_hifigan_fastpitch.riva:None\
             --sample_rate=44100 \
             --preprocessor.g2p_ignore_ambiguous=true \
             --phone_set="ipa" \
             --voice_name=tts_spanish \
             --wfst_tokenizer_model=/servicemaker-dev/tokenize_and_classify.far \
             --wfst_verbalizer_model=/servicemaker-dev/verbalize.far \
             --num_speakers=174 \
             --preprocessor.max_sequence_length=1000 \
             --chunk_length=350 \
             --num_mels=350\
             --num_samples_per_frame=1120 \
             --chunkerFastPitch.max_sequence_idle_microseconds=30000 \
             --chunkerFastPitch.max_queue_delay_microseconds=30000 \
             --denoiser.fade_length=30000
sft peft merge
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add Cython build requirement to pyproject.toml
State: closed | Pull request: yes
Labels: name: stale

# What does this PR do ?

Add Cython as a build requirement to pyproject.toml.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
This is related to https://github.com/NVIDIA/NeMo/issues/6854 and solves the issue for me in Python 3.9 and Python 3.10. Instead of requiring the user to manually install cython beforehand, why not specify it as a build requirement as per [PEP-0518](https://peps.python.org/pep-0518/).
[WIP] Speech translation code pushed to upstream
State: closed | Pull request: yes
Labels: name: TTS

name: stale

name: ASR

name: NLP

name: common

# What does this PR do ?

Speech Translation model based on Conformer encoder and Transformer encoder added to upstream.

**Collection**: ASR, TTS, NLP

# Changelog 
- Added ConformerEnc+TransformerDec model to ASR
- Support for mixed speech-to-text and text-to-text (with TTS) training
- Replicate results / release models from IWSLT 2023

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Merging T5 PEFT methods into a simple tuning/eval script
State: closed | Pull request: yes
Labels: name: stale

name: NLP

name: CI

# What does this PR do ?

This PR merges the following T5 PEFT methods:
- adapters
- lora
- p-tune
- ia3

The changes in T5-related classes make it easier to merge the PEFT methods. All of the methods have been tested on Squad-100 subset (not optimal, but will try to run with optimal params also): https://wandb.ai/nvidia/davidm_nemo-t5_peft_tests?workspace=user-dmosallanezh

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
The tuning can be used as:

```bash
python /data/NeMo/examples/nlp/language_modeling/tuning/megatron_t5_peft_tuning.py \
    trainer.precision=32 \
    model.language_model_path=BASE_MODEL \
    model.data.train_ds=['TRAIN_DATA'] \
    model.data.validation_ds=['VAL_DATA'] \
    exp_manager.create_wandb_logger=True \
    exp_manager.wandb_logger_kwargs.project='PROJ' \
    exp_manager.wandb_logger_kwargs.name='NAME' \
    model.optim.lr=4e-3 \
    model.tensor_model_parallel_size=1 \
    model.pipeline_model_parallel_size=1 \
    trainer.devices=1 \
    exp_manager.exp_dir='/data/t5_peft' \
    ~model.optim.sched \
```

and the eval script can be used as:

```bash
python /data/NeMo/examples/nlp/language_modeling/tuning/megatron_t5_peft_eval.py \
    trainer.precision=32 \
    data.test_ds=['TEST_DATA'] \
    data.global_batch_size=64 \
    data.micro_batch_size=4 \
    language_model_path=BASE_MODEL \
    peft_model_file=SAVED_NEMO_FILE \
    pred_file_path=t5-peft-preds.txt
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[bugfix] avoid the random shuffle of phoneme and tone tokens.
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

The order of outputs of `dict.values()` is usually shuffled so that the tokenizer and the model will be confused to match a correct sound for a specific integer index. You can tell the diff in between as below,

**CTC Training Loss Curves**
The upper and bottom curves are referring to the current implementation and bugfix, respectively.
![W B Chart 6_12_2023, 12_08_47 PM](https://github.com/NVIDIA/NeMo/assets/1646669/9e6f99a0-c668-4eff-ad77-135d6a238d6f)

**Phoneme Token List**
* current random order: `['#Q', '#VE', '#IE', '#X', '#Y', '#ZH', '#S', '#EI', '#ANG', '#V', '#O', '#H', '#D', '#F', '#T', '#G', '#OU', '#UI', '#UN', '#B', '#IU', '#I', '#C', '#AO', '#IN', '#L', '#^', '#M', '#ING', '#VN', '#Z', '#SH', '#EN', '#J', '#P', '#K', '#CH', '#AI', '#A', '#ENG', '#R', '#W', '#U', '#E', '#AN', '#N', '#ER', '#ONG']`
* bugfix: `['#A', '#AI', '#AN', '#ANG', '#AO', '#B', '#C', '#CH', '#D', '#E', '#EI', '#EN', '#ENG', '#ER', '#F', '#G', '#H', '#I', '#IE', '#IN', '#ING', '#IU', '#J', '#K', '#L', '#M', '#N', '#O', '#ONG', '#OU', '#P', '#Q', '#R', '#S', '#SH', '#T', '#U', '#UI', '#UN', '#V', '#VE', '#VN', '#W', '#X', '#Y', '#Z', '#ZH', '#^']`

**Audio Examples**
* current: https://drive.google.com/file/d/1w7SmNobbwx5g44M2uXff2Mrbkmy2lQAH/view?usp=drive_link
* bugfix: https://drive.google.com/file/d/1eROMHI5m4X1AZid-gryBnH99YGa85OI7/view?usp=drive_link

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
NeMo installation fails due to outdated abandoned dependency on youtokentome
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

NeMo installation fails with Python >=3.9 because `youtokentome` does not properly specify it's build dependency on `cython`. See this issue [in their repo.](https://github.com/VKCOM/YouTokenToMe/issues/94). There is also [a PR](https://github.com/VKCOM/YouTokenToMe/pull/108) on that repo to fix that but the author seems to have abandoned the project, its last release was in 2020.

**Steps/Code to reproduce bug**

Trying to install `nemo_toolkit` using Python 3.9 or above.


**Expected behavior**

Installation should succeed.

**Environment overview (please complete the following information)**

 - Environment location: GCP
 - Method of NeMo install:`poetry add nemo_toolkit[asr]`

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- Debian
- PyTorch 1.13.1 and >2.0
- Python 3.9, 3.10, 3.11

**Additional context**

This is particularly annoying given that the whole package depends on `youtokentome` (see `requirements_common.txt`). So even if users like me only care about the ASR part, `youtokentome` must be installed.
LID: several random samples for long file
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Use several random samples for long files

**Collection**: ASR

# Changelog 
Added parameters:
- segment_duration (float): random sample duration in seconds
- num_segments (int): number of segments of file to use for majority vote

# Usage

```python
lang_model = nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained(model_name="langid_ambernet")
lang = lang_model.get_label(filename, segment_duration = np.inf, num_segments = 1, random_seed = None)
```

**PR Type**:
- [V] New Feature


## Who can review?
@fayejf
[TTS] Create EnCodec training recipe
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

This is an experimental draft for the EnCodec training recipe based on https://github.com/NVIDIA/NeMo/compare/encodec?expand=1

The PR is quite large, but has been minimized to what is necessary to run the training end to end and get a good result. It is still missing some features from the original EnCodec code that may be added in later. Including:

- Data augmentation in data loader
- Multi-resolution training (might not need to support as a feature)
- Streaming /w causal convolutions (might not need to support as a feature)
- Gradient balancer (might be unnecessary if we tune model and scale losses properly).

Note that the VQ implementation does not work with FP16 (loss always becomes NaN). I have not tested the distributed training.

**Collection**: [TTS]

# Changelog 
- Add EnCodec training recipe

# Usage

Example script invocation

```bash
python /home/rlangman/Code/NeMo/examples/tts/encodec.py
--config-path=/home/rlangman/Code/NeMo/examples/tts/conf/encodec
--config-name=encodec.yaml
max_epochs=100
weighted_sampling_steps_per_epoch=1000
batch_size=32
trainer.devices=1
log_dir=/home/rlangman/exps/dns4_jamendo_24khz/EnCodec/test_vq/samples
exp_manager.exp_dir=/home/rlangman/exps/dns4_jamendo_24khz
exp_manager.create_wandb_logger=true
+exp_manager.version=test_vq
+exp_manager.wandb_logger_kwargs.project=encodec_dns4_jamendo_24khz
+exp_manager.wandb_logger_kwargs.name=test_vq
+train_ds_meta.dns4.manifest_path=/home/rlangman/data1/data/dns4/train_manifest.json
+train_ds_meta.dns4.audio_dir=/home/rlangman/data1/data/dns4/audio_24khz
+train_ds_meta.dns4.sample_weight=0.5
+val_ds_meta.dns4.manifest_path=/home/rlangman/data1/data/dns4/val_manifest.json
+val_ds_meta.dns4.audio_dir=/home/rlangman/data1/data/dns4/audio_24khz
+log_ds_meta.dns4.manifest_path=/home/rlangman/data1/data/dns4/log_manifest.json
+log_ds_meta.dns4.audio_dir=/home/rlangman/data1/data/dns4/audio_24khz
+train_ds_meta.jamendo.manifest_path=/home/rlangman/data1/data/jamendo/train_manifest.json
+train_ds_meta.jamendo.audio_dir=/home/rlangman/data1/data/jamendo/audio_24khz
+train_ds_meta.jamendo.sample_weight=0.5
+val_ds_meta.jamendo.manifest_path=/home/rlangman/data1/data/jamendo/val_manifest.json
+val_ds_meta.jamendo.audio_dir=/home/rlangman/data1/data/jamendo/audio_24khz
+log_ds_meta.jamendo.manifest_path=/home/rlangman/data1/data/jamendo/log_manifest.json
+log_ds_meta.jamendo.audio_dir=/home/rlangman/data1/data/jamendo/audio_24khz
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
[Bugfix] Fix a bug in filtering checkpoints
State: closed | Pull request: yes

# What does this PR do ?

When we cannot find `self.monitor` in `checkpoint`, `checkpoint.find(self.monitor)` should be -1 so `index` should be `len(self.monitor)` according to L98.

With the current code, checkpoints such as `mp_rank_00_model_weights.ckpt` would go to the True condition and cause an error in Line 103 because  it cannot find the correct `value` to be converted to float.
https://github.com/NVIDIA/NeMo/blob/5c3ed943d8b23ae32719f8fd0e51fca5801596d3/nemo/utils/callbacks/nemo_model_checkpoint.py#L98-L103.

**Collection**: [Note which collection this PR will affect]

- NeMo/nemo/utils/callbacks/nemo_model_checkpoint.py

# Changelog 


# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Should fix https://github.com/NVIDIA/NeMo/issues/6796
Add API docs for NeMo Megatron
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Adds API documentation.

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[NMT] Riva deployment issues of model built with nemo:23.03
State: closed | Pull request: no
Labels: name: bug

name: stale

I understand the this is not the primary issue tracker for Riva (I've posted an issue in the forum https://forums.developer.nvidia.com/t/riva-2-11-0-nemo-23-03-unable-to-deploy-nmt/256169).

Since I can not pinpoint from where the issue arises, I am posting here as well.

Trying to deploy a nemo NMT AAYN model built using `nemo:23.03`, the deploy fails. Riva does not load, the reported error is
```
UNAVAILABLE: Invalid argument: model 'nmt-classifier', tensor 'log_probs': the model expects 3 dimensions (shape [-1,-1,64000]) but the model configuration specifies 3 dimensions (an initial batch dimension because max_batch_size > 0 followed by the explicit tensor shape, making complete shape [-1,-1,32000])
```

I've managed to track down the issue to line 96 in `/usr/local/lib/python3.8/dist-packages/servicemaker/triton/nmt.py`, which sets a hardcoded value of `[-1,32000]` for the `log_probs` output in the classifier. If I change the value in the classifier `.pbtxt` by hand, the model loads and works as it should.

However, I'm unable to find where does the number of 64000 in my case come from. Is it from the target language tokenizer (in my case its size is 64000 tokens)?
Diarization returning only 2 speakers with 3 in the audio
State: closed | Pull request: no
Labels: name: stale

Hello everyone,
I'm working with speaker diarization to identify 2 or more speakers in the same audio. When I try that returns the same results, something like 2 speaker in a audio when have 3.

For that I using Neural Diarizer with titanet_large or my own model to speaker_embedding and 'diar_msdd_telephonic' to mssd model.
Parameters:
```
config.diarizer.manifest_filepath = 'data/input_manifest.json'
config.diarizer.out_dir = output_dir # Directory to store intermediate files and prediction outputs
pretrained_speaker_model = 'finetune-Epochs_2-Batch_32-sgd-Lr_0.011823477534453206-Scale_60-Margin_0.07095099595312822-Date_18-05-2023_10-26-20-917.nemo'
config.diarizer.speaker_embeddings.model_path = pretrained_speaker_model

config.diarizer.speaker_embeddings.parameters.window_length_in_sec = [1.5,1.25,1.0,0.75,0.5] #[1.5,1.25,1.0,0.75,0.5]
config.diarizer.speaker_embeddings.parameters.shift_length_in_sec = [0.75,0.625,0.5,0.375,0.1] #[0.75,0.625,0.5,0.375,0.1]
config.diarizer.speaker_embeddings.parameters.multiscale_weights= [0.5,0.5,0.5,0.5,0.5] #[1,1,1,1,1]

config.diarizer.msdd_model.model_path = 'diar_msdd_telephonic' # Telephonic speaker diarization model 
config.diarizer.msdd_model.parameters.sigmoid_threshold = [0.03] # Evaluate with T=0.7 and T=1.0
config.diarizer.msdd_model.parameters.infer_batch_size = 64
config.diarizer.msdd_model.parameters.seq_eval_mode = True 
config.diarizer.msdd_model.parameters.split_infer = True
config.diarizer.msdd_model.parameters.diar_window_length = 10
config.diarizer.msdd_model.parameters.overlap_infer_spk_limit = 8
```

I tried to change sigmoid_threshold between 0.02 - 0.7, diar_window_length between 5-50 and batch_size between 64-16 to improve the output

Here an example:
[diarized_1.zip](https://github.com/NVIDIA/NeMo/files/11708476/diarized_1.zip)
Just 2 speakers were recognized in a audio with 3 speakers.

Why can't I return 3 speakers with this parameters? How can I improve this results? Maybe I did something wrong? Maybe audios need to be cleaned up, doing that without noise?
Thanks in advance
Support for LLaMa-type models?
State: closed | Pull request: no
Labels: name: stale

**Is your feature request related to a problem? Please describe.**

With the recent PR adding support for the MPT-7b model, I was wondering if someone has figured out how to work with LLaMa-type models, such as Vicuna

**Describe the solution you'd like**

A solution similar to this PR for LLaMa-type models 
https://github.com/NVIDIA/NeMo/pull/6804

**Describe alternatives you've considered**

Triton/FasterTransformer can now support LLaMa so hopefully this is possible for Nemo

**Additional context**

Huggingface Transformers also supports LLaMa-architecture
https://huggingface.co/docs/transformers/main/model_doc/llama
temp_outputs/pred_rttms/mono_file.rttm File not found. Randomly happens, have to restart container.
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

The bug is at a random interval we will suddenly get file not found or is not directory temp_outputs/pred_rttms/mono_file.rttm
Will work just fine hundreds of times and will randomly throw this.

**Steps/Code to reproduce bug**

Deploy as normal, using whisper + diarization (https://github.com/MahmoudAshraf97/whisper-diarization). Nothing special and it will work many times and just randomly will have that file missing. 

**Expected behavior**

Should not have a file missing...

**Environment overview (please complete the following information)**

Docker using nvcr.io/nvidia/pytorch:23.04-py3
If you have another container that nemo would work in then I am all ears.


**Additional context**

GPU: A100 
CUDA 11.7 with driver 525.XX
fix for negative indices in gpt dataset
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

This is required when the number of samples is very small compared to the batch size. The magnitude of the negative index can be greater than the sample index array.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Bring back Talk Net 2
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**
Talk Net 2 was deprecated but no comparable model took its place.

After experimenting with the various models, we have yet to find anything in open source land that compares to Talk Net 2. 

FastSpeech2 is *okay* but is subjectively worse (and objedtively has .25 lower MOS), especially with spacing and prosody.

Is there any reason support was dropped and the model was deprecated?

Because we have to use a very old version of NeMo to continue to support TalkNet2, it's kind of version locked us to some specific stale and conflicting dependencies within our stack.

Thanks to Pony Preservation Project, Webaverse and other open source projects, TalkNet might have been the most popular NeMo TTS model in production. There are notebooks, colab, extensive training materials built up by communities around it, for example: https://github.com/bycloudai/TalkNET-colab

Happy to make a PR, unless there are specific technical reasons why it was deprecated. In the meantime we're all living off a fork, notably this one: https://github.com/SortAnon/NeMo
Correct typos in FC text
State: closed | Pull request: yes

# What does this PR do ?

Fix some text in FC post

**Collection**: [Docs]

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
layer unfreeze
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
sharded manifests updated docs
State: closed | Pull request: yes
Labels: name: documentation

name: ASR

# What does this PR do ?

Adds a short section on sharded manifests in the docs

**Collection**: [ASR]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ X] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
added fc-xl, xxl and titanet-s models
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add FC-XL RNNT and CTC models, FC-XXL (1.1 B) transducer model and TitaNet-S model

**Collection**: ASR

# Changelog 
- Add pre trained models to code
- Add to docs
- Add FC configs

# Usage

```python
# FC Transducer BPE models
from nemo.collections.asr.models import EncDecRNNTBPEModel
fc_xl_rnnt = EncDecRNNTBPEModel.from_pretrained('stt_en_fastconformer_transducer_xlarge') 
fc_xxl_rnnt = EncDecRNNTBPEModel.from_pretrained('stt_en_fastconformer_transducer_xxlarge')

# FC CTC BPE models
from nemo.collections.asr.models import EncDecCTCModelBPE
fc_xl_ctc = EncDecCTCModelBPE.from_pretrained('stt_en_fastconformer_ctc_xlarge') 

# TitaNet-S model
from nemo.collections.asr.models import EncDecSpeakerLabelModel 
speaker_model = EncDecSpeakerLabelModel.from_pretrained('titanet_small')
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [x] Documentation
Post for Fast Conformer
State: closed | Pull request: yes

# What does this PR do ?

Add GH post for Fast Conformer

**Collection**: [ASR]

# Changelog 
- Adds blogpost for fast conformer
- Fixes issue with the font not loading
- removes mathjax file when supported from CDN


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Sharded manifests setting in config files
State: closed | Pull request: yes
Labels: name: stale

name: ASR

# What does this PR do ?

Makes sharded manifests default for train dataloader in ASR config files

**Collection**: ASR

# Changelog 
Adding these flags to the train_ds section of config files:
shard_manifests: False
defer_setup: False

# Usage
Set these flags to true if you wish to use the sharded manifest feature

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation
- [X ]  Config

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Upperbound Numpy to < 1.24
State: closed | Pull request: yes

# What does this PR do ?

Adds upperbound to Numpy temporarily.
Related Issue : https://github.com/NVIDIA/NeMo/issues/6824

**Collection**: [Core]

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Concat samples
State: closed | Pull request: yes
Labels: name: stale

name: ASR

name: common

# What does this PR do ?

Adds [limited] support for runtime concatenation of samples

**Collection**: ASR

# Changelog 
- The implementation is limited to the ConcatDataset class. 
- There is no sample normalization
- samples are joined with a zero pause

# Usage
* You can potentially add a usage example below

```python
# Add these parameters
train_ds.is_concat: True  # need to enable the concat dataset
train_ds.concat_samples: True  # enable sample concatenation
train_ds.concat_samples_count_as_one: True  # count samples as per real number or  as one
train_ds.concat_samples_max_length: 22 # max length of concatenated sample
train_ds.concat_samples_min_length: 16 # min length of concatenated sample
train_ds.concat_samples_joining_pause: 0.1 # joining pause
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [X ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Dockerfile apex build command fails
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

The Dockerfile contains the following command to build apex

```
# TODO: Remove once this Apex commit (2/24/23) is included in PyTorch
# container
RUN git clone https://github.com/NVIDIA/apex.git && \
  cd apex && \
  git checkout 57057e2fcf1c084c0fcc818f55c0ff6ea1b24ae2 && \
  pip3 install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--fast_layer_norm" --global-option="--distributed_adam" --global-option="--deprecated_fused_adam" ./
```
But this fails due to https://github.com/NVIDIA/apex/issues/1636

I had to follow the suggestions in the apex issue, i.e. add the following lines on the top of `apex/contrib/layer_norm/ln.h`

```
include <stdint.h>
include <stdio.h>
```