renwei2024/nemo-github-issues huggingface dataset:
[TTS] Add STFT and SI-SDR loss to audio codec recipe
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Make audio codec losses more configurable, and add STFT and SI-SDR loss definitions.

**Collection**: [TTS]

# Changelog 
- Rename encodec.yaml to encodec_24000.yaml to include sampling rate, and modify parameters to better match the original paper.
- Add audio_codec_24000.yaml where we can start creating config for optimized NeMo audio codec recipe. 
- Separate out aggregate mel loss into separate L1 and L2 losses.
- Add parameter to control mel dimension for different resolutions.
- Implement multi-resolution log STFT loss.
- Implement SI-SDR loss, being a slightly modified version of the torchmetrics implementation but with masking support.
- Make all losses optional during train_step, only computing and optimizing on them if their loss_scale is non-zero. Except L1 mel loss which I expect will always be used, so I can use it as the logged "t_loss" value.
- Compute all losses during validation step for monitoring and validation.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

# Additional Information
The new configuration places no weight on time domain loss, scales the mel dimension proportional to the resolution (like DAC), and adds STFT loss (like univnet) to further improve accuracy. 

These are what the validation losses look like when training with different losses: mel fixed dimension, mel scaled dimension, STFT, and (mel scaled & STFT).

![loss_mel](https://github.com/NVIDIA/NeMo/assets/9942053/3f1e4db6-ec3f-4359-8789-86506227315e)

![loss_si_sdr](https://github.com/NVIDIA/NeMo/assets/9942053/1613c838-56c9-41aa-a2a6-8cd8bffa1b3a)

![loss_time_domain](https://github.com/NVIDIA/NeMo/assets/9942053/4339cd9f-beca-4007-b0ed-032fb1e8585f)

![loss_stft](https://github.com/NVIDIA/NeMo/assets/9942053/35d16738-82c7-41a1-bb36-70313a03dde7)
Add an option to disable manual GC during validation
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Disable manual GC during the validation routine. This is valid only when `gc_interval` is set larger than 0.
To disable, set `NEMO_MANUAL_GC_IN_VALIDATION=0`

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add onnx export for Tacotron2 model
State: closed | Pull request: yes
Labels: name: TTS

name: stale

# What does this PR do ?

1. Make Tacotron2 able to be exported to ONNX format files.
2. Create an example of how to use these ONNX files to do inference

The code is mainly learned from [NVIDIA/DeepLearningExamples](https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/SpeechSynthesis/Tacotron2/exports/export_tacotron2_onnx.py)

# Changelog 
- Add support for Tacotron2 model to export to onnx format

# Usage
For testing, we could install necessary packages, check out this PR and run the example script:
```bash
pip install onnxruntime

cd NeMo
PYTHONPATH=. python3 examples/tts/export/infer_tacotron2_onnx.py 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

@blisc @XuesongYang 

# Additional Information
* Related to #6999
[Draft] Add support for Tacotron2 model to export to ONNX format
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

1. Make Tacotron2 able to be exported to ONNX format files.
2. Create an example of how to use these ONNX files to do inference

The code is mainly learned from [NVIDIA/DeepLearningExamples](https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/SpeechSynthesis/Tacotron2/exports/export_tacotron2_onnx.py)

# Changelog 
- Add support for Tacotron2 model to export to onnx format

# Usage
For testing, we could install necessary packages, check out this PR and run the example script:
```bash
pip install onnx onnxruntime

cd NeMo
PYTHONPATH=. python3 examples/tts/export/infer_tacotron2_onnx.py 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?



# Additional Information
* Related to #6999
Fix sft dataset truncation
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Replace assert with warning if we don't have enough token to truncate.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] Add dataset to path of logged artifacts
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Modify callbacks to save artifacts into dataset specific folders, instead of saving artifacts from all datasets into 1 folder. This helps organization when training on many datasets, and also handles the case when multiple datasets have files with the same name.

**Collection**: [TTS]

# Changelog 
- Pass dataset names through TTS data loaders
- Save artifacts under epoch_{x}/{dataset_name}/
- Add initial logging for FastPitch and HiFi-GAN recipes

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Add strategy as ddp_find_unused_parameters_true for glue_benchmark.py
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
RNN-T confidence and alignment bugfix (#7381)
State: closed | Pull request: yes
Labels: name: ASR

* new frame_confidence and alignments lists are now always created after the while loop

Signed-off-by: Aleksandr Laptev <alaptev@nvidia.com>

* tests added

Signed-off-by: Aleksandr Laptev <alaptev@nvidia.com>

---------

Signed-off-by: Aleksandr Laptev <alaptev@nvidia.com>
(cherry picked from commit 6012ca2f4034cadb2f3186899ffc9ef8490472cb)

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Resuming from last checkpoint is broken with tensor parallelism
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When running an experiment where the model is using tensor parallelism, the `exp_manager.resume_if_exists` option fails to load the last checkpoint when resuming the experiment.

**Steps/Code to reproduce bug**

* Launch an experiment with a TP>1 model and stop it after saving the first checkpoint
* Try to resume the experiment, and notice the warning:

```
`[NeMo W 2023-09-19 01:42:57 exp_manager:591] There were no checkpoints found in checkpoint_dir or no
checkpoint folder at checkpoint_dir:  /path/to/checkpoints. Training from scratch.`
```


**Expected behavior**

It should resume from the last checkpoint saved.

**Environment overview (please complete the following information)**

 - Environment location: Docker
 - Method of NeMo install: from Docker image (was taken from a recent nightly build)

**Additional context**

At first glance I suspect the issue came from https://github.com/NVIDIA/NeMo/pull/7116/files due to changing `rglob` into `glob` (cc @ericharper) ==> since the .ckpt files are under the `mp_rank_xy` subfolders, they are not found by `glob`.

(if the fix is just to change it back to `rglob` I'm happy to submit a PR, but I assume there's a reason why it was changed)
Fix None dataloader issue in PTL2.0
State: closed | Pull request: yes
Labels: name: core

# What does this PR do ?

With PTL2.0, returning None from the *_dataloader methods is no longer supported (reference - https://github.com/Lightning-AI/lightning/issues/17154). This change causes an issue with ASR decoding when we don't provide a valid trainer.validation_ds.manifest_filepath or trainer.test_ds.manifest_filepath. For example running the following code:
`python NeMo/examples/asr/asr_ctc/speech_to_text_ctc_bpe.py --config-path=../conf/citrinet/ --config-name=citrinet_1024.yaml model.tokenizer.dir=data/tokenizer/tokenizer_bpe_v1024/ model.tokenizer.type="bpe" model.train_ds.manifest_filepath=data/LibriSpeech/train_clean_100.json model.validation_ds.manifest_filepath=data/LibriSpeech/dev_clean.json model.train_ds.batch_size=16 +accelerator="gpu" +strategy="ddp" +devices=1 trainer.max_epochs=1 exp_manager.checkpoint_callback_params.monitor="train_loss"`

throws the error - 
`TypeError: An invalid dataloader was returned from EncDecCTCModelBPE.val_dataloader(). Found None.`

The following PR fixes this issue by making the default return value of val_dataloader and test_dataloader as `[]`

**Collection**: All

# Changelog 

```
  def val_dataloader(self):
      if self._validation_dl is not None:
          return self._validation_dl
      else: # None dataloader no longer supported in PTL2.0
          return []

  def test_dataloader(self):
      if self._test_dl is not None:
          return self._test_dl
      else: # None dataloader no longer supported in PTL2.0
          return []
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Add strategy as ddp_find_unused_parameters_true for glue_benchmark.py
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
fix load dist ckpt in peft
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Output .ctm of Speech Data Simulator has channel and spk_id swapped 
State: closed | Pull request: no
Labels: name: stale

https://github.com/NVIDIA/NeMo/blob/2cc09425aba3e9b3cfdba43a3188eaef58227055/nemo/collections/asr/data/data_simulation.py#L1057

But according to https://web.archive.org/web/20170119114252/http://www.itl.nist.gov/iad/mig/tests/rt/2009/docs/rt09-meeting-eval-plan-v2.pdf it should be: 

```
<SOURCE><SP><CHANNEL><SP> <BEG-TIME><SP><DURATION><SP><TOKEN><SP>
<CONF><SP><TYPE><SP><SPEAKER><NEWLINE>
```
Fix checking of cuda/cpu device for inputs of Decoder
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do?
When running the below snippet from the [document](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/tts_en_tacotron2) on a machine with GPU
```python
# Load Tacotron2
from nemo.collections.tts.models import Tacotron2Model
spec_generator = Tacotron2Model.from_pretrained("tts_en_tacotron2")

# Load vocoder
from nemo.collections.tts.models import HifiGanModel
vocoder = HifiGanModel.from_pretrained(model_name="tts_en_hifigan")

# Generate audio
import soundfile as sf
import torch
with torch.no_grad():
    parsed = spec_generator.parse("You can type your sentence here to get nemo to produce speech.")
    spectrogram = spec_generator.generate_spectrogram(tokens=parsed)
    audio = vocoder.convert_spectrogram_to_audio(spec=spectrogram)

# Save the audio to disk in a file called speech.wav
if isinstance(audio, torch.Tensor):
    audio = audio.to('cpu').numpy()
sf.write("speech.wav", audio.T, 22050, format="WAV")
```
It works well.
But if change it to using CPU:
```python
spec_generator = Tacotron2Model.from_pretrained("tts_en_tacotron2").to("cpu")
...
vocoder = HifiGanModel.from_pretrained(model_name="tts_en_hifigan").to("cpu")
```
The snippet will report the error:
```
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! 
```
The reason is the code directly sets input to GPU when it's available.

# Changelog 
- Fix checking of cuda/cpu device for inputs Decoder

# Usage
```python
# Load Tacotron2
from nemo.collections.tts.models import Tacotron2Model
spec_generator = Tacotron2Model.from_pretrained("tts_en_tacotron2").to("cpu")

# Load vocoder
from nemo.collections.tts.models import HifiGanModel
vocoder = HifiGanModel.from_pretrained(model_name="tts_en_hifigan").to("cpu")

# Generate audio
import soundfile as sf
import torch
with torch.no_grad():
    parsed = spec_generator.parse("You can type your sentence here to get nemo to produce speech.")
    spectrogram = spec_generator.generate_spectrogram(tokens=parsed)
    audio = vocoder.convert_spectrogram_to_audio(spec=spectrogram)

# Save the audio to disk in a file called speech.wav
if isinstance(audio, torch.Tensor):
    audio = audio.to('cpu').numpy()
sf.write("speech.wav", audio.T, 22050, format="WAV")
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

## Who can review?
@blisc @okuchaiev @titu1994 

# Additional Information
This fix is learned from the [PyTorch forum](https://discuss.pytorch.org/t/how-to-check-if-model-is-on-cuda/180/4?u=robindong)
Confused - Is Fastpitch 2 the Fastpitch model in the NGC catalogue?
State: closed | Pull request: no
Labels: name: TTS

So i found this example from nvidia:

https://pytorch.org/hub/nvidia_deeplearningexamples_fastpitch/

I reimplemented it in my environment, but I am little confused, because the title says "Fastpitch 2". Is this the same as the Fastpitch models in the NGC catalogue (for example this: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/tts_es_multispeaker_fastpitchhifigan) architecture wise?

There is also this model for riva which is called Fastpitch 1.1: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/speechsynthesis_english_fastpitch

Beside multiple speakers and language I do not understand what the difference between this models is (Except riva being faster of course). Can someone elaborate this naming convention. Should I use the Fastpitch 2 model from the example and retrain it for my purposes to get the best performance or are the fine tuned models in the NGC catalogue already fastpitch2 models?
Speech Simulator, update README.md: output_path --> output_manifest_filepath
State: closed | Pull request: yes

I think the correct option for 

```bash
python <NeMo base path>/scripts/speaker_tasks/create_alignment_manifest.py \
  --input_manifest_filepath <Path to train_clean_100.json manifest file> \
  --base_alignment_path <Path to LibriSpeech_Alignments directory> \
  --output_path train-clean-100-align.json \
  --ctm_output_directory ./ctm_out \
  --libri_dataset_split train-clean-100
```

is `--output_manifest_filepath` not `--output_path` in the current version.
Fix adding positional embeddings in-place in FFTransformerDecoder
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

I believe this PR fixes the issue of FFTransformerDecoder altering the input tensor by adding positional embeddings in-place.

**Collection**: TTS

# Changelog 
- change `inp += pos_emb` to `inp = inp + pos_emb` to prevent changing the input tensor

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
pytorch lightning strategy = None prompt error during trainer config
State: closed | Pull request: no
Labels: name: bug

name: stale

For trainer through pytorch lightning strategy = None prompt error.

```
ValueError: You selected an invalid strategy name: `strategy=None`. It must be either a string or an instance of `pytorch_lightning.strategies.Strategy`. Example choices: auto, ddp, ddp_spawn, deepspeed, ... Find a complete list of options in our documentation at https://lightning.ai/
```
A clear and concise description of what the bug is.

To reproduce :
- Running notebook on strategy = None with pytorch lightning version 2.0.7 
-https://github.com/NVIDIA/NeMo/blame/83d6614fbf29cf885f3bc36233f6e3758ba8f1e3/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb#L478
- Run https://github.com/NVIDIA/NeMo/blame/83d6614fbf29cf885f3bc36233f6e3758ba8f1e3/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb#L494

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.


**Expected behavior**
https://github.com/NVIDIA/NeMo/blame/83d6614fbf29cf885f3bc36233f6e3758ba8f1e3/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb#L494 pytorch trainer should be able to configured without issue.

**Environment overview (please complete the following information)**

 - Environment location: AWS
 - Method of NeMo install:  pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[asr]

**Environment details**

- PyTorch version

**Additional context**
Fix sft dataset tokenization
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

* Prevent split, tokenize, and truncation if not exceed `max_seq_length`.
* Remove `_` if we don't have leading space before. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Add rope dynamic linear scaling
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add dynamic position interpolation for `seq length > learned length`
Learned length is `pre-trained max_position_embeddings * seq_len_interpolation_factor`

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[ASR] Confidence measure -> method renames
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Renames confidence `measure` back to `method` and removes warnings.

**Collection**: [Note which collection this PR will affect]
ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?
@Kipok 

# Additional Information
* Related to # (issue)
[draft] NeMo Mcore Falcon variants support 
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Conversion script for HuggingFace Falcon checkpoint (1b-rw/7b-rw/7b/40b/180b) to NeMo format
Support Falcon models in mcore.

**Collection**: NLP

# Changelog 
- add conversion script

# Usage
* See docstring in the script added.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Way to export/save intermediate logit outputs
State: closed | Pull request: no

Hello, I am currently training the Conformer-CTC model for ASR.

I am interested in how logit prediction changes throughout training, and I am wondering if there is any way to modify the code so that my model saves logits and targets data as a pickle file for the end of each validation step. I am aware that I can append each (logits, targets) in `validation_step_outputs` and then save them in `on_validation_epoch_end()`. However, I am a bit scared to overwrite the function in `ctc_bpe_models.py`. It would be great if there is a safer way to handle such a situation.
Fix failure of ljspeech's get_data.py
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do?

When running `python scripts/dataset_processing/tts/ljspeech/get_data.py --data-root <your_local_dataset_root>` to get the data of LJSpeech as the [document](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/tts/datasets.html#ljspeech), it reported error:
```
FileNotFoundError: [Errno 2] No such file or directory: 'lj_speech.tsv extracted from the readme file in the dataset. You can also download the file from https://github.com/NVIDIA/NeMo-text-processing/blob/main/nemo_text_processing/text_normalization/en/data/whitelist/lj_speech.tsv'
```
Solution: The default value of argument `--whitelist-path` of `get_data.py` should be `None` (so the get_data.py will automatically download the lj_speech.tsv) instead of a long string.

# Changelog 
- Fix failure of ljspeech's get_data.py.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

## Who can review?

@blisc @okuchaiev @titu1994
Draft of fix for #7428
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Draft fix for #7428

**Collection**: [nlp]

# Changelog 
- Fix handling of `limit_val_batches` which could lead to crashes or computing validation error on less data than intended

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Fixes #7428

This is a draft as I am unsure this is the "right" way to fix it, and more work is needed to finish it. However, I would like to request feedback from NeMo devs to know whether this is heading in the right direction. Some things to think about:
* Probably that other classes need to be modified as well, beyond MegatronGPTModel
* Changing the length of the validation dataloader might have unintended consequences
* The MegatronPretrainingRandomSampler might need some fix as well
* I think (at first glance) that the dataloaders are not properly implementing `drop_last` since it seems to be based on `micro_batch_times_data_parallel_size` rather than the global batch size (I didn't try to fix it in this PR)
* I haven't tested at all my fix for the float `limit_val_batches`
* Does the test set also need something similar?
* ... (this is definitely rushed => there's a good chance I may have missed more stuff)
Validation does not work as expected due to incorrect handling of `limit_val_batches`
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

There are at least two issues with how `limit_val_batches` is handled in the current master branch:
1. `trainer.limit_val_batches` is multiplied by `get_num_microbatches()` [here](https://github.com/NVIDIA/NeMo/blob/e3365ead822b4355a6391dd458d3457c441252ac/nemo/collections/nlp/models/language_modeling/megatron_base_model.py#L213C40-L213C47) even if it is a float (e.g., 1.0), which can make the code crash immediately after [here](https://github.com/NVIDIA/NeMo/blob/e3365ead822b4355a6391dd458d3457c441252ac/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py#L993) because we end up with a float > 1.0
2. Because the length of the validation dataloader is computed in terms of global batches ([here](https://github.com/NVIDIA/NeMo/blob/e3365ead822b4355a6391dd458d3457c441252ac/nemo/collections/nlp/data/language_modeling/megatron/data_samplers.py#L85)), PyTorch Lightning will cap the number of validation steps to this number if `limit_val_batches * get_num_microbatches()` is higher, and we won't do the intended number of validation steps. Worse, this can lead to a crash due to incomplete batches.

Here's an example of what can go wrong because of (2):
* Consider the case where the validation set has 127 samples, `global_batch_size=16`, `micro_batch_size=2`, `data_parallel_size=2` and `limit_val_batches=6`
* In that case we have `get_num_microbatches() == 4`
* The multiplication of `limit_val_batches` by `get_num_microbatches()` sets `trainer.limit_val_batches=24`
* The length of the validation dataloader is `127 // 16 == 7`
* As a result, instead of processing 24 validation microbatches, we only process 7. This ends up causing a `StopIteration` in-between the `on_validation_batch_start` and `on_validation_batch_end` callbacks, which is caught by PTL but prevents the validation_batch timer from stopping properly, which triggers a crash on the next validation batch (`RuntimeError: Cannot start timer = 'validation_step_timing' since it is already active`)


**Steps/Code to reproduce bug**

I'm seeing this issue with custom code that I can't easily share now, but it should be easy to repro with any megatron GPT training loop.

**Environment overview (please complete the following information)**

 - Environment location: Docker (I think the image is taken from a recent nightly build)
 - Method of NeMo install: I tried both the version in my Docker image and the current master

**Additional context**

I'll put up a draft PR with a potential direction to solve these issues (see #7429)
Fix CustomProgressBar for resume
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

- Fix the CustomProgressBar for cases when we are resuming training from a checkpoint. Currently when we are resuming from a checkpoint total value in `iter_num/total` of the progress bar is the number of remaining training steps instead of total training steps which leads to the progress bar displaying` iter_num/?` when iter_num > remaining training steps. This PR fixes the issue by reseting the total with `max_steps`.
- Also, in case of multiple epochs the PR updates the current step number (n) to be the current step number within an epoch (default was current step across an epoch and this can lead to an errored progress bar).

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix resume from checkpoint in exp_manager
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Adding Megatron NMT tuning -> config updates
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Minor updates to fix Megatron NMT tuning issues on the latest models.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix resume from checkpoint in exp_manager
State: closed | Pull request: yes

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
HF StarCoder to NeMo conversion script
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Conversion script for HuggingFace StarCoder checkpoint to NeMo format.

**Collection**: NLP

# Changelog 
- Add conversion script
- Update Jenkinsfile

# Usage
See docstring in the script added.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
Starcoder support
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Conversion script for HuggingFace StarCoder checkpoint to NeMo format.

**Collection**: NLP

# Changelog 
- Add conversion script
- Update Jenkinsfile

# Usage
See docstring in the script added.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.