renwei2024/nemo-github-issues huggingface dataset:
ASR Language Modeling
State: closed | Pull request: no
Labels: name: bug

name: stale

name: ASR

**Describe the bug**
Aim: To Imporve the ASR model Accuracy with the help of LM.
ASR Model Used: Nvidia Nemo: Conformer-CTC-BPE.nemo
As per my understanding steps to implement ASR Language Modeling
1. Create KenLM Library.
2. Install Beam Search Decoder
3. Train N-Gram LM using ASR Model and KenLM Library. #output of step 3 is we have train N-Gram Model which can be used in beam search decoder on top of ASR model. (Decoder used PyCTCdecoder)
4. Evaluate by beam search decoder and N-Gram LM ##To tune parameter of beam search decoder. 
My question is after performing this steps how can I used my trained N-Gram Model with beam search decoder on top of ASR model.




**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model
Fix multi rank finetune for ASR
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Fix multi node fine-tuning script by adding rank based delay

**Collection**: [ASR]

# Changelog
- Add some delay to other ranks for the pretrained model call

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Change confidence parameters in the test
State: closed | Pull request: yes
Labels: name: cherry-pick

# What does this PR do ?

There was a change of the hybrid models and the default parameters of confidence don't work well for them on the test data.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix in the confidence ensemble test
State: closed | Pull request: yes

# What does this PR do ?

There was a change of the hybrid models and the default parameters of confidence don't work well for them on the test data. There was also a bug in one parameter not being correctly renamed.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Replace strategy='dp'/None with 'auto'
State: closed | Pull request: yes
Labels: name: ASR

name: NLP

name: Speaker Tasks

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Change confidence parameters in the test
State: closed | Pull request: yes

# What does this PR do ?

There was a change of the hybrid models and the default parameters of confidence don't work well for them on the test data.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Update docs: readme, getting started, ASR intro
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Update the docs a bit. Some key ASR info will be more prominent now.

**Collection**: Docs (General and ASR)

# Changelog 
- Update README
- Update 'Introduction' in docs - contains more up-to-date code example
- Update 'asr/intro.rst' to explain some key things about ASR

# Usage
N/A

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [x] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
add pleasefixme marker for potential failed nightly tests.
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Associated to https://github.com/NVIDIA/NeMo/pull/7627. Add pleasefixme marker as an temporal solution.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
How to get timestamp pretrained RNNTBPEModel
State: closed | Pull request: no

asr_model = nemo_asr.models.EncDecRNNTBPEModel.from_pretrained("nvidia/stt_en_conformer_transducer_large")
words_dict, word_ts_dict = {}, {}
hypotheses = asr_model.transcribe(paths2audio_files=[AUDIO_FILENAME], return_hypotheses=True)
Use builder in more examples
State: closed | Pull request: yes
Labels: name: stale

name: NLP

# What does this PR do ?

Apply Trainer Builders to more example scripts.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[Test][ASR] Adding a mask estimator which can process an arbitrary number of chan…
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

This PR adds a multichannel mask estimator which can process flexible number of channels.
It also fixes some bugs related to PTL 2.0 and librosa version.

**Collection**: ASR

# Changelog 

- added example config `beamforming_flex_channels.yaml`
- add `max_utts` `process_audio.py`
- updated `librosa.get_duration` to use `path`, since `filename` is deprecated
- bumped `librosa` requirement to `0.10.0` in `requirements_asr.py`
- added `MaskEstimatorFlexChannels` in `audio_modules.py`
- added various multichannel modules in `multichannel_modules.py`, including `ParametricMultichannelWienerFiler` and `ReferenceChannelEstimatorSNR`
- added unit tests for multichannel modules

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
different config file
State: closed | Pull request: no
Labels: name: bug

in this line:
https://github.com/NVIDIA/NeMo/blob/c1881f2105a0b65cb9513bcd85456b430ff81bb2/examples/asr/asr_ctc/speech_to_text_ctc_bpe.py#L76

Should it be conformer_ctc_bpe config file instead of citrinet config file?
ONNX Export for GPT models fails
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

Using NeMo container 23.06 and the [nemo_gpt1.3B_fp16.nemo](https://huggingface.co/nvidia/nemo-megatron-gpt-1.3B/blob/main/) file located in the NVIDIA HuggingFace Hub, the following command fails to execute:

"python3 examples/nlp/language_modeling/megatron_export.py \
    model_type=gpt \
    onnx_model_file=nemo_gpt1.3B_fp16.onnx \
    gpt_model_file=nemo_gpt1.3B_fp16.nemo"

With the error:

`TypeError: Can't instantiate abstract class ModelPT with abstract methods list_available_models, setup_training_data, setup_validation_data`

It seems NeMo is failing on the line:

`ModelPT.restore_from("/path/to/.nemo")`


**Steps/Code to reproduce bug**

1. Pull NeMo container [23.06](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo/tags)
2. Pull the megatron-gpt1.3B model using `wget https://huggingface.co/nvidia/nemo-megatron-gpt-1.3B/resolve/main/nemo_gpt1.3B_fp16.nemo`
3. Execute:

python3 examples/nlp/language_modeling/megatron_export.py \
    model_type=gpt \
    onnx_model_file=nemo_gpt1.3B_fp16.onnx \
    gpt_model_file=nemo_gpt1.3B_fp16.nemo

**Expected behavior**

The script runs and produces a valid `.onnx` file at the desired location.

**Environment overview (please complete the following information)**

 - Fresh NeMo container 23.06

**Environment details**

**Additional context**
Fix vad & speech command tutorial - onnx
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix vad & speech command tutorial - onnx
State: closed | Pull request: yes
Labels: name: ASR

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Fix import error of megatron.core in tests
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

After the standard installation of NeMo, running `pytest tests/` in code repo will report the error:
```
______________________________________________________________________________________ ERROR collecting tests/collections/nlp/test_flash_attention.py ______________________________________________________________________________________
ImportError while importing test module '/home/robin/try1/NeMo/tests/collections/nlp/test_flash_attention.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../miniconda3/envs/nemo/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/collections/nlp/test_flash_attention.py:19: in <module>
    from megatron.core import ModelParallelConfig
E   ImportError: cannot import name 'ModelParallelConfig' from 'megatron.core' (/home/robin/miniconda3/envs/nemo/lib/python3.10/site-packages/megatron/core/__init__.py)
```

The `requirements/requirements_nlp.txt` fixed version of `megatron_core` to `0.2.0`. But in this version, the Megatron-LM doesn't even have `ModelParallelConfig`.
The fix is learning from `nemo/collections/nlp/models/language_modeling/megatron/gpt_model.py`. By using `ApexGuardDefaults` to replace `ModelParallelConfig`, the `pytest tests/collections/nlp/test_flash_attention.py` could pass without import error.

**Collection**: tests

# Changelog 
- Fix import error of megatron.core in tests

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation


## Who can review?

@MaximumEntropy @ericharper @ekmb @yzhang123 @VahidooX @vladgets @okuchaiev @XuesongYang
Bias is not propagated when using TE
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

When transformer_engine is enabled. NeMo doesn't pass the bias flag to the [AutocastTransformerLayer](https://github.com/NVIDIA/NeMo/blob/2cc09425aba3e9b3cfdba43a3188eaef58227055/nemo/collections/nlp/modules/common/megatron/transformer.py#L1014) in [ParallelTransformer](https://github.com/NVIDIA/NeMo/blob/2cc09425aba3e9b3cfdba43a3188eaef58227055/nemo/collections/nlp/modules/common/megatron/transformer.py#L846) so bias is always defaulted to True, despite the user trying to set it to False
conversion issue fix
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Swa
State: closed | Pull request: yes
Labels: name: stale

name: NLP

name: CI

# What does this PR do ?

Adds support for Sliding Window Attention (SWA) in Flash Attention 2 to Nemo

**Collection**: NLP

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Support Falcon Variants (7B/40B/180B) in Mcore NeMo
State: closed | Pull request: yes
Labels: name: NLP

name: CI

# What does this PR do ?

Conversion script for HuggingFace Falcon checkpoint (1b-rw/7b-rw/7b/40b/180b) to NeMo format, as well as NeMo to HuggingFace format.
Support Falcon parallel attention and new decoder architecture by adding falcon_decoder_layer and falcon_spec,
these will make use of mcore's latest spec system. 

Limitation: although the conversion script works for 1b-rw/7b-rw, Alibi is currently not supported in mcore GPT, so expect suboptimal generation results from these two models that use Alibi specifically. This limitation will NOT affect other 7b/40b/180b Falcon variants. 

**Collection**: [Note which collection this PR will affect]

# Changelog 
- add bi-directional HF <-> NeMo conversion script
- Falcon architecture implemented using mcore spec system and mcore layers.

# Usage
* See docstring in the conversion script on how to use. 

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
int counter logging shouldn't be reduced across ranks 
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

It appears that Nemo is using the wrong PTL API for logging aggregate int counters:

```
$ grep '\.log(' | grep global 
nemo/collections/nlp/models/language_modeling/megatron_finetune_model.py:            
self.log('global_step', self.trainer.global_step, prog_bar=True, rank_zero_only=True, batch_size=1)
[...]
$ grep '\.log(' | grep global | grep -v float | wc -l
10
```
same happens to other counters like `consumed_samples`

Currently PTL emits a warning:

```
site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:212: UserWarning: You 
called `self.log('consumed_samples', ...)` in your `training_step` but the value needs to be floating 
point. Converting it to torch.float32.
pytorch_lightning/trainer/connectors/logger_connector/result.py:212: UserWarning: You called 
`self.log('global_step', ...)` in your `training_step` but the value needs to be floating point. 
Converting it to torch.float32.
```

Leading to both - an incorrect data type in the log and a wasteful reduction across ranks, even though all ranks should have the same counter value.

I first asked about this at PTL and their detailed answer on how to do it correctly is here:
https://github.com/Lightning-AI/lightning/issues/18739#issuecomment-1753263694

tldr: for non-reduced counters the call should be: `trainer.logger.log_metrics()` and not `self.log()` 

Thank you.
[TTS] replace default ddp with ddp_find_unused_parameters_true
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
[TTS] replace default ddp with ddp_find_unused_parameters_true
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: Speaker Tasks

name: CI

name: common

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Shinwool/upstream from bionemo utils scheduler
State: closed | Pull request: yes
Labels: name: core

name: stale

# What does this PR do ?

Adds a LR scheduler policy (LinearWarmupHoldDecayPolicy) that has following properties:
- Phase 1: linear warmup from 0 to the optimizer's learning rate
- Phase 2: constant phase
- Phase 3: linear decay to 'min_lr'
- Phase 4: constant phase at 'min_lr'
A visual representation of above is included in code.

**Collection**: N/A

# Changelog 
- Added class for LR scheduler policy (LinearWarmupHoldDecayPolicy)
- Added unit test for above

# Usage

# Before your PR is "Ready for review"
**Pre checks**:
- [v] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [v] Did you write any new necessary tests?
- [v] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [v] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Order of scheduler/optimizer calls
State: closed | Pull request: no
Labels: name: bug

name: stale

I have the same problem as in #6869 with punctuation_capitalization_train_evaluate.py

UserWarning: Detected call of lr_scheduler.step() before optimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before lr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule.
Error executing job with overrides

NeMo: 1.21.0
OS version: Ubuntu 18.04
PyTorch version: '2.1.0+cu121'
Python version: Python 3.11.5
Add publications section to website
State: closed | Pull request: yes

# What does this PR do ?

Adds a new publications section to the NeMo website

**Collection**: [Docs, Core, ASR, TTS, NLP]

# Changelog 
- Add a pseudo "blog" to incorporate all our publications
- Update some internal overrides to support useful redirects
- Make the `Continue reading` button have a right arrow plus be surrounded by a box to ensure users note where to click to go to the next page

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Add find_unused_parameters_true for text_classiftn and punctuation_capitalization
State: closed | Pull request: yes
Labels: name: cherry-pick

name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
Text memmap index file cascade structure
State: closed | Pull request: no
Labels: name: bug

name: stale

MR https://github.com/NVIDIA/NeMo/pull/6683 added functionality to avoid collisions but in the case of bionemo the shared data on daraco OCI is structured in a way where every .csv has a .idx in the same directory. We would benefit by having a flag https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/data/language_modeling/text_memmap_dataset.py#L474 where if True use the non cascade logic and the default can be set to False. This would allow bionemo to reuse its previously done indices and not introduce errors that the cascade introduction was designed to fix.

Current with Nemo 1.20
index_mapping_dir = /data/uniref202104_esm2/uf50/train 
fn = /data/uniref202104_esm2/uf50/train/x048.csv 
idx_fn = /data/uniref202104_esm2/uf50/train/data/uniref202104_esm2/uf50/train/x048.csv.idx

Desired: Rest of Bionemo / nemo 1.18
index_mapping_dir = /data/uniref202104_esm2/uf50/train 
fn = /data/uniref202104_esm2/uf50/train/x048.csv 
idx_fn = /data/uniref202104_esm2/uf50/train/x048.csv.idx


BIONEMO_IMAGE="nvcr.io/nvidian/cvai_bnmo_trng/bionemo-esm2-001:dev"
generalized chat sft prompt
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

In this PR, it genialized the chat SFT dataset that it can use customized turn start/end tokens by using chat_prompt_tokens config. e.g.
```yaml
    chat_prompt_tokens:  # special tokens for the chat prompts, a dictionary of {token_type: token}. note that some tokenizer may combine the characters at the junction between {end_of_turn}{turn_start}. e.g. '<im end><im start>', the '><' sometimes is merged to be a single token. This is not supported, try to avoid
      system_turn_start: '<extra_id_0>'
      turn_start: '<extra_id_1>'
      label_start: '<extra_id_2>'
      end_of_turn: "\x0A"  # \0x0A is '\n'
      end_of_name: "\x0A"  # \0x0A is '\n'
```
after this change, the LM is not required to have "extra_id" special tokens any more to use chat SFT dataset. In this PR, also expanded the unit test to cover more LM tokenizers.

Another feature added is to overwrite the prompt_template config with the chat prompt format.
[ASR] Fix type error in jasper
State: closed | Pull request: yes
Labels: name: cherry-pick

name: ASR

# What does this PR do ?

Currently when running VAD marblenet you sometimes randomly get the error:

```
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1514, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/nemo/collections/asr/parts/submodules/jasper.py", line 351, in forward
    self.update_masked_length(x.size(2), device=lens.device)
  File "/usr/local/lib/python3.10/dist-packages/nemo/collections/asr/parts/submodules/jasper.py", line 374, in update_masked_length
    self.lens, self.max_len = _masked_conv_init_lens(self.lens, max_len, self.max_len)
RuntimeError: _masked_conv_init_lens() Expected a value of type 'Tensor' for argument 'original_maxlen' but instead found type 'int'.
; ((Tensor, Tensor))
Cast error details: Unable to cast 64 to Tensor
terminate called without an active exception
```

This happens because in this line of code, it assigns self.max_len to an integer value:

https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/parts/submodules/jasper.py#L378

Whereas in all other code paths it assigns it to a tensor:

https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/parts/submodules/jasper.py#L321

https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/parts/submodules/jasper.py#L374
https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/parts/submodules/jasper.py#L231
https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/parts/submodules/jasper.py#L234

**Collection**: [ASR]

# Changelog 
- Convert max_len to tensor.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Fix issues with Dockerfile
State: closed | Pull request: yes
Labels: name: cherry-pick

name: CI

# What does this PR do ?

Fix an issue with Dockerfile not able to find correct value of ffmpeg.

**Collection**: [Core]

# Changelog 
- Revert version of ffmpeg
- Fix rw mode of mount

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
[TTS] Add dataset to path of logged artifacts
State: closed | Pull request: yes
Labels: name: TTS

# What does this PR do ?

Merging https://github.com/NVIDIA/NeMo/pull/7462 to r1.21.0

Modify callbacks to save artifacts into dataset specific folders, instead of saving artifacts from all datasets into 1 folder. This helps organization when training on many datasets, and also handles the case when multiple datasets have files with the same name.

**Collection**: [TTS]

# Changelog 
- Pass dataset names through TTS data loaders
- Save artifacts under epoch_{x}/{dataset_name}/
- Add initial logging for FastPitch and HiFi-GAN recipes

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation
Fix issues with Dockerfile
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Fix an issue with Dockerfile not able to find correct value of ffmpeg.

**Collection**: [Core]

# Changelog 
- Revert version of ffmpeg
- Fix rw mode of mount

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
Add find_unused_parameters_true for text_classiftn and punctuation_capitalization
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
conversion issue fix
State: closed | Pull request: yes
Labels: name: NLP

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
megatron-core was not found
State: closed | Pull request: no
Labels: name: bug

python punctuation_capitalization_train_evaluate.py
gives me
[NeMo W 2023-10-05 22:24:59 nemo_logging:393] megatron-core was not found. Please see the NeMo README for installation instructions: https://github.com/NVIDIA/NeMo#megatron-gpt.
I can't understand how can I install megatron-gpt?
There is an instruction how to install Apex. But I already have Apex.
Or maybe I need to install Transformer Engine?

Thanks in advance.
Installation from source ./reinstall.sh fails with Failed to build mecab-python3
State: closed | Pull request: no
Labels: name: bug

Ubuntu 18.04
python 3.11.5
torch 2.11.0
conda
I have installed MeCab via pip3 install mecab-python3.

      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/MeCab
      copying src/MeCab/cli.py -> build/lib.linux-x86_64-cpython-311/MeCab
      copying src/MeCab/__init__.py -> build/lib.linux-x86_64-cpython-311/MeCab
      running build_ext
      error: [Errno 2] No such file or directory: 'mecab-config'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for mecab-python3
  Running setup.py clean for mecab-python3                                                                                      
Successfully built nemo-toolkit
Failed to build mecab-python3
ERROR: Could not build wheels for mecab-python3, which is required to install pyproject.toml-based projects

Thanks in advance.