renwei2024/nemo-github-issues huggingface dataset:
where is 'get_aishell_data.py'
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

A clear and concise description of what the bug is.

**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model
port UnivNet to NeMo TTS collection
State: closed | Pull request: yes

includes 2 commits:

1. portning UnivNet based on [https://github.com/mindslab-ai/univnet](url). Also includes `scripts/dataset_processing/get_libritts_data.py` for LibriTTS training.
2. Currently HiFi-GAN from main have two `output_types`, where one should be `input_types`.
NeMo cache model files in hard-coded location without using environment variables
State: closed | Pull request: no

https://github.com/NVIDIA/NeMo/blob/e987e178ecb42f1e8411c59e80628cd67af03f79/nemo/core/classes/common.py#L722

NeMo store and load models from hard-coded folder (Path.home()/.cache/torch/NeMo/)

Is it possible to change it to a location dependent on some environment variable, similar to a way it is done for torch.hub (https://pytorch.org/docs/stable/hub.html#where-are-my-downloaded-models-saved) or transformers (https://huggingface.co/transformers/installation.html?highlight=transformers_cache#caching-models)?
Error when downloading the Persian data set of Mozilla Common Voice
State: closed | Pull request: no
Labels: name: bug

Hi,
Thanks for your great repo. I want to download the Mozilla Common Voice (MCV) **Persian** data set via the following code:
[get_commonvoice_data](https://github.com/NVIDIA/NeMo/blob/main/scripts/dataset_processing/get_commonvoice_data.py)
I have used below parameters:
`VERSION = "cv-corpus-7.0-2021-07-21"`
`LANGUAGE = "fa"`
but I got the following error:
```
  File "scripts/get_commonvoice_data.py", line 121, in process_files
    with open(csv_file) as csvfile:
FileNotFoundError: [Errno 2] No such file or directory: 'datasets/fa/CV_unpacked/cv-corpus-7.0-2021-07-21/fa/train.tsv'
```
In addition, I have read the above from the below tutorial:
[ASR_CTC_Language_Finetuning](https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/asr/ASR_CTC_Language_Finetuning.ipynb#scrollTo=VDDiXCiPimr_)  
would you please help me to address this problem?
Thanks
Issues with saving checkpoints
State: closed | Pull request: no
Labels: name: bug

Hi! I'm trying to train tacotron 2 from scratch.

Running in current main branch with pytorch-lightning==1.5.0
```
PYTHONPATH="$(pwd)" python examples/tts/tacotron2.py \
train_dataset=train_data.json \
validation_datasets=validation_data.json \
+exp_manager.checkpoint_callback_params.save_top_k=1 \
+exp_manager.checkpoint_callback_params.monitor=loss
```

Results in correct training without checkpointing. Downgrading to pytorch-lightning==1.4.2 fixes this issue.
LJSpeech alignment scripts fixed for latest MFA
State: closed | Pull request: yes

The latest releases of MFA breaks extract_ljspeech_phonemes_and_durs.sh

Example:
```
~/NeMo/scripts/dataset_processing/ljspeech$ mfa thirdparty
usage: mfa [-h]
           {version,align,adapt,train,validate,g2p,train_g2p,model,train_lm,train_dictionary,train_ivector,classify_speakers,create_segments,transcribe,configure,history,annotator,anchor}
           ...
mfa: error: argument subcommand: invalid choice: 'thirdparty' (choose from 'version', 'align', 'adapt', 'train', 'validate', 'g2p', 'train_g2p', 'model', 'train_lm', 'train_dictionary', 'train_ivector', 'classify_speakers', 'create_segments', 'transcribe', 'configure', 'history', 'annotator', 'anchor')

~/NeMo/scripts/dataset_processing/ljspeech$ mfa download acoustic english
usage: mfa [-h]
           {version,align,adapt,train,validate,g2p,train_g2p,model,train_lm,train_dictionary,train_ivector,classify_speakers,create_segments,transcribe,configure,history,annotator,anchor}
           ...
mfa: error: argument subcommand: invalid choice: 'download' (choose from 'version', 'align', 'adapt', 'train', 'validate', 'g2p', 'train_g2p', 'model', 'train_lm', 'train_dictionary', 'train_ivector', 'classify_speakers', 'create_segments', 'transcribe', 'configure', 'history', 'annotator', 'anchor')
``` 

Latest version uses conda-forge and also changed "mfa download" to "mfa model download".
Described in 
https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner/issues/349#issuecomment-966610658

This is related to TTS (@blisc?)
_torchtext.so: undefined symbol: _ZTVN5torch3jit6MethodE
State: closed | Pull request: no
Labels: name: bug

Сan't import nemo.collections.asr in google colab notebook https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/asr/ASR_with_NeMo.ipynb. 
This worked on 11/11/2021, starting today (12/11/2021), the error appears:
ImportError: /usr/local/lib/python3.7/dist-packages/torchtext/_torchtext.so: undefined symbol: _ZTVN5torch3jit6MethodE
remove cfg in _load_model_state in nlp_models
State: closed | Pull request: yes

We patched cfg issue in modelPT level in https://github.com/NVIDIA/NeMo/pull/3170.
Revert the cfg arg in `_load_model_state` in nlp_models in https://github.com/NVIDIA/NeMo/pull/3149 since it will break the fix and is deprecated in PTL. 

Signed-off-by: fayejf <fayejf07@gmail.com>
Fix cfg issue for r1.5.0
State: closed | Pull request: yes

Fix issues with config saving TypeError: __init__() missing 1 required positional argument: 'cfg' when trying to do model.load_from_checkpoint(XYZ)

related to https://github.com/NVIDIA/NeMo/pull/3058 and https://github.com/PyTorchLightning/pytorch-lightning/discussions/7525
Fix bug with max_steps in modelPT.py according to PTL 1.5.0
State: closed | Pull request: yes

Fix bug in modelPT.py according to this PR in PTL 1.5.0: 
https://github.com/PyTorchLightning/pytorch-lightning/commit/0e20119d24c410c9004a9d0c6ce0505914c1dc02

Signed-off-by: Oktai Tatanov <oktai.tatanov@gmail.com>
nemov1.4.0 : RuntimeError: Error(s) in loading state_dict for EncDecCTCModelBPE: 	size mismatch for preprocessor.featurizer.window: copying a param with shape torch.Size([40]) from checkpoint, the shape in current model is torch.Size([400]).
State: closed | Pull request: no

I am training a conformer model (size medium)  on hindi dataset of about 501 hrs after training for few epoch I am getting ?? in the predicted transcription. When I am trying to load checkpoint for inference I am getting size mismatch for preprocessor error can someone help or any insights to help with this?
<img width="1155" alt="Screen Shot 2021-11-11 at 6 36 03 PM" src="https://user-images.githubusercontent.com/80812299/141303234-f923f8e5-0d4e-462e-a7d0-ec00376a0f14.png">
Notebook bugfixes
State: closed | Pull request: yes

Fixes notebook bugs for r1.5.0 release:

Bug 3415701
https://nvbugswb.nvidia.com/NVBugs5/redir.aspx?url=/3415701 [ML-NeMo][Tesla A30][NLP] Zero_Shot_Intent_Recognition.ipynb failed with FileNotFoundError: [Errno 2] No such file or directory:


Bug 3415458
https://nvbugswb.nvidia.com/NVBugs5/redir.aspx?url=/3415458 [ML-NeMo][Tesla A30][NLP] Relation_Extraction-BioMegatron.ipynb failed with ValueError: megatron-lm BERT models have been deprecated in NeMo 1.5+


Bug 3415486
https://nvbugswb.nvidia.com/NVBugs5/redir.aspx?url=/3415486 [ML-NeMo][Tesla A30][NLP] Token_Classification-BioMegatron.ipynb failed with ValueError: megatron-lm BERT models have been deprecated in NeMo 1.5+


Bug 3415335
https://nvbugswb.nvidia.com/NVBugs5/redir.aspx?url=/3415335 [ML-NeMo][Tesla A30][NLP] Entity_Linking_Medical.ipynb failed with MisconfigurationException: You have asked for `amp_level='O1'` which is not supported with `amp_backend='native'`.
Improve data pipeline for punctuation capitalization model and make other useful changes
State: closed | Pull request: yes

1. `PunctuationCapitalizationDataset`
  - Sort input sequences before packing them into batches to reduce padding
  -  Batching is done in `PunctuationCapitalizationDataset`, not in `DataLoader`
  -  Parallelize features creation
  -  Exclude `'input_mask'`, `'loss_mask'`, `'segment_ids'` from dataset cache. Now these arrays are created in dataset constructor
  - `PunctuationCapitalizationDataset` now yields dictionaries, not tuples
2. Tarred dataset class and instruments for tarred dataset creation.
3. Classes for punctuation and capitalization configs
4. Now parameters `max_seq_length`, `use_cache`, `num_workers`, `pin_memory`, `drop_last` can be passed in `train_ds`, `validation_ds`, `test_ds` and in these sections they have higher priority.
5. Add separate metrics for every evaluation dataset.

Closes #3125
Streaming inference for Conformer Transducer ASR model
State: closed | Pull request: no

Hi. Im want to developing a streaming interface for my trainned Conformer Transducer ASR model. I have only see tutorials about  streaming for CTC model here:
https://github.com/NVIDIA/NeMo/blob/stable/tutorials/asr/Online_ASR_Microphone_Demo.ipynb
https://github.com/NVIDIA/NeMo/blob/stable/tutorials/asr/Streaming_ASR.ipynb
My question is can i apply this tutorial for Transducer model ? I do not know it is efficient or not to use this method to streaming with transducer model. If its not, can someone give me a direction ? Thank you (I already know about riva and tensorrt for production, but I just want developing the streaming method on python for now)
index out of dimension in sd_model.diarize() "Speaker_Diarization_Inference"
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I'm using "Speaker Diarization Inference" but 'vad_marblenet' wasn't enough, I decided to do the Fine-tune training on the vad model as in the example of "Speech_Commands.ipynb" which is mentioned at the end of the Voice_Activity_Detection notebook.
After the model's Fine-tune I tested it, and the following error came back


```
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/tmp/ipykernel_704/3722763614.py in <module>
----> 1 sd_model.diarize()

/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/clustering_diarizer.py in diarize(self, paths2audio_files, batch_size)
    352 
    353             self._setup_vad_test_data(manifest_vad_input)
--> 354             self._run_vad(manifest_vad_input)
    355         else:
    356             if not os.path.exists(self._speaker_manifest_path):

/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/clustering_diarizer.py in _run_vad(self, manifest_file)
    201                 log_probs = self._vad_model(input_signal=test_batch[0], input_signal_length=test_batch[1])
    202                 probs = torch.softmax(log_probs, dim=-1)
--> 203                 pred = probs[:, 1]
    204                 if status[i] == 'start':
    205                     to_save = pred[:-trunc]

IndexError: index 1 is out of bounds for dimension 1 with size 1
```


**Steps/Code to reproduce bug**

```
pretrained_vad = 'vad_marblenet_Fine_Tune.nemo'
pretrained_speaker_model = 'ecapa_tdnn'
```
default config.diarizer cell values

**Expected behavior**

Execution of VAD and embendding calculations

**Environment overview (please complete the following information)**

 - Environment location: [Docker]

**Environment details**

NVIDIA docker image.
Cannot unset both `source_processor` and `target_processor` for NMT model inference.
State: closed | Pull request: yes
Labels: name: bug

If I omit both parameters `--target_lang` and `--source_lang` when running script `nmt_transformer_infer.py`, then both `target_processor` and `source_processor` attributes of model `MTEncDecModel` are set as during the model training.

This behaviour appears because 
1) `source_processor` and `target_processor` are set while [restoring from checkpoint](https://github.com/NVIDIA/NeMo/blob/a089791db1d4be51da2cd7cdec2eb822ca6b588c/examples/nlp/machine_translation/nmt_transformer_infer.py#L154),
2) `source_processor` and `target_processor` are [reset](https://github.com/NVIDIA/NeMo/blob/a089791db1d4be51da2cd7cdec2eb822ca6b588c/nemo/collections/nlp/models/machine_translation/mt_enc_dec_model.py#L858) only if at least one of parameters `--source_lang`, `--target_lang` are provided.

To produce the problem you need
1) find `.nemo` file for `MTEncDecModel` model which was trained with source or target processor,
2) add some debug logging [here](https://github.com/NVIDIA/NeMo/blob/a089791db1d4be51da2cd7cdec2eb822ca6b588c/nemo/collections/nlp/models/machine_translation/mt_enc_dec_model.py#L763) to see if processing is performed,
3) launch script `nmt_transformer_infer.py` without parameters `--source_lang` and `--target_lang`.
Unable to run the notebook on Google Colab
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Cannot run the notebook 

**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.
Try running this notebook: https://colab.research.google.com/drive/1-QmFrSHsnirtnu6gB0Tx-JQAXWDtm1yE?usp=sharing

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model
a quick fix for issue #3094 index out-of-bound when truncating long text to max_seq_length
State: closed | Pull request: yes

@okuchaiev

This is a PR for fixing issue #3094 where the index will be out-of-bound when truncating long text ext to max_seq_length in text_classification_dataset.py

Signed-off-by: bugface <alexgre@ufl.edu>
pyt21.11 compatibility: (1) enable nvfuser for bf16 (2) change of api…
State: closed | Pull request: yes

The current NeMo code will crash at PyTorch21.11. 
This update will crash at PyTorch21.10 so should be made sure this is packaged with PyTorch21.11.

Signed-off-by: Sangkug Lym <slym@nvidia.com>
Punctuation data set uses too much memory
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Punctuation datasets cannot be constructed with more than a few million examples (depending on the max length). Even these small datasets can consume a huge amount of memory while preprocessing (100's of GB) and while training (10's of GB per process). 

This issue was mentioned in passing in https://github.com/NVIDIA/NeMo/issues/2569 though it was not the main issue discussed there and thus not fixed.

Assuming it is an acceptable idea to load the entire dataset into memory in the first place, these are the points where there seems to be an excessive use of memory.

The function with issues (`get_features`):
https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/data/token_classification/punctuation_capitalization_dataset.py#L34 

The main issues:
1. Why save `all_segment_ids`? It appears there is nowhere this is set to anything but a constant 0. `__getitem__` can deal with it.
2. Why pad to the max length? `collate_fn` can deal with that, and could do so much more efficiently (by only padding to the max length of the batch).
3. Loss mask and input mask can be generated by `__getitem__` or `collate_fn`
4. Subtokens mask could be generated later as well, albeit not trivially.


**Expected behavior**

Should be able to use a dataset with 10's of millions of lines.
Inconsistent model inference time
State: closed | Pull request: no

Recently, I want to measure the inference time, I design two cases, 
First I read model input from files, 
Second, each model input from the standard input.

While I found first case inference time is about 70 ms except the first call, the second case starts to be highly inconsistent, ranging from 72 ms to 483 ms. 

Why this happened ? How can I make the second case inference time keep consistent ?

```python
model = TokenClassificationModel.restore_from(restore_path=checkpoint_path)
# first case code
for line in open(txt_file, 'r'):
    torch.cuda.synchronize()
    start = time.perf_counter()
    result = model.infer_one_text(text)
    torch.cuda.synchronize()
    elapsed_time = time.perf_counter() - start
    print(f'elapsed_time:{elapsed_time:.6f}', flush=True)

# first case log
elapsed time:0.289745
elapsed time:0.072184
elapsed time:0.060471
elapsed time:0.062084
elapsed time:0.063837
```

```python
# second case code
model = TokenClassificationModel.restore_from(restore_path=checkpoint_path)
while True:
    text = input("text:")
    torch.cuda.synchronize()
    start = time.perf_counter()
    result = model.infer_one_text(text)
    torch.cuda.synchronize()
    end = time.perf_counter() - start
    print(f'elapsed time: {end:.6f}', flush=True)

# second case log
elapsed time:0.483897
elapsed time:0.089466
elapsed time:0.450116
elapsed time:0.072568
elapsed time:0.469203
```
ImportError: cannot import name 'QConfig' from 'torch.ao.quantization'
State: closed | Pull request: no
Labels: name: bug

**Bug description**

Cannot import `asr` from `nemo.collections`

**Steps/Code to reproduce bug**
```bash
# Fresh install
git clone https://github.com/NVIDIA/NeMo.git
cd NeMo/
conda create -n nemo python=3.8
conda activate nemo
./reinstall.sh 

# Then
python

# in the python terminal:
from nemo.collections import asr as nemo_asr
```

Log:
```bash
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/Projects/NeMo/nemo/collections/asr/__init__.py", line 15, in <module>
    from nemo.collections.asr import data, losses, models, modules
  File "/home/user/Projects/NeMo/nemo/collections/asr/losses/__init__.py", line 15, in <module>
    from nemo.collections.asr.losses.angularloss import AngularSoftmaxLoss
  File "/home/user/Projects/NeMo/nemo/collections/asr/losses/angularloss.py", line 18, in <module>
    from nemo.core.classes import Loss, Typing, typecheck
  File "/home/user/Projects/NeMo/nemo/core/__init__.py", line 16, in <module>
    from nemo.core.classes import *
  File "/home/user/Projects/NeMo/nemo/core/classes/__init__.py", line 16, in <module>
    from nemo.core.classes.common import (
  File "/home/user/Projects/NeMo/nemo/core/classes/common.py", line 32, in <module>
    from nemo.utils import logging
  File "/home/user/Projects/NeMo/nemo/utils/__init__.py", line 22, in <module>
    from nemo.utils.lightning_logger_patch import add_memory_handlers_to_pl_logger
  File "/home/user/Projects/NeMo/nemo/utils/lightning_logger_patch.py", line 18, in <module>
    import pytorch_lightning as pl
  File "/home/user/Tools/miniconda3/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/__init__.py", line 20, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
  File "/home/user/Tools/miniconda3/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/callbacks/__init__.py", line 26, in <module>
    from pytorch_lightning.callbacks.quantization import QuantizationAwareTraining
  File "/home/user/Tools/miniconda3/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/callbacks/quantization.py", line 30, in <module>
    from torch.ao.quantization import QConfig
ImportError: cannot import name 'QConfig' from 'torch.ao.quantization' (/home/user/Tools/miniconda3/envs/nemo/lib/python3.8/site-packages/torch/ao/quantization/__init__.py)

```

**Expected behavior**
Expected: being able to import without error

**Environment overview (please complete the following information)**
- installed with `reinstall.sh`

- also tried to reinstall different version of pytorch:
`conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch`
and getting exactly the same error
 
**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS: ubuntu 20.04
- pytorch version (default from reinstall.sh) 1.10.0+cu102
- python version: 3.8.12
- nvidia driver: 470.82.00
Multiple ASR Fixes to SPE tokenization
State: closed | Pull request: yes

# Changelog
- Fix issue with transcribe building more workers than samples
- Fix SPE tokenizer vocab being built incorrectly for tokenizers with BOS EOS and PAD
- Update tokenizer script to designate tokenizer dir with BOS EOS and PAD token.

# Breaking change
- SPE tokenizer based models, when requested for their vocab via model.tokenizer.tokenizer.get_vocab() will return special tokens + order of tokens will be different than previous result.
Enable param groups for ModelPT optimizer
State: closed | Pull request: yes

Adds `.setup_optimizer_param_groups` method to `ModelPT`. The `.setup_optimization` method will give optimizers `self._optimizer_param_groups` instead of `self.parameters()`.

The `.setup_optimizer_param_groups` method will by default set `self._optimizer_param_groups=self.parameters()`.

This enables NeMo models to specify custom param groups so that, for example, per layer learning rates can be used.

In particular, this enables Megatron-LM style weight decay where LayerNorm and Bias weights are not decayed.
Training of machine translation model fails if config parameter `trainer.max_epochs` is used instead of `trainer.max_steps`.
State: closed | Pull request: yes
Labels: name: bug

Fixes error which occurs if use `trainer.max_epochs` instead of `trainer.max_steps`. Got this error with `enc_dec_nmt.py` script.
```
Traceback (most recent call last):
  File "enc_dec_nmt.py", line 133, in main
    trainer.fit(mt_model)
  File "/home/apeganov/anaconda3/envs/megatron/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 640, in fit
    self._call_and_handle_interrupt(self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule)
  File "/home/apeganov/anaconda3/envs/megatron/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 592, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/apeganov/anaconda3/envs/megatron/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 670, in _fit_impl
    self._run(model)
  File "/home/apeganov/anaconda3/envs/megatron/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1049, in _run
    self.accelerator.setup(self)
  File "/home/apeganov/anaconda3/envs/megatron/lib/python3.8/site-packages/pytorch_lightning/accelerators/gpu.py", line 46, in setup
    return super().setup(trainer)
  File "/home/apeganov/anaconda3/envs/megatron/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 92, in setup
    self.setup_optimizers(trainer)
  File "/home/apeganov/anaconda3/envs/megatron/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 364, in setup_optimizers
    optimizers, lr_schedulers, optimizer_frequencies = self.training_type_plugin.init_optimizers(
  File "/home/apeganov/anaconda3/envs/megatron/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 250, in init_optimizers
    return trainer.init_optimizers(model)
  File "/home/apeganov/anaconda3/envs/megatron/lib/python3.8/site-packages/pytorch_lightning/trainer/optimizers.py", line 35, in init_optimizers
    optim_conf = self.call_hook("configure_optimizers", pl_module=pl_module)
  File "/home/apeganov/anaconda3/envs/megatron/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1398, in call_hook
    output = model_fx(*args, **kwargs)
  File "/home/apeganov/NeMo/nemo/core/classes/modelPT.py", line 575, in configure_optimizers
    self.setup_optimization()
  File "/home/apeganov/NeMo/nemo/core/classes/modelPT.py", line 463, in setup_optimization
    if self._trainer.distributed_backend is None:
AttributeError: 'Trainer' object has no attribute 'distributed_backend'
```
Training of machine translation model fails if config parameter `trainer.max_epochs` is used instead of `trainer.max_steps`.
State: closed | Pull request: yes
Labels: name: bug

Replace `trainer.distributed_backend` with `trainer.accelerator` everywhere in the repo, because `distributed_backend` is [renamed ](https://pytorch-lightning.readthedocs.io/en/latest/generated/CHANGELOG.html?highlight=distributed_backend#id119) into `accelerator` in `pytorch-lightning==1.0.5`.

Closes #3104
Modify speaker input
State: closed | Pull request: yes

This PR brings a major revamp to speaker diarization inference, with following are notable changes:
* Input to offline_diarization.py and offline_diarization_with_asr.py expects manifest file ( Now no longer need to create individual files for audio paths, rttms, oracle speakers and uem filepaths)
* With oracle_vad option, CLusteringDiarizer automatically computes oracle_manifest without being fed as the separate manifest file
* Updated yaml file and is now more clear with many options for VAD, Speaker Embeddings and Clustering 
* Separated clustering and scoring modules
* Improved pyannote usage with uris passed as uniq_key values
* Updated Speaker_Diairization_Inference.ipynb and ASR_with_Speaker_Diarization.ipynb 
* Removed duplicate functions from diarization utils and avoided recalculation of diarization score 
* Added Jenkins Test for diarization with asr 
* Updated README and DocStrings
Attribute `batch_dim_index` is not working in `WER`.
State: closed | Pull request: yes
Labels: name: bug

Currently `WER` works incorrectly if `batch_dim_index=1`.

The outputs for `batch_dim_index=1` and `batch_dim_index=0` in the following example have to be equal.
Code to reproduce the error:
```python
from string import ascii_lowercase
from typing import List

import torch

from nemo.collections.asr.metrics.wer import WER


def string_to_ctc_tensor(txt: str, vocabulary: List[str]) -> torch.Tensor:
    # This function emulates how CTC output could like for txt
    blank_id = len(vocabulary)
    char_to_ind = dict([(vocabulary[i], i) for i in range(len(vocabulary))])
    string_in_id_form = [char_to_ind[c] for c in txt]
    ctc_list = []
    prev_id = -1
    for c in string_in_id_form:
        # when character is repeated we need to insert CTC blank symbol
        if c != prev_id:
            ctc_list.append(c)
        else:
            ctc_list.append(blank_id)
            ctc_list.append(c)
        prev_id = c
    return torch.Tensor(ctc_list).unsqueeze(0)


def reference_string_to_tensor(txt: str, vocabulary: List[str]) -> torch.Tensor:
    # Reference tensors aren't produced by CTC logic
    char_to_ind = dict([(vocabulary[i], i) for i in range(len(vocabulary))])
    string_in_id_form = [char_to_ind[c] for c in txt]
    return torch.Tensor(string_in_id_form).unsqueeze(0)


vocabulary = [" "] + list(ascii_lowercase) + ["'"]
wer = WER(vocabulary=vocabulary, batch_dim_index=1, use_cer=False, ctc_decode=True)

prediction, reference = ['a b c', 'a c c']
predictions_tensor = string_to_ctc_tensor(prediction, vocabulary)
targets_tensor = reference_string_to_tensor(reference, vocabulary)
targets_tensor.transpose_(0, 1)
predictions_tensor.transpose_(0, 1)
wer(predictions=predictions_tensor, targets=targets_tensor, target_lengths=torch.tensor([len(reference)]))
print("WER for batch_dim_index=1:", wer.compute()[0])

vocabulary = [" "] + list(ascii_lowercase) + ["'"]
wer = WER(vocabulary=vocabulary, batch_dim_index=0, use_cer=False, ctc_decode=True)

prediction, reference = ['a b c', 'a c c']
predictions_tensor = string_to_ctc_tensor(prediction, vocabulary)
targets_tensor = reference_string_to_tensor(reference, vocabulary)
wer(predictions=predictions_tensor, targets=targets_tensor, target_lengths=torch.tensor([len(reference)]))
print("WER for batch_dim_index=0:", wer.compute()[0])
```
Output:
```
[NeMo I 2021-10-31 02:33:21 wer:225] 
    
[NeMo I 2021-10-31 02:33:21 wer:226] reference:a
[NeMo I 2021-10-31 02:33:21 wer:227] predicted:a
WER for batch_dim_index=1: tensor(0.)
[NeMo I 2021-10-31 02:33:21 wer:225] 
    
[NeMo I 2021-10-31 02:33:21 wer:226] reference:a c c
[NeMo I 2021-10-31 02:33:21 wer:227] predicted:a b c
WER for batch_dim_index=0: tensor(0.3333)
```
Also updated docstrings.
NMT checkpoint averaging
State: closed | Pull request: yes

This PR adds convenience scripts to checkpoint average any `.nemo` file (regardless of the class).
Python script is heavily based on code by @MaximumEntropy and @titu1994.

# Usage

The script `checkpoint_averaging.py` builds a `.nemo` file with average weights over multiple `.ckpt` files (assumes `.ckpt` files in same folder as `.nemo` file).
The input is a list of folders that contains a .nemo file (or a list of .nemo files).
The output of the script (per input folder / `.nemo` file) is a file named `*-averaged.nemo` where `*.nemo` is the original `.nemo` file.

# Example

Usage example for building `*-averaged.nemo` for a given `.nemo` file:

```
NeMo/scripts/checkpoint_averaging/checkpoint_averaging.py my_model.nemo
```

Usage example for building *-averaged.nemo files for all results in sub-directories under current path:

```
find . -name '*.nemo' | grep -v -- "-averaged.nemo" | xargs NeMo/scripts/checkpoint_averaging/checkpoint_averaging.py
```
Method in text_classification/text_classification_dataset.py cannot truncate long text to right length
State: closed | Pull request: no
Labels: name: bug

in text_classification/text_classification_dataset.py

```python
if max_seq_length > 0 and len(sent_subtokens) + 1 > max_seq_length:
    sent_subtokens = sent_subtokens[:max_seq_length]
    too_long_count += 1

sent_subtokens.append(tokenizer.sep_token)
sent_lengths.append(len(sent_subtokens))
```

The line 2 in the above code from /text_classification_dataset.py should be
```python
sent_subtokens = sent_subtokens[:max_seq_length-1]
```

otherwise, after append sep token, the total token length will be longer than max and cause the following error:
```
/tmp/pip-req-build-g2m34a_4/aten/src/ATen/native/cuda/Indexing.cu:690: indexSelectLargeIndex: block: [647,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
```
pytorch-lightning version is specified wrong at requirements_lightning.txt 
State: closed | Pull request: no
Labels: name: bug

requirements_lightning.txt requires pytorch-lightning version to be higher than 1.4.9 though it's the latest stable version and it's not possible. Probably meant pytorch-lightning>=1.4.9 instead
![image](https://user-images.githubusercontent.com/17128248/139324215-dc3a360c-de70-4471-89ec-96521222891b.png)
![image](https://user-images.githubusercontent.com/17128248/139324329-abaf315d-191c-43db-b33a-7c1b2dfed0d5.png)
Error generating graphs using plot
State: closed | Pull request: no
Labels: name: bug

Hey guys,
I'm having this problem with some audios when I try to generate your graphs using the plot in the script "Speaker_Diarization_Inference"

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-89-9cae07b448e9> in <module>()
      6     'outputs/vad_outputs/overlap_smoothing_output_median_0.875/Audio_Caio_Dias16K.median',
      7     path2groundtruth_rttm_files[0],
----> 8     per_args = config.diarizer.vad.postprocessing_params, #threshold
      9     ) 
     10 

3 frames
/usr/local/lib/python3.7/dist-packages/nemo/collections/asr/parts/utils/vad_utils.py in plot(path2audio_file, path2_vad_pred, path2ground_truth_label, offset, duration, threshold, per_args)
    749     if path2ground_truth_label:
    750         label = extract_labels(path2ground_truth_label, time)
--> 751         ax2.plot(np.arange(len_pred) * FRAME_LEN, label, 'r', label='label')
    752 
    753     ax2.plot(np.arange(len_pred) * FRAME_LEN, pred, 'b', label='pred')

/usr/local/lib/python3.7/dist-packages/matplotlib/axes/_axes.py in plot(self, scalex, scaley, data, *args, **kwargs)
   1645         """
   1646         kwargs = cbook.normalize_kwargs(kwargs, mlines.Line2D)
-> 1647         lines = [*self._get_lines(*args, data=data, **kwargs)]
   1648         for line in lines:
   1649             self.add_line(line)

/usr/local/lib/python3.7/dist-packages/matplotlib/axes/_base.py in __call__(self, *args, **kwargs)
    214                 this += args[0],
    215                 args = args[1:]
--> 216             yield from self._plot_args(this, kwargs)
    217 
    218     def get_next_color(self):

/usr/local/lib/python3.7/dist-packages/matplotlib/axes/_base.py in _plot_args(self, tup, kwargs)
    340 
    341         if x.shape[0] != y.shape[0]:
--> 342             raise ValueError(f"x and y must have same first dimension, but "
    343                              f"have shapes {x.shape} and {y.shape}")
    344         if x.ndim > 2 or y.ndim > 2:

ValueError: x and y must have same first dimension, but have shapes (811,) and (812,)
```

I'm using the default settings and already tried to run changing some parameters

All audio samples I use are at 16000 Hz
in .wav format

This is a sample in this format that had this problem
[Audio_Caio_Dias.zip](https://github.com/NVIDIA/NeMo/files/7436513/Audio_Caio_Dias.zip)
eval_beamsearch_ngram.py not working
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

eval_beamsearch_ngram.py not working because logprobs argument is not defined in rnnt_model.transcribe
**Steps/Code to reproduce bug**

run eval_beamsearch_ngram.py as described in  https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/asr_language_modeling.html#evaluate-by-beam-search-decoding-and-n-gram-lm

**Expected behavior**

If I remove logprobs argument, it produces string and makes an error because it is fed to softmax.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model
AssertionError: args is already initialized
State: closed | Pull request: no

Hi Team,
I am using biomegatron-bert-345m-uncased for training my nemo model, after setting the config.model.language_model.pretrained_model_name ="biomegatron-bert-345m-uncased" in the config file.

When I  am trying to restore the model .nemo model using the command 
model = TextClassificationModel.restore_from(restore_path=config.cfg['model_path']['NEMO_BIO_MODEL']) it is throwing below error as shown in the image as AssertionError: args is already initialized.
![args error](https://user-images.githubusercontent.com/45452951/139025458-00685d52-e4e4-4b27-8551-b6d7b09a1b5d.PNG)

I have tried initializing config file parameters, also tried with suggestions made in this blog: https://githubmemory.com/repo/NVIDIA/NeMo/issues/2222 but still getting issue when I am trying to restore the model but so far no use

Waiting for your kind response
Fails when load 'ecapa_tdnn' model configs
State: closed | Pull request: no
Labels: name: bug

Hi everyone, i am trying train ecapa_tdnn model but can't load model with [ecapa_tdnn.yaml config](https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_tasks/recognition/conf/ecapa_tdnn.yaml)

This is error log:
```
Traceback (most recent call last):
  File "speaker_veri_training.py", line 61, in <module>
    main()
  File "speaker_veri_training.py", line 57, in main
    model = nemo_asr.models.ExtractSpeakerEmbeddingsModel(cfg=config.model, trainer=trainer)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/label_models.py", line 380, in __init__
    super().__init__(cfg=cfg, trainer=trainer)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/label_models.py", line 92, in __init__
    super().__init__(cfg=cfg, trainer=trainer)
  File "/opt/conda/lib/python3.8/site-packages/nemo/core/classes/modelPT.py", line 120, in __init__
    self.setup_training_data(self._cfg.train_ds)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/label_models.py", line 152, in setup_training_data
    self._train_dl = self.__setup_dataloader_from_config(config=train_data_layer_config)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/label_models.py", line 109, in __setup_dataloader_from_config
    augmentor = process_augmentations(config['augmentor'])
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/parts/preprocessing/perturb.py", line 832, in process_augmentations
    augmentation = perturbation_types[augment_name](**augment_kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/parts/preprocessing/perturb.py", line 401, in __init__
    self._manifest = collections.ASRAudioText(manifest_path, parser=parsers.make_parser([]), index_by_file_id=True)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/common/parts/preprocessing/collections.py", line 193, in __init__
    for item in manifest.item_iter(manifests_files):
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/common/parts/preprocessing/manifest.py", line 68, in item_iter
    for manifest_file in manifests_files:
TypeError: 'NoneType' object is not iterable
```

The manifest file is true and still working with [SpeakerNet config](https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_tasks/recognition/conf/SpeakerNet_recognition_3x2x512.yaml) but got error when using ecapa_tdnn config.

I am using docker container nemo:1.4 what download from ngc
GPU: DGX
vocab_size in tokenizer for Citrinet
State: closed | Pull request: no

Hi,
I used **stt_en_citrinet_1024** as a based model for fine-tuning with the Persian audio dataset. In your idea what is the best tokenizer for this? (**wpe** or **spe**). if **spe** is good what is the better type? (**bpe**, **unigram**, **char**, **word**). Also, I want to know about the vocab_size. What is the maximum number of vocab_size for tokenizer? If the vocab_size is increased, so the WER decreased?
Thanks

**Environment overview**

    Environment location: Google Colab
    Method of NeMo install: !pip install nemo_toolkit[asr]
    NeMo version: r1.2.0
    Learning Rate: 1e-2

**Environment details**

    OS version : "Ubuntu20.04.3 LTS"
    PyTorch version : "1.9.1"
VAD Data Set
State: closed | Pull request: no
Labels: name: feature

Hey guys

I've been tinkering with the "Voice_Activity_Detection.ipynb" script, and I've had some questions about mounting the data set.

The first, is it really necessary to use human voices speaking a specific word?
And would I have to keep feeding this base by adding new voices speaking other words?

Would I get a good result just doing this training with noise audios?
Add PUBLICATIONS.md
State: closed | Pull request: yes

# Changelog
- Adds publications.md to include research articles that utilize NeMo.

# Format 

* Years are sorted in descending order, articles within the same year are sorted by month in ascending order. 
* Articles may contain Arxiv submissions, but reviewed articles are preferred. 
* In cases where Arxiv articles are accepted at a venue, the month/year of the **accepted paper** should be used to assign its position on the list (not its arxiv submission date).

```

# <Domain>

## (Optional) <Subdomain>

<details>
  <summary>YEAR</summary>
  
  * [Name of article](Link to Article)
  
</details>

--------

```

Signed-off-by: smajumdar <titu1994@gmail.com>
Beam Search Decoder with LM (10 gram)
State: closed | Pull request: no
Labels: name: bug

Hi, I am using 10 gram LM (from kenlm). But when I use BeamSearchDecoderLM, I am getting this error:


```
terminate called after throwing an instance of 'lm::FormatLoadException'
  what():  kenlm/lm/vocab.cc:40 in void lm::ngram::{anonymous}::ReadWords(int, lm::EnumerateVocab*, lm::WordIndex, uint64_t) threw FormatLoadException because `memcmp(check_unk, "<unk>", 10)'.
Vocabulary words are in the wrong place.  This could be because the binary file was built with stale gcc and old kenlm.  Stale gcc, including the gcc distributed with RedHat and OS X, has a bug that ignores pragma pack for template-dependent types.  New kenlm works around this, so you'll save memory but have to rebuild any binary files using the probing data structure.
Aborted (core dumped)
```



What can be the problem? @VahidooX  do you have a any suggestions?
Perform VAD for my audio with this model
State: closed | Pull request: no
Labels: name: bug

Hello, i want to try the code for test one wav file:
python NeMo/examples/asr/vad_infer.py  --vad_model=vad_marblenet.nemo --dataset=/fullpath/to/manifest/ --out_dir='frame/demo' --time_length=0.63

but i don't know what is /fullpath/to/manifest/ , it is json file ? 
i construct one json file, The contents are as follows：
{"audio_filepath": "/source/c0/speech/data/google_dataset_v2/google_speech_recognition_v2/bird/b65f6015_nohash_0.wav", "duration": 0.63, "label": "infer", "text": "_", "offset": 0}
but get error. 
So, i want to know how to construct /fullpath/to/manifest/ for my one wav file, Thank you for your guidance!
Enable stateful decoding of RNNT over multiple transcribe calls
State: closed | Pull request: yes

# Changelog
- Enable transcribe method to store hypothesis and reuse them for stateful multi step predictions
- RNNT `transcribe` now takes `partial_hypotheses` as optional input, provided in returned hypothesis from previous transcribe call
- Add abstract `batch_concat_states` and  `batch_copy_states` support for RNNT models to abstract away state management.
- Force all greedy and beam inference to pack results into Hypothesis objects. They now contain by default the 
  - `y_sequence` : tensor form of int ids of tokens
  - `dec_state` : CPU tuple of decoder states
  - `score`: logprob score of current y_sequence
  - (optional) `alignments`: 2d dangling matrix of token alignments
 
# Example 

```python

model = EncDecRNNTModelBPE.restore_from(...)

decoding= model.cfg.decoding
decoding.strategy = 'greedy'
model.change_decoding_strategy(decoding)

# First transcription, stateless
part_hyp, _ = model.transcribe([path to audio files], batch_size=4, return_hypotheses=True)

# Second transcription, restored state
hyp, _ = model.transcribe([path to continuation of audio files], return_hypotheses=True, partial_hypothesis=part_hyp)

```
Replicate quartznet results 
State: closed | Pull request: no

Hello! 

I am training Quartznet 5x5 to do some experiments of my own, after I get the baseline. 

I want to first replicate results with WSJ. What are some hyperparameters you recommend for training with a small number of GPUs. Now I am training with 3 GPUs, with a learning rate of 0.05, batch 32 with DDP in PyTorch lighting, fprecision 16 and number of epochs 1000. The other hyperparameters are similiar to Quartznet bigger models that show in the paper. Also, can I set up a number of iterations to train for each batch that you recommend?

I would wanna know how fast I can converge to 11/12 of WER greedy in the test set for WSJ. I got already to around 24% of WER for validation using only 100 epochs, with 3 GPUs using learning rate of 0.01.      

Thanks!
Carlos
no module named 'nemo'
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Yesterday I ran the ipynb and it could work. But today, I'm not sure why it just couldn't work

**Steps/Code to reproduce bug**

I literally just ran the same codes from here https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/nlp/Question_Answering_Squad.ipynb#scrollTo=dzqD2WDFOIN-

it gave me this and during the installing, this happened, but It did not happen yesterday

![image](https://user-images.githubusercontent.com/56108692/138214509-f75d9c29-79c4-4adc-a3f8-b0d15561b152.png)

![image](https://user-images.githubusercontent.com/56108692/138214298-1fa4a1bd-dad7-42d4-afc9-c8be98a85e43.png)


**Expected behavior**

I expected the whole ipynb to run successfully which happened yesterday.

**Environment overview (please complete the following information)**

 - Environment location: [google Collab]
 - Method of NeMo install: [install from source given]. the exact command !python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]

Add any other context about the problem here.
Example: GPU model
Decoding hangs with pretrained models (conformer transducer, contextnet)
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

The decoding hangs with some pretrained models, no error, it just freezes at some point. Killing the process, re-running the same command gets the same behavior.
I have observed it with different models on different data: `stt_en_conformer_tranducer_large.nemo`, `stt_en_contextnet_256.nemo`.

**Steps/Code to reproduce bug**

```bash
python examples/asr/transcribe_speech.py \ model_path=models/downloaded/conformer_transducer/stt_en_conformer_transducer_medium.nemo \
dataset_manifest=some_data_to_transcribe.json \
batch_size=2  \
output_filename=transcription_nemo_conformer_transducer_medium.json
```
The resulting end of the log is something like the following:

```
Transcribing:   0%|          | 0/65903 [00:00<?, ?it/s][NeMo W 2021-10-20 09:26:58 patch_utils:49] torch.stft() signature has been updated for PyTorch 1.7+
    Please update PyTorch to remain compatible with later versions of NeMo.
Transcribing:  33%|███▎      | 21583/65903 [30:25<52:12, 14.15it/s]
```
and at every trial, the step where it gets stuck is the same. Looking at the data in that batch (assuming batches are read in the order of the json input file) there is nothing specific and removing some files from the json list does not help. So maybe nemo reshuffles the order of files to be decoded?

Is there a way to get more detailed logs of why the freeze may happen?

**Environment overview (please complete the following information)**

- installed with conda
- version 1.4.0

**Additional context**
GPU model: RTX 3090