renwei2024/nemo-github-issues huggingface dataset:
vad_infer.py script fails with `IndexError` if option `--dont_auto_split` is not set
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

`NeMo/examples/asr/vad_infer.py` script fails with `IndexError` if option `--dont_auto_split` is not set and at least 1 `.wav` file is long enough so that `nemo.collections.asr.parts.utils.vad_utils.prepare_manifest()` function split the `.wav` file.

**Steps/Code to reproduce bug**

```
mkdir -p ~/debug_data
wget http://i13pc106.ira.uka.de/~jniehues/IWSLT-SLT/data/eval/en-de/IWSLT-SLT.tst2019.en-de.tgz -O ~/debug_data/IWSLT-SLT.tst2019.en-de.tgz
tar xzf ~/debug_data/IWSLT-SLT.tst2019.en-de.tgz -C ~/debug_data/
cd ~/NeMo/examples/asr/
wget https://raw.githubusercontent.com/NVIDIA/NeMo/feat/asr/iwslt_audio_to_nemo_format/scripts/dataset_processing/prepare_iwslt_audio_data.py
python prepare_iwslt_audio_data.py -a ~/debug_data/IWSLT.tst2019/wavs/ -t ~/debug_data/IWSLT.tst2019/IWSLT.TED.tst2019.en-de.en.xml -o ~/debug_data/IWSLT.tst2019/manifest.json
python vad_infer.py --dataset ~/debug_data/IWSLT.tst2019/manifest.json --out_dir ~/debug_data/IWSLT.tst2019/vad --vad_model vad_marblenet
```


**Expected behavior**

No errors

**Environment overview (please complete the following information)**

 - Environment location: Bare-metal
 - Method of NeMo install: pip install nemo_toolkit[all]

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 20.04.2 LTS
- PyTorch version: 1.8.1
- Python version: 3.8.10
Error in create_tarred_transformer_lm_dataset.py
State: closed | Pull request: no
Labels: name: bug

**Bug description**

When trying to prepare data for ASR neural LM training, using the script `scripts/asr_language_modeling/neural_rescorer/create_tarred_transformer_lm_dataset.py` getting the following error (and a few preceding log lines):
```
[Parallel(n_jobs=-2)]: Done  68 out of 128 | elapsed:   10.6s remaining:    9.3s
[Parallel(n_jobs=-2)]: Done  81 out of 128 | elapsed:   11.4s remaining:    6.6s
[Parallel(n_jobs=-2)]: Done  94 out of 128 | elapsed:   12.0s remaining:    4.3s
[Parallel(n_jobs=-2)]: Done 107 out of 128 | elapsed:   12.6s remaining:    2.5s
[Parallel(n_jobs=-2)]: Done 120 out of 128 | elapsed:   13.5s remaining:    0.9s
[Parallel(n_jobs=-2)]: Done 128 out of 128 | elapsed:   14.1s finished
Traceback (most recent call last):
  File "./scripts/asr_language_modeling/neural_rescorer/create_tarred_transformer_lm_dataset.py", line 318, in <module>
    main()
  File "./scripts/asr_language_modeling/neural_rescorer/create_tarred_transformer_lm_dataset.py", line 299, in main
    chunk_paths, chunk_lens = __tokenize_text(
  File "./scripts/asr_language_modeling/neural_rescorer/create_tarred_transformer_lm_dataset.py", line 174, in __tokenize_text
    chunk = np.asarray(chunk, dtype=np.int64)
  File "/home/pe.honnet/Tools/miniconda3/envs/nemo/lib/python3.8/site-packages/numpy/core/_asarray.py", line 102, in asarray
    return array(a, dtype, copy=False, order=order)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Read 1000000 lines: : 1048576 lines [00:18, 57331.13 lines/s]
```

**Steps/Code to reproduce bug**
Command run as follows
```bash
python ./scripts/asr_language_modeling/neural_rescorer/create_tarred_transformer_lm_dataset.py \
  --text_path=training_data.txt \
  --data_root=./data/prep_nn_lm \
  --tokenizer_model=conformer_ctc_large/tokenizer/tokenizer.model \
  --tokenizer_vocab_file=bert_uncased_vocab.txt \
  --tokenizer_name='bert-base-uncased'
```
I have tried using the conformer ctc tokenizer vocab as well with the same error.

**Environment overview**

 - nemo installed with `reinstall.sh` script, on CentOS 8.0
 
**Environment details**

commit used: 034941566 (HEAD -> main, origin/main, origin/HEAD) Merge bugfixes and doc updates from 1.2.0 to main (#2533)
Speech Data Explorer filtering on text throws TypeError if an int or float is used as filter parameter
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
On the Speech Data Explorer, on the "Samples" page, using an `int` or a `float` value to filter text by results in the Flask app throwing a TypeError such as: 

```
10.2.48.115 - - [23/Jul/2021 14:40:13] "POST /_dash-update-component HTTP/1.1" 500 -
[2021-07-23 14:44:01,300] ERROR in app: Exception on /_dash-update-component [POST]
Traceback (most recent call last):
  File "/home/kathyreid/.local/lib/python3.8/site-packages/flask/app.py", line 2070, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/kathyreid/.local/lib/python3.8/site-packages/flask/app.py", line 1515, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/kathyreid/.local/lib/python3.8/site-packages/flask/app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/kathyreid/.local/lib/python3.8/site-packages/flask/app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/kathyreid/.local/lib/python3.8/site-packages/dash/dash.py", line 1096, in dispatch
    response.set_data(func(*args, outputs_list=outputs_list))
  File "/home/kathyreid/.local/lib/python3.8/site-packages/dash/dash.py", line 1017, in add_context
    output_value = func(*args, **kwargs)  # %% callback invoked %%
  File "data_explorer.py", line 565, in update_datatable
    data_view = [x for x in data_view if filter_value in str(x[col_name])]
  File "data_explorer.py", line 565, in <listcomp>
    data_view = [x for x in data_view if filter_value in str(x[col_name])]
TypeError: 'in <string>' requires string as left operand, not float
```
Here is a visual: 

![Screenshot from 2021-07-23 14-45-33](https://user-images.githubusercontent.com/114158/126738670-5bff6749-3d56-4812-8004-7b5a9a70765d.png)

**Steps/Code to reproduce bug**

* Go to [IP address that SDE is running through on Flask/samples]
* In the Filter row of the table, enter an `int` or `float` value in the `text` column
* The terminal in which Flask is running will throw the error 

**Expected behavior**

I expect SDE to filter for numbers - so that we can inspect a dataset to see if it has been properly normalized. 

**Environment overview (please complete the following information)**

* Environment, bare metal localhost running Ubuntu Linux 20.04 LTS, running Python 3.8.10
* Install method - git clone from GitHub, SDE is spawned with 
* `$ python3 data_explorer.py ../../scripts/dataset_processing/commonvoice_train_manifest.json  `

Add any other context about the problem here.

* I am evaluating datasets using SDE
Transcript test during training
State: closed | Pull request: no

Hi,

Do you still support version 0.10.1? I have a question about. I have been training a dataset in Portuguese in this version and in the results folder I see several checkpoints (decoder and encoder) being generated during training, specifically every 1000 steps.
Can I count on them to take my transcription tests? As an eventual evolution test, for example, or should I just wait for all epochs to complete and get the last decoder and encoder?

I have this question because during training I do transcription tests, but the result varies a lot. There are times that are great, there are times that are terrible.
Update requirements.txt to include tqdm and pandas as deps for SDE and plotly
State: closed | Pull request: yes

This PR adds `tqdm` as a dependency for the Speech Dataset Explorer; it fails as below without it. 

```
✔ ~/NeMo/tools/speech_data_explorer [main|✔] 
23:02 $ python3 data_explorer.py 
Traceback (most recent call last):
  File "data_explorer.py", line 37, in <module>
    import tqdm
ModuleNotFoundError: No module named 'tqdm'
✘-1 ~/NeMo/tools/speech_data_explorer [main|✔] 
23:02 $ pip install tqdm
Defaulting to user installation because normal site-packages is not writeable
Collecting tqdm
  Downloading tqdm-4.61.2-py2.py3-none-any.whl (76 kB)
     |████████████████████████████████| 76 kB 325 kB/s 
Installing collected packages: tqdm
Successfully installed tqdm-4.61.2
```
getting out of memory error for finetuning
State: closed | Pull request: no

I am finetuning existing Citrinet model (stt_en_citrinet_1024). We have around 130h of transcribed T-mobile data.
The test and evaluation sets are fixed: test(3.53h) and evaluation(5.45h). I can't use all the train data because of out of memory error. This is the type of error that I'm getting: 
`RuntimeError: CUDA out of memory. Tried to allocate 434.00 MiB (GPU 0; 39.59 GiB total capacity; 37.22 GiB already allocated; 356.19 MiB free; 37.44 GiB reserved in total by PyTorch)`
The most amount of data that I could use was (90.29h) of data (max_duration = 65s).
I am using 4gpus (40g each). I reduced the batch_size to 16 and precision to 16.
I also, used the ctc-segmentation to split long audio files and the corresponding transcripts into shorter fragments. So, instead of 90h of data I can use 66h of data most with max_duration = 66. (not that much improvement on increasing the train size)
Any suggestion about how I can increase the train data? ( we are still not covering 1/3 of data)
Error when running eval_beamsearch_ngram.py
State: closed | Pull request: no
Labels: name: bug

Hi,
I started looking into LM with Nemo. I was able to create a kenlm  model using train_kenlm.py script.
Then I tried to run eval_beamsearch_ngram.py using a fine-tuned model we trained, kenlm model, and test manifest, but got error.
Could you tell us what to do to fix this? By the way there is no log file created, and we didn’t modify those NVIDIA scripts.
Thank you so much!
 
I followed: 
https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/asr_language_modeling.html
 
python eval_beamsearch_ngram.py --nemo_model_file ../../models/model_2021-07-19_12-01-41_4GPU.nemo \
--input_manifest ../../manifests/test.jsonl \
--kenlm_model_file kenlm_model_2021-07-19_12-01-41 \
--acoustic_batch_size 32 \
--preds_output_folder output \
--decoding_mode beamsearch_ngram \
--beam_width 64 128 \
--beam_alpha 1.0 \
--beam_beta 1.0 0.5
 
Traceback (most recent call last):
  File "eval_beamsearch_ngram.py", line 345, in <module>
    main()
  File "eval_beamsearch_ngram.py", line 217, in main
    asr_model = nemo_asr.models.EncDecCTCModelBPE.restore_from(
  File "/NeMo/nemo/core/classes/modelPT.py", line 479, in restore_from
    return cls._default_restore_from(restore_path, override_config_path, map_location, strict, return_config)
  File "/NeMo/nemo/core/classes/modelPT.py", line 430, in _default_restore_from
    instance = cls.from_config_dict(config=conf)
  File "/NeMo/nemo/core/classes/common.py", line 471, in from_config_dict
    instance = cls(cfg=config)
  File "/NeMo/nemo/collections/asr/models/ctc_bpe_models.py", line 98, in __init__
    raise ValueError("`cfg` must have `tokenizer` config to create a tokenizer !")
ValueError: `cfg` must have `tokenizer` config to create a tokenizer !
Reproducing stt_en_conformer_ctc_medium_ls training details
State: closed | Pull request: no
Labels: name: feature

Hello,

I'm trying to train conformer_ctc_medium model from scratch. I use the config file from the pretrained model stt_en_conformer_ctc_medium_ls and only change the manifest_filepath and batch_size configuraion (I changed the batch_size to 16 to fit my GPU).

After training 42 epoches, the WER on librispeech validate data (about 15%) is still much higher than the pretrained stt_en_conformer_ctc_medium_ls model. 

May I know the trained epoch of the pretrained model stt_en_conformer_ctc_medium_ls ? Or is there any other configuration or training process I should pay attention to acheive similar results as the pretrained model ?

Thank you very much!
Unable to install nemo on ubuntu
State: closed | Pull request: no
Labels: name: bug

Getting this error
 ERROR: Command errored out with exit status 1:
     command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-J5oGxi/nemo-toolkit/setup.py'"'"'; __file__='"'"'/tmp/pip-install-J5oGxi/nemo-toolkit/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-weXhSc
         cwd: /tmp/pip-install-J5oGxi/nemo-toolkit/
    Complete output (6 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-J5oGxi/nemo-toolkit/setup.py", line 106
        f'--use-parentheses --line-width={__LINE_WIDTH} -rc -ws'
        ^
    SyntaxError: invalid syntax
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Not able to import NeMo on macOS (ModuleNotFoundError: No module named 'nemo')
State: closed | Pull request: no
Labels: name: bug

I tried following ways to install Nemo:
1. `pip3 install nemo_toolkit`
2. `pip3 install git+https://github.com/NVIDIA/NeMo.git `
3.  Running `git clone git+https://github.com/NVIDIA/NeMo.git ` and then `cd NeMo` and finally `pip3 install .`

The above mentioned commands are successfully executed without any errors.
After running these commands on terminal, when I run python `import nemo`  it shows me `ModuleNotFoundError: No module named 'nemo'`

OS version - macOS Big Sur 11.4
Python version - 3.9.6
Updated ctc-segmentation Python package version
State: closed | Pull request: yes

This PR updates the `ctc-segmentation` Python package used for the CTC segmentation tool from version 1.1.0 to 1.4.0. The update provides a bugfix, as well as sane defaults, which results in better alignments.

I would also recommend the latest version, however, the interface changed at two points:

1. As not every ASR architecture uses subsampling, I replaced the `subsampling` and `frame_duration` parameters with a new parameter `index_duration`. This parameter can simply be calculated as `frame_duration_ms * subsampling_factor / 1000`.
2. The text can now be given as a token ID sequence obtained from the tokenizer (which is already integrated into Nemo). The text input was chosen for our initial implementation, because our ASR model had long tokens and it made sense to scan for partial tokens. As the pre-trained Nemo models are already character-based, scanning for partial tokens is not necessary; using this method will improve alignment speed and probably also generate better alignments (because the pre-processing of the text may be better with a tokenizer).

@ekmb
Extending the neural TN/ITN models for other languages
State: closed | Pull request: yes

https://github.com/NVIDIA/NeMo/pull/2415 was mainly targetting English. This PR extends it to support other languages.

- [x] Testing the new code on Russian
- [x] Testing the new code on German

Results on Google Russian dataset and internal German dataset:
- Russian Duplex System. TN accuracy: 96.20%. ITN accuracy: 85.57%
- German Duplex System. TN accuracy: 94.38%. ITN accuracy: 79.89%
Version comparison 1.0.2 x 0.10.1
State: closed | Pull request: no

Hi

I've been doing transcription training for the Portuguese language for a few months and so far I'm getting great results. But I came across a question when performing a comparison of results from version 0.10.1 and 1.0.2 with the same dataset.
For this comparison I respectively used the scripts:
**(Version 0.10.1)** 
Container https://ngc.nvidia.com/catalog/containers/nvidia:nemo_asr_app_img and notebook 'English to Spanish Cross Language Finetuning'
**(Version 1.0.2)** 
Notebook https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/asr/10_ASR_CTC_Language_Finetuning.ipynb

In their respective parameters I used the recommended values:
**(Version 0.10.1)**
```
# training parameters 
project.set_am_num_gpus(1)
project.set_am_batch_size(42)
project.set_am_num_epochs(50) # as an example we use 1 epoch but in reality we finetuned the model for 256 epochs
project.set_am_learning_rate(0.0015) # lr in finetuning an order of magnitude smaller than training from scratch
project.manifest.am.train_params.amp_opt_level = 'O1'

# optimizer novograd
project.manifest.am.train_params.beta1 = 0.95
project.manifest.am.train_params.beta2 = 0.25 

project.manifest.am.train_params.lr_policy = "CosineAnnealing"
project.manifest.am.train_params.weight_decay = 0.002
project.manifest.am.train_params.warmup_ratio = 0.12
```

**(Version 1.0.2)**
```
# Setup train, validation, test configs
with open_dict(cfg):    
  # Train dataset  (Concatenate train manifest cleaned and dev manifest cleaned)
  cfg.train_ds.manifest_filepath = train_manifest_cleaned
  cfg.train_ds.labels = list(train_dev_set)
  cfg.train_ds.normalize_transcripts = False
  cfg.train_ds.batch_size = 28
  cfg.train_ds.num_workers = 8
  cfg.train_ds.pin_memory = True
  cfg.train_ds.trim_silence = True

  # Validation dataset  (Use test dataset as validation, since we train using train + dev)
  cfg.validation_ds.manifest_filepath = test_manifest_cleaned
  cfg.validation_ds.labels = list(train_dev_set)
  cfg.validation_ds.normalize_transcripts = False
  cfg.validation_ds.batch_size = 8
  cfg.validation_ds.num_workers = 1
  cfg.validation_ds.pin_memory = True
  cfg.validation_ds.trim_silence = True

with open_dict(char_model.cfg.optim):
  char_model.cfg.optim.lr = 0.01
  char_model.cfg.optim.betas = [0.95, 0.5]  # from paper
  char_model.cfg.optim.weight_decay = 0.001  # Original weight decay
  char_model.cfg.optim.sched.warmup_steps = None  # Remove default number of steps of warmup
  char_model.cfg.optim.sched.warmup_ratio = 0.05  # 5 % warmup
  char_model.cfg.optim.sched.min_lr = 1e-5
```

For all values I'm following the recommendations of the scripts themselves. The only change was that BatchSize could only use 28 in the most recent script and 42 in the oldest script. (That's really a question, does the latest version of NeMo use more VRam? For these tests I'm using an Nvidia T4 - 16GB).
After doing the training on both scripts, I noticed that the older version has 30% better accuracy. Is there any explanation for this? Could BatchSize be the reason?
Text to speech module
State: closed | Pull request: no
Labels: name: feature

I have seen in tutorials for audio translation, 
https://github.com/NVIDIA/NeMo/blob/main/tutorials/AudioTranslationSample.ipynb
It has 3 modules speech to text, translation and text to speech. But have seen it uses the IPython audio function for converting the text to audio. I was wondering is there any native module for converting text to speech.
Quartz ASR to jarvis gibbrish output
State: closed | Pull request: no
Labels: name: bug

Hey Tensors,
I have a Quartz model trained on NeMO and I am bit confused on some for the things-

1. When nemo model is converted to enemo, it first converts to [onnx](https://github.com/NVIDIA/NeMo/blob/main/scripts/export/convasr_to_enemo.py#L60). But there are 3 onnx model generated. why? 
2. While enemo to jmir conversion, quartz.onnx(1 onnx model) is used only. why?

The conversion is happened successfully but when I ran test, it gives output gibberish. Output is going continued. It did not stop.
I can not understand the reason behind.

https://forums.developer.nvidia.com/t/jarvis-hindi-model-giving-gibberish-output/183813
@okuchaiev
Unable to find  install_ctc_decoder.sh file In offline ASR 
State: closed | Pull request: no

In order to get the timestamps of each word, I am using Offline asr ( https://colab.research.google.com/github/NVIDIA/NeMo/blob/r1.0.0b3/tutorials/asr/Offline_ASR.ipynb#scrollTo=I9eIxAyKHREB) for my experiment but I am unable to find the install_ctc_decoder.sh in scripts.

Can you suggest any other alternative so that I can use this tutorial for my experimentation purpose?

Thanks in Advance.
bert-case-chinese is not supported 
State: closed | Pull request: no
Labels: name: bug

using bert-case-chinese as pre-trained model, but has this issue:

[NeMo I 2021-07-14 13:41:17 bert_module:54] Restoring weights from /data/cmrc2018/bert-base-chinese/pytorch_model.bin
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
    return func()
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 347, in <lambda>
    lambda: hydra.run(
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 107, in run
    return run_job(
  File "/opt/conda/lib/python3.8/site-packages/hydra/core/utils.py", line 127, in run_job
    ret.return_value = task_function(task_cfg)
  File "/tlt-nemo/nlp/question_answering/scripts/train.py", line 118, in main
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/nlp/models/question_answering/qa_model.py", line 61, in __init__
    self.bert_model = get_lm_model(
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/nlp/modules/common/lm_utils.py", line 92, in get_lm_model
    model.restore_weights(restore_path=checkpoint_file)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/nlp/modules/common/bert_module.py", line 79, in restore_weights
    assert len(pretrained_dict) == len(model_dict)
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tlt-nemo/nlp/question_answering/scripts/train.py", line 140, in <module>
  File "/opt/conda/lib/python3.8/site-packages/nemo/core/config/hydra_runner.py", line 98, in wrapper
    _run_hydra(
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 346, in _run_hydra
    run_and_report(
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 237, in run_and_report
    assert mdl is not None
AssertionError
2021-07-14 06:41:18,254 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.





Training with TLT, backend is Nemo bugs

git clone https://huggingface.co/bert-base-chinese

tlt question_answering train \
                        -e /specs/question_answering/train_bert.txt \
                        -g 1  \
                        -k 'tlt_encode' \
                        -r /results/question_answering/train \
                        training_ds.file=/data/cmrc2018/squad-style-data/cmrc2018_train.json \
                        validation_ds.file=/data/cmrc2018/squad-style-data/cmrc2018_dev.json \
                        trainer.max_epochs=2 \
                        trainer.val_check_interval=1.0 \
                        training_ds.num_workers=4 \
                        validation_ds.num_workers=4

[train_bert.txt](https://github.com/NVIDIA/NeMo/files/6816666/train_bert.txt)
Remove multiplication of sample rate for pydub indexing
State: closed | Pull request: yes

When loading segments, specified with offset, files that are not supported by SoundFile (i.e. filetypes that are non-seekable such as GSM encoded wavs) the script resorts to using [pydub](https://github.com/jiaaro/pydub). 

Pydub uses millisecond indexing. Fix to remove incorrect multiplication with sampling rate at offset.
Replace deprecated imp with newer importlib machinery
State: closed | Pull request: yes

`imp` module has been deprecated since 3.4. Since we do not support older versions of python, we should use supported functions. 

Unfortunately, there is no direct equivalent to `load_source()` function. While `exec_module` loads necessary objects into `package_info` object, it's not possible to use `package_info` in the import statement directly. 

I could not find any way to use this module in the import statements without hacks, but I must admit that I am not very well versed in the python import system. 

The resulting code is not clean, but I want to use this PR a starting point for further discussion.


Signed-off-by: Aleksei Kalinov <alekseia@nvidia.com>
Citrinet model TensorRT Inference
State: closed | Pull request: no
Labels: name: bug

Hi Nvidia Team,

I have converted the Nemo-based Citrinet(for ASR) Model to ONNX then to TRT, but I am stuck at writing an inference code for the TRT model. 

Having said that, while I am doing an Inference in Nemo, there is a single line of code for Inference. So, I am not sure about the TRT inference for the Converted TRT model. it will really help me if you can share some materials on TRT inference for Citrinet models(converted from Nemo).

Thanks and Regards,
Darshan C G
Integration of corcua dataset library
State: closed | Pull request: no
Labels: name: feature

Hi,

I wanted to notify you about my speech dataset handling tool [corcua](https://gitlab.com/Jaco-Assistant/corcua). The task it solves is loading datasets from different formats and converting them to any framework-customized format we can use for training STT networks. It was built with the goal of making the addition of new datasets as easy as possible. Currently it supports 34 datasets, many in German, but 12 of them are multilingual containing 3 or more languages.

NeMo's `.jsonl` format is already supported, so the integration should be very easy.

I hope this is interesting for your project.
Greetings
Daniel
audio_to_melspec_precessor does not return proper MelSpectrogram
State: closed | Pull request: no

```python
import torchaudio
from nemo.collections.tts.models.base import Vocoder

audio, sample_rate = torchaudio.load(audio_path) # sample_rate = 22050
vocoder = Vocoder.from_pretrained('tts_melgan')

audio_len = torch.tensor(audio.size(1)/22050).view(1,1)
gen_mel, _ = vocoder.audio_to_melspec_precessor(audio.cuda(), audio_len.cuda())
plt.imshow(gen_mel.cpu().detach().numpy().squeeze())
plt.show()
```

![image](https://user-images.githubusercontent.com/21142903/125405170-0cf73980-e3d5-11eb-8b25-ebd6fe04d4e8.png)


This code returns a blank(constant) spectrogram.
How do I get the Mel spectrogram for vocoder training/inference?
Add ignore_id/pad_id for classification Acc and sCELoss for seq of label models
State: closed | Pull request: yes

- [x] Add `pad_id` for `SmoothedCrossEntropyLoss` to ignore label=pad_id in labels. In most cases pad_id is max_label+1.
Handel general cases, and special case that has padding label e.g. 5, but the model output of shape pad_id, e.g. [0,1,2,3,4]
- [x] Add `ignore_id` for `TopKClassificationAccuracy` to Ignore label=ignore_id in labels and corresponding indices in predictions. Typically used for excluding padding_id.
- [x] Add tests for `TopKClassificationAccuracy`
- [x]  Add tests for  `SmoothedCrossEntropyLoss`

Signed-off-by: fayejf <fayejf07@gmail.com>
Test audio samples for transcribe
State: closed | Pull request: no
Labels: name: feature

By the demo given for transcribe, it has been given only for Russian text to English translation. Is there any way to get german voice samples for testing? 
I could not access the files in https://nemo-public.s3.us-east-2.amazonaws.com

Link for Russian text is 
Get example from: https://nemo-public.s3.us-east-2.amazonaws.com/mcv-samples-ru/common_voice_de_19034087.wav
Speaker diarization example produces negative offsets
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Using [speaker_diarize.py](https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_recognition/speaker_diarize.py ) results in a negative offset for a file in a manifest, which leads to an error in dataloader:
```

  File "speaker_diarize.py", line 45, in main
    sd_model.diarize()
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/nemo/collections/asr/models/clustering_diarizer.py", line 341, in diarize
    self._run_vad(manifest_vad_input)
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/nemo/collections/asr/models/clustering_diarizer.py", line 193, in _run_vad
    for i, test_batch in enumerate(tqdm(self._vad_model.test_dataloader())):
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/tqdm/std.py", line 1185, in __iter__
    for obj in iterable:
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/nemo/collections/asr/data/audio_to_label.py", line 476, in vad_frame_seq_collate_fn
    return _vad_frame_seq_collate_fn(self, batch)
  File "/home/dev/anaconda3/envs/oMeN/lib/python3.8/site-packages/nemo/collections/asr/data/audio_to_label.py", line 245, in _vad_frame_seq_collate_fn
    audio_signal = torch.stack(audio_signal)
RuntimeError: stack expects a non-empty TensorList
```

Resulting manifest: 
```
{"audio_filepath": "test.wav", "duration": 2.2756875, "label": "infer", "text": "_", "offset": -0.15}
```
**Steps/Code to reproduce bug**

Command:
python speaker_diarize.py   diarizer.paths2audio_files=files.txt   diarizer.vad.model_path="vad_telephony_marblenet"   diarizer.vad.threshold=0.7   diarizer.speaker_embeddings.model_path="speakerverification_speakernet"   diarizer.out_dir="demo"

Audio file with manifest:
[reproduce.zip](https://github.com/NVIDIA/NeMo/files/6799603/reproduce.zip)


**Expected behavior**
No negative offsets in manifests.


**Environment overview (please complete the following information)**

Nemo version 1.1.0
Pytorch version 1.9.0
DDP-related(?): all CUDA-capable devices are busy or unavailable
State: closed | Pull request: no

**Describe the bug**

"speech_to_text_bpe.py"(ASR example) with ddp (8-gpus) failed to run.
It produced a "RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable"

**Steps/Code to reproduce bug**

After generating the manifest for LibriSpeech dataset  and tokenization, I ran the following script in a command line
(Below codes are from the comments in examples/asr directory, actually)

python speech_to_text_bpe.py \
    model.train_ds.manifest_filepath=librispeech/train_all.json \
    model.validation_ds.manifest_filepath=librispeech/dev_all.json \
    model.tokenizer.dir=librispeech/tokenizer_spe_bpe_v5000 \
    model.tokenizer.type=bpe \
    trainer.gpus=-1 \
    trainer.accelerator="ddp" \
    trainer.max_epochs=100 \
    model.optim.name="adamw" \
    model.optim.lr=0.001 \
    model.optim.betas=[0.9,0.999] \
    model.optim.weight_decay=0.0001 \
    model.optim.sched.warmup_steps=2000
    exp_manager.create_wandb_logger=True \
    exp_manager.wandb_logger_kwargs.name="exp.tst" \
    exp_manager.wandb_logger_kwargs.project="prj.tst"

**Expected behavior**

It should run without errors

**Environment overview (please complete the following information)**

 - Environment location: NVIDIA V100 x8, docker 
 - Method of NeMo install: from source (git-cloned on 10 July, installed with reinstall.sh)

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 18.04.5 LTS
- PyTorch version: 1.7.1+cu110
- Python version: 3.8.5

**Additional context**

Add any other context about the problem here.
Example: GPU model

---
No progress after installing Nemo. :( 
Any help would be appreciated.
Make parallel version of ConvASREncoder
State: closed | Pull request: yes

The change introduces an encoder that can stack and execute several similar Jasper blocks side by side. 

This implementation can be used to define and train CarneliNet.

Underlying ParallelBlock supports tower dropout to train models that can be reconfigured during inference time.

# Usage

## Config 
Define proper target for encoder.  Use multiple values of kernel sizes to define the towers. The rest of the parameters will be shared across all of them.
```
          ...
  encoder:
    _target_: nemo.collections.asr.modules.ParallelConvASREncoder

          ...
    jasper:
      - filters: 384
        kernel: [11, 13, 15, 17]
        aggregation_mode: "dropout"
        block_dropout: 0.1
        ...
        
```

## Code 

Just call necessary model with correct config, as usual.
```
cfg = OmegaConf.load(cfg_path)
asr_model = nemo_asr.models.EncDecCTCModelBPE(cfg=cgf, trainer=trainer)
```
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
State: closed | Pull request: no
Labels: name: bug

Command line executed

docker run --gpus all -it --rm -v abc:/NeMo --shm-size=8g -p 8888:8888 -p 6006:6006 --ulimit 
memlock=-1 --ulimit stack=67108864 --device=/dev/snd nvcr.io/nvidia/pytorch:21.05-py3

Getting error message:
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

checking if nvidia driver is installed

nvidia-smi 

Thu Jul  8 14:23:48 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.27       Driver Version: 465.27       CUDA Version: 11.3     |
Unable to download pretrained model
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
`ValueError` when trying to import pretrained model.

**Steps/Code to reproduce bug**

```python
import nemo.collections.asr as nemo_asr

model = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")
[NeMo I 2021-07-06 22:51:41 cloud:66] Downloading from: https://api.ngc.nvidia.com/v2/models/nvidia/nemospeechmodels/versions/1.0.0a5/files/QuartzNet15x5Base-En.nemo to /root/.cache/torch/NeMo/NeMo_1.1.0/QuartzNet15x5Base-En/2b066be39e9294d7100fb176ec817722/QuartzNet15x5Base-En.nemo
[NeMo I 2021-07-06 22:51:42 cloud:79] Download from cloud failed. Attempt 1 of 3
[NeMo I 2021-07-06 22:51:42 cloud:79] Download from cloud failed. Attempt 2 of 3
[NeMo I 2021-07-06 22:51:42 cloud:79] Download from cloud failed. Attempt 3 of 3
ValueError: Not able to download url right now, please try again.
```


**Expected behavior**

Model is downloaded without error.

**Environment overview (please complete the following information)**

 - Environment location: Databricks
 - Method of NeMo install: Tried with both pip magic `%pip install nemo_toolkit['all']` and directly in Databricks.

**Environment details**

- Databricks runtime 8.3 ML
- PyTorch==1.8.1
- Python version==3.8.8

**Additional context**

I thought this was similar to [#2555](https://github.com/NVIDIA/NeMo/issues/2255) but I have confirmed that my Databricks cluster has internet access.
Change Setting for Training in Portuguese
State: closed | Pull request: no
Labels: name: feature

Hi,
I'm doing a training with 10_ASR_CTC_Language_Finetuning, I was wondering if you would have a reference of these settings, ![image](https://user-images.githubusercontent.com/49493322/124511430-7ad7ab80-ddac-11eb-9ed0-45b64c1e180a.png)
which values would be better for a training in Portuguese?
Hyperpeter Optimization with ray tune and pytorch lightning
State: closed | Pull request: no
Labels: name: feature

It would be helpful to add Hyperpeter Optimization with ray tune and pytorch lightning for different nemo models (ASR/TTS) 
This will help in finding the best set of paramters when training using Nemo on different sorts of data.  

This can make the training more straightforward and can be of an advantage compared to other repos.
Thank you!
Citrinet_1024 config file 
State: closed | Pull request: no

Hi,

I am trying to finetune the Citrinet_1024 nemo model first on 50K chunks of the Indian accent English-language dataset using 1 gpu.
Since [here](https://github.com/NVIDIA/NeMo/tree/main/examples/asr/conf/citrinet) is no config file for citrinet_1024. So I changed the filters 512 to 1024, set the dropout to 0.01 and changed the model name from "Citrinet-512-8x-Stride" to "Citrinet-1024-8x-Stride".

Is that sufficient which I did above or need to change something else in the config file?

The config file which I modified is attached below in txt format;
[citrinet_1024.txt](https://github.com/NVIDIA/NeMo/files/6759687/citrinet_1024.txt)
Changes for ONNX conversion of Conformer models
State: closed | Pull request: yes

* Brought in the needed changes from Vahid's earlier PR: https://github.com/NVIDIA/NeMo/pull/2359/files
* Tested conversion with scripts/export/convasr_to_single_onnx.py
* Tested the converted large, medium and small models with dev-clean and WER is very close (within 0.01%) of correspondiong .nemo models
What is the pre-trained model to be used for Mandarin TTS in NeMo?
State: closed | Pull request: no
Labels: name: feature

**Describe the bug**
Hi,

Today after I have installed NeMo, I have tried the ASR both in English and Chinese -- it all works very well!!

Also I did a quick tts test on the English based on following setting  -- it works!

# Download and load the pretrained tacotron2 model
spec_generator = SpectrogramGenerator.from_pretrained("tts_en_tacotron2")
# Download and load the pretrained waveglow model
vocoder = Vocoder.from_pretrained("tts_waveglow_88m")

Could you please let me know what is the pre-trained model to be used for the Mandarin TTS? Also can tts_waveglow_88m is also used for that case?

Thanks a lot
Jimmy


**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version        : ubuntu 18.04
- PyTorch version  1.9.0+cu102
- Python version:   3.6

**Additional context**

Add any other context about the problem here.
Example: GPU model
Model QuartzNet15x5Base-Zh was not found
State: closed | Pull request: no
Labels: name: documentation

**Describe the bug**

Hi, 

I am new to NeMo. After I have installed the NeMo in my dGPU laptop, I have written a simple test to test the ASR.

1. first I have test the .wav file in English ("Where is the bus station?")

quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En").cuda()
print("\nGot the TEXT")
english_text = quartznet.transcribe([Audio_sample])
print("\n")
print(english_text)    --- it prints the "where is the bus station" ----Good thing

2. Then: I have test the .wav file in Chinese in same word where is the bus station in Chinese
 
quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-Zh").cuda()

But I have got: FileNotFoundError: Model QuartzNet15x5Base-Zh was not found.

Per NeMo user guide, QuartzNet15x5Base-Zh is supported model.

Can you let me know how to solve the problem?

Thanks
Jimmy




**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version:       Ubuntu 18.04
- PyTorch version  1.9.0+cu102
- Python version:   3.6

**Additional context**

Add any other context about the problem here.
Example: GPU model: RTX 2080Ti
Fine-tune stt_en_conformer_ctc_large
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I am trying to fine-tune **stt_en_conformer_ctc_large** model using the following:

`asr_model = nemo_asr.models.EncDecCTCModelBPE.from_pretrained(model_name="stt_en_conformer_ctc_large")`

Then I set the training params:

```
# not working: 
from omegaconf import OmegaConf, open_dict
    
params = asr_model.cfg 
new_opt = copy.deepcopy(params.optim)
new_opt.lr = 0.0001

params['train_ds']['manifest_filepath'] = train_manifest_path
params['train_ds']['batch_size'] = 8


params['validation_ds']['manifest_filepath'] = dev_manifest_path
params['validation_ds']['batch_size'] = 8

# Use the smaller learning rate we set before
asr_model.setup_optimization(optim_config=new_opt)

# Point to the new training data for fine-tuning
asr_model.setup_training_data(train_data_config=params['train_ds'])


# Point to the new validation data for fine-tuning
asr_model.setup_validation_data(val_data_config=params['validation_ds'])
```

Received error:

```
~/NeMo/nemo/collections/asr/models/ctc_models.py in setup_training_data(self, train_data_config)
    423             # If it's an int, we assume that the user has set it to something sane, i.e. <= # training batches,
    424             # and don't change it. Otherwise, adjust batches accordingly if it's a float (including 1.0).
--> 425             if isinstance(self._trainer.limit_train_batches, float):
    426                 self._trainer.limit_train_batches = int(
    427                     self._trainer.limit_train_batches

AttributeError: 'NoneType' object has no attribute 'limit_train_batches'

```

However, it **works** when I load the config from the config_bpe.yaml:
```
from omegaconf import OmegaConf, open_dict

# ! wget -P configs/ https://raw.githubusercontent.com/NVIDIA/NeMo/main/examples/asr/conf/citrinet/config_bpe.yaml --no-check-certificate
    
params = OmegaConf.load('configs/config_bpe.yaml')
    
new_opt = copy.deepcopy(params.model.optim)
new_opt.lr = 0.0001

params['model']['train_ds']['manifest_filepath'] = train_manifest_path
params['model']['train_ds']['batch_size'] = 8


params['model']['validation_ds']['manifest_filepath'] = dev_manifest_path
params['model']['validation_ds']['batch_size'] = 8

# Use the smaller learning rate we set before
asr_model.setup_optimization(optim_config=new_opt)

# Point to the new validation data for fine-tuning
asr_model.setup_validation_data(val_data_config=params['model']['validation_ds'])


# Point to the data we'll use for fine-tuning as the training set
asr_model.setup_training_data(train_data_config=params['model']['train_ds'])

```

What is the correct procedure?
Adding training_step to parameters in ModelPT breaks DP accelerator
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
Similar to https://github.com/pytorch/pytorch/issues/8637, ModelPT class adds training_step to its parameters [here](https://github.com/NVIDIA/NeMo/blob/main/nemo/core/classes/modelPT.py#L159). Because of that, when training with 'dp' accelerator workers call the same instanced function instead of copies on different devices.

**Steps/Code to reproduce bug**
Create any model inherited from ModelPT.
Fit with torch lightning trainer and 'dp' accelerator.
Check module id [here](https://github.com/PyTorchLightning/pytorch-lightning/blob/4af8eff0a13309b2e030c423db91156b1e869fac/pytorch_lightning/overrides/base.py#L84) and in your training_step - they will differ.

**Expected behavior**
No function should be a parameter in the class.

**Environment overview (please complete the following information)**

 - Environment location: My own machine
 - Method of NeMo install: pip install nemo v1.0.2

**Environment details**

- OS version Ubuntu 18.04
- PyTorch version 1.9.0
- Python version 3.8.3

**Additional context**

I understand that NeMo doesn't support DP accelerator, but this is a very hard to trace bug that can be easily fixed and can help someone in the future.
Link updates in docs and notebooks and typo fix
State: closed | Pull request: yes

Typo for Voice Activi**t**y Detection notebooks/links fix
Update links in docs for branches to **stable** 
Update links in notebooks for branches to **stable**

Will submit a PR to update links in **main** branch to main once this PR has been merged to r1.1.0 and main
Please make sure to update **stable** before release
Run Offline_ASR (python notebook) decode with a GPU
State: closed | Pull request: no
Labels: name: feature

Hello,

I'm using the Offline_ASR notebook testing some large and small values for width, alpha and beta in 'beamsearch', all is correct and returning good results, but needs so much time to return those results. This script is running in Colab.
Have some way to run this notebook or only the beamsearch part with a GPU, or just accelerate in general?
Run multiple model inference on a single GPU concurrently
State: closed | Pull request: no

Hello nemo team,

I have a queue of audio files waiting to be transcribed. when I run the citrinet model on them I see there's still free memory on the gpu hence my question - is there a way and how, to run multiple nemo models (can be all citrinet for example, but also others) concurrently on a single gpu? 

btw unrelated: is there a compatible pretrained LM model for bpe models (citrinet)? also is downsampling needed/important for audio files higher than 16kHz ? (22, 41..) so far it seems to handle them well.

Thank you 😊
`run_sample.sh` opens too many files
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

The data preparation script `run_sample.sh` errors out pretty frequently, caused by opening too many files at the same time and hitting the process file limit. This seems to arise from the 'prepare_data.py' script which is called

**Steps/Code to reproduce bug**

(This seems to stem from the use of multiprocessing, so this may not be consistently reproducible)
1. Create a data directory with ~1000 files
2. Run the script
3. Script fails before completion of step 1 (`prepare_data.py`), with an OS Error

**Expected behavior**

Script should complete successfully

**Environment overview (please complete the following information)**

 - Environment location: Local dev system
 - Method of NeMo install: install from source (as described in readme, in a Conda environment)

**Environment details**

- OS version: Linux 5.12.9-arch1-1 x86_64 GNU/Linux
- PyTorch version: 1.9.0
- Python version: 3.9.5

**Additional context**

I was able to solve the issue by switching the use of `multiprocessing.Process` with `multiprocessing.Pool` (and other changes required to use it). Not sure if it is a robust solution, or if it can be solved in other ways
Fine-tune ASR language model
State: closed | Pull request: no

How to fine-tune **nemo:asrlm_en_transformer_large_ls LM** [here](https://ngc.nvidia.com/catalog/models/nvidia:nemo:asrlm_en_transformer_large_ls) 

I found this training script here (https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/transformer_lm.py), but I did not know how to use it? can you give more instruction.