renwei2024/nemo-github-issues huggingface dataset:
Language model refactoring
State: closed | Pull request: yes

1. Language model training code refactored to be compatible with the latest changes in NMT.
2. Perplexity metric refactored to take into account samples of different length in a batch.

This pull request **introduces 2 alerts** when merging ed136b9b2a340c1c3e5ffa8c0e609e5d3de52d01 into b4040fb37350ae86b64a5f53be911371d7a3879d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-55223cc7bf93f881bcbeaac5e0bdbb46892d0188)

**new alerts:**

* 2 for Unused import

This pull request **introduces 2 alerts** when merging f6bdef3a27fc55767864fa6d0da5d49362305cb0 into fe7efb6adc536b3e9ca3937e15fdcb68dacdcec7 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-962097c676a5aa27d4d893aa597d90dd3643a05a)

**new alerts:**

* 2 for Unused import

This pull request **introduces 2 alerts** when merging 65219f584018a7de49d0285448252c5c86fa9659 into 26e5d3a9b3cc934db0004b1dd9263fb7bce9a964 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-aa3235736ac34a5f12121be893fdb010c59b694d)

**new alerts:**

* 2 for Unused import

This pull request **introduces 1 alert** when merging 40f5e5566e019d995480647aaa6cda1b9db8623f into 26e5d3a9b3cc934db0004b1dd9263fb7bce9a964 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d29acaf67c9c1608a52caa7d934dd37ef16f1e17)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 3318718c96564ec08c86ab8aefba97d54cb2f401 into 26e5d3a9b3cc934db0004b1dd9263fb7bce9a964 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-595add236d3cea5440abf2fc37617128c2722511)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging c022b0bcae924cf4bbedeea5a24b5bbd739e3f82 into 5316f2bcd0e326d336cb35bd8222b00914a5b77f - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-95ab97aa35785384fe258687344c8fcb2f4d5d77)

**new alerts:**

* 1 for Unused import

> Thanks! Just one last change, can you please rename the dataset in `translate_ddp.py` as well?https://github.com/AlexGrinch/NeMo/blob/lm_training/examples/nlp/machine_translation/translate_ddp.py#L23. I should add a CI check for this at some point.

Hmm, I missed it. Now it should be consistent.
Modified convert_to_tarred_audio_dataset.py to accept another argument also as input
State: closed | Pull request: yes

The newly added argument(dataset_dir) is concatenated with the path obtained from the manifest file. This solves the problem of FileNotFoundError as an absolute path is given for the datasets.

This pull request **introduces 2 alerts** when merging 1ce64a75065b53909e042395c27bf1a35f68309d into b4040fb37350ae86b64a5f53be911371d7a3879d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-8c2c9c4c374e615c273837a54858c8611874091b)

**new alerts:**

* 1 for Unused local variable
* 1 for Unused import

Could you sign the commits (instructions are provided in the dco bot output), and explain your reasoning for dataset paths argument?

Tarred datasets flatten all paths, and you can create them from a provided dataset directory and write then to another directory, so this flag is odd. 

Created merge request for wrong branch
[NMT] Add support for multiple validation and test dataloaders
State: closed | Pull request: yes

Specify multiple validation (or test) dataloaders by inputting a list of src (tgt) files.

# Usage
Replace `src_file_name` and `tgt_file_name` with a list of file paths.
```
    model.validation_ds.src_file_name=/data/wmt14-en-de.src \
    model.validation_ds.tgt_file_name=/data/wmt14-en-de.ref \
```
Becomes:
```
    model.validation_ds.src_file_name=[/data/wmt13-en-de.src,/data/wmt14-en-de.src] \
    model.validation_ds.tgt_file_name=[/data/wmt13-en-de.ref,/data/wmt14-en-de.ref] \
```
When using `val_loss` or `val_sacreBLEU` for the `exp_manager.checkpoint_callback_params.monitor` then the 0th indexed dataloader will be used as the monitor. This ensures backwards compatibility.

Additionally, any validation dataloader can now be used as a monitor by appending the index as below:
```
    exp_manager.checkpoint_callback_params.monitor=val_sacreBLEU_dl_index_1
```
Multiple test datasets works exactly the same way as validation datasets, simply replace `validation_ds` by `test_ds` in the above examples.
I get an incompatible library version error 
State: closed | Pull request: no
Labels: name: bug

I run the code in https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/06_Voice_Activiy_Detection.ipynb, however I got again an error while runing below cell:

  1 # NeMo's "core" package
----> 2 import nemo
3 # NeMo's ASR collection - this collections contains complete ASR models and
4 # building blocks (modules) for ASR
5 import nemo.collections.asr as nemo_asr

11 frames
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/apply_func.py in ()
23 TORCHTEXT_AVAILABLE = importlib.util.find_spec("torchtext") is not None
24 if TORCHTEXT_AVAILABLE:
---> 25 from torchtext.data import Batch
26 else:
27 Batch = type(None)

ImportError: cannot import name 'Batch' from 'torchtext.data' (/usr/local/lib/python3.7/dist-packages/torchtext/data/init.py)

and I got another error when I run below cell:
AttributeError Traceback (most recent call last)
in ()
----> 1 exp_dir = exp_manager(trainer, config.get("exp_manager", None))

1 frames
/content/MyVADNeMo/nemo/utils/exp_manager.py in get_log_dir(trainer, exp_dir, name, version, explicit_log_dir, use_datetime_version)
428 if trainer.logger.save_dir:
429 if exp_dir:
--> 430 raise LoggerMisconfigurationError(
431 "The pytorch lightning trainer that was passed to exp_manager contained a logger, the logger's "
432 f"save_dir was not None, and exp_dir ({exp_dir}) was not None. If trainer.logger.save_dir "

AttributeError: 'Trainer' object has no attribute 'is_slurm_managing_tasks'.

Thanks in advance!

(1) Please try running it Google's Colab (which would have a clean environment and should work)
(2) If you try running it locally, I'd recommend creating a new conda environment with Python 3.8 (and without Pytorch, Torchtext, etc.). This way when you do "pip install ..." correct versions of torch and torchtext should be installed.
(3) Also, you can try using our container: ```docker run --gpus all -it --rm --shm-size=8g \
-p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit \
stack=67108864 --device=/dev/snd nvcr.io/nvidia/nemo:1.0.0rc1```
Keyword spotting in NEMO
State: closed | Pull request: no
Labels: name: feature

I am contacting you today to request information about detecting keywords from audio using NEMO ASR.
I am very interested in finding out if NEMO supports keyword spotting.

Regards,
Melika Bahman-Abadi

Hi @MelikaBahmanabadi Thanks for your question.
We have Voice Activity Detection support ([tutorial](https://github.com/NVIDIA/NeMo/blob/r1.0.0rc1/tutorials/asr/06_Voice_Activiy_Detection.ipynb)) in NeMo. For KWS I think you can start with that and train on data to have a classier to handle `the keyword vs. others`. 

if you are interested multi-word keyword spotting. You can use the combination of VAD and Keyword detection. If you are interested in this approach, have a look at the [03 tutorial](https://github.com/NVIDIA/NeMo/blob/r1.0.0rc1/tutorials/asr/03_Speech_Commands.ipynb) that talks about keyword detection training and finetuning on the new dataset, and [04 tutorial](https://github.com/NVIDIA/NeMo/blob/r1.0.0rc1/tutorials/asr/04_Online_Offline_Speech_Commands_Demo.ipynb)  that talks about VAD+keyword detection inference.
Itn punctuation
State: closed | Pull request: yes

fixed punctuation for itn in SH to be recognized as word
fixed punctuation for tn in SH to be recognized as word
added tests
simplified deployment route


This pull request **introduces 2 alerts** when merging da62342a19182f09fee4ec3d31e5b23bd0d7004c into e7977fd52de17f27befb4d36237584373adfe154 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-28e3ff6a1ceac31e742f1feca442469642e2da8b)

**new alerts:**

* 2 for Unused import
Add End-to-End TTS Models
State: closed | Pull request: yes

- Add the TTS base class: `TextToWaveform` for use with end to end models
- Add two end-to-end models:
  - FastPitchHifiGanE2EModel: An end-to-end model combining FastPitch and HiFiGan
  - FastSpeech2HifiGanE2EModel: : An end-to-end model combining FastSpeech2 and HiFiGan

This pull request **introduces 12 alerts** when merging af437cf4df8f2897529e1aa911b9288daa67d555 into 0b495ccbfc3bade395678fd3d0524febb799a9cb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-bd47bbdee5cc95948826a8844a4ff81bc1d84c0b)

**new alerts:**

* 9 for Unused import
* 2 for Unused local variable
* 1 for Signature mismatch in overriding method

This pull request **introduces 3 alerts** when merging 61e0b4e803e11cca3a0e39c86bea9317ed37d103 into 0b495ccbfc3bade395678fd3d0524febb799a9cb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-c25dcc72d84a600981873466023c1a620d05414d)

**new alerts:**

* 2 for Unused import
* 1 for Signature mismatch in overriding method

This pull request **introduces 2 alerts** when merging 2cb00c22a46de35f96d6ca191107cf9483d569be into b4040fb37350ae86b64a5f53be911371d7a3879d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-536a2ddac195c789efbf6f2bc3256e1cdbc13639)

**new alerts:**

* 2 for Signature mismatch in overriding method
bug fix for handling empty lines in input text
State: closed | Pull request: yes

This fixes the empty line issue of `nemo_text_processing/inverse_text_normalization/run_predict.py` discussed in https://github.com/NVIDIA/NeMo/issues/2094

@yzhang123 

@taufique74 first of all, thanks for catching the bug and PR! Since we would prefer to keep the line breaks which fits more the user's expectations i will replace this PR my this one https://github.com/NVIDIA/NeMo/pull/2101
And close this one.
Inverse Text Normalization doesn't work for upper-cased words
State: closed | Pull request: no
Labels: name: bug

name: TN/ITN

**Bug Description**

Inverse Text Normalization text-processing fails to normalize a verbal form if it has a word with an upper case.
While I understand that the transcriptions are usually all-lower cased words, it'd be nice if we lower the text before passing it to `inverse_normalizer`.

**Steps to reproduce bug**

Let's create a text file `input.txt` containing the following information:
```text
one thousand and seventy four
One thousand and seventy four
one Thousand and seventy four
one thousand And seventy four
one thousand and Seventy four
one thousand and seventy Four
```
if we run `python run_predict.py --input=input.txt --output=output.txt`, the  generated `output.txt` will contain the following results:
```text
1074
One thousand and 74
one Thousand and 74
1000 And 74
1000 and Seventy four
1070 Four
```

So only the first line that contains all lower-cased words was correctly normalized.


**Environment overview**

- Environment location: Cloud(AWS)
- Method of NeMo install: from source


**Environment details**

- OS version: Ubuntu 18.04.5 LTS
- PyTorch version: 1.8.1+cu102
- Python version: 3.7


@yzhang123 

@taufique74 this is true, we do not handle upper cased words at this point, since we assume its the output of the asr model. please lower case your input
Inverse Text Normalization text-processing breaks when there's an empty line the input text
State: closed | Pull request: no
Labels: name: bug

name: TN/ITN

**Bug Description**

While running `python run_predict.py --input=input.txt --output=output.txt`, it breaks saying `pywrapfst.FstOpError: Operation failed`. I noticed there is an empty line in `input.txt`, and after removing that empty line the error was gone.


**Steps to reproduce the bug**

Let's first create a text file `input.txt` with some examples along with an empty line
```bash
$ echo "one hundred and twenty four" >> input.txt
$ echo "" >> input.txt
$ echo "may third nineteen sixty" >>
```

Now if we run `python run_predict.py --input=input.txt --output=output.txt`, it will throw an error.
Here is the complete traceback:
```python-traceback
Traceback (most recent call last):
  File "/home/ubuntu/prtc/itn/NeMo/nemo_text_processing/inverse_text_normalization/inverse_normalize.py", line 209, in inverse_normalize_nemo
    text = inverse_normalize(input, verbose=verbose)
  File "/home/ubuntu/prtc/itn/NeMo/nemo_text_processing/inverse_text_normalization/inverse_normalize.py", line 169, in inverse_normalize
    tagged_text = select_tag(tagged_lattice)
  File "/home/ubuntu/prtc/itn/NeMo/nemo_text_processing/inverse_text_normalization/inverse_normalize.py", line 125, in select_tag
    tagged_text = pynini.shortestpath(lattice, nshortest=1, unique=True).string()
  File "extensions/_pynini.pyx", line 464, in _pynini.Fst.string
  File "extensions/_pynini.pyx", line 509, in _pynini.Fst.string
_pywrapfst.FstOpError: Operation failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_predict.py", line 77, in <module>
    inverse_normalizer_prediction = inverse_normalizer(data, verbose=args.verbose)
  File "/home/ubuntu/prtc/itn/NeMo/nemo_text_processing/inverse_text_normalization/inverse_normalize.py", line 211, in inverse_normalize_nemo
    raise Exception
Exception
```



**Environment overview**

 - Environment location:  Cloud(AWS)
 - Method of NeMo install: from source

**Environment details**

- OS version: Ubuntu 18.04.5 LTS
- PyTorch version: 1.8.1+cu102
- Python version: 3.7

Hey @yzhang123 
Let me know if you'd be interested in a PR, it's a quick fix.

@taufique74 sure go ahead! thanks!

closing since https://github.com/NVIDIA/NeMo/pull/2101 is merged. Please re-open if the bug persists
generalized export to both itn and tn
State: closed | Pull request: yes

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

This pull request **introduces 1 alert** when merging f5bf7fe4827595ac8b290ca8e002357ac4e29814 into 88af7a24a7ebb4418cb51c328e2c03aed8388e03 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b5d3cd734613603214c2207eab3527795ce634af)

**new alerts:**

* 1 for Unused import
WFST Normalization support for emails/electronic
State: closed | Pull request: yes


This pull request **introduces 2 alerts** when merging 6383d773b8cef1cb0206d9cdadec9b0abe610cf4 into 98bdf7a16e27cf4157f0ec0fcf78e6b0d6373c02 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-f4cf5d868dae323cc31e029daacb46a2d10f1e31)

**new alerts:**

* 2 for Unused import
added more date formats
State: closed | Pull request: yes

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

This pull request **fixes 1 alert** when merging e6ae53a4b3dde3e74a3b34778b6a0ffcf50a81f8 into cc1553b253e9d17369c9771ccd1af196bfac70a5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-f9683a00beb9844ce1192d72bb3702e9183e045d)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging 85839109d19b8c09e818f096ca0660b9c22bc09f into cc1553b253e9d17369c9771ccd1af196bfac70a5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-7d08219436246b8357c295a4d6f5f684a6ce96d0)

**fixed alerts:**

* 1 for Unused import
configure input capitalization
State: closed | Pull request: yes

* adding configuration to distinguish between lower_cased and cased input, affects whitelist, word classes
* added test
* added pynini sentence predict test to jenkins
* deleted sentence boundary exception

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

This pull request **introduces 1 alert** and **fixes 2** when merging 0e2ecc61c2b1caa9eac17b42025c9df2b9bad6cf into 4ce74f134ca5c318f421a54e5fbd571879bbb418 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-3895299e38b5f30ef6af355511dca3a9d47a19f1)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for Unused import

This pull request **introduces 1 alert** when merging 5d65457fd92da253990cc2ac2b4842d2924ba25d into fbfe24bc548cd88c1f87fa66b00b12ddfbdeab29 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-8a62b94a4caaf0cc2e95c2c739b8b7cbc516a4c4)

**new alerts:**

* 1 for Unused import
Can't Install nemo_text_processing
State: closed | Pull request: no
Labels: name: bug

name: TN/ITN

I'm following the instruction in
`https://github.com/NVIDIA/NeMo/blob/main/tutorials/text_processing/Text_Normalization_Tutorial.ipynb`

Running code on a docker container `nvcr.io/nvidia/nemo:1.0.0rc1`

There are no errors when running the suggested install code:
```
BRANCH = 'main'
!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]
```
Many lines of `Requirement already satisfied:...`

However, the `import nemo_text_processing` fails:
```
ModuleNotFoundError: No module named 'nemo_text_processing'
```


@mschonwe we changed a few things regarding normalization since 1.0.0rc.
 please refer to RC tutorial https://github.com/NVIDIA/NeMo/blob/r1.0.0rc1/tutorials/tools/Text_Normalization_Tutorial.ipynb which is tested to work in nvcr.io/nvidia/nemo:1.0.0rc1.

Thank you.  Got it working following the updated tutorial.

@mschonwe  we will be soon changing text normalization to WFST just like inverse text normalization, just FYI
100% training WER on custom data
State: closed | Pull request: no

Hello,

I am finetuning the QuartznetBase-en5x5 model on my own dataset, but I am getting  trainingWER  100% and training loss is flatted after few steps although val_WER  and val_loss is decreasing gradually

**Dataset Details**
train Dataset consists of 169630 files totaling 410.18 hours
Val_ Dataset consists of   25036 files totaling 84.82 hours
Audio data is of length between( 5-60secs) and made sure that there are no samples without text output

I'm using the file [speech_to_text.py](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/speech_to_text.py) just made a little changes and added the following line before  _'trainer.fit'_


` asr_model.restore_from("QuartzNet15x5Base-En.nemo")`


**other details are:**
        trainer.max_epochs=200\
        model.train_ds.batch_size=16\
        model.validation_ds.batch_size=16 \
        model.optim.lr=0.00015 \
        +model.validation_ds.num_workers=2  \
        +model.train_ds.num_workers=10 \
        +trainer.precision=16 


**nemo version = 1.0.0rc1**

. Could you please troubleshoot where could the problem be?

It is normal to get 100% WER for first few epochs. How many epochs did train?

Also, there might be a mismatch between the datasets. For example, is the sampling rate in your data the same as it is in the LibriSpeech dataset?

around 50 epoch, after 50 epochs its crashing
> It is normal to get 100% WER for first few epochs. How many epochs did train?



> Also, there might be a mismatch between the datasets. For example, is the sampling rate in your data the same as it is in the LibriSpeech dataset?

sample rate is same but when I make a test run on lesser data i.e. around 200 hours at that time it was decreasing



![image](https://user-images.githubusercontent.com/46400775/115661535-6c700c80-a2f2-11eb-8a2d-bdfbb7b5b6b4.png)


Blue = ~200hours data
pink = ~400 hours data

60s is unnecessarily long. We typically use max 16.7s. Btw, by default, it may ignore everything longer than that.
Such long fragments doesn't help with training too much but will require large amount of GPU ram.

Also, what is your character set in your transcriptions? For QuartzNet, it is lowercase En letters and apostrophe ('), no punctuation or numbers.

Try lowering your learning rate and increasing batch size too.

Several things - disable spec augment, reduce Weight Decay to 0 and see if the model can learn with no regularization first. 
The fact that 200 hours works and 400 doesn't means that your learning rate is too high - with 200 hours you have roughly half the number of steps per epoch so warmup and all other parameters should be adjusted.
VAD speech time intervals
State: closed | Pull request: no

Hi, I want to extract the speech time intervals from the audios using VAD (vad_marbelnet) model. Is there a direct method (function) to do it without modifying the config file?

Thanks for your question.
Yes. Have a look at **_VAD diarization_** part in diarization [tutorial](https://github.com/NVIDIA/NeMo/blob/r1.0.0rc1/tutorials/speaker_recognition/Speaker_Diarization_Inference.ipynb )  which will generate VAD RTTM-like json file that could suit your request.


Yes, thank you for the response.
Fix and enable DALI tests
State: closed | Pull request: yes

Signed-off-by: Joaquin Anton janton@nvidia.com

This pull request **introduces 1 alert** when merging 760f3012e9f086af1d1893d0a1c378bd2b2aabab into bee43e840837d7e30e17a65b972e9fe89ecb9ca5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-7b0cbe4483c3653568bbc31df8b151ec2086aefc)

**new alerts:**

* 1 for Unused local variable
How can I drop the last layer to get output as hidden states in terms of feature extraction?
State: closed | Pull request: no

**Describe your question**
Hi, I'm working on the Megatron model how can I drop the last layer to get output as hidden states in terms of feature extraction?





This question is a little bit vague. Would you mind expanding on what you're trying to do? For example, what NeMo model are you working with and what would you like to do with the hidden states?
how to convert .pt models  into .nemo format[Question]
State: closed | Pull request: no


 I have following  blog [Jump-start Training for Speech Recognition Models in Different Languages with NVIDIA NeMo](https://developer.nvidia.com/blog/jump-start-training-for-speech-recognition-models-with-nemo/#respond) and successfully finetune the model on my custom data. Now I have to test the models but in checkpoint directory I have  3 .pt files as output (JasperDecoderForCTC-STEP-80000.pt   JasperEncoder-STEP-80000.pt  trainer-STEP-80000.pt)

I am confuse how would I convert them into .nemo format or is there any other way to use those models for inference.


code : [quartznet.py](https://nemo-public.s3.us-east-2.amazonaws.com/transfer-learning-04-2020/quartznet.py)
nemo version = 0.11


Also it will be helpful if anyone can share tutorial for finetuning on latest release


We have completely changed NeMo since .11. We highly recommend to start using our latest stable version. Please see our User Guide: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/ and our Tutorials: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/starthere/tutorials.html for how to get started with NeMo.
Update preproc_mt_data.py tarred dataset
State: closed | Pull request: yes

If just a metadata_file is passed in through the config, it won't enter the correct elif statement

I think @ericharper may have had this in place on purpose to make sure both metadata and tarfiles were provided and not just the metadata. I'm fine with providing just the metadata JSON file (even better would be to allow specifying preproc_out_dir to avoid specifying tokenizers as well). 

Can you also handle the case where `cfg.train_ds.get('tar_files')` is not None and `cfg.train_ds.get('metadata_file')` is None and throw an error in that case since we always require the metadata file.

> I think @ericharper may have had this in place on purpose to make sure both metadata and tarfiles were provided and not just the metadata. I'm fine with providing just the metadata JSON file (even better would be to allow specifying preproc_out_dir to avoid specifying tokenizers as well).
> 
> Can you also handle the case where `cfg.train_ds.get('tar_files')` is not None and `cfg.train_ds.get('metadata_file')` is None and throw an error in that case since we always require the metadata file.

Yes, I can handle that case as well.
Is there multi-lingual support for NeMo? I am looking particularly for a Hindi model.
State: closed | Pull request: no

Hello,

Does NeMo ASR support Hindi and other languages? Will there be any tutorial/doc released any time soon on how to use NeMo ASR for multi lingual training?

yes, we support other languages https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/results.html 
But at the moment, we don't have Hindi ASR model.
Add Aliases to Model Cards
State: closed | Pull request: yes

 - Adds new `aliases` property to PretrainedModelInfo
 - Allow models to use `from_pretrained()` with aliases

Example:
```python
PretrainedModelInfo(
    pretrained_model_name="tts_en_tacotron2",
    location="https://api.ngc.nvidia.com/...nemo",
    description="model description",
    class_=cls,
    aliases=["Tacotron2-22050Hz"],
)
model.from_pretrained("tts_en_tacotron2")
model.from_pretrained("Tacotron2-22050Hz")  # Both will load the same model
```
[Question] Quantizing QuartzNet5x5 Reduces Inference Speed 
State: closed | Pull request: no

**Describe your question**

I am quantizing QuartzNet5x5 model that I use for ASR. I followed the following example: `/examples/asr/quantization/speech_to_text_quant_infer.py`

My code is as follows: 
```
quant_modules.initialize()
params['model']['encoder']['quantize'] = True

self.first_asr_model = nemo_asr.models.EncDecCTCModel(cfg=DictConfig(params['model']))
self.first_asr_model = self.first_asr_model.load_from_checkpoint(self.model_to_load)
self.first_asr_model.cpu()
self.first_asr_model.eval()
```

Code runs without errors. I am getting lots of line saying:
```
Input is fake quantized to 8 bits in QuantConv1d with axis None!
Weight is fake quantized to 8 bits in QuantConv1d with axis 0!
Creating max calibrator
Creating max calibrator
```

Then I get the result from model. Result is correct, however, inference is a slowed to 0.33 seconds average from 0.29 second average.

Am I missing something here?


**Environment overview (please complete the following information)**

 - Environment location: Docker
 - Method of NeMo install: Tested installing from source, also tested pip install

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 20.04 and Windows 10
- PyTorch version: 1.7.1 and 1.9
- Python version: 3.6 and 3.8

**Additional context**

Tested on CPU. Tried setting num_threads to 1 and 2.


Hi, Ulucsahin

The `speech_to_text_quant_infer.py` script is meant for evaluating numerical property and accuracy of the quantized network, not speed/throughput. It is expected to be slower because it runs emulated quantization.

The script can generate an ONNX graph with quantization nodes. To deploy the quantized model for optimal performance, it will need TensorRT 8.0 (Refert this [GTC talk](https://gtc21.event.nvidia.com/media/Quantization%20Aware%20Training%20in%20PyTorch%20with%20TensorRT%20%5BS31653%5D/1_qdvvff64) for more detail) to compile the model to int8 I/O and compute. `speech_to_text_quant_infer_trt.py` is the script to run the quantized model by TensorRT and get optimal performance. Note that quantization features are designed for GPU and TensorRT only. To run the quantized model on CPU, it will need a different inference engine that supports the model
PyTorch container  20.11-py3 contains pytorch version 1.8 not 1.7.1
State: closed | Pull request: no
Labels: name: bug

NVIDIA's PyTorch container version 20.11-py3 contains 
torch                     1.8
but "version currently does not support Pytorch 1.8.0"





this is not a bug. The container version looks like 1.8.0"something" - where "something" is commit id pre 1.8.0 release but a little after 1.7.1
[Question]
State: closed | Pull request: no

**Describe your question**
Is it possible to use our own VAD or Speaker Embedding models in speaker diarization? If so, how one can do it properly? 
Many thanks. 


Thanks for your question. 
Of course! You can change the `diarizer.vad.model_path` and `diarizer.speaker_embeddings.model_path` in [config file](https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_recognition/conf/speaker_diarization.yaml) to use your own models.
Entity Linking 
State: closed | Pull request: yes

This PR adds a new model, dataset, loss, training example code and notebook demo for entity linking. The loss and model are an implementation of the "Self-Alignment Pretraining for Biomedical Entity Representations" paper.

This pull request **introduces 18 alerts** when merging aed6db426bd95772754fe44ad564645ee494a9c1 into 179d49ff4f1998df60f8ccefa6ddb8a5f95ad674 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-ea22ab685b5a0f77bf123ef856320f3ae6314173)

**new alerts:**

* 16 for Unused import
* 1 for Redundant assignment
* 1 for Wrong number of arguments in a call

This pull request **introduces 18 alerts** when merging 6f7912bcc5d53fd03105582eed850f1945082b2f into 179d49ff4f1998df60f8ccefa6ddb8a5f95ad674 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e06c749c33bf16ec168789a203ee5c44e26719c2)

**new alerts:**

* 16 for Unused import
* 1 for Redundant assignment
* 1 for Wrong number of arguments in a call

Please ping me offline - let's put the tiny dataset on AWS rather than GitHub repo.
Regarding DCO, when you commit, please add "-s" flag to do signoff on your commits. To sign already committed commits, checkout DCO's bot output https://github.com/NVIDIA/NeMo/pull/2042/checks?check_run_id=2309052982 


This pull request **introduces 18 alerts** when merging 0ebb5d40f18a73e9464e8cec6e31a43d978097d5 into 52121810b428e011666e656d313b42c954def26b - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-ba0cf96127b08d3fce0f2d80a181abe2afdac28a)

**new alerts:**

* 16 for Unused import
* 1 for Redundant assignment
* 1 for Wrong number of arguments in a call
Docker can't be build using last Dockerfile
State: closed | Pull request: no
Labels: name: bug

Trying to build Nemo Docker image fails when using the last Dockerfile
(Also when trying do use also other docker file in the zip at  https://github.com/NVIDIA/NeMo/releases/tag/v1.0.0rc1)


Step 12/29 : WORKDIR /tmp/nemo
 ---> Running in 9b5b7d241aee
Removing intermediate container 9b5b7d241aee
 ---> e5fe9b0fabe1
Step 13/29 : COPY requirements .
COPY failed: file not found in build context or excluded by .dockerignore: stat requirements: file does not exist





https://github.com/NVIDIA/NeMo/pull/2054

Using the last DockerFile still get same problem 

Using /opt/conda/lib/python3.8/site-packages
Finished processing dependencies for pytorch-quantization==2.1.0
Removing intermediate container 46d355fa5012
 ---> 481e54a4d4dd
Step 12/29 : WORKDIR /tmp/nemo
 ---> Running in 52bbd41fe5e9
Removing intermediate container 52bbd41fe5e9
 ---> e7f41ad30f24
Step 13/29 : COPY requirements .
COPY failed: file not found in build context or excluded by .dockerignore: stat requirements: file does not exist


I am unable to reproduce - please used built and published docker for 1.0.0rc1

Are you using ` docker build - < Dockerfile` ?  @asher-bs 

> "This will read a Dockerfile from STDIN without context. Due to the lack of a context, no contents of any local directory will be sent to the Docker daemon. Since there is no context, a Dockerfile ADD only works if it refers to a remote URL."

Please use `docker build -f Dockerfile -t <container_name> .` as suggested in  https://github.com/NVIDIA/NeMo/pull/3223
Citrinet model with LM to reduce the WER for microphone  recorded audio
State: closed | Pull request: no

Hi 
I am using stt_en_citrinet_1024 model and able to get good transcript ,  I am using the recorded audios with microphone and WER is varying from 3.5% to 15%. This has names of person and place, how to include the words in the model.

any suggestions with following aspects

1. Preprocessing the audio 
2. using LM model for decoding ( for citrinet is there any implementation in nemo) 
3. Post processing steps say spell correction 

looking for inputs .





You could finetune Citrinet using the same Tokenizer on the specific domain (if there is sufficient data). 

If you have some noise files, noise robust training, the same method as QuartzNet can be applied to Citrinet. 

For preprocessing, there should be monochannel 16Khz wav files. We find that attempting signal denoising before inference will generally not do much better, and sometimes will do worse based on the artifacts introduced. 

For language modelling with Citrinet (and BPE models in general), we plan to release code snippets to build custom KenLM model and run beam search through similar steps as the offline asr notebook. However there are some significant differences and we have not compiled a clean script for such a task yet. I will try to prioritize that in the coming weeks. 

There is also Transformer based rescoring that can further boost offline WER reduction, though that pipeline is not ready yet.

@AlexGrinch is there any ETA (within some months?) that you expect to have the pipeline for transformer based rescoring?

Thanks you for response.
We would try to write the script with LM for BPE models . Any inputs or leads are much appreciated. 


@VahidooX If you have a rough draft, could you create a gist and share here when it's ready ? We can clean it up in the actual PR

Can someone please share some details on this ? waiting for response.

Created a PR for adding the feature of training and evaluating n-gram KenLM on top of BPE-base ASR models. It still needs the documentations. https://github.com/NVIDIA/NeMo/pull/2066

The PR to support N-gram LM for ASR models is merged : https://github.com/NVIDIA/NeMo/pull/2066
It can do grid search for beam search decoder's hyperparameters to fine-tune them. The scripts support both character-level and BPE-level models. You may read more here: https://github.com/NVIDIA/NeMo/blob/main/docs/source/asr/asr_language_modelling.rst
 
You need to install the beam search decoders and KenLM to use this feature.

Thank you very much.
NaN values on Arm Processor - QuartzNet [Question]
State: closed | Pull request: no

**Describe your question**

Hello, I trained an ASR model for speech to text conversion. My model has decent accuracy and I want to use this model on a device with ARM Cortex-A72 processor.

I am doing inference using EncDecCTCModel.transcribe. Everything is fine on Windows and Ubuntu with x86 architecture. However, when doing inference on device with Arm processor, I am getting either a single character transcription or an empty transciption. I checked the input to see if audio file was loaded correctly and it was correct. 

I set the seed to 0 on all my devices to check array values and I tracked down the issue to following line  in nemo/collections/asr/models/ctc_models.py:
`encoded, encoded_len = self.encoder(audio_signal=processed_signal, length=processed_signal_length)`

encoded values are same on arm and x86 devices with very small differences, however, I am getting lots of "nan" values in log_tensor values on arm device in the next line:
`log_probs = self.decoder(encoder_output=encoded)`

I am guessing my result gets messed up due to these values. I tried with different environments but the problem persists. 

Is there a way to make this work on Arm processors? I don't have gpu on my device with arm processor so I have to do the computation on Arm processor itself (raspberry pi 4).

I am fine with alternate solutions such as exporting the model (if that is going to solve it).


**Environment overview (please complete the following information)**

 - Environment location: Tested on Raspberry Pi 4, Windows and Ubuntu devices. Tested both with and without docker.
 - Method of NeMo install: Tested pip installation and building from source.

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 20.04, Windows 10
- PyTorch version: Tested both 1.7.1 and 1.9
- Python version: 3.8

**Additional context**

Add any other context about the problem here.
CPU Model: Arm Cortex-A72, Intel i5



I am not sure what kind of precision is used for computation by your Arm processor. 
Is it IEEE float32? Then it should work. However, if it is something else, then precision-related issue would be likely.
Compute SE in float32 for stability in AMP training.
State: closed | Pull request: yes

Squeeze-and-Excitation block introduced instabilities to Citrinet when training with mixed precision.
This change avoids computing SE in float16 all together.

As SE block does not take much of computational time, the time and memory benefits of other computations done in float16 can allow to train model faster.  

The runs on NGC and multinode cluster confirm that there is no performance degradation or instabilities when using fp16 with current patch. The speedup is 15% and memory savings are around 3Gb for 21M parameter model.

~~Can you open a new PR?~~ nvm.

I'm looking forward for this!  I'm hoping to train a citrinet model from scratch soon on A100 GPUs which could leverage really well mix-precision training.  Is there an ETA for this patch to land in the main trunk?

As discussed with Jason and Som, we're leaving the workaround for the SE with AMP right now. Once I get to repro the exact issue, we can follow up with PyTorch team.

@jprobichaud, I am a bit absorbed by watching GTC this week :) 
Hopefully, the patch will land by the beginning of the next week.
Make pretrained models download path configurable
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**

Directory where pretrained model is downloaded is hardcoded to
`cache_dir = Path.joinpath(Path.home(), f'.cache/torch/NeMo/NeMo_{nemo.__version__}/{filename[:-5]}')` in https://github.com/NVIDIA/NeMo/blob/main/nemo/core/classes/common.py#L652. 
So model download fails in environments where $HOME is not writeable.

**Describe the solution you'd like**

Provide option to specify path to download pretrained model. May be via environment variable or a parameter?

**Describe alternatives you've considered**

To unblock myself, I have currently created a change in my fork to specify download path via environment variable.
https://github.com/jarivk/NeMo/commit/f2e1c57e400b71ddcb0c76499c0cd7f29893ba9c

**Additional context**

None


@titu1994 do you see any problems with us making `cache_dir` into an argument?

This seems like a good plan, let's check for existence of global override in os, use that otherwise default to current default. 

This is now supported. 

> This is now supported.

Hello! I understand that we can now change the location of default cache directory where the pre-trained models are downloaded to, is that correct? I can't find how to do that in the documentation, could you help me with that?

Hmm true it's burried in documentation. You can use NEMO_CACHE_DIR="SOME PATH" in your environment env (maybe bashrc or similar)

Its broken again for downloading from hub: https://github.com/NVIDIA/NeMo/blob/main/nemo/core/classes/common.py#L843

Getting Huggingface cache dir will solve that. We don't explicitly set that as it can interfere with other users setup
Add asymmetric padding support to Jasper Block
State: closed | Pull request: yes

# Changelog
- Adds `future_context` parameter to JasperBlock to enable asymmetric padding for the convolution layers.
- Adds several tests for JasperBlock components
- Adds docstring to explain how to use asymmetric padding + potential pitfalls.

This pull request **fixes 4 alerts** when merging 592b26b4f30a57a3fec5afa470e1e56ffdf51d04 into e70231dd920468e2d8562aefd43d01d7572e135e - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-9d3db6c91cb004379050ced8d60b6fc51287ad21)

**fixed alerts:**

* 4 for Explicit export is not defined

This pull request **fixes 4 alerts** when merging 6903649e28099087209ab2be695ed0eeb19dcd89 into e70231dd920468e2d8562aefd43d01d7572e135e - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-09d705fc6399f0d3b71f03c15e2df2c80f74a3d5)

**fixed alerts:**

* 4 for Explicit export is not defined

This pull request **fixes 4 alerts** when merging 6c840c9413291c46186d1adf6db5708be8c82b6e into 6962f75547e7ac2bb576f4e5a862b38d815ed79d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b0ac067e68f941242473b127a6710e2dd344272e)

**fixed alerts:**

* 4 for Explicit export is not defined

This pull request **fixes 4 alerts** when merging 3c787a70c479e87bd2ae6fb170ba60cad9957017 into 2f52c361941431dc1ee963b330be9c12a7d048b5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a65ad4b1f4e4f1735d8476ad3e1dae107eb8b51c)

**fixed alerts:**

* 4 for Explicit export is not defined

This pull request **fixes 4 alerts** when merging 3bdaf0aa4ef9593b4ce1ca49bbce22eb6b8dc994 into 6ef979353257986722f51fa78a75f4385c5e0fe2 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-58243b0f0116aea5e304a51a53b4ed3d4dcc2fa3)

**fixed alerts:**

* 4 for Explicit export is not defined
Corrects support for subclass overriding "target" in Serialization
State: closed | Pull request: yes

# Changelog
- After class resolution of `target`, if calling `cls` is subclass of resolved `imported_cls`, then defer to subclass `cls` instantiation instead of the imported target class.
- Add tests to check these scenarios

Signed-off-by: smajumdar <titu1994@gmail.com>
[Question] Unable to continue training upon checkpoint restoration: TypeError: 'NoneType' object is not iterable
State: closed | Pull request: no

Hi,

I am trying to fine-tuning my model, but after a successful restore of my checkpoint, i keep on stumbling on this error: TypeError: 'NoneType' object is not iterable

I am a beginner and is using Nemo on Google Colab. I am following the tutorial provided in one of your notebooks. Below is the code on how i restored my checkpoint and how i started the training:

----------------------------------------------------------------------------------------------
import pytorch_lightning as pl
trainer = pl.Trainer(gpus=1, max_epochs=50) 

from omegaconf import DictConfig
params['model']['train_ds']['manifest_filepath'] = train_manifest
params['model']['validation_ds']['manifest_filepath'] = test_manifest

my_model = nemo_asr.models.EncDecCTCModel.restore_from('my_checkpoint.nemo')

trainer.fit(my_model)

----------------------------------------------------------------------------------------------

How i saved my checkpoint:
.
.
.



checkpoint_path = "my_checkpoint.nemo"
my_model.save_to(checkpoint_path)

----------------------------------------------------------------------------------------------

This was the error produced:
.
.
.
.



TypeError                                 Traceback (most recent call last)
<ipython-input-11-2116e952a23e> in <module>()
      1 # Start training!!!
----> 2 trainer.fit(my_model)

6 frames
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/connectors/data_connector.py in _with_is_last(self, iterable)
     43         """Pass through values from the given iterable with an added boolean indicating if this is the last item.
     44         See `https://stackoverflow.com/a/1630350 <https://stackoverflow.com/a/1630350>`_"""
---> 45         it = iter(iterable)
     46         last = next(it)
     47         for val in it:

TypeError: 'NoneType' object is not iterable

----------------------------------------------------------------------------------------------
I was able to load my checkpoint and get the original WER, but i could not proceed to further train it.

I have tried extracting my .nemo checkpoint file and find that it holds another my_checkpoint.nemo zip file that contains two other files which is model_config.yaml and model_weights.ckpt.

Am I loading the correct checkpoint file type? It says NoneType object is not iterable, I am guessing there is a problem with the format of my checkpoint?

Thank you in advance.

**Environment overview (please complete the following information)**

 - Environment location: Colab
 - Method of NeMo install: Pip install

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:

- PyTorch version: 1.7.1
- Python version: Python 3

**Additional context**
- none

can you please try with the latest main branch and re-open if the issue still exists
Fix the versioning on multi-node training.
State: closed | Pull request: yes

Used resume_if_exists instead of checking the Slurm to void creating the version folders. So whenever resume_if_exists  is enabled, we should skip creating the versions otherwise training can not continue.

> 
> 
> Please update the docstring of exp_manger so that it matches the new behaviour.
> Please add a note that the behaviour of this argument changed as of version 1.0.0.

Added to the docstring.
Talknet bug fix & TalkNet Training tutorial 
State: closed | Pull request: yes

* Add has_audio parameter to speech_collate 
    * @blisc I wouldn't like to write new `speech_collate` for TalkNet dataset, let's reuse old version with `has_audio` parameter?
* Log warning in `setup_optimizaton` when Trainer is not specified 
* Update TalkNet models that were without trainer in constructor
* TalkNet Training tutorial notebook
    * This version of notebook has 6 TODOs. It is related to branch name, link to new version of paper and pretrained models that are currently not in NGC (@stasbel, please, add them and we can finish it). Without fix, we **shouldn't** merge it. 

Signed-off-by: Oktai Tatanov <oktai.tatanov@gmail.com>
remove experimental decorator
State: closed | Pull request: yes

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

This pull request **introduces 9 alerts** and **fixes 4** when merging 63ba7e6532c47bed6b853b7f8162f42b7748aefc into 7a820ae7b2d5bc9a744e26b1b05b9d75cece20bf - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-25ee3bdaba69ccc0c2cd060900014c9fd6b95a66)

**new alerts:**

* 6 for Unused import
* 2 for Modification of dictionary returned by locals\(\)
* 1 for Signature mismatch in overriding method

**fixed alerts:**

* 2 for Unused import
* 1 for Mismatch in multiple assignment
* 1 for Explicit export is not defined
Fix and enable DALI tests
State: closed | Pull request: yes

Signed-off-by: Joaquin Anton <janton@nvidia.com>

This pull request **introduces 1 alert** when merging af670381c65ac34248df3ca92a7decefe00e40f8 into 00c5429902c5bf8a1ff3efbdff16aa8935ace8d9 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-85e49b514fdccac71e3d647f3e3a3c42454826a7)

**new alerts:**

* 1 for Unused import

Closing in favor of #2077
Actually install NeMo requirements in Docker.
State: closed | Pull request: yes

When building docker container locally: 
```
#17 [nemo-deps 10/12] RUN for f in $(ls requirements/*.txt); do pip install ...
#17 0.515 ls: cannot access 'requirements/*.txt': No such file or directory
#17 DONE 0.6s

```

The files are copied not into a separate `/tmp/nemo/requirements/` folder but directly into `/tmp/nemo/`, so we either have to create new directory first or install directly from where we copied.




Signed-off-by: Aleksei Kalinov <alekseia@nvidia.com>