renwei2024/nemo-github-issues huggingface dataset:
[Question] ASR Correction output
State: closed | Pull request: no

Hello, 

I'm trying to train ASR models for different languages. The problem is that the output contains many mistakes. Do you have any idea to correct them.

Thanks in advance,
Sincerely.

The best bet it to train on more data.
If you do not have more data, then try fine-tuning from pretrained model (even from another language) by re-using acoustic encoder.
And, of course, you will need to play/experiment with hyperparameters
How can i the change the provided classes on the mapping into my own [Token_Classification_Named_Entity_Recognition]
State: closed | Pull request: no

Hi, I am using the Token_Classification_Named_Entity_Recognition tutorial for a project.

By default, the dataset provided these labels: 
LOC = Geographical Entity
ORG = Organization
PER = Person
GPE = Geopolitical Entity
TIME = Time indicator
ART = Artifact
EVE = Event
NAT = Natural Phenomenon

How can I change the classes in the provided mapping such that it can use my labels and text file instead?

(My labels include: label_names = ['O', 'B-Product', 'B-Processor', 'B-WindowVersion','B-GPU','B-Display','B-MemoryStorage','B-Storage','B-Weight','B-Warranty','B-Price'] )

Attached is the error I received.
![gh2](https://user-images.githubusercontent.com/45226851/98646694-ad8ce980-236e-11eb-82e5-6ef86bf05d07.PNG)


If your labels are not a subset of the provided labels, it's recommended to use [this script](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/token_classification.py). Note, you `shouldn't` use a pre-trained TokenClassificationModel, instead start fine-tuning with the original BERT model. When you provide a path to your data directory, the text to int labels mapping will be created automatically based on the training data.

The data directory should follow the format described in the script or in the [tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Token_Classification_Named_Entity_Recognition.ipynb). 

I now have loaded the files into `DATA_DIR = "DATA_DIR"` but I still get the same error though it's a different tag that is caught (B-GPU above but now B-Price)

I also stopped the pre-trained TokenClassificationModel and ran the original BERT model
![gh4](https://user-images.githubusercontent.com/45226851/98708429-177fb000-23bc-11eb-954e-bc13afec15d3.PNG)


Could you please try following the `Model Training From Scratch` section of the [tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Token_Classification_Named_Entity_Recognition.ipynb) and initialize your model with `model = nemo_nlp.models.TokenClassificationModel(cfg=config.model, trainer=trainer)`?
This should create labels mapping from your training data.

If you're running locally, it might be easier to use the training script with the default config: 
`python examples/nlp/token_classification/token_classification.py model.dataset.data_dir=<PATH_TO_DATA_DIR>`

If the issue persists, please add the output of the model initialization call here.

I got it to work now thank you!
Update tarred dataset conversion script
State: closed | Pull request: yes

Bugfix for if maximum duration set to None
Added minimum duration filter

This pull request **introduces 6 alerts** when merging 57bbdd1984e6529cb18f138488bad775e10a19b1 into c0fedd91e316a4ba95ac6b14330ff92844c56097 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-fc89a4569b49ec7de84b75ce25de8500ede41b9d)

**new alerts:**

* 4 for Unused import
* 2 for Syntax error
[Question] Difference between `blank` and `space` character in ASR quartznet?
State: closed | Pull request: no

How to differentiate the meaning of these two characters? Which one represents silence, gap for next word or the end of word? Thanks in advance :)

`space` character is a regular delimiter between words [` `]. Usually, it is also returned for silent segments. 
`blank` symbol is a special extra character in CTC modeling that allows representing double characters, for example. It is removed from final decoded transcript. More details could be found in the following great paper: https://distill.pub/2017/ctc/ .

That's very helpful, thanks
[Question] How may i save the model to my own file in colab?
State: closed | Pull request: no

After running trainer.fit(model_from_scratch) i see that the models are actually saved in 

" Epoch 0: val_loss reached 0.38735 (best 0.38735), saving model to /content/nemo_experiments/token_classification_model/2020-11-07_03-05-01/checkpoints/token_classification_model---val_loss=0.39-epoch=0.ckpt as top 3"

How can I access this model or save it into my own local file (training_session_folder)?

I need to use the model to pass in a text file and predict the labels

p.s. I'm a beginner pls help T.T
![gi1](https://user-images.githubusercontent.com/45226851/98432545-3d762d80-20fa-11eb-94a5-b632a846d481.PNG)
![gi2](https://user-images.githubusercontent.com/45226851/98432548-3f3ff100-20fa-11eb-8eee-baf372b8537f.PNG)



You can use google.colab library to download files off of google colab - https://stackoverflow.com/questions/48774285/how-to-download-file-created-in-colaboratory-workspace.

Instead of `example.txt` in that snippet, you can provide the full path to the checkpoint instead and it should (hopefully) download the file to your local drive.

[EDIT]
Or the more modern way of simply using the inbuilt file browser to the left on Google colab - following the second comment on the above link

Sorry I think what I meant to say is to save a copy of the model into training_session_folder on google colab as it by default does into nemo_experiments 

For that, you can change the experiment manager config - the `exp_dir` to be precise. Set the exp_dir to your training path and it should save model checkpoints there. 

![gh1](https://user-images.githubusercontent.com/45226851/98622686-03e53280-2345-11eb-87f8-572f47284df4.PNG)
i did this not sure if it is correct

Sure that's fine too, but you want a file name inside model_file, so it should be `/content/training_session_folder/ner_model.pt`

You can more easily just use model.save_to(model_file) and then use TokenClassificationModel.restore_from(model_file) for more easy saving and restoring of Nemo models. 

okay thank you so much for replying, really appreciate your help!!
Add support for replicated shard strategy for tarred datasets
State: closed | Pull request: yes

# Changelog
- Support `replicate` strategy for sharding of tarfiles under ddp for Tarred WebDataset.
- Add config line to all ASR models

Signed-off-by: smajumdar <titu1994@gmail.com>

As in remove the configs shard_strategy = scatter portion? We could drop them, but it reduces visibility. Practically, it should be fine, since the shard strategy is not used unless is_tarred is set 

Edit: I do agree that the configs are going quite big though. 

I was worried about it potentially causing some confusion, since it's not explicitly named in relation to the tarred dataset. How do you feel about a `null` option like for `tarred_audio_filepaths` when `is_tarred` is False?

Oh that's a good option. None by default switches to the current scatter strategy and otherwise the user sets it explicitly. I'll make the changes 

Another is to name the parameter `tarred_shard_strategy` 

Either sounds good to me

Thanks !
[Question]Quantized QuartzNet
State: closed | Pull request: no
Labels: name: feature

I am trying to quantize a QuartzNet 15x5 encoder like this:
```python
jasper_encoder = jasper_encoder.cpu()
jasper_encoder.eval()
jasper_encoder.qconfig = torch.quantization.get_default_qconfig('qnnpack')
torch.quantization.propagate_qconfig_(jasper_encoder)
jasper_encoder = torch.quantization.prepare(jasper_encoder, inplace=False)
jasper_encoder = torch.quantization.convert(jasper_encoder)
```
But, unfortunately, the encoder is not fully quantized, but only some RuLu. I also tried to quantize MaskedConv1d separately, but unfortunately even in this case the Conv1d layer was not quantized
This is what I get when I do print (jasper_encoder.encoder) after quantizing:
```
Sequential(
  (0): JasperBlock(
    (mconv): ModuleList(
      (0): MaskedConv1d(
        (conv): Conv1d(64, 64, kernel_size=[33], stride=[2], padding=(16,), dilation=[1], groups=64, bias=False)
      )
      (1): MaskedConv1d(
        (conv): Conv1d(64, 256, kernel_size=(1,), stride=(1,), bias=False)
      )
      (2): BatchNorm1d(256, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
    )
    (mout): Sequential(
      (0): QuantizedReLU()
      (1): Dropout(p=0.0, inplace=False)
    )
  )
...
```
As you can see from this output, neither Conv1d nor BatchNorm1d were quantized. Could you help me with quantizing QuartzNet if it is possible to quantize this model

**Environment overview (please complete the following information)**

 - Environment location: Bare-metal
 - Method of NeMo install: pip install

**Environment details**
 - OS version: Ubuntu 18.04 LTS
 - PyTorch version : 1.6.0
 - Python version: Python 3.6
 - NeMo version: 0.10.1


I don't think they can help with this
Pytorch need to support Conv1d and BatchNorm1d in order for you to quantize it.

The only way you can do it is by converting Conv1d to Conv2d(same for BatchNorm) using [this](https://github.com/NVIDIA/NeMo/blob/c90e3aaee67a73dc1c2b0c0614766246e403e1c1/nemo/utils/export_utils.py#L53)

That code not working right now, you need to change some stuff to make it to work

we are actually working on adding quantization support, but it is not there yet.

I apologize, I was wrong with the PyTorch version, I used 1.4.0 but have now updated to 1.6.0
The problem with Conv1d is gone as it is supported in version 1.6.0. Also the issue with BatchNorm1d has been resolved (https://github.com/pytorch/pytorch/issues/41642#issuecomment-668115459). However, now, I have a problem with ReLU.
For some reason, not all ReLUs are quantized and, most likely because of this, at the trace stage in the line https://github.com/NVIDIA/NeMo/blob/v0.10.1/nemo/collections/asr/parts/jasper.py#L425 throws "RuntimeError: _Map_base :: at" exception
If necessary, I can send the full output of the architecture after quantization
VAD offline inference 
State: closed | Pull request: no

Hello!
I ran into the VAD Inference problem. I cannot use offline inference if I do not have an output or input of a device, because the pyaudio requires a device. Are there any solutions?
I just want to feed a track into the model and get a predicted time. To do this, I'll just rewrite the callback function, but how to get around the problem with the lack of devices?

Hello @4nton-P 

- I just want to feed a track into the model and get a predicted time. To do this, I'll just rewrite the callback function, but how to get around the problem with the lack of devices?
It is designed for the demonstration in the tutorial you don't have to use it to do inference for VAD.

We're updating the inference pipeline here`https://github.com/NVIDIA/NeMo/pull/1384` and will be updated and merged soon. It will help you to perform **AUDIO -> MANIFEST -> VAD OUTPUT TXT -> VAD RTTM-like TABLE**

I will let you know once it merged. Hope it helps. 


> Hello @4nton-P
> 
> * I just want to feed a track into the model and get a predicted time. To do this, I'll just rewrite the callback function, but how to get around the problem with the lack of devices?
>   It is designed for the demonstration in the tutorial you don't have to use it to do inference for VAD.
> 
> We're updating the inference pipeline here`https://github.com/NVIDIA/NeMo/pull/1384` and will be updated and merged soon. It will help you to perform **AUDIO -> MANIFEST -> VAD OUTPUT TXT -> VAD RTTM-like TABLE**
> 
> I will let you know once it merged. Hope it helps.

Thanks, I'll wait

@4nton-P  The PR is merged. Feel free to play with the scripts. Questions and feedbacks are very welcome!
I'm closing this issue. Please reopen or open a new one with questions. Thanks!
Fix perplexity
State: closed | Pull request: yes

Fix for issue https://github.com/NVIDIA/NeMo/issues/1289 . Move code to the branch in the original repo for easier collaboration with @yzhang123 

This pull request **fixes 1 alert** when merging 95dfdd4e791577dfdcef46bb40b29ea6fc654767 into 1ac828df423fbcec1b34c650b3a20266bb133dde - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-49f4b5e0cd5f2f08dc1ab6bb582037306b47e847)

**fixed alerts:**

* 1 for Unused import
Add GAN-based TTS Vocoder
State: closed | Pull request: yes

- Adds the fullband MelGAN model as described in https://arxiv.org/abs/2005.05106
- Remove empty *_step() from ModelPT
- Split UniGlowLoss into glow portion and STFTLoss portion
- Simply stft_exact_pad option in FilterBankFeatures


ToDos
- [x] ~Enable FP16~. No currently a priority.
- [x] Add NeMo Loss subclass
- [x] Update Docs
- [x] ~Update TTS tutorials~, can't until I upload the .nemo files

This pull request **introduces 6 alerts** when merging 1a32266b9ae76f036aac21b58486d9e9c54ebe40 into 4de7bd8394fb7e809e04ee8bcadcf9a1711ef497 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-f4743e03b4be34541d11903c6d31e9d4f4d054b4)

**new alerts:**

* 3 for Unused import
* 1 for Signature mismatch in overriding method
* 1 for Unused local variable
* 1 for Suspicious unused loop iteration variable

This pull request **introduces 3 alerts** when merging 31efecac501313a482045fa84f4c9de37b1c0a43 into 4de7bd8394fb7e809e04ee8bcadcf9a1711ef497 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-213f5a3610ecb1c635b1218b7044c6fe988b1630)

**new alerts:**

* 1 for Signature mismatch in overriding method
* 1 for Unused import
* 1 for Wrong number of arguments in a class instantiation

This pull request **introduces 1 alert** when merging 1d83b438b70c1b622a1fdb773462fc4dfda432d8 into 4de7bd8394fb7e809e04ee8bcadcf9a1711ef497 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-da57d6a36fa0609eaa3daebd4bebdf361ce395c7)

**new alerts:**

* 1 for Signature mismatch in overriding method

This pull request **introduces 1 alert** when merging ea45a80522c99426c3e70465af8bb5a087b52624 into 38c854da3f553b6577d6f8b08490e026b53b7ea3 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-8cbef5ba362dbc3ec63a86ec04acafd90e0fc464)

**new alerts:**

* 1 for Unused import
ASR Post Processing with language model with new Nemo version
State: closed | Pull request: no

How can we deploy ASR post processing on the new Nemo Version 1.0.0b1 ?
I found this [tutorial](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/v0.11.0/nlp/asr-improvement.html#asr-improvement) and this [page](https://github.com/NVIDIA/NeMo/tree/fa68d336cfefbbbb1849ff1b6ef454149f45234d/examples/nlp/asr_postprocessor) but they don't seem to work with the new Nemo Version


**Environment overview (please complete the following information)**

 - Environment location: Bare-Metal with GPU / GCP
 - Method of NeMo install: pip install


**Environment details**

- OS version: Ubuntu 20.04.1 LTS
- PyTorch version : 1.6
- Python version:  Python 3.7.6




sorry this feature have not been ported yet.

@okuchaiev is it wip or will take some time?

This is available in Nemo. Please refer to Nemo docs for ASR Language Modelling.
Segmentation fault (core dumped) when set create_tb_writer=True
State: closed | Pull request: no
Labels: name: bug

Dear all,
I have following problem when I run speech2text file:

**Setup NeuralModuleFactory to control training nf = nemo.core.NeuralModuleFactory( create_tb_writer=True, ...)*
The system will raise an error: 
################################################################################
### WARNING, path does not exist: KALDI_ROOT=/mnt/matylda5/iveselyk/Tools/kaldi-trunk
###          (please add 'export KALDI_ROOT=<your_path>' in your $HOME/.profile)
###          (or run as: KALDI_ROOT=<your_path> python <your_script>.py)
################################################################################

Segmentation fault (core dumped)
**Environment overview (please complete the following information)**

 - Environment location: anaconda, nemo v0110
 - Method of NeMo install: pip install nemo_toolkit[all]

**Environment details**

Otherwise, please provide:
- OS version: Ubuntu 16.04.5 LTS
- PyTorch version: pytorch 1.6.0
- Python version: Python 3.7.7

**Additional context**
CUDA Version: 10.1
GPU: GeForce GTX 1080

Please show me where was wrong!  
Thank you very much!
Cao Hong Nga


Thanks for the issue. Please upgrade your version of NeMo to 1.0.0b*.

See https://github.com/NVIDIA/NeMo/issues/1091 for a discussion on what has changed.
Update checkpointing to support model parallel
State: closed | Pull request: yes

TODO

This pull request **introduces 1 alert** when merging f0345ed4dbf94798b1375f4798b997ebd1115f22 into 99ed9c2ae5c840a8723f236530ccb1abe8c75377 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-fa795c7f3253cb10f0b090983cf620dbc7125e51)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging d7da994dbae189c1e02a6f51efe924046ca2f03b into 99ed9c2ae5c840a8723f236530ccb1abe8c75377 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-9bea83783937cb7e953f7b8a603b137f54e21238)

**new alerts:**

* 1 for Unused import

This pull request **fixes 2 alerts** when merging 0e0f7cfcf8579705f3e7dad5d6f54340e0ecb8f0 into 4de7bd8394fb7e809e04ee8bcadcf9a1711ef497 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-8aa805f1579286cee43cd4b310d05532f97d9e0b)

**fixed alerts:**

* 2 for Unused local variable
[Question] Does TTS modules (for example Tacotron2) support fine-tuning like ASR ? 
State: closed | Pull request: no

**Is fine-tuning with TTS not supported yet?**

Am working on transfer learning with TTS models From English to the Arabic language. I saw an example of fine-tuning using ASR Quartznet model mentioned [here](https://github.com/NVIDIA/NeMo/issues/1107#issuecomment-686005952).
I am using this pre-trained model:

`tacotron2= nemo_tts.models.Tacotron2Model.from_pretrained("Tacotron2-22050Hz")`

I found that Tacotron model doesn't support  `model.change_vocabulary()`  and it raises receptions when I am using `model.setup_training_data()`.

Nemo version: 1.0.0b1
Method of NeMo install: `pip install nemo_toolkit[all]==1.0.0b1`


Hello, any update on this issue?

I have the same problem, I would like to perform a fine-tuning of Tacotron 2 model with a different set of labels (characters/phonemes in input to the model).

I'm also interested in simple fine-tuning without changing the set of labels, is this feature already available in Nemo? I didn't find the option to start the training from a checkpoint.

Thanks!

Tacotron is not easily finetunable, we suggest using other supported TTS models which support finetuning.
[Question] Conformer pre-trained models
State: closed | Pull request: no

Thanks authors for publishing ASR sources. I know that QuartzNet & Jasper are comparable to some SOTA model for S2T tasks. Recently, Google published "Conformer" encoder network and achieved very good performance on LibriSpeech. I saw that, that network source is released on new Nemo version. Could you please release pre-trained model for this?

I would also be very interested!

Hi, first thanks for developing this repo. It is quite useful. I am also interested in the pretrained conformer model. It would be helpful if this could also be added to ngc

Thanks,

We have the plan to release publicly some checkpoints with Conformer-CTC, and maybe even Conformer-RNNT in our next release in March.

> We have the plan to release publicly some checkpoints with Conformer-CTC, and maybe even Conformer-RNNT in our next release in March.

Thanks Vahid, this would be very helpful

Thanks @VahidooX

I see there are now pretrained conformers for the CTCEncDecBPE models [here](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/results.html#automatic-speech-recognition-models). Are there any checkpoints anywhere for the char version? I found the yaml file for training conformer + char but can't seem to find any pretrained models or checkpoints. Thanks!

We do not have any plan to release any publicly available checkpoints trained with char-level encoding. In our experiments, accuracy of char-level models was worse than BPE-level models. Generally when you use downsampling, BPE-based models may work better. The current released checkpoints for Conformer-CTC are trained with just LS, and we have the plan to release some better checkpoints trained on larger datasets in the following month.

@VahidooX , any plans on releasing Conformer-RNNT pre-trained weights? It would be helpful if you could share any sort of rough timeline as to when can we expect it.  

It is on our roadmap and very likely that we release some RNNT models by the end of July, maybe even sooner. We are also going to release some Conformer-CTC models trained on larger datasets this month.

Conformer-CTC checkpoints published:
https://ngc.nvidia.com/catalog/collections/nvidia:nemo_asr

Is there anywhere a public table with librispeech test-other results?

Model Cards at https://ngc.nvidia.com/catalog/collections/nvidia:nemo_asr require nvidia login. On the side, I think this is baseless restriction of access.

You may find the results on test-other in their corresponding model cards. Unfortunately we don't have a table with all results in the same place. You don't need to create accounts to get access to NGC, there is a button labelled "guest" on left bottom side of the login page. If you do it just once, you get access to all public pages on NGC.
Fix another hanging bug in text classifiction model.
State: closed | Pull request: yes


This pull request **introduces 2 alerts** when merging 260e83eac7de1938e07b5665c52b5d5f26906d50 into 536cf58e010a5ce389ed16ce52acce80e5524a1d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-7330a0019def9291ed2c5484a46b6fc4616b32f2)

**new alerts:**

* 2 for Unused import
Remove call to optimizer from the model initialization and added upda…
State: closed | Pull request: yes

Removed call to optimizer from the model initialization and added update data directory function

@vladgets do you still want to merge this?

> @vladgets do you still want to merge this?

Yes, I do

> > @vladgets do you still want to merge this?
> 
> Yes, I do

Should this go to main or v1.0.0b2 @okuchaiev? It's needed for tlt
How to use/port quartzNet model on mobile? [Question]
State: closed | Pull request: no

Can I deploy NeMo trained ASR on mobile platform (Android/iOS)? If yes then what are the steps? Is there a working demo?

> Can I deploy NeMo trained ASR on mobile platform (Android/iOS)? 
- Yes!

> If yes then what are the steps?
- Every NeMo model is inherited from ```torch.nn.Module``` , therefore you should checkout Pytorch tutorials about deploying models on mobile. For example, you can export QuartzNet ASR models into .onnx file for deployment.

> Is there a working demo?
- Unfortunately, we don't have an open-sourced working demo yet.

@okuchaiev Thanks for the reply.

I am looking for some answers in order to deploy the NeMo model on mobile.

1. How to export Nemo quartzNet model to .onnx?
2. When we deploy it on mobile then what are the libraries required (does it have some dependency on 3rd part libraries apart from pytorch code)?
3. What will be inference code in mobile?

@rahul-hiker you can follow the instruction: https://pytorch.org/mobile/home/
I hope it helps you deploying Nemo model on mobile
[Question] How do I infer the speaker recognition model/module at 1.0.0b1.
State: closed | Pull request: no

I have trained a speaker recognition model and save the checkpoints. Even can restore it using the LightningModule.load_from_checkpoint() class method. But couldn't able to perform the next steps after it. 

Please suggest how do I perform the next steps i.e then predict the speaker on unseen data using trained model at  at 1.0.0b1 version ?


@uditdixit11, 
If you have finetuned or trained all together with a new speaker recognition model, you could test or infer by two steps as shown similarly in [this script](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/speech_to_text_infer.py) . 
1. set up test data and 
2. get log_prob by calling forward loop of trained speaker model. 
If it's confusing let me know, I will add a infer script to current examples. 

Thanks @nithinraok

Please add the infer script to the current examples. It will be really helpful to us.

@nithinraok  Would appreciate an infer script as well, working on it myself right now.

Did you try passing test_ds to yaml file to speaker_reco.py by commenting out trainer.fit with your trained .nemo model? 

Anyway will post a script very soon for your reference. 

Somehow missed your response @uditdixit11, please find the script requested at this [link](https://github.com/NVIDIA/NeMo/blob/afbe52cb726eeb0c8262779299f4cc4354a9625d/examples/speaker_recognition/speaker_reco_infer.py). Let me how it goes. @ShantanuNair 

Hi! It appears in the script setting up test data loader and load from test data is a necessary step, in which process audio data were somehow converted to a tensor and can be passed to the `forward()` function as an argument, I was wondering if there is a way to get around this? For example, I just have a running program that takes input audio streams, might there be a way to process the audio stream to a required tensor form and pass it directly to the `forward()` function without having to write to an intermediate test manifest and load from the test data? Has anyone tried this? Help would be appreciated! 

You may look at this discussion to get an idea: https://github.com/NVIDIA/NeMo/discussions/2248 

> You may look at this discussion to get an idea: #2248

Thank you! @nithinraok
about japanese
State: closed | Pull request: no

Thank you very much for opening such a good tool. Is nemo effective for Japanese speech recognition?

Yes, NeMo is not English-only. You should be able to train good Japanese models if you have Japanese training data. Of course, knowledge of Japanese helps too :) 

> Yes, NeMo is not English-only. You should be able to train good Japanese models if you have Japanese training data. Of course, knowledge of Japanese helps too :)

Thank you for your answer. It seems that there is no automatic speech recognition pre-training model for Japanese
[WIP] ASR inference script to output hypotheses in sclite formats
State: closed | Pull request: yes


This pull request **introduces 2 alerts** when merging 9c213ff48acc513dfd3b02654fa92fdd54b93341 into 75774cf24755bddb393e3b1a4cbcc1921daf8898 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-29dea0dd8c242cb81d2215a8c6323dfb9f2b3347)

**new alerts:**

* 2 for Unused local variable
fixed the hanging bug of text classification model.
State: closed | Pull request: yes

Dropped the evaluation with another trainer in the same script.

This pull request **introduces 1 alert** when merging 0a42bc5c87cce6e6bcbe1ccff4772a6caca25a9c into 17fc2541172dd3d781bfa0b5a1fa41576d4bfe21 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-25e2feefaf1a084c11c7c502283923afe3fa5df8)

**new alerts:**

* 1 for Unused import
Is there a tutorial for data augmentation using transfer learning on pretrained QuartzNet?
State: closed | Pull request: no

This doc: https://developer.nvidia.com/blog/develop-smaller-speech-recognition-models-with-nvidias-nemo-framework/ mentions that 

> The QuartzNet-15×5 model was trained on the LibriSpeech training dataset with two types of data augmentation: speed perturbation and Cutout

If I transfer-learn the English QuartzNet on another language, is there an easy way to add

- Speed perturbation
- Cutout
- Random background noise

to the training loop?  I know how to do this in MONAI but it's not as clear in the NeMo design whether there are audio transforms that can be added into the training loop, that is, I don't see the equivalent in NeMo of the MONAI Compose and transform loader stack: https://docs.monai.io/en/latest/transforms.html

Here is a tutorial on data augmentation https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/05_Online_Noise_Augmentation.ipynb
Is there a Jupyter tutorial notebook for BERT ASR postprocessing?
State: closed | Pull request: no

BERT ASR postprocessing is described here: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/v0.10.1/nlp/asr-improvement.html

Is there a notebook in the GitHub to demo this in combination with transfer learning of QuartzNet15x5 for a new language?

It would be very helpful to show how to transfer-learn the Bert for a transfer-learned new language QuartzNet.

Also...what happened to all the files referenced in the docs, for example `examples/nlp/asr_postprocessor`

After some digging, I found them buried in an old version.  I hope this doesn't mean this doesn't work any more:

https://github.com/NVIDIA/NeMo/tree/fa68d336cfefbbbb1849ff1b6ef454149f45234d/examples/nlp/asr_postprocessor

Actually never mind, I can just train using other text-to-text notebooks as a new language pair, the pair being ASR output to ground truth.

Hey, they've explained in other issues - https://github.com/NVIDIA/NeMo/issues/1156, https://github.com/NVIDIA/NeMo/issues/1126 that it hasn't been ported yet, but looks like they have some workarounds to adding a Language Model for incorporating context. They might want to consider adding it to their docs since it seems to pop up often.



> Actually never mind, I can just train using other text-to-text notebooks as a new language pair, the pair being ASR output to ground truth.

Also, would you mind elaborating a bit on your process? Did it help with results in your case?

Hi @ShantanuNair, they had this script in older releases, they just deleted it without explanation.

I will elaborate, with this caveat: It didn't help.

The method was just to take the predicted output from my NeMO Quartz15x15 trained ASR net, pair it with the gold translation, and train a new net.  The new net was this Seq2Seq model from Ben Trevett: https://github.com/bentrevett/pytorch-seq2seq/blob/master/6%20-%20Attention%20is%20All%20You%20Need.ipynb

I suppose I could have done the equivalent  model in NeMo but they [deleted the example](https://github.com/NVIDIA/NeMo/tree/fa68d336cfefbbbb1849ff1b6ef454149f45234d/examples/nlp/asr_postprocessor).    For the NIST exercise I was doing, I needed to train from scratch, and the (deleted) example only showed starting from a pretrained model.  I already knew the Ben Trevett code so I went with that.

The score after adding this "afterburner" net trained on the mistakes of the ASR net was only negligeably higher than without it.
FileNotFoundError: Make sure that your data follows the standard format supported by JointIntentSlotDataset.
State: closed | Pull request: no
Labels: name: bug

To Author:
Now I am using your intent_slot_classification.py. The model could be trained, however, the FileNotFoundError was shown. I just discovered you add the inference part. 
![ERROR](https://user-images.githubusercontent.com/8537280/97674424-772ab100-1ac8-11eb-8dd3-444bf947f27b.PNG)


Jonathan Sung
Fix http 500 ngc v1.0.0b2
State: closed | Pull request: yes

Try and wait in case NGC download fails.

This pull request **introduces 1 alert** when merging 4f0baa600e628c3bc964be157a258d7dea590259 into 0c575303f49dc0060ad1ff13f3ec9f6e2dcbc381 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d58640ed1f19ca480d854b6e6914fa40c21d5cc6)

**new alerts:**

* 1 for Except block handles &#39;BaseException&#39;

This pull request **introduces 1 alert** when merging 87b10c1890a5170bc752316be41e9b6d1dc5c9c0 into 0c575303f49dc0060ad1ff13f3ec9f6e2dcbc381 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-07e82da68288ebddb7b2d77a76ed3fc2850929d0)

**new alerts:**

* 1 for Except block handles &#39;BaseException&#39;

looks like it fails on code formatting though

This pull request **introduces 1 alert** when merging cb0a5176876f80cacb571f0e761535b2bdb90f20 into 0c575303f49dc0060ad1ff13f3ec9f6e2dcbc381 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-cefe62200e4bdc5bc858569e3d4ff3c0ac446297)

**new alerts:**

* 1 for Except block handles &#39;BaseException&#39;
Updated ambiguous placeholder text
State: closed | Pull request: yes

Line 29 in `examples/nlp/glue_benchmark` says `trainer.gpus="[<CHANGE_TO_GPU_YOU_WANT_TO_USE>]` which is not correct since this parameter sets the number of GPUs to use during training. 

I have updated the placeholder text accordingly to say `trainer.gpus="[<CHANGE_TO_NUMBER_OF_GPUS_TO_USE>]`.

No, that flag can accept a list of GPU ids. https://pytorch-lightning.readthedocs.io/en/stable/trainer.html#gpus 
So that placeholder looks right to me.
Inverse text normalization in NeMo
State: closed | Pull request: no
Labels: name: TN/ITN

**Describe your question**
Are you guys planing to add **Inverse text normalization** models in NeMo. Something [like this](https://docs.aws.amazon.com/transcribe/latest/dg/how-numbers.html).

If anyone have worked on this before can you share some resources where I can study about this.
Thanks :)

No. Currently NeMo ASR models transcribe 55 as "fifty five". 
In fact, if En normalization is tuned on (default) in data layer, then 55 will be converted to fifty five.

Keep in mind that there could be many valid ways to say, for example, 2020 differently - you can say "two thousands twenty" or "twenty twenty". 

Hi @okuchaiev thanks for your response. I was thinking that this could be an NLP task rather than expecting ASR to generate such results similar to punctuation and capitalization model where it do some post-processing to the enhance the ASR results further more.

Found a related
paper: https://isca-speech.org/archive/Interspeech_2017/pdfs/1274.PDF
blogpost: https://machinelearning.apple.com/research/inverse-text-normal

but there seems to be no code/pretrained models present for this.

@rohitgr7 Stay tuned, we are working on this :)
In the meantime, we have released a tool for text normalization, check it out here: https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/tools/Text_Normalization_Tutorial.ipynb

@yzhang123 Awesome!! eagerly waiting for this :smiley: 

@rohitgr7  check out this PR https://github.com/NVIDIA/NeMo/pull/1797 :)
Please reopen this issue if you have followup questions

thanks, @yzhang123. Will check it out :)

@rohitgr7 and let us know your feedback, or if anything does not work :)
Add patches for Notebooks (Model Primer, ASR)
State: closed | Pull request: yes

# Bugfixes
- NeMo Model Primer updated to use PTL 1.x api
- Fix WER calculation for ASR notebook
- Speech commands notebook update to use `-last.ckpt` instead of `-end.ckpt` as `-end.ckpt` no longer is created.

Edit: Colab updated some internal format, hence the large diff. No content has been changed.

Signed-off-by: smajumdar <titu1994@gmail.com>

are these fixes to 1.0.0b2? Why is this sent to main?
v1.0.0b2 to main
State: closed | Pull request: yes

* pin to v1.0.0b2

Signed-off-by: ericharper <complex451@gmail.com>

* pin to v1.0.0b2

Signed-off-by: ericharper <complex451@gmail.com>

* pin to v1.0.0b2

Signed-off-by: ericharper <complex451@gmail.com>

* pin to v1.0.0b2

Signed-off-by: ericharper <complex451@gmail.com>

* pin to v1.0.0b2

Signed-off-by: ericharper <complex451@gmail.com>

* pin to v1.0.0b2

Signed-off-by: ericharper <complex451@gmail.com>

* hard coding nemo version after kernel restart

Signed-off-by: ericharper <complex451@gmail.com>

* pin to v1.0.0b2

Signed-off-by: ericharper <complex451@gmail.com>

* pin to v1.0.0b2

Signed-off-by: ericharper <complex451@gmail.com>

* pin to v1.0.0b2

Signed-off-by: ericharper <complex451@gmail.com>

* pin to v1.0.0b2

Signed-off-by: ericharper <complex451@gmail.com>

* pin to v1.0.0b2

Signed-off-by: ericharper <complex451@gmail.com>

* pin to v1.0.0b2

Signed-off-by: ericharper <complex451@gmail.com>

* change --end to --last

Signed-off-by: ericharper <complex451@gmail.com>
Model saving/restoring using EFF
State: closed | Pull request: yes

 - [x] EFF dependency in requirements.txt (0.1.0 release)
 - [x] Modified `ModelPT.save_to()`
 - [x] Modified `ModelPT.restore_from()`
 - [x] `ModelPT.__set_eff_save()`/`Model.__use_eff_save()` static methods for manual enabling/disabling saving using EFF (flag set to `True` by default).
 - [x] unit test - polished test adding artifact
 - [x] unit test - testing checkpoint encryption when using _eff_save_to()/_eff_restore_from() with provided encryption key.

`ModelPT.save_to()` logic:
 - if EFF present and __use_eff_save flag set: save .nemo using EFF
 - else: save .nemo using the old method

`ModelPT.restore_from()` logic:
 - if EFF present: try to restore the model using EFF, if fails: try to restore using the old method
 - else: restore .nemo using the old method

In short:
 - Restore will be able to load both the existing .nemo files and the new EFF-based .nemo files
 - All newly saved .nemo files will have EFF under the hood.

Note that from the content of the archive perspective, the new .nemo archive is a superset of the old format, i.e. the old .nemo file content:
```
 + model_config.yaml
 + model_weights.chkt
 + … (additional files a.k.a. artifacts)
```
The new .nemo file content:
```
 + manifest.yaml (<- this file is the only difference!)
 + model_config.yaml
 + model_weights.chkt
 + … (additional files a.k.a. artifacts)
```
Remove experimental automodel
State: closed | Pull request: yes


This pull request **introduces 1 alert** when merging 9c18170f1ed653b80bdf62dc30c810dbf3e5b87f into 48620bb1369dc72dcac7b9a4e75b113f1cc55d36 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-29b3095947becb5ca10111ca89fa4e93061b71fb)

**new alerts:**

* 1 for Unused import

is this a fix? should we send this to v1.0.0b2 as well?

> is this a fix? should we send this to v1.0.0b2 as well?

both v1.0.0b2 and main are Ok, it's not a bug just removing these to avoid confusion for users that any model from get_lm() works with all NLP tasks.

changed base to v1.0.0b2
Reduce CI time NLP
State: closed | Pull request: yes

1. set exp_manager to `null` to reduce CI run time for most of the NLP examples - @VahidooX, @vladgets, @yzhang123  please check if it's ok to do for your scipts
2.Reorganized Megatron-LM tests, now we're testing:
- single GPU megatron loading from config and checkpoint
- 2 GPU model parallel 
- 2 GPU loading from ngc

Signed-off-by: ekmb <ebakhturina@nvidia.com>
Script to download commonvoice dataset
State: closed | Pull request: yes

Ties to #568, it would be nice to have a script here to download common voice (and be language agnostic). The script works in a similar fashion to the librispeech script. Let me know if theres any issues. We have a blog post coming out where I had to format CV for NeMo :)

@SeanNaren when you commit, could you please do "git commit -s ..." to "sign" your commits.
For the already committed commits, DCO's bot output tells you https://github.com/NVIDIA/NeMo/pull/1361/checks?check_run_id=1321341664 how to sign already committed commits. 

The -s flag means: "certifies that committer has the rights to submit this work under the same license and agrees to a Developer Certificate of Origin (see http://developercertificate.org/ for more information)". We thought that asking for this is an easier ask then asking every external contributor to sign some sort of CLA.


Will do! I think I made a mess with the DCO commands, will fix...

@okuchaiev should all be done!
BeamSearchDecoderWithLM, TypeError: Number of output arguments
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Traceback (most recent call last):
  File "speech_to_text_beamsearch.py", line 141, in <module>
    main()  # noqa pylint: disable=no-value-for-parameter
  File "speech_to_text_beamsearch.py", line 122, in main
    res = beam_search_lm.forward(log_probs=log_probs, log_probs_length=log_probs_len)
  File "/opt/conda/lib/python3.6/site-packages/nemo_toolkit-1.0.0b2-py3.6.egg/nemo/core/classes/common.py", line 513, in __call__
    instance._attach_and_validate_output_types(output_types=output_types, out_objects=outputs)
  File "/opt/conda/lib/python3.6/site-packages/nemo_toolkit-1.0.0b2-py3.6.egg/nemo/core/classes/common.py", line 161, in _attach_and_validate_output_types
    len(out_container), len(output_types)
TypeError: Number of output arguments provided (4) is not as expected (1)



**Steps/Code to reproduce bug**

My batch size is 4, I'm trying to use beam search decoder

![image](https://user-images.githubusercontent.com/33082061/97324583-a946d380-187a-11eb-844f-299b8df402a4.png)


**Expected behavior**

When I remove typecheck [here](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/modules/beam_search_decoder.py#L87), it works fine.



Changing [here](https://github.com/NVIDIA/NeMo/blob/6796faa62e9d9c42ee86f860f1a79d8db15561ae/nemo/collections/asr/modules/beam_search_decoder.py#L57) `return {"predictions": NeuralType(('B', 'T'), PredictionsType())}` to `return {"predictions": [NeuralType(('B', 'T'), PredictionsType())]}` (note the square brackets) also helps due to [this](https://github.com/NVIDIA/NeMo/blob/6796faa62e9d9c42ee86f860f1a79d8db15561ae/nemo/core/classes/common.py#L285) check, as the comment "the provided NeuralTypes {} should enable container support (add '[]' to the NeuralType definition)" [here](https://github.com/NVIDIA/NeMo/blob/6796faa62e9d9c42ee86f860f1a79d8db15561ae/nemo/core/classes/common.py#L296) suggests. What is a correct way to fix this output type definition?
Ensure `onnx_opset_version` consistency.
State: closed | Pull request: yes

The `attach_onnx_to_onnx` method merges two onnx graphs. It uses
`onnx.helper.make_model`. That has an `opset_import` optional argument.
If this is not set, it will use whatever the default for your onnx
version is.

I found this playing around with exporting `quartznet` in the `main` branch on the `nvcr.io/nvidia/pytorch:20.09-py3` container. While the `encoder` and `decoder` `.onnx` models were opset version 12 (as specified in the `export` method), the single model `.onnx` was opset version 13. This change causes it to use the same opset version as model1.
For example: https://github.com/NVIDIA/NeMo/blob/d061dfd7970b95790d713b24f7874511129833e5/nemo/collections/asr/models/ctc_models.py#L503

I think, given the context, it is safe to assume that `model1` and `model2` will have the same opset version?

Thank you!
how to save a fine tuned model and load it back?[Question]
State: closed | Pull request: no

hi
I am trying to save the model after fine tuning by using:  torch.save(quartznet,'fine_tuned2.pt')
but I get this error:
AttributeError: Can't pickle local object 'FilterbankFeatures.__init__.<locals>.<lambda>'
how can I save the model and load it back again?
thanks
Niloo


You can use .save_to/.restore_from methods to save to and restore from ".nemo" file. 
".nemo" file is a simply ".tar.gz" file (you can rename it to tar.gz and unpack) with checkpoint weights, configuration yaml which was used and (for some models) additional artifacts such as tokenizer's vocabulary, etc.
TypeError while trying to run the GLUE Benchmark notebook in Google Colabs.
State: closed | Pull request: no
Labels: name: bug

**Description**

The following error is reported when trying to run the GLUE benchmark notebook in Google Collab.
`TypeError: __init__() got an unexpected keyword argument 'accelerator'`

**Steps/Code to reproduce bug**

1. Open https://github.com/NVIDIA/NeMo.
2. Open the [GLUE benchmark notebook](https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/nlp/GLUE_Benchmark.ipynb) in Google Collab.
4. Run the required cells until the following code block is reached.
```# lets modify some trainer configs
# checks if we have GPU available and uses it
cuda = 1 if torch.cuda.is_available() else 0
config.trainer.gpus = cuda

config.trainer.precision = 16 if torch.cuda.is_available() else 32

# for mixed precision training, uncomment the line below (precision should be set to 16 and amp_level to O1):
# config.trainer.amp_level = O1

# remove distributed training flags
config.trainer.accelerator = None

# setup max number of steps to reduce training time for demonstration purposes of this tutorial
config.trainer.max_steps = 128

trainer = pl.Trainer(**config.trainer)
```

**Expected behavior**

The GLUE benchmark notebook should run without any errors.

**Environment overview (please complete the following information)**

 - Environment location: Google Collab
 - Method of NeMo install: `pip install nemo-toolkit[nlp]==1.0.0b1`

**Environment details**

- OS version: MacOS Catalina 10.15.7
- PyTorch version: 1.6.0+cu101
- Python version: Python 3.6.9

@Fackor I had this error while working with `pytorch_lightning==0.9.0`. Upgrading it to the `pytorch_lightning==1.0.3` solved the issue for me.

If you still want to work with `pytorch_lightning==0.9.0`, you should change   `accelerator: ddp` to `distributed_backend: ddp` in [the config file](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/glue_benchmark/glue_benchmark_config.yaml)

```
  # accelerator: ddp
  distributed_backend: ddp
```

Hope that this helps :) 

Hi, thanks for the issue. We have been working to pin the tutorial notebooks linked from the README to the correct version of NeMo and the correct version of the notebook. These changes are going into the next minor release. 

Please try the notebooks here: https://github.com/NVIDIA/NeMo/tree/v1.0.0b2 and let us know if you still have issues.
IntentSlotClassificationModel exportable
State: closed | Pull request: yes

Signed-off-by: Sergei Nikolaev <snikolaev@nvidia.com>

Thanks Tom,
at this moment it's unavoidable, but let's talk to Vlad - the owner

What are the questions to me? 

How can we instantiate the model without providing so much data?

To make inference with a trained model basically we need only Dictionary of Intent and Slots, to know how to interpret the result numbers. They are 2 small text files stored in the data directory that is used for training. I can also add them inside of Config files, if it is better. For testing of training we need some training data as for any other model.

No, just to build a graph and export it to ONNX.
Fix multi dataloader and ASR logging + patch asr wer calculation
State: closed | Pull request: yes

# Changelog
- Patch multidataloader to add all of the keys from the validation / test steps.
- Add the default `val/test` log results, where default is designated by `val_dl_idx` or `test_dl_idx` (optional keys in the *_ds configs which default to the 0th dataloader).
- Patch ASR wer computation. 
  - It was computing wer two times, which impacts execution time and causes logs to be printed twice.
- Patch all subclasses of `training_step` to automatically remove all log dicts and simply use the new logging api in place.

Signed-off-by: smajumdar <titu1994@gmail.com>
Talknet tutorial
State: closed | Pull request: yes


This pull request **introduces 1 alert** when merging eaf6cf589e86f83acb2f48c1e6dc831e1533af89 into cd55594e0f60a4bcbaf6d0e4196c1404591e0076 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-9ee197eb87b278c5e13fc11f091b00114da5f04b)

**new alerts:**

* 1 for Unused import
Patch to Model Parallelism after PTL 1.0.3 Upgrade
State: closed | Pull request: yes

Hooks that we were previously using are now gone.

We were able to initialize model parallel and configure ddp from `setup` hook.

This pull request **introduces 1 alert** when merging 77f3867cb8ed29de19137c14b7606acbd6ed58fd into 53b31f7b902f96e3b964214373c328b3fa2d3606 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-fdbaf1bef5bfb2c8b4b97e7eda74c8c0cb5d0a77)

**new alerts:**

* 1 for Unused import
.nemo to single .onnx CLI converter script for ASR models
State: closed | Pull request: yes

Signed-off-by: Sergei Nikolaev <snikolaev@nvidia.com>

Now we can do
```
$ python convasr_to_single_onnx.py --nemo_file SpeakerNet.nemo --onnx_file SpeakerNet.onnx --model_type speaker
```