renwei2024/nemo-github-issues huggingface dataset:
Bug fixes to EncDec MT model
State: closed | Pull request: yes

Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>

This pull request **introduces 1 alert** when merging 6b8bc699ec29a04c8be043eacebdfd0974c576e7 into 601b6789bcbfcd6721b9d27aee6008c9e44ba42f - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-c5dcb90b11d16b32e076d0ab25205640cc2cda01)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 7b26b1f79ba86fcad8c583bd6c4bff77329b31de into 601b6789bcbfcd6721b9d27aee6008c9e44ba42f - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5247e13878060b391ad23279b875e0b3f8c171b9)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging f00dc0547561f2ec845f29f46429a1905a893a8a into 601b6789bcbfcd6721b9d27aee6008c9e44ba42f - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-789113e3c8e5c194db43ce417b744d587b56b4be)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging eef3f5a703581dbc91c0165834e2aac6e7e3da3a into 601b6789bcbfcd6721b9d27aee6008c9e44ba42f - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-0eae7364a7eef0ad12a12a9293f0b406e74d1311)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 93e793f110ffbcea7529ced39aeb43c2187ccb20 into 601b6789bcbfcd6721b9d27aee6008c9e44ba42f - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e6f88c94021589987c87e9507dd02046c7328ca2)

**new alerts:**

* 1 for Unused import
Add WaveGlow Denoiser
State: closed | Pull request: yes

- Adds new GlowVocoder
- Removed the SqueezeWaveDenoiser module
- Denoisers are now moved into the GlowVocoder class with support for WaveGlow and SqueezeWave
- WaveGlow and SqueezeWave's denoisers are now enabled by default in their convert_spectrogram_to_audio() functions
Patch refactoring bug for RNNT Char decoding + Update configs
State: closed | Pull request: yes

# Changelog
- Patch the abstract definition of RNNT decoding
- Update configs to use 8x stride to reduce memory and execution time

Fixes https://github.com/NVIDIA/NeMo/issues/1520

Signed-off-by: smajumdar <titu1994@gmail.com>
Broken tokenizer ref in AbstractRNNTDecoding in rnnt_wer.py 
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

On [line 209 of rnnt_wer.py](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/metrics/rnnt_wer.py#L209) there is a reference to `self.tokenizer.ids_to_text(prediction)` which errors out because there is no tokenizer. This blocks the example script in speech_to_text_rnnt.py below at the validation/sanity check stage. Possibly an artifact of refactoring for BPE? What is the intended NeMo way of incorporating the tokenizer?

I changed self.tokenizer to an explicitly defined CharTokenizer and it fixed the error. 

**Steps/Code to reproduce bug**

```
        python examples/asr/speech_to_text_rnnt.py \
        model.train_ds.manifest_filepath="<path to train dataset>" \
        model.validation_ds.manifest_filepath="<path to validation dataset>" \
        trainer.gpus=0 \
        trainer.max_epochs=50
```

**Expected behavior**

Sanity check run of RNN-T.


Yup that's a bug during the refactor, I'll send a patch in sometime. 

Added a PR https://github.com/NVIDIA/NeMo/pull/1521, it might take some time to get merged so you could update the code locally in the mean time (its a one liner at the end of the PR). 

Btw, just a heads up. The configs for RNNT are just template references for Librispeech - they were not trained nor validated on Librispeech. AN4 has a different token set compared to librispeech (no special characters, no numbers etc) so the `model.labels` would need to be updated for AN4.

Great! Thanks for the heads up.
[feature request] [proposal] Always track CER in addition to WER for speech recognition tasks
State: closed | Pull request: no
Labels: name: bug

Currently one has to choose whether `use_cer` or not, and the choice is exclusive. Both are useful as the target metric, CER is more smooth.

I support this change. Can you share code for how you validate using CER? I'm using the validate code from the ASR with NeMo tutorial and it does something like 

```
for test_batch in pb(model.test_dataloader()):
          test_batch = [x.cuda() for x in test_batch]
          xs, x_lens, ys, y_lens = test_batch
          log_probs, encoded_len, greedy_predictions = model(
              input_signal=xs, input_signal_length=x_lens)
          # Notice the model has a helper object to compute WER
          model._wer.update(greedy_predictions, ys, y_lens)
          _, wer_num, wer_denom = quartznet._wer.compute()
          wer_nums.append(wer_num.detach().cpu().numpy())
          wer_denoms.append(wer_denom.detach().cpu().numpy())
```

and since `_WER` is already instantiated in quartznet I'm not sure how to turn CER on. 

I think you need to pass `use_cer = True` to `_wer` constructor, maybe this can be configured via the yaml config. Unfortunately, I don't have the final code, as we found it's not practical to have just WER xor CER.

What you could do is to compute the CER a-posteriori, but for that you need to change logging to include the epoch/iteration number. That could be a separate request :)

I figured this out and will note it here in case anyone wants to use CER in the future. Note `quartznet` is the variable name of my model so you may need to change it if your model has a different name. 

The easiest way is to toggle between CER and WER by setting `quartznet._wer.use_cer`,  
`quartznet._wer.use_cer=True` will switch it to use CER, `quartznet._wer.use_cer=False`  will switch it back to WER. 

Here is what the constructor looks like for WER and how you can override
```
from nemo.collections.asr.metrics.wer import WER
quartznet._wer = WER(
            vocabulary=quartznet.decoder.vocabulary,
            batch_dim_index=0,
            use_cer=True,
            ctc_decode=True,
            dist_sync_on_step=True,
            log_prediction=True,
        )
```

Since `log_every_n_steps` doesnt appear to apply to validation as discussed in #1518, you can use this to turn off the output logs if printing every single line is causing problems. This can also be done directly with `quartznet._wer.log_prediction=False` 

Hope this helps someone!

Edit: Also would like to say I'm very much in favor of including WER and CER by default. 

Since .metrics are calculates on cpu we prefer to preserve only one metric during training. Offline it is much easier to evaluate models with large batch sizes as well.
Update every ASR config to new Hydra format
State: closed | Pull request: yes

# Changelog
- Update every ASR config to new hydra 1.x format

Heh let's hope this is the last time we ever need such a heavy update to configs

🤞  still, I expect one more change related to this project:
https://github.com/pytorch/hydra-torch
once it will be mature enough...

@tkornuta-nvidia I miss you!

My goal is to have the initial versioned release by the end of January 😉 .

> @tkornuta-nvidia I miss you!
> 
> My goal is to have the initial versioned release by the end of January 😉 .

Hi @romesco sorry for being absent, I was involved in a project that consumed 120.5% of my bandwidth. :( replied on Zulip...
Fastspeech2
State: closed | Pull request: yes


This pull request **introduces 2 alerts** when merging 51ac7df55e15ff6e742b6761ef5ee336208f4c94 into 97b1c96b64a64ebd45ca75409347bcf5514c0fb9 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-832bc76fc446076c1c275776064deb558da6fc7e)

**new alerts:**

* 2 for Unused import
[Question] how to sold Jasper ASR Loss: inf or Gradient overflow issue?
State: closed | Pull request: no

Hello,
I train Jasper ASR but the loss is inf and Gradient overflow. How should I solve it.
I already refer to the issue #637 and see that it is already fixed as suggested for the current version but my model still has the same issue.
Thank you


try very small learning rate first. And / or learning rate policy with a longer warmup period. Then see if you can increase them.
fix for fast/slow tokenizers compatibility
State: closed | Pull request: yes

Signed-off-by: ekmb <ebakhturina@nvidia.com>

This pull request **fixes 1 alert** when merging 8ad7e9a62e964dcab1adc45a20f76b4bf09347d3 into e54a3b030e11bcc64327e10bed7ea70724651260 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-656a0fc1adcb486ec3bbd4aec5fbac97896b62f7)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging d48d9f0e34cb7174c68bcc63ee4be0e556dbe361 into e54a3b030e11bcc64327e10bed7ea70724651260 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-49a1ec54d0073b27936b6de1a4379124e8936a00)

**fixed alerts:**

* 1 for Unused import
Update model config automatically inside ModelPT
State: closed | Pull request: yes

# Changelog
- Immediately convert all Hydra 0.x configs to Hydra 1.x configs inside constructor of ModelPT
- Add tests to ensure that no backward incompatible changes exist
- Update every ASR side config at once - other domains can update as per convenience.

Hey @titu1994 I am trying to deal with some circular/lazy imports in EFF and noticed in my NeMo Cookbook unit tests I have 
`asr_model()` fixture with old structure - and I took this fixture from NeMo.

So this is a reminder - please update the structure (params etc.) also down there (i.e. in NeMo unit tests) :)

I actually left the unit tests intact in order to properly test that the convertion occurs properly - ie old model config to new model config. I think it's easier to replicate the fixture in EFF ? At least until all domains of Nemo have moved into new config style ?

NLP Jenkins test changes look to me.

Thanks!

> I actually left the unit tests intact in order to properly test that the convertion occurs properly - ie old model config to new model config. I think it's easier to replicate the fixture in EFF ? At least until all domains of Nemo have moved into new config style ?

EFF is not the problem here, what I think is this might be misleading to some people, i.e. when they will see this won't they simply think "ok, let me copy and adapt"... (I did this in EFF, so you know ;) )

Let's update it once we have uniformly updated all configs.
How to get encoder output as audio feature?
State: closed | Pull request: no

Hi all,
How can I get the encoder part output in NeMo? Basically, I'm assuming that this part extracts audio features from the raw sample. I'm trying to do some sentiment analysis with those audio features. However, the examples provided use the whole encoder-decoder pipeline. Thanks in advance.

Both EncDecCTCModel and it's bpe counterpart are pytorch modules - so you will need to write your own function that is based off of the validation_step method inside these two models.
Add tutorial entry for external language model
State: closed | Pull request: yes

I asked this before I found the tutorial notebook: https://github.com/NVIDIA/NeMo/issues/1503

This question of how to add external language models is very common, so now that it is supported I think it would be worthwhile to add it to the chart of the main README - I'd guess you will all get fewer questions that way as well.

Figured I'd save others some time and go on and submit this small PR.

@okuchaiev Is there going to be any more work on external LMs? I'm asking as this seems like that notebook is not very direct to productionize, especially since we're moving some compute to the CPU (is this not an area where we could use the GPU effectively?). 

There's also the issue where the timestamps generated, which are used for beam search, skew heavily for longer files (not talking about the offset inherent to the model), on which I am working on right now, and will create a PR for. Right now, given a longer file, the timestamps are offset from the words more and more later in the transcription.
[Question] External LM and Beam Search Notebook/Guide?
State: closed | Pull request: no

I saw that the v1.0.0b2 tagged release says "Added beam search and external LM rescoring support for character-based CTC ASR models."
Is there a notebook on how to do this, or for now is it easiest to dig around in the code?


For anyone looking at this, I dug into the PRs and found this notebook added:

https://github.com/NVIDIA/NeMo/blob/v1.0.0b2/tutorials/asr/Offline_ASR.ipynb
Add quantization support
State: closed | Pull request: yes

Uses nvidia's quantization toolkit https://gitlab-master.nvidia.com/TensorRT/Tools/pytorch-quantization.

Added two scripts for ASR model, one for calibration, one for post training quantization.

usage
```shell
python examples/asr/speech_to_text_calibrate.py --asr_model QuartzNet15x5Base-En --dataset data/librispeech/eval/librispeech-dev-other.json --batch_size 32
```

```shell
python examples/asr/speech_to_text_infer.py --asr_model QuartzNet15x5Base-En-max-32.nemo --dataset data/librispeech/eval/librispeech-dev-other.json --wer_tolerance 0.1012 --batch_size 64
```

ONNX export will fail because opset13 is not supported in Pytorch yet.

@okuchaiev 

This pull request **introduces 1 alert** when merging 9d1ffa4eb969bae0402026edecde7b854df92928 into d15c09333d009f82ad6fb5996480c18fe7fc4d2b - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-562341bd8982145a858e9b5512c28f2b95117bd1)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 6e397dbb41dff55689553bb51831cce4befa2ecb into d15c09333d009f82ad6fb5996480c18fe7fc4d2b - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-8872285ed60f86bacfe040d8c6bd294d439c7e7a)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 61f0d6c893132ec41ea8b9aea30b426fe4f9f71f into d15c09333d009f82ad6fb5996480c18fe7fc4d2b - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2b31a83c8e8220d848bbd1d65a1f37970bd4e5d0)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 42a5ef04a8471611aacef0239af5999db8a7985b into d15c09333d009f82ad6fb5996480c18fe7fc4d2b - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-bb339da58d978f5bfd003d5ac5855ded6f2bcb4e)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 389ae70cfb9460900085d757d42c531ecce53dfc into 6e349b98a0e9380e6c342762648b0ab7a0292d25 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-4a8ca62bae17c23ffa729f4c1b077d34a24be06a)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 5642078997aa3ddbba7bab88c0132d7121752b65 into 787faa5e314328b42e89aa189a7ee9e2a31d32df - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a1ab256209ef17ca9cbdc3bf367e64ed7a6572d4)

**new alerts:**

* 1 for Unused import

@skyw It seems I can not access  https://gitlab-master.nvidia.com/TensorRT/Tools/pytorch-quantization.
Docker image "v1.0.0b2 not found"
State: closed | Pull request: no
Labels: name: bug

Docker image "v1.0.0b2 not found"

Error message:

docker pull nvcr.io/nvidia/nemo:v1.0.0b2
Error response from daemon: manifest for nvcr.io/nvidia/nemo:v1.0.0b2 not found: manifest unknown: manifest unknown

The docker image has been published and should be available now.
Add transcription functionality to ASR Classification models
State: closed | Pull request: yes

# Changelog
- Refactor TopKClassification accuracy to make `top_k` a property + perform top_k operation seperately
- Add tests for transcription scenarios

Signed-off-by: smajumdar <titu1994@gmail.com>

For transcription, duration is a dummy value that's not precalculated. It would cost too much compute to precompute the durations of all the samples simply for logging.

Hmm thinking about the vad case, it doesn't work with this dataloader at all. So feel free to push changes to this branch (or an addendum PR to this one) to add vad transcription. 

I think it would be cleaner to have a `transcribe_vad` or equivalent and properly accept or precompute durations for such batched segment inference as in vad case

Hmm VAD is another case and we have a vad_infer and we have vad_frame_seq_collate_fn to handle this.

> For transcription, duration is a dummy value that's not precalculated. It would cost too much compute to precompute the durations of all the samples simply for logging.

emm I didn’t mean precompute the durations of all the files for logging. Just wondering If we have a 5 sec audio, then what if we want my inference window to be 1.28s or 1.0s?

This PR performs offline inference only - and is meant to consume the entire audio sequence and emit one label per sample only. That's why we should have a seperate method to perform segmented (aka infer every T seconds) that is not part of this PR. 

The .transcribe() method comes from the parent ASRModel and as such needs to have similar semantics - offline full scale transcription / classification over an entire sample at once go. 

> The .transcribe() method comes from the parent ASRModel and as such needs to have similar semantics - offline full scale transcription / classification over an entire sample at once go.

Got it. Thanks for the explanation!  I thought it gonna perform streaming/average window inference so I was wondering did I miss anything from dataloader for it. 

Yes, let's add another method (let's think of a good name, maybe transcribe_streaming or transcribe_segments or something like that) to perform actual stream classification over limited time segment. It's useful to have a method to do it rather than rely on notebook all the time. 

> Yes, let's add another method (let's think of a good name, maybe transcribe_streaming or transcribe_segments or something like that) to perform actual stream classification over limited time segment. It's useful to have a method to do it rather than rely on notebook all the time.

Yes good idea! We have sliced/vad_frame_seq_collate_fn now. It's gonna be awesome to have unified **streaming transcribe** function for classification models, label models, and etc. So we can have more functionality coming in online/offline such as smoothing the inference results by using average window.
[Question] : Docker image  nvcr.io/nvidia/nemo:v1.0.0b2 not found
State: closed | Pull request: no


Docker image "v1.0.0b2 not found"

Error message:

docker pull nvcr.io/nvidia/nemo:v1.0.0b2
Error response from daemon: manifest for nvcr.io/nvidia/nemo:v1.0.0b2 not found: manifest unknown: manifest unknown


on the same machine, it works for "nvcr.io/nvidia/nemo:v1.0.0b1 "

closing as duplicate of https://github.com/NVIDIA/NeMo/issues/1498
Patch for NLP vocab artifacts
State: closed | Pull request: yes

Changed setup_tokenizer logic so that if the model is being restored it will try to use the vocab file "tokenizer.vocab_file" first.

If "tokenizer.vocab_file" doesn't exist, then it will fall back to cfg.vocab_file.

If no `vocab_file` is given, we will try to get the vocab from the tokenizer and then register the `vocab_dict` and the `vocab_file`.
When writing the `vocab_dict` and `vocab_file` that we get from the tokenizer we write them to a cache directory based on a hash of the `vocab_dict`.

TODO: create docstring to clarify the flow

shouldn't tokenizer_model be handled similarly?

> shouldn't tokenizer_model be handled similarly?

ok, got it.
ASR Inference from Tensor instead of file[Question]
State: closed | Pull request: no

I would like to do inference on a tensor of audio samples instead of a file as I am inferring on audio passed via microphone and saving/loading adds extra latency. The only inference method I've been able to find for quartznet is `transcribe` which takes a list of files. Is there an easy way to infer on tensors? Or is my best bet to look at `nemo.collections.asr.models.ctc_models.transcribe` and replicate for tensors instead of files? 


All NeMo models are pytorch modules, so it should be easy enough to look at their validation/test step code and replicate for tensors. 

The easiest possible way might be to simply copy paste the validation step code and modify it use individual tensors instead of a batch from a data loader. Make sure to use no_grad()!

Thank you, I can handle that! Closing the issue but I'll share the code if I implement it.
[Question]Token_Classification_NER output with model_from_scratch.evaluate_from_file
State: closed | Pull request: no

How do I let `evaluate_from_file` read into my entire text file and produce the output?

`model_from_scratch.evaluate_from_file(
    text_file= 'Imported_data.txt',
    output_dir='.',
)`

Apparently, it's only reading the first line of my text file as shown below.

![importText](https://user-images.githubusercontent.com/45226851/99944804-7840bc80-2dae-11eb-8379-35193ea74676.PNG)
![exportText](https://user-images.githubusercontent.com/45226851/99944797-770f8f80-2dae-11eb-9419-46cb254d67a2.PNG)

The intended usage of the `evaluate_from_file()` is to run the final evaluation. The predictions for all examples are stored in a single line.

It's recommended to use `get_predictions()`.  Note, each word in the input sequence could be split into one or more tokens. In our implementation, the prediction for the first token of the input was used to label the whole word. 

[PR 1487](https://github.com/NVIDIA/NeMo/pull/1487/files) modified `get_predictions()` to read the input from a file and save the inference result, usage:
`model.add_predictions('Imported_data.txt', output_file='Exporting_data.txt')`

Please pull from the latest main to see the changes.
[Question] How can I predict the class for a wav file for an EncDecClassificationModel object?
State: closed | Pull request: no

I need the equivalent of transcribe function of EncDecCTCModel objects for EncDecClassificationModel objects.

The current implementation seems to only taken into batch of wav files and its quite confusing to follow along. 

PR https://github.com/NVIDIA/NeMo/pull/1497 addresses this. I'll update here once it's merged. 

This is merged now, closing issue. Please let us know if the solution is insufficient for your requirements
[Question]Loading NeMo token classification model for named entity recognition
State: closed | Pull request: no

I did the Token_Classification_Named_Entity_Recognition tutorial, saved the model as .nemo and now I'm trying to load up the model on another notebook.

[](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Token_Classification_Named_Entity_Recognition.ipynb)

I saved it with pytorch's suggestion,
` torch.save(model.state_dict(), PATH)` 
and trying to load with:
`model = TheModelClass(*args, **kwargs) model.load_state_dict(torch.load(PATH)) model.eval()`

But I don't have the model class, how else may I do it?

(edit)
Do I have to instantiate with,
`model_from_scratch = nemo_nlp.models.TokenClassificationModel(cfg=config.model, trainer=trainer)`
to do 
`model = model_from_scratch`
`model.load_state_dict(torch.load(model_file)`
[Question] Can I get a pretrained model to predict phonemes rather than text?
State: closed | Pull request: no

My use-case involves using nemo for new languages that have different phones and phoneme to text mappings. I would like to learn what options are available to do this, but one that came to mind was if `QuartzNet15x5Base-En` could predict phonemes, then I can have a layer that does the phone mapping from english to the new language which then makes the final phone to text mapping in the new language a lot easier.

The tutorial `01_ASR_with_NeMo.ipynb` does a good job explaining how transfer learning can be done with Nemo. Thanks
[Question] example of ConformerEncoder class 
State: closed | Pull request: no
Labels: name: feature

Are you going to provide examples on how to use the newly added ASR decoders, like ConformerEncoder? 

You may use this example to use the Conformer encoder with CTC-loss:
https://github.com/NVIDIA/NeMo/blob/main/examples/asr/experimental/conformer/conformer_linear_decoder_bpe.yaml

We have the plan to release some checkpoints trained with Confomer-CTC in our next release in March, maybe even Conformer-RNNT.
Added storage of intent and slot labels inside of the config and did …
State: closed | Pull request: yes

Added storage of intent and slot labels inside of the config and did correction in prediction function


@vladgets and/or @tkornuta-nvidia could you please resolve merge conflicts?

Hey @okuchaiev  I am not responsible for this PR.

All the TLT-related changes to this model that I need were incorporated in #1477 .

@vladgets it is your shot on what you want to do with this.

Since Tomasz did his changes in the code, I will just cancel this PR request.
Enable `default` pytorch initialization mode for ConvASREncoder/ConvASRDecoder modules
State: closed | Pull request: yes

# Changelog
- Adds `init_mode=None` option which will preserve the default pytorch initialization for the `ConvASREncoder` and `ConvASRDecoder` modules

Signed-off-by: smajumdar <titu1994@gmail.com>

@titu1994, please merge when ready
AudioText leaks memory for large-scale training datasets
State: closed | Pull request: no
Labels: name: bug

If I understand well, it appends and stores paths to audio files in a list:
https://github.com/NVIDIA/NeMo/blob/8d506d5e47367643328375e7e5b0ff6643ddb82b/nemo/collections/asr/parts/collections.py#L90

If this object is then scattered to all DataLoader workers, it leads to leaking Linux shared memory (measured by PSS) and gets some data loader threads killed by OOM killer halting the training process. Full explanation of the bug is in: https://github.com/pytorch/pytorch/issues/13246.

Large-scale training datasets contain a lot of audio files and hence a lot of audio path strings, this exacerbates the problem

One solution is to use Manager objects as proposed by @snakers4 in https://github.com/pytorch/pytorch/issues/13246#issuecomment-725488844

Another solution is to pack strings into tensors as done in my gist: https://gist.github.com/vadimkantorov/86c3a46bf25bed3ad45d043ae86fff57#file-tensorbackeddictarray-py-L32

Thanks, @vadimkantorov for reporting this. Will look into the solution and test it out. 

Indeed, when @squire-tomsk investigated stubbing out `.data` in the UserList to our share-safe list holder in https://gist.github.com/vadimkantorov/86c3a46bf25bed3ad45d043ae86fff57, the memory leak goes away. The other solution with Manager'd ShareableDict objects may require Python 3.8, but it probably should be fine.

Also if supporting the full UserList interface is not necessary, it would be best to get rid of it, since slicing would require special care with these shareable list hacks.

The leak will be even worse when persistent workers are used

@titu1994 Is the issue fixed? Or superseded by another issue? Could you please provide any clarifying information about issue closing? Thank you!

@vadimkantorov were you able to solve the issue using your tensorbacked arrays? I tried replacing audio_files in the __init__ function with a StringArray, but it crashes due to high cpu and memory load, probably when iterating through it

Another option is to switch to [tarred datasets](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/datasets.html#tarred-datasets). 

@samehraban btw do you know if TarDatasets from webdataset/in nemo support caching an index file? so that one does not need to always do a linear scan to discover what's storred in the tarball 

@vadimkantorov Do you mean a single cache for all the tar files? I don't think so.

A single index for all tar files, or even an index-per-tarfile... For if we have offset+count index, individual files can be read from tarfile by just doing seek
[Question] why is _test_dl defiend in setup_training_data() and setup_validation_data() at qa_model.py
State: closed | Pull request: no

https://github.com/NVIDIA/NeMo/blob/e82ff1863bf750673400592a8220d63dff51933a/nemo/collections/nlp/models/question_answering/qa_model.py#L282

**Describe your question**
NeMo version is '1.0.0b2'

This is a question about setup_training_data() and setup_validation_data() nemo/collections/nlp/models/question_answering/qa_model.py

I wonder why test data loader is set as None if Dataloader config or file_path for the training or validation is missing.
[Bug] Ensure lr scheduler uses limit_train_batches when working out max steps
State: closed | Pull request: yes

When `limit_train_batches` is different to the default, this is not included in the inferred calculation of max_steps. This leads to the incorrect schedule being used when `limit_train_batches` is used.

This PR adds `limit_train_batches` to the calculation to ensure the schedule is correct!

Hi, thanks for the bugfix! Could you fix the style issues by running `python setup.py style --scope=nemo/ --fix` ? Oh theres also a style issue in tests so another pass of  `python setup.py style --scope=tests/ --fix` 

@SeanNaren Thanks for the test fix, youll need to run style fixes as mentioned above to pass the tests.

Thanks @titu1994! Should be fixed :) 

Perfect, thanks !

Thanks @SeanNaren!

Thank you guys :)
Confidence measures for Quartznet predictions
State: closed | Pull request: no

Hello, 

I am interested in getting the word level confidences from Quartznet model. What would be the best approach to obtain some proxy values to word level confidences? Taking the logit sequences and normalizing them by length seems like the most straight forward approach but I am not sure how reliable this would be. Any suggestions or ideas would be appreciated.

I would also be interested in this

I am interested in this as well. 
For now, I'm using a simple approach by averaging the confidence score of each letter per word, resulting in a confidence score per word.
You can use [parlance](https://github.com/parlance/ctcdecode) to get a timestamp per letter with the beam search.

> I am interested in this as well.
> For now, I'm using a simple approach by averaging the confidence score of each letter per word, resulting in a confidence score per word.
> You can use [parlance](https://github.com/parlance/ctcdecode) to get a timestamp per letter with the beam search.

How do you get the confidence score of each letter?

> > I am interested in this as well.
> > For now, I'm using a simple approach by averaging the confidence score of each letter per word, resulting in a confidence score per word.
> > You can use [parlance](https://github.com/parlance/ctcdecode) to get a timestamp per letter with the beam search.
> 
> How do you get the confidence score of each letter?
averaging the score config score per letter


We don't support this yet, but we will let you know if we add support in the future.
[Question] Jasper/Quartznet ASR - Recommended length per file is between 5 and 25 seconds. Why?
State: closed | Pull request: no

**Describe your question**

Recommended length per file is between 5 and 25 seconds for most models. I'm curious where/how we arrive at this limitation, and what this means for results I get from running model.transcribe() on a audio longer than 25s. Some insight into where/why we have this limitation would be appreciated.


**Environment overview (please complete the following information)**

 - Environment location: [Cloud(Collab)]
 - Method of NeMo install: [from git with branch == v1.0.0b2]


Not an expert, but I'll try to help (assuming you're talking about ASR). I believe the recommendation is due to most of the training data being in that range. Since the ASR models, Quartznet and Jasper, are fully convolutional, there's no RNN and I believe this means longer than 25s shouldn't cause too much of a problem, but give it a shot and see what kind of word error rate/accuracy you get.

The potential problems I see with...
**Clips < 5 seconds** 
Not having enough context may cause problems. The receptive field of the convolutional ASR models is at least a few seconds, so really short bursts may cause a lot of the input to the model to be zeros (padding) resulting in weird results. Not sure if this actually happens

**Clips > 25s**
I would guess this isn't a problem for cpu inference. On GPU (inference, and especially training) you'd be limited to very small batch sizes and that could potentially cause problems. 

Hope that helps and holds you over until a more experienced contributors is able to answer your question. 

@rbracco Thank you!! That's what I thought regarding a lack of RNN meaning >25s shouldn't be a problem. I ran with it and it seems okay, but still giving rather unusable output. I haven't run any issues with batch size since I'm testing out transcriptions with a single audio file using the pretrained QuartzNet15x5Base-En and haven't done any additional training yet. 

```
oh i meat five workers yesrano it's crazy so that's what i' saying it depends on what you consider my team um my research team within nhat product there's two of us that are like um full on workers and then we have some lab assistants that help out ofon the brigger product
```

Seems like beam-searching via a Language Model would help but I feel the quality is still lacking even without an LM. 

@okuchaiev Any comments regarding the original question? 

A bit late, but better late than never. The above comment is correct, Clips over 20 seconds of length would exhaust GPU memory, forcing tiny batch sizes and reducing training efficiency. Clips of any duration less than that is fine, but if its too short, then either the model wont have enough context to emit tokens.
RNN-T decoding: question about greedy batch inference
State: closed | Pull request: no

As `blank_mask` contains ones where `time_idx >= out_len` and `k == self._blank_index` and zeros otherwise, we can replace the following code:

https://github.com/NVIDIA/NeMo/blob/5b6dd3850129898be47cf0d65587897ec45a5b59/nemo/collections/asr/parts/rnnt_greedy_decoding.py#L451-L454

with the following code (without masked_fill):

```python
for kidx, ki in enumerate(k):
    if blank_mask[kidx] == 0:
        label[kidx].append(ki)
```

It allows to remove `time_mask` at all. The following block:

https://github.com/NVIDIA/NeMo/blob/5b6dd3850129898be47cf0d65587897ec45a5b59/nemo/collections/asr/parts/rnnt_greedy_decoding.py#L391-L392

Can be squeezed to:

```python
blank_mask = time_idx >= out_len
```

Is it correct? @titu1994 



Yes, this is correct. I didn't think about reusing blank_mask for the label decoding step. Would you like to submit a PR to improve this for both greedy decoding implementations?
[Question] I trained a model with Token_Classification_NER tutorial and saved it as .pt file how can I load up and use it in Pycharm? (or any python env to recommend)
State: closed | Pull request: no

What are the imports or anything I may need? I have pip install nemo_toolkit[nlp] installed already.

Do I need to (2. Define and initialize the neural network) as mentioned in https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_models_for_inference.html

How may I do it?

Appreciate any help T.T Thanks in advance


You may use the .pt file as any other Pytorch checkpoint or you can save your model as a .nemo file. The .nemo file will have everything needed (except datasets) to use your model. Please see the NeMo Primer tutorial for more information: https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.0b2/tutorials/00_NeMo_Primer.ipynb#scrollTo=g91FE8mlMcnh

okay thank you!

I got the file as .nemo now, how can I just pass in a text file with only some sentences inside to have it predict the labels for me?

As it does in this part of the tutorial,
`queries = [
    'we bought four shirts from the nvidia gear store in santa clara.',
    'Nvidia is a company.',
    'The Adventures of Tom Sawyer by Mark Twain is an 1876 novel about a young boy growing '
    + 'up along the Mississippi River.',
]
results = pretrained_ner_model.add_predictions(queries)

for query, result in zip(queries, results):
    print()
    print(f'Query : {query}')
    print(f'Result: {result.strip()}\n')`

@181473h To load the pre-trained model:
model = TokenClassificationModel.restore_from("path_to_.nemo")
model.add_predictions(queries)

If you need to continue training, please follow the setup in the example: https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/token_classification.py#L89
[Question]Does anyone try to optimize quartznet inference through tensorrt?
State: closed | Pull request: no

**Describe your question**

I follow [this link](https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.0b2/tutorials/asr/01_ASR_with_NeMo.ipynb) to try to achieve quartznet inference optimization through tensorrt. I have installed onnxruntime-gpu-tensorrt and use tensorrt provider in onnxruntime. I did observe high gpu usage and gpu memory ocupation. But the inference time is a little slower than without tensorrt. I heard that some neural network can't be optimized by tensorrt. So is quartznet that case? Does anyone try to optimize quartznet inference through tensorrt? If yes, can you share some data with me? Thanks.
Dataset creation tool based on CTC-segmentation
State: closed | Pull request: yes

Adding a tool for an ASR dataset creation.  
Long audio files and corresponding transcripts are segmented with the algorithm described in [CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition](https://arxiv.org/pdf/2007.09127.pdf).  

Provided functionality:
- a complete pipeline from audio and text pre-processing, including minimal normalization needed for proper alignment with [ctc_segmentation package](https://github.com/lumaku/ctc-segmentation) to manifest generation and verification utils
- support for parallel segmentation of multiple files

A tutorial on how to use the tool is included.
How to get word level timestamps with External Language Model?
State: closed | Pull request: no

How can we get word level timestamps for the output words of external LM?

Since the word timestamps logic given in Offline_ASR.ipynb works on Greedy CTC decoder. Number of words in Greedy decoder and Beam Search decoder are usually different so we can't use this logic with Beam Search Decoder.

Is there any way to do this? I have seen some code related to timesteps in source code of ctcdecode package, do I need to try to implement greedy decoder logic into source code of ctcdecode to return timestamps and build it again? Or there is any function/package already available?

Please help.

@anujsuchchal  Did you come across a solution for this?

@ShantanuNair Yes. For this I am using another CTC Beam Search Decoder package by [Parlance](https://github.com/parlance/ctcdecode). 
This returns timesteps of characters as well in addition to Beam results which you can use to align Audio with Transcript. Hope this helps.
Closing this issue now.




@ShantanuNair and @anujsuchchal I've tried CTC decoder package by parlance to get the time stamp for the output for conformer-CTC model. I'm getting output but unable to understand how to get the timestamp for each word. Can you guys help me with this?


@dbkalaria did you ever manage to figure out how to use the timestamp information to calculate word-level timestamps? I am stuck on the same problem too.

@abarcovschi Yes I was able to extract word-level timestamp information although it was not that accurate and I had tried that about 2 years ago, didn't know whether they had added that feature or not as there was no direct implementation at that time. 

FYI, I tried different options like vosk, whisper, etc and ended up using whisper model for my task.

I was able to extract word-level timestamp information in the following way:

Using `transcribe_speech.py` with the `compute_timestamps=True` argument, the `transcriptions` object is a list of `Hypothesis` objects that have a `timestep` instance variable, which is a dictionary that has a `word` field, which is a list of dictionary objects that contain the start and end timestep (offset) for each word in the transcription text.

The start and end offset for each word can be converted to the start and end time, in seconds, for each word in the audio file using the following code:

```
# create config object from config yaml file of the model.
cfg = Config(os.path.join(args.model_folder_path, "hparams.yaml"))

# get the seconds duration of a timestep at output of the Conformer.
time_stride = 4 * cfg.get("cfg/preprocessor/window_stride")

# for each word dict:
start_time = word['start_offset'] * time_stride
stop_time = word['end_offset'] * time_stride
```

**NOTE:** I am using a Conformer model here, so the time_stride has a multiplication factor of 4, I am not sure if it is different for other types of ASR models.
[Question] Why 256 Epoch for German and 512 for Russian in Transfer learning (Based on Paper)?
State: closed | Pull request: no

Question
First of all, thank you guys specially @okuchaiev for this magnificent project on Conversational A.I. Good job Guys.

I am using NeMo **ASR** for **Persian** language mainly because of being able to use **transfer learning** due to lack of labeled data. I got 40% WER on my validation data after working on NeMo for about 3 months. For more information, I am using Quartznet15x5 (En) and used transfer learning for another language. _But there is 2 main question here:_

1. According to the [Paper ](https://arxiv.org/abs/2005.04290): "The Russian model was trained for 512 epochs while the Spanish and German ones for 256 epochs."
But I am stuck on the 40% WER after almost 15th epoch and it is not changing even after 50 epochs. **Am i experiencing overfitting here?**
On the other hand, based on the main use of transfer learning, there is no need to train the per-trained model for this **such large number of epochs.** Please let me know what is the reason of this strategy of training.

2. **If this is overfitting, How can i regularize it?** 
(Can Dropout help me with that? ) If YES, is there any advice for setting that?

Environment overview:
-Environment location: Jupyter Notebook (Cloud Services)
-Method of NeMo install: !python -m pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[all]

Environment details
-OS version : Ubuntu 18.04
-PyTorch version : 1.6.0+cu101
-Python version : Python 3.6.9
-GPU model: GTX 1080 Ti

Can you share more info about your training process?

-How many hours of training data do you have?
-What is the size of the Persian vocab? Did you change the decoder to use the new vocab? 
-What is the loss doing? Is it falling to a certain level and stalling? What's the lowest loss you get and can you share a graph if possible? 

If you have a colab notebook I'd be happy to take a look. I also discovered a possible flaw in the initialization of the decoder weights in NeMo as discussed in #1462. Using the standard initialization my transfer learning frequently fails to converge. 

I had more success by doing the following:

```
quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")
quartznet.change_vocabulary(new_vocabulary=en_ipa_vocab)
print(quartznet.decoder)
```

![92c750a436cccb6429eba20cdf16f5a9](https://user-images.githubusercontent.com/47190785/99440937-3d89e080-28e5-11eb-9c62-d103db67aeab.png)


Then manually reinitialize the first layer of the decoder so that it uses PyTorch's default Kaiming Uniform initialization instead of NeMo's Xavier Uniform. Replace <N_CLASSES> with the number of symbols in your vocab (+1 for the blank token). It is the same value that has a red box around it in my decoder above.  

```
quartznet.decoder.decoder_layers[0] = nn.Conv1d(1024, <N_CLASSES>, kernel_size=1, stride=1)
```

Hope this helps and let me know if you have any questions.
Added new features to Speech Data Explorer
State: closed | Pull request: yes

Added new features to Speech Data Explorer:
 - backend paging/sorting/filtering (allows to work with large datasets)
 - support for assigning a port
 - minor tweaks to UI

Signed-off-by: Vitaly Lavrukhin <vlavrukhin@nvidia.com>

This pull request **introduces 2 alerts** when merging 918f97f81514c0f243321e81eb73adcd49c18cc3 into a711ca1ad2987797b1e80ac9b0ffd7479c0a4730 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a817d9e013ff7a00f492166aeb5bf1830ad2cba6)

**new alerts:**

* 2 for Except block handles &#39;BaseException&#39;
Register BERT model config for Jarvis
State: closed | Pull request: yes

This PR adds the `.register_bert_model` and `.register_tokenizer` methods to `NLPModel` and lifts the `_setup_tokenizer` method to `NLPModel`. 

For HuggingFace and Megatron BERT models the BERT config will be written to `pretrained_model_name_encoder_config.json` and added to the `.nemo` archive.

For tokenizers, the vocab file and tokenizer model will be added to the `.nemo` archive. If no vocab file is provided, and using HuggingFace AutoTokenizer, then the `vocab_dict` will extracted from the tokenizer object and the `vocab_file` will be created from the `vocab_dict` and both saved to the `.nemo` archive. 

This gives Jarvis the necessary information to generate a TRT Engine from the PyTorch checkpoint.

Calling `.register_bert_model` from PTL `.setup` hook so that all models subclassed from `NLPModel` will register their bert model (if possible).

awesome!!

This pull request **introduces 2 alerts** when merging 12122b266f59f46f76414748c73da675f830ec03 into a711ca1ad2987797b1e80ac9b0ffd7479c0a4730 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-cf2265463b3732f561da98b7d2b7b43d39332d52)

**new alerts:**

* 2 for Unused import

This pull request **introduces 2 alerts** when merging 46b256d87db223294478f98dac0619c4d964452a into a711ca1ad2987797b1e80ac9b0ffd7479c0a4730 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-9b15b496347258ec100ac86d6420295375459224)

**new alerts:**

* 2 for Unused import

This pull request **introduces 2 alerts** when merging 176b240529c2007c0807ae1c46a053532b1dccd8 into a711ca1ad2987797b1e80ac9b0ffd7479c0a4730 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-18f8083ea32be718223835aa8f576a38940e5d19)

**new alerts:**

* 2 for Unused import

Once everyone is happy with this PR I can make changes to other NLP models that need to register artifacts for Jarvis.

> @ericharper please change `nn_config.json` to ``bert_config.json` and we're good to go :)

I vote for `pretrain_config.json`. 
Rest LGTM.



Had a discussion with @tkornuta-nvidia and best if we go with 
`megatron_config.json` and `bert_config.json` for now.

This pull request **introduces 8 alerts** when merging 0b9e8446a6b69fc1f90d184897c5bb04d09acccc into fb802c524d784da616e7fe7788255ef89520682d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-59ed07db28c65458b07356418e534e132426a4e9)

**new alerts:**

* 8 for Unused import

This pull request **introduces 8 alerts** when merging 6a376ba3ccd63a4ddc83f0cb9204f789e8bbd7ac into afdfe6879589d2c21aa73f84def1f53d9b8a3211 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e5650899bbd05ec43832c812ad13807f1d87a2f9)

**new alerts:**

* 8 for Unused import

This pull request **introduces 8 alerts** when merging 72eb208ae724dfbc291218622f30c48505fa0136 into afdfe6879589d2c21aa73f84def1f53d9b8a3211 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-3e269e05c2e79901f9d205c8dde2c16a7aeda9a5)

**new alerts:**

* 8 for Unused import
Update NeMo core for melgan
State: closed | Pull request: yes

Split off updates to nemo core that are required for #1418 

Changes:
- Remove empty *_step() methods from ModelPT
- Treat None inside of ElementType's parameters as a Void that matches anything
  - Use case being removing the default 16kHz sample_rate from AudioSignal and replacing it with None. This allows a module with type AudioSignal() to match ANY AudioSignal regardless of sample_rate

Minor Changes:
- Add Singleton NeuralType
- Update wrap_training_step such it is doesn't require a dictionary to be passed
- Other cosmetic changes