renwei2024/nemo-github-issues huggingface dataset:
How to use asr model for streaming
State: closed | Pull request: no
Labels: name: feature

Hello folks

I used nvidia quartznet for asr and it is taking wav files to transcribe. How to do real time streaming asr model

Thanks

How to do real time streaming using Citrinet asr model? Will it be the same way as of Quartznet?

> How to do real time streaming using Citrinet asr model? Will it be the same way as of Quartznet?

do you know how to implement  quartznet for real time streaming??



> > How to do real time streaming using Citrinet asr model? Will it be the same way as of Quartznet?
> 
> do you know how to implement quartznet for real time streaming??

No, I didn't try yet by myself.

For streaming you need FrameAsr like from this notebook https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/02_Online_ASR_Microphone_Demo.ipynb or use Nvidia Jarvis

> For streaming you need FrameAsr like from this notebook https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/02_Online_ASR_Microphone_Demo.ipynb or use Nvidia Jarvis

Yes, I tried this earlier but it didn't work for CitriNet model.

What you mean by didn't work? Did you use correct stats? https://github.com/NVIDIA/NeMo/issues/1203
And from what I know CitriNet use different stride,you might need to take that into account, I don't have experience with CitriNet yet

> What you mean by didn't work? Did you use correct stats? #1203
> And from what I know CitriNet use different stride,you might need to take that into account, I don't have experience with CitriNet yet

Correct, It is working for Quartznet model, but no output was coming for Citrinet model. I wanted to know what could be the reason of it. @khursani8 thanx for your response, I'll be checking for different stride.
change tensor interface to B,T,D between mel generator and vocoder 
State: closed | Pull request: yes

updates for  TTS models to generate mels with output of type B, T, D and all vocoders to take as input B,T,D shapes to enable continues memory when grabbing chunks of spectrogram at inference time.

Please hold off merge of this until we make sure neural type information is included in nemo files.

> Please hold off merge of this until we make sure neural type information is included in nemo files.

@ryanleary @junkin Should we try to merge this into NeMo? Or do you still want to block for now?

@blisc @junkin and @ryanleary  what's the status of this? Do we need this PR? If not, someone, please close
sgdqa update data directories for testing
State: closed | Pull request: yes

- [x ] is backwards compatible
- [x ] fixes loading previously trained .nemo model for testing

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

This pull request **introduces 2 alerts** when merging 265bf1cf644ca545c07361b0c69b139176a04c16 into cfe2548d4db9dcc4ca393ee63604139c9f681cd8 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-daa3ed728ff039b73f402e72c8d2b2aa4efa1369)

**new alerts:**

* 2 for Unused exception object
how to use language model in train
State: closed | Pull request: no
Labels: name: feature

We are trying to train models with a language model '.arpa', but reading the documentation and trying some scripts we can't get anything but the same result without the language model. I think we are doing something wrong. Do you have any tip for this question, another documentation or something like this to help us? Do we just use at the inference or not?

Regards

what models are you training?

We are using own dataset, with audios (wav) and .txt, in Portuguese. Like a small LibriSpeech

Language model is trained separately from the acoustic model such as QuartzNet. Then you can use LM during beam search process to improve the results. Please refer to this doc https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/asr_language_modeling.html 

Thanks @okuchaiev, this help us a lot and confirm something that we was thinking. 
Now we tried to use the same example in this doc and we got that:
![image](https://user-images.githubusercontent.com/47003942/121403939-d17ad280-c931-11eb-8cb5-7b4af8f08098.png)

We use beam search with lm:
beam_search_with_lm = nemo_asr.BeamSearchDecoderWithLM(
        vocab=labels,
        beam_width=beam_width,
        alpha=alpha,
        beta=beta,
        lm_path=lm_path,
        num_cpus=max(os.cpu_count(),1))
        
I don't know if have anything wrong

The documentation above links to a utility script that will perform beam search for you - please try with that. The KenLM binary built for character models via that script should be evaluated using that script only. 

Thank you all for help, we got something news and got some results.
Now, I just want one more help. About the parameters, beam_width, alpha and beta. Do have a documentation about that, blog or something that I can study about, to understand more how it work?

Beam width = beam size. Larger beam size will take longer for marginally better results. 
Alpha = LM weight. 
Beta = Length penalty. 
We perform a sparse grid search over alpha and beta, keeping beam width fixed.
timestamp and score per word or sentence
State: closed | Pull request: no

We are doing training/transcription tests using NeMo for the Portuguese language and we are getting great results. But now I want to evolve the transcripts in order to get the confidence rate, timestamp of each word or sentence. Is there any tool inside NeMo that already does this autonomously?
Just for comparison, other transcribers return Json similar to this:

"words": [
     {​​​​​​​​
     "end_time": 0.39077207,
     "score": 66,
     "start_time": 0.272892,
     "text": "a"
      }​​​​​​​​,
     {​​​​​​​​
     "end_time": 1.1700554,
     "score": 94,
     "start_time": 0.40797436,
     "text": "internet"
     }​​​​​​​​
]

We don't (yet) have something that sophisticated as an API that emits word level confidence. However, we do have CTC alignments available. You can use the CTC model.transcribe(..., return_hypotheses=True) argument, which will return a hypothesis object with a value in it called `alignment` - that describes what token the model predicted at each timestep of the Acoustic sequence. Note that this will include the CTC blank token (which is by conversion the last token in the decoding vocabulary).

You also get the log probabilities, so I suppose you could perform softmax at each timestep to compute the actual confidence per token predicted. 

Start and end times can be estimated as well depending on token boundaries and reversing the timesteps count by number of stride steps the model performs. Note that QuartzNet is a 2x stride model where as Citrinet is an 8x stride model. 

@titu1994 
Is there any way to calculate the word timestep after applying beamsearch decoding on top of  EncDecCTCModelBPE  citrinet_1024.

I tried [parlance ](https://github.com/parlance/ctcdecode) solution but took a very long time to execute, maybe because of the large vocabulary size.
[BUGFIX] OmegaConf forward compatibility
State: closed | Pull request: yes

Apply updates where OmegaConf is breaking with 2.1.0.

See #2316 for the original PR to v1.0.0.

Note, we have to upper bound OmegaConf as Hydra 1.0.6 is not compatible with OmegaConf 2.1.0. When the new version of Hydra is released we should remove the upper bound.
patch quantization
State: closed | Pull request: yes

Hi @titu1994 
I've further tested the quantization part code and I get the below results:

First, calibrate the model:
```bash
python speech_to_text_calibrate.py --asr_model=./QuartzNet15x5-Zh-NR.nemo  --dataset=/raid/data/data_aishell2/dev_manifest.json --dont_normalize_text
```
Then, test the quant model:
```bash
python3 speech_to_text_quant_infer.py --asr_model=./QuartzNet15x5-Zh-NR-max-256.nemo --dont_normalize_text --use_cer --onnx --dataset=/raid/data/data_aishell2/dev_manifest.json
```
I get a CER of  7.34%.

Finally, to test with trt with the generated onnx model:
```bash
python3 speech_to_text_quant_infer_trt.py --asr_model=./QuartzNet15x5-Zh-NR.nemo  --asr_onnx=./QuartzNet15x5-Zh-NR-max-256.onnx --dont_normalize_text --use_cer --qat --dataset=/raid/data/data_aishell2/dev_manifest.json --batch_size=256
```
I get a CER of 9.08% with batch_size=256,  8.06% with batch_size=256 & sort dev set by audio length,  7.95% with batch_size=1.

I know it is normal to have 0.01%~0.05% error tolerance due to the padding of Quartznet. But the above result shows that there might be some issues with the script or the generated trt engine file.

(The size of the generated engine file is 24MB, which seems reasonable and the original model trt engine file is 95MB.)

I build my environment by starting from `nvcr.io/nvidia/pytorch:21.05-py3` + tensorrt 8 EA + nemo_toolkit[all].

Not sure if I made any mistakes in the above steps. Please correct me.




This pull request **introduces 2 alerts** when merging f139082ed38998ea2474a09dc2f8759b68625969 into 29b6dd8890d5c196ded96351607ff6642406e75a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-90934cd23206306438231b8b7dab39a60d08f70b)

**new alerts:**

* 2 for Unused import

@titu1994 Could you help check the above issue ?

I'm not particularly familiar with the quantization aware training methods. Will request someone from team to help with this.
Changing Language Model in QuatzNet5*15
State: closed | Pull request: no

Hi
I use QuartzNet15-5 for fine-tuning. Based on this [paper](https://arxiv.org/pdf/1910.10261.pdf) I understand that QuartzNet15-5 uses 6-gram and T-XL language models. How can I change the language model in QuartNet15-5? For instance, using Bert.

**Environment overview**

Environment location: Google Colab

Method of NeMo install: !pip install nemo_toolkit[asr]

NeMo version: 1.0.0

Learning Rate: 1e-3

**Environment details**

OS version : "Ubuntu20.04.3 LTS"
PyTorch version : "1.7.1"
 

BERT is a masked LM and different from regular Transformer LM. Using a maksed LM model to score a sentence is not as efficient as Transformer. You need to call it X times and X is equal to the number of tokens in a candidate while Transformer just needs one pass. For instance, if average of your candidates have 30 tokens, BERT is 30 times slower than Transformer. It makes BERT very expensive for rescoring while the improvement may not be that much significant. There are some papers which have tried to address this issue like this one:

https://assets.amazon.science/cc/54/980cb7d74f93849b49b9d5c42466/masked-language-model-scoring.pdf
Is it possible to use BeamSearchDecoderWithLM() method in asr in a Windows 10 system?
State: closed | Pull request: no
Labels: name: feature

I tried using the above feature in my Windows system, then I found that I need to execute the bash file in 
`scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh `

I also installed [swig for windows](http://www.swig.org/download.html), [wget for windows](https://eternallybored.org/misc/wget/) & [make for windows](https://sourceforge.net/projects/gnuwin32/files/make/3.81/) 
additionally for running the wget & make commands in the .sh files.

After installing all these, I am getting an error while I am trying to run the above `install_beamsearch_decoders.sh ` inside the nemo folder.
`ModuleNotFoundError: No module named '_swig_decoders'`

Earlier, I was also getting error `ModuleNotFoundError: No module named 'swig_decoders'` but I resolved that by installing make & wget, so the command is able to generate swig_decoder.py file, but now I am getting `ModuleNotFoundError: No module named '_swig_decoders'` which is another python module file.

I have attached the log file with this. As per my understanding, this is happening mainly because in the log there comes an error
```
cl : Command line warning D9002 : ignoring unknown option '-O3'
cl : Command line warning D9002 : ignoring unknown option '-std=c++11'
```

[swig-error.txt](https://github.com/NVIDIA/NeMo/files/6602117/swig-error.txt)


However, I would appreciate if you can check the log file and see what is the problem if in case I am wrong. Also let me know if this is impossible to use in Windows system.

Thanks for the help!

We dont support Windows 10 in any capacity at the moment. 

I face the same problem in Colab, I installed all prerequisite.
Import Error on tutorial of Relation_Extraction-BioMegatron
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
Hi, 

I have an error about tutorial of relation_extraction-Biomegatron.
In Colab, there was no problem to install and to test.

Step/Code : from nemo.collections import nlp as nemo_nlp
During import process, import error was happened like this.
 ImportError: libhdf5.so.101: cannot open shared object file: No such file or directory

- Environment location: Linux centOS, Jupyter notebook.
- OS version: Linux CentOS
- PyTorch version:  1.7.1 (from nemo toolkit)
- Python version: 3.6.10
- GPU model (V100 on server)

Best Regards.
[ImportError.docx](https://github.com/NVIDIA/NeMo/files/6606953/ImportError.docx)
Added JSON manifest's support to transcribe_speech.py
State: closed | Pull request: yes

Extended `transcribe_speech.py`
- added support for JSON dataset manifest as input (to transcribe datasets for analysis in SDE)
- modified output format to JSON manifest (to store filenames with ASR transcripts)

Signed-off-by: Vitaly Lavrukhin <vlavrukhin@nvidia.com>

This pull request **introduces 1 alert** when merging 6ffaa42f691b38c7012ee5ff3c8c6efd538c4546 into c8131785cb8980ddb76c088eb32baf91c3c32495 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2551329ce0f2e7e1a90f3b651cfa199021015c86)

**new alerts:**

* 1 for Unused import
fix_cluster_small_sample
State: closed | Pull request: yes

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

This pull request **introduces 1 alert** when merging 8660f43394b1b601d799acc123041db49f050566 into d23ee3e3d5ed2fd17608b736ebae2b415a258a9e - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5d92866e1f3bf62f7a9d6f0212f49c6c64e67338)

**new alerts:**

* 1 for Variable defined multiple times
Fastpitch export
State: closed | Pull request: yes

Dependant on https://github.com/NVIDIA/NeMo/pull/2278


This pull request **introduces 3 alerts** and **fixes 1** when merging e99c2fbea67b64a856bd9d59420ad32249fde875 into d23ee3e3d5ed2fd17608b736ebae2b415a258a9e - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-f7d8f6f310df97f17db8897a9e541b9001de6c60)

**new alerts:**

* 3 for Unused import

**fixed alerts:**

* 1 for Unused import
update quantization
State: closed | Pull request: yes

Signed-off-by: slyned <slyned@nvidia.com>

Add CER support

But the results of speech_to_text_quant_infer_trt.py and speech_to_text_quant_infer.py are a little different.

I've also tried to use speech_to_text_quant_infer_trt.py to do inference with NeMo models downloaded from ngc without adding --qat,  but the results also are not consistent with speech_to_text_infer.py.
Adding new Models releases on NGC.
State: closed | Pull request: yes

Adding the following models:

https://ngc.nvidia.com/catalog/models/nvidia:nemo:asrlm_en_transformer_large_ls
https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_small_ls
https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_medium_ls
https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_ctc_large_ls

Added a new section for language models for ASR in the checkpoint section.
ASR improvements
State: closed | Pull request: yes

# Changelog
- Update Citrinet configs.
- Improved error message when numba is not installed.
- Add tarred dataset support for all supported filetypes.
- Ignore extra parameters for layer-normed LSTM.
- Correct deprecated call signature for OmegaConf.

@okuchaiev fyi - updating the signature for config update to remove deprecated API call for OmegaConf in ModelPT.
Neural Clustering model
State: closed | Pull request: yes


This pull request **introduces 26 alerts** when merging 426bfeebea7528ea52a4ab130076546f2a37c7e6 into 4781d1744765fafdcd1238d8d4a9c3f94057aa23 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5739c24c7c6ab00b4184aecc25110c9ddffdeb5a)

**new alerts:**

* 14 for Unused import
* 8 for Unused local variable
* 2 for &#39;import \*&#39; may pollute namespace
* 1 for Illegal raise
* 1 for Variable defined multiple times
AttributeError: module 'nemo.collections.nlp.models' has no attribute 'EntityLinkingModel'
State: closed | Pull request: no

Hi Team,
As per Entity linking tutorial, release = **1.0.0rc1**, does not have **EntityLinkingModel** available.
We are getting error - 
AttributeError: module 'nemo.collections.nlp.models' has no attribute 'EntityLinkingModel'.

Could you please help us with the issue?

EntityLinkingModel is a new model and was not a part of 1.0.0rc1. You may use the main branch or wait for the release of 1.0.0 which is going to happen very soon.
Adding neural rescorer and its documentations
State: closed | Pull request: yes

This PR would add a script to evaluate a neural language model (Transformer) trained with
`examples/nlp/language_modeling/transformer_lm.py' as a rescorer for ASR systems.
Given a trained TransformerLMModel `.nemo` file, this script can be used to re-score the beams obtained from a beam
search decoder of an ASR model.

USAGE:
1. Obtain `.tsv` file with beams and their corresponding scores. Scores can be from a regular beam search decoder or
   in fusion with an N-gram LM scores. For a given beam size `beam_size` and a number of examples
   for evaluation `num_eval_examples`, it should contain (`num_eval_examples` x `beam_size`) lines of
   form `beam_candidate_text \t score`. This file can be generated by `scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py`.
   
2. Rescore the candidates:
    python eval_neural_rescorer.py
        --lm_model=[path to .nemo file of the LM]
        --beams_file=[path to beams .tsv file]
        --beam_size=[size of the beams]
        --eval_manifest=[path to eval manifest .json file]
        --batch_size=[batch size used for inference on the LM model]
        --alpha=[the value for the parameter rescorer_alpha]
        --beta=[the value for the parameter rescorer_beta]
       --scores_output_file=[the optional path to store the rescored candidates]

You may find more info on how to use this script at:
https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/asr_language_modeling.html


This pull request **introduces 1 alert** when merging 3f1390819c64cd3be2bee2a150775db6fab8ef99 into 43add2a37812a4786e6229322f7cf30ac2103313 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-772b7f88024564503202282a9d0d11fb42a5b969)

**new alerts:**

* 1 for NotImplemented is not an Exception

what is the `eval_manifest.json` file supposed to contain?
I want to rescore sentences with a gpt2 language model, any guide?
Regularizing during fine-tuning?
State: closed | Pull request: no

Hello! Thank you again for the amazing work, I really appreciate it. I am fine-tuning a pre-trained QuartzNet model and wanted to ask what you'd recommend for regularization. I have updated the dropout from the pre-trained QuartzNet default 0.0 to 0.2 using: 

cfg = copy.deepcopy(quartznet.cfg)
print(len(cfg['encoder']['jasper']))
for i in range(0,18):
  cfg['encoder']['jasper'][i]['dropout'] = 0.2
print(OmegaConf.to_yaml(cfg))
quartznet2 = quartznet.from_config_dict(cfg)

But this seems to just cause loss to explode? 

Thanks!

You should not apply dropout to the first or final Encoder layer. Also 0.2 is quite high - it's only required for training for an elongated period. For short finetuning runs, a small value of 0.1 is sufficient. Also weight decay should be 0.001 or lower when using dropout to prevent over regularization. 

Thanks! In that case, probably makes sense for me to just leave dropout at 0 and just use an increased weight decay from the .0001 default? Thanks!

You could refer to the QuartNet paper, generally we recommend 0.001 to 0.0001 range of weights decay, tending to the higher side. 

Thank you!
Audio Norm
State: closed | Pull request: yes

- added jenkins tests for the normalization_with_audio script (text only, text + audio, manifest)
- all default normalization unit tests will be executed for normalization with audio as well
- added serial class to the default normalization (deterministic): using single digits by default
- addressed [this](https://github.com/NVIDIA/NeMo/issues/2266) and partially [this](https://github.com/NVIDIA/NeMo/issues/2267) (`k` -> `kay` is not supported yet)
- more verbalization options added for cardinals and whitelist
- now all possible verbalization options are provided not only the first shortest 100

This pull request **introduces 1 alert** when merging b5663d69d6300672c44fe37a32927240ffeb55ef into bee43e840837d7e30e17a65b972e9fe89ecb9ca5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a4416774459ffc898579744c1c2ba41bbcca79c4)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 760e3d5b0467e9d7045611b5878d86bb91cc7e27 into 29b6dd8890d5c196ded96351607ff6642406e75a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-de34885ba44068ab8cd60ec32eebe832ec6bda2c)

**new alerts:**

* 1 for Unused import
Fix ExpManager Issues and FastPitch
State: closed | Pull request: yes


> LGTM. Just wondering why typecheck is being disabled there.

For some reason, doing `super().forward()` uses it's own input_types but not the super() module's input_types

> For some reason, doing `super().forward()` uses it's own input_types but not the super() module's input_types

That's expected. super().forward() is just calling super()'s method instead of overridden one, but your object is still of derived class and resolution of all method calls from within forward() will not change.
diarization example
State: closed | Pull request: no
Labels: name: bug

Hi, I was trying the notebook for diarization from the main branch with max and orcle speakers. I noticed that for the default an4.wav file the confusion rate has increased to 0.4 whereas in the previous releases .1.r.c it used be zero meaning perfect diarization. May I know what has changed and how can we get better accuracy on the example wav file?

Hi @sciai-ai , thanks for the question. We are aware of this limitation with new NMSE Clustering schema we introduced. It has parameters that limits its performance for smaller samples, and works wonderfully for larger audio samples. 

In notebook, we used small audio sample hence the incorrect labels. While we push a fix for it, If you would like to use previous clustering algorithm you may just replace the following line  
https://github.com/NVIDIA/NeMo/blob/bee43e840837d7e30e17a65b972e9fe89ecb9ca5/nemo/collections/asr/parts/utils/speaker_utils.py#L259  with

`cluster_method = SpectralClusterer(min_clusters=2, max_clusters=NUM_speakers)    
`
`cluster_labels = cluster_method.predict(emb)`
Added onnxruntime check of exported ONNX, bumped up default ONNX opset
State: closed | Pull request: yes

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

This pull request **fixes 1 alert** when merging 28bba40745713ccee63d6df39b28c15e5eb65c57 into bee43e840837d7e30e17a65b972e9fe89ecb9ca5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e9686dd3724e0a070a08ca4aea2cbc3518107c62)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging 73ae063885a5867b658685609129657131bf82e8 into bee43e840837d7e30e17a65b972e9fe89ecb9ca5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1aaf2421fba0f23d27b272e7c934c4fc9bb41ee5)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging b38e49ff1dfdbb7012d67b3fffcc987525d04beb into 4781d1744765fafdcd1238d8d4a9c3f94057aa23 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d5b8fa1fdb9f0103202ad493fbbb0e7a8d01855f)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging de3dd297ef9f31bbb536744d4c8257c6480a9fb0 into 8ce322431a6e20c2f82e93d8a917fe2e43ad7708 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-8ccd53f48de6ae5a663d4a222ad3ab537b85da52)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging 5557e2e0f898f9e148c1be885a78f1abef967449 into d23ee3e3d5ed2fd17608b736ebae2b415a258a9e - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e7af2d36f7beb394a62aa25f2579fd162fc0a7da)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging b5e4516ca2fdd8726d26934221b79b5dba4aec0d into d23ee3e3d5ed2fd17608b736ebae2b415a258a9e - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-3538ed7c1ac1590ae8f1edec3e3632c34d272536)

**fixed alerts:**

* 1 for Unused import
Add an NVIDIA DALI implementation of AudioToMelSpectrogramPreprocessor
State: closed | Pull request: yes

Adds an option `use_dali` to `AudioToMelSpectrogramPreprocessor` to choose between the native torch implementation of the audio preprocessing pipeline or a DALI based implementation.

Adds unit tests to verify that the results produced by both implementations are equivalent.

Removes code related with previous attempts to integrate with NVIDIA DALI. 

This pull request **introduces 7 alerts** when merging d95d5d521e8ce34b24793e8587a66c1eaeb1d925 into bee43e840837d7e30e17a65b972e9fe89ecb9ca5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-37ade276ee5798ce168ea17564133c5501e5e918)

**new alerts:**

* 4 for Unused import
* 2 for First parameter of a method is not named &#39;self&#39;
* 1 for Unused local variable

This pull request **introduces 1 alert** when merging 9002b4daff2956fda9e65035b2cb145626887f76 into bee43e840837d7e30e17a65b972e9fe89ecb9ca5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-0767ff2f37952d52429a366b751eee2a01f544b0)

**new alerts:**

* 1 for Unreachable code

This pull request **introduces 1 alert** when merging ff4ac0444bc4feaf108dab29e65cd22fe28db159 into bee43e840837d7e30e17a65b972e9fe89ecb9ca5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-fd490f27420186d5c834ebc7dc7115a88adb9302)

**new alerts:**

* 1 for Unreachable code

Closed for the time being while other approaches are tested
Applying CutOut in evaluation
State: closed | Pull request: no

I'm trying to deliberately worsen the outputs of an existing pre-trained model. Specifically, I was hoping to apply CutOut to the audio data so that the model outputs will be missing some tokens / frequencies.

My implementation thus far is to just change to `.spec_augmentation` attribute of the model.

`import nemo.collections.asr as nemo_asr
jasper = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="stt_en_jasper10x5dr")
spectrogram_aug = nemo_asr.modules.SpectrogramAugmentation(rect_masks=1000, rect_time=1200, rect_freq=500) # huge values to see if anything changes
jasper.spec_augmentation = spectrogram_aug
transcripts = jasper.transcribe(paths2audio_files=[
    'LibriSpeech_sample.flac', # random audio files
    'CHiME5_sample.wav',
])
`

However, I found that the transcripts do not change, suggesting that the SpectrumAugmentation is not being applied to data in evaluation. Even when the`jasper.training` is set to `True`, there is no change in the outputted transcripts. Am I missing something? Thank you!

> SpectrumAugmentation is not being applied to data in evaluation

All spec augment is disabled by default in evaluation steps (validation and testing). https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/models/ctc_models.py#L542-L543

model.transcribe() forcibly uses model.eval() - which again disables model.training to False. This is by design - we do not allow spec augment to be used during inference.

If its a requirement, you can copy paste the transcribe() method and simply force the model.training = True before calling the data loaders - but note that batch norm will be active and your results will be incorrect as compared to the original model.transcribe() (ofc the spec augment will also affect outputs).

Another approach is subclass EncDecCTCModel and override the forward method to always apply spec augment instead of looking at the model.training property. 

That makes sense! I had to re-bind the forward method instead of subclassing since from_pretrained returns the model directly, but same idea. Thank you for the prompt help!
Add support for Numba CUDA optimized SpecAugment
State: closed | Pull request: yes

# Changelog
- Adds support for numba cuda kernel to perform SpecAugment when following criterion are met:
  - Numba version is >= 0.53.1
  - Input tensor device = CUDA
  - Lengths of the input tensors are provided (handled by NeMo, user's responsibility if called separately)
  - User provided flag `use_numba_spec_augment` is set to True (True by default)
- In such a scenario, if the above conditions are met, SpecAugment() will be replaced by SpecAugmentNumba()
- Users can disable this kernel by setting the flag `use_numba_spec_augment` to False.
notebook for training nmt models
State: closed | Pull request: no
Labels: name: feature

Hi, I've been trying to use nemo for neural machine translation but i keep hitting a roadblock since the only for of intro notebook on this subject is the text2sparql.  Any intro notebook on the matter would be very helpful

please have a look at the NMT docs: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/r1.0.0rc1/nlp/machine_translation.html
NormalizerWithAudio  - Allow plural Serial verbalizers
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**

Pluralized verbalizers of Serial objects are not available in the results from NormalizerWithAudio.normalize_with_audio(), but a large number of plural serials exist in English eg "B-52s" should allow for a verbalizer of "B fifty twos" in addition to "B fifty two S".  The correctly pluralized serial should be discernible with the CER and audio transcript. 

These appear often in the financial, military, and other realms (think "401ks" "1099s" "W2s" etc).

**Describe the solution you'd like**

Add a parameter to the Serial FST. If it ends in "s" (or a plural form), offer a plural form of serial[-2] to the verbalizers as an option (and drop serial[-1]).

This may be non-trivial when serials end in a letter eg "401k" --> "four oh one kays" may be more useful than "four oh one ks"

**Describe alternatives you've considered**

The usual regex preprocessing which should be more efficiently implemented in FSTs?

**Additional context**

```python
from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio

audio_normalizer = NormalizerWithAudio(input_case='cased')

texts = [
    "Twelve B-17s were totally destroyed.",
    "8 401ks were emptied.",
]

for text in texts:
    normalized_texts = audio_normalizer.normalize_with_audio(text, verbose=False)
    print('\n'.join([x for x in normalized_texts]))

    assert "eight four oh one kays were emptied." in normalized_texts or "Twelve B seventeens were totally destroyed" in normalized_texts
```
Normalizer and NormalizerWithAudio fail with dashed tokens eg "550-pound bomb"
State: closed | Pull request: no
Labels: name: bug

name: TN/ITN

**Describe the bug**

Example sentence fragments for text normalizer:
"It had 12-inch guns and a 21-knot cruising speed."
"Each 550-pound bomb exploded."

**Steps/Code to reproduce bug**

```python
from nemo_text_processing.text_normalization.normalize import Normalizer
from nemo_text_processing.text_normalization.normalize_with_audio import NormalizerWithAudio

normalizer = Normalizer(input_case='cased')
audio_normalizer = NormalizerWithAudio(input_case='cased')

texts = [
    "It had 12-inch guns and a 21-knot cruising speed.",
    "Each 550-pound bomb exploded.",
]

for text in texts:
    normalized_text = normalizer.normalize(text, verbose=False)
    print(normalized_text)

    normalized_texts = audio_normalizer.normalize_with_audio(text, verbose=False)
    print(normalized_texts)
```
The results from normalized_texts will be something like (notice the spaces between each letter in the word for the units):
"five hundred fifty p o u n d bomb"
and
"twenty one k n o t cruising speed"

It seems like the sparrow/pynini rules don't allow for dashed text input followed by a unit and interpret the unit as an abbreviation. The CER calculation in NormalizerWithAudio doesn't repair this because the results from normalize_with_audio() don't ever contain the expected outputs.

**Expected behavior**

"five hundred fifty pound bomb"
"twenty one knot cruising speed"

**Environment overview (please complete the following information)**

 - Environment location: Bare-metal
 - Method of NeMo install: pip install from source, main branch

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version Ubuntu 20.10
- PyTorch version 1.7.1
- Python version 3.8.5

I'm trying to track down where in pynini or the nemo rules this is happening and if I find it I will follow-up.

The taggers pick these examples up as Serials and not as Measures. The options with the spaces between the letters make sense if it's a Serial.

I think a modification to the tagger for Measure that allows the dash between Cardinal and Unit may do the trick. If I can figure out the syntax...

Yes, this is a known issue, the fix is a part of the upcoming update. 
I'll update this bug when the PR is ready.

I'll be following the progress closely and am happy to help if I can get more familiar with pynini -- is this repo the best place to file / discuss similar issues?
Update conformer recipes
State: closed | Pull request: yes


how about we remove links to non-ready checkpoints and merge this?

Agreed. Everything else seems ready to merge 

> 
> 
> how about we remove links to non-ready checkpoints and merge this?



> 
> 
> how about we remove links to non-ready checkpoints and merge this?

Dropped the links for new models and reverted to older checkpoints.
TTS User Guide
State: closed | Pull request: no
Labels: name: bug

Please note the TTS user guide quick start seems to have two issues (https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/tts/intro.html):

1) More easily solvable: spec_generator = SpectrogramGenerator.from_pretrained("tts_en_tacotron2") <- the variable should be named spec_gen to match the lines that follow

2) The last line:
sf.write("speech.wav", audio.to('cpu').numpy(), 22050)

This gives the error "RuntimeError: Error opening 'speech.wav': Format not recognised." I believe this is an error with soundfile, see: https://github.com/bastibe/python-soundfile/issues/203

EDIT: For issue 2, to fix this in Colab, simply create audio2 = audio.transpose(), which works. May just be an issue with the version of soundfile imported by Colab. 



1) will be fixed in our upcoming release
Thanks for finding 2. I'll push a fix.

Thanks! On #2, this may just be an issue with what version Colab imports (so users not using Colab might not get the error with the current NeMo code, not sure?)

Should be fixed now. If not, please open another issue.
Can a language model trained using transformer_lm be used with beam_search_decoder?
State: closed | Pull request: no

I have seen a pipeline is coming for this, but just wanted to check in on current state, thanks so much!

You may have N-gram LM with beam search decoder or just beam search decoder without N-gram to produce some top candidates, then use the Transformer LM as a Neural Rescorer to rescore the candidates. Then the scores from Neural Rescorer can get weighted average with scores from beam search decoder to produce the final scores.

Please take a look here:
https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/v1.0.0/asr/asr_language_modeling.html

Thank you!
'num_speakers' missing from config file for speaker diarization
State: closed | Pull request: no

`num_speakers` is missing from `NeMo/examples/speaker_recognition/conf/speaker_diarization.yaml`

Which causes the tutorial for ASR_with_SpeakerDiarization.ipynb to break on this code cell:

```
from nemo.collections.asr.models import ClusteringDiarizer
oracle_model = ClusteringDiarizer(cfg=config);
oracle_model.diarize();
```

For the following error:
```
ConfigAttributeError: Key 'num_speakers' is not in struct
	full_key: diarizer.num_speakers
	reference_type=Any
	object_type=dict

```

nemo.__version__ = '1.0.0rc1'



Hi @cjameyson , 
You are probably using a notebook from main branch but codebase version `1.0.0rc1`, 
we changed `num_speakers` to `oracle_num_speakers` which takes number of speakers per input session else None, also included `max_num_speakers` option. 
Refer to this section for more info:
https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/speaker_diarization/configs.html 
https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_recognition/conf/speaker_diarization.yaml 

I see, thanks @nithinraok!
Cant download model
State: closed | Pull request: no
Labels: name: feature

Please help me resolve this. I am not able to download the model and get an error later that cant download from url.


![error11](https://user-images.githubusercontent.com/50978534/119278039-31c7ff80-bc23-11eb-99ba-059a106d2f99.PNG)



Was able to download just now. Could you retry - maybe some issue with the server in the last attempt?

I am getting the same error again and again. I am actually downloading it on a cluster. That could be the reason?

If the cluster does not have access to the internet (other than the local intranet), then yes.
100% WER on twice fine-Tuning with Persian audio using NEMO QuartzNet15*5
State: closed | Pull request: no
Labels: name: bug

**Fine-tuning on restored checkpoint has WER 100%  with Persian audio**
Hi 
I fine-tuned the Quartz Net 15*5 model with 235 hours dataset of Persian Mozilla Common Voice. When I tested the fine-tuned model with Persian Common Voice It worked well with WER 35%. So I saved the checkpoint of it.
Now I restore that checkpoint. I start fine-tune it with another Persian dataset in new domain. The audios of new dataset are recorded telephony audio. I changed the sample rate of new dataset from 8KHZ to 16KHZ and convert each telephony audio to different chunks due to long duration and lack of RAM problem in Colab tarring process.
After I fine-tuned the checkpoint  with new dataset I test it but the test result in transcription is not good any more. The model can not transcribe any audio even one correct word. In fact it transcribe some alphabet not a complete word or sentence.
What`s your suggestion for new fine-tuning with telephony Persian audio dataset on the mentioned checkpoint?

**Environment overview**

- Environment location: Google Colab

- Method of NeMo install: !pip install nemo_toolkit[asr]

- NeMo version: 1.0.0

- Learning Rate: 1e-5


**Environment details**

- OS version : "Ubuntu20.04.3 LTS"
- PyTorch version : "1.7.1"
- Dataset used for first Fine-Tuning: "Persian Common Voice" (Mozilla) and duration: 235 hours
- Dataset used for first Fine-Tuning: "Persian Telephony Audio" and duration: 22 minutes

Does the loss reduce to earlier levels when you combine datasets? How many finetuning steps are performed?

> Does the loss reduce to earlier levels when you combine datasets? How many finetuning steps are performed?

Yes. The loss reduces. I test different epochs such as 5,10,20,50,100, and 200 but the results are the same.

It is hard to debug without the dataset. The fact that loss reduces properly means the model is training - however it cannot be known why predictions are not correct at test time. I would suggest training with just the smaller dataset first to see if it learns at all - long clips might simply be causing wer to jump since model is unable to learn both short and long audio.

Also can you try combining the dataset and training from scratch if that works
TalkNet 2?
State: closed | Pull request: no

[This recent paper](https://arxiv.org/abs/2104.08189) is titled "TalkNet 2". But the model is referred to as "TalkNet" in the actual text (no 2), and recent pull requests here mention "TalkNet 1.0". 

Is there a difference between the TalkNet discussed in the paper, and the TalkNet that's implemented here?
Update FastPitch
State: closed | Pull request: yes

Adds the alignment framework to Fastpitch (AKA FastPitch 1.1 from https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/FastPitch)

This pull request **introduces 1 alert** when merging e64d265f4be1ff2f93a15bec7937e84e96091137 into 9c75c348bc8137717ca79a34a93c50028ffdfd38 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-815154e263c9e74ca0cef21d3ca85e9d209c2060)

**new alerts:**

* 1 for Unused import

This pull request **introduces 2 alerts** when merging 079080be385286604fb4ed4bac2100f872351607 into 05438876740468e1582793b72039e9865ee259a5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-7aa0faae55022f6d913e93f3f514cb9f0e297d0e)

**new alerts:**

* 2 for Unused import

This pull request **introduces 2 alerts** when merging 83aaba1adb8bb97c6bcd3f47585d34a05cdfa01c into 05438876740468e1582793b72039e9865ee259a5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-0411a589b49fc4b7594842efc3bfa51b01e4bf09)

**new alerts:**

* 2 for Unused import

This pull request **introduces 3 alerts** when merging 3b6b30f18d4880d5b278be79000e133157a2343a into b94c7a17accf40e4235b9f746caaf8d9ea5d20af - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-ca4d08fbb88e8c21bc37755f6b305c8cf7479a75)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging fcf0168bb17c5795218135d58d545cdf68134d84 into d23ee3e3d5ed2fd17608b736ebae2b415a258a9e - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a4170c432eda390dec7d02f858893eb2caff576c)

**new alerts:**

* 3 for Unused import
loss equals to nan when I use rir and noise augmentor
State: closed | Pull request: no
Labels: name: bug

Hi guys,
I am trying to finetune a quartznet model using rir noise augmentor. During the training step, the loss become nan and at the end when I save the model, I have a file with clearly nothing on it ( Size: 132K instead of the usual 69 Mo size).
Does anyone have an idea about it?
Thanks in advance.
With kind regards.

I have the same issue. When I am trying to train with Background Noises, the ctc-loss becomes nan. Also i checked spectrogram, it contains nans. So, I guess, the bug comes from preprocessor or comes from augmentation script.
@amejri Do you find the problem? 

I found the bug. In my case, It comes from the data. When the voice energy is high and the noise energy is very small. The multiplication gain becomes very large which raises nan's in mel spectrogram.

Did you manually inspect the spectrograms? Or a pass through the data post spectrogram generation to read the min max ?

Yes, I manually set NaN checker.

@naymaraq could you share with us how you fix the problem please ? I didn't solve it yet.
Support multiple models being instantiated in same execution scope
State: closed | Pull request: yes

# Changelog
- Model's are now attached with a one off global unique id (guid) which is registered into the global AppState for maintanence.
- AppState tracks whether the model was restored via a nemo file or via constructor and monitors an ever growing list of model restoration paths (If the model constructor was used, its restoration path is None).
- During registration of artifacts, each model registers its own copy of artifacts (via self)
- During save to, the restoration path is retrieved from AppState and artifacts are copied from the restoration file. If restoration path is None, assume local path artifacts
- During restore from, the restoration path is first registered in an ever growing list. Then the model is constructed (which allocates a GUID and registers this GUID with the restoration path).

# Bugfix
- Fixes a bug which prevents a model constructed via constructor to call save_to multiple times.

Signed-off-by: smajumdar <titu1994@gmail.com>

This pull request **introduces 1 alert** when merging a788ea5a7364e462abf1caf5c09bdcdd1751ee41 into 9c75c348bc8137717ca79a34a93c50028ffdfd38 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-43861f4a74f04ebba08b42ab74d68064cf672f88)

**new alerts:**

* 1 for Unused local variable

This pull request **introduces 1 alert** when merging b6570a95e471971781d3e6def93f87762983de80 into 9c75c348bc8137717ca79a34a93c50028ffdfd38 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-18a5ed1dd2475d6de64d6fc685c0f5614d9b5c05)

**new alerts:**

* 1 for Unused local variable
Offline asr notebook bug fix
State: closed | Pull request: yes

- [x] Remove .cpu().numpy()
- [x] Add -y when apt-get install swig
- [x] Check and install ipywidgets for tdqm/notebook 

@fayejf  Hi! I have just tried to run the script and it failed at e = np.exp(logits - np.max(logits)) for the wrong argument type.
Why did you remove converting torch.tensor into numpy array? 
detach().cpu().numpy() is still needed for the method to run properly.


With the main branch, transcribe() method will return numpy arrays of logits instead of torch tensors to preserve gpu memory after transcription has occurred. 

Could you paste the error trace here ?

I added detach().cpu().numpy() to logits and it ran without errors.
Would you like me to rerun it without converting and show the error?

Yes, please. Which version/branch are you using? 

I am using current main, not sure which is the latest version. I modified the script a bit (cut the analysis with librosa and plots from it, as the server where I am running the script does not support it).
Without correcting logits, trace is as follows:

```
Transcribing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  9.30it/s]
Transcript: "i cannot follow you she said"
Transcribing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 10.81it/s]
Traceback (most recent call last):
  File "transcribe_audio_with_timestamps.py", line 110, in <module>
    probs = softmax(logits)
  File "transcribe_audio_with_timestamps.py", line 98, in softmax
    e = np.exp(logits - np.max(logits))
  File "<__array_function__ internals>", line 6, in amax
  File "/usr/local/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 2706, in amax
    keepdims=keepdims, initial=initial, where=where)
  File "/usr/local/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 85, in _wrapreduction
    return reduction(axis=axis, out=out, **passkwargs)
TypeError: max() received an invalid combination of arguments - got (out=NoneType, axis=NoneType, ), but expected one of:
 * ()
 * (name dim, bool keepdim)
      didn't match because some of the keywords were incorrect: out, axis
 * (Tensor other)
 * (int dim, bool keepdim)
      didn't match because some of the keywords were incorrect: out, axis
```


This should be working with main branch as well.

Could you run

```
cd NeMo
./reinstall.sh
```
and try again? 

Ok, I fetched the main branch, recompiled and it worked. So I guess I didn't have some new changes, not sure when they appeared. Thanks for pointing it out!
Update "last" Checkpoint
State: closed | Pull request: yes


This pull request **introduces 1 alert** when merging 833a085c8b95e54bf7d759348a16155301265237 into 41e42f892772586d7ff7bba72562b2075418e442 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-cc228ac717f94501424640dde1bd7a5eab44f95a)

**new alerts:**

* 1 for Wrong number of arguments in a call