renwei2024/nemo-github-issues huggingface dataset:
Alias Swish to SiLU 
State: closed | Pull request: yes

# Changelog
- Alias `Swish` activation to `nn.SiLU` which is mathematically identical
- Update activations to inplace execution if possible for `ConvASREncoder`

Signed-off-by: smajumdar <titu1994@gmail.com>

This pull request **introduces 1 alert** when merging 7b5bebeebaca9a722f2ab6b3b1ea8947aa99fa8b into f7d3a72e1b5da664e5e1c5ac0855a9e29d32601d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-29052614e94cdee0d929259ae93e0a7126cc6347)

**new alerts:**

* 1 for Unused import
Manifest files error with custom data for finetuning ASR models
State: closed | Pull request: no

Hi, I am using the ASR with NeMo tutorial notebook on colab ([https://colab.research.google.com/github/NVIDIA/NeMo/blob/r1.0.0rc1/tutorials/asr/01_ASR_with_NeMo.ipynb](url). It works fine for the an4 dataset. But I am trying to do this with the VOiCES dataset (https://iqtlabs.github.io/voices/). As per the tutorial I think I only need to supply the manifest files in the proper format for training and testing. I am running this on colab with the `QuartzNet15x5Base-En`

I have formatted the manifest files in which they specify but while running
`from omegaconf import DictConfig`
`params['model']['train_ds']['manifest_filepath'] = train_manifest`
`params['model']['validation_ds']['manifest_filepath'] = test_manifest`
`first_asr_model = nemo_asr.models.EncDecCTCModel(cfg=DictConfig(params['model']), trainer=trainer)`

I keep getting the JSONDecodeError: Expecting value: line 2 column 1 (char 1).

Below I have attached my manifest file screenshot and the manifest file screenshot generated in the tutorial, they both appear to be identical in format so I'm wondering what seems to be the problem. I am guessung specifying "text" or "duration" in different orders shouldn't matter.

an4 dataset manifest screenshot
![an4 manifest](https://user-images.githubusercontent.com/64498789/118785567-9a734d00-b8ae-11eb-8e8a-434e39498061.JPG)

voices dataset manifest screenshot
![voices manifest](https://user-images.githubusercontent.com/64498789/118786675-b0cdd880-b8af-11eb-9258-1381b48ba1ce.png)


Is there something else I need to change to fine tune with my dataset?

Any help would be appreciated.

Thanks.



Could you post the full error log. Hard to tell where exactly it fails to decode the json - beginning or end. Could you make sure your voices manifest does not have any additional space or new lines at end end ?

This is the zip file for the VOiCES dataset. I have checked the end and beginning and they don't seem to have any blank spaces.

[test_index.zip](https://github.com/NVIDIA/NeMo/files/6525850/test_index.zip)


It looks like you have two extra empty lines at the end of your manifests. Please remove them and try again.

Good catch. I tried parsing it with python and didn't notice the extra line at the end. Removing it so there's just 1 blank line fixed it locally. That would be the fix @varunsingh3000 

Worked like a charm. Thanks @VahidooX and @titu1994
audio based normalization
State: closed | Pull request: yes

Text Normalization conditioned on audio: multiple normalization options considered, the best match selected based on WER.
Python support only (no SH).

This pull request **introduces 5 alerts** when merging 1c377a54d1e6839552177764d8cda01625ee4d02 into f0b3624dc1aee9670d85b82251ad4e488a622bc7 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-9e99806aca1eb1e8466bb5af4cc9ae1424e9a890)

**new alerts:**

* 4 for Unused import
* 1 for Nested loops with same variable

This pull request **introduces 5 alerts** when merging a9fc90e15755844988cbcd6c7a2cb9630424c79d into 202ccc902937b10bd9a0f37d49f8b8dc29e3ca4d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5240fe671cd93a037439119ede238dafcdb30dfe)

**new alerts:**

* 4 for Unused import
* 1 for Wrong number of arguments in a class instantiation
Add Russian language
State: closed | Pull request: yes

Add Russian yaml, class RUCharParser() and relative path support in __parse_item() 
Add "manifest_line_parser" argument and one more parser: __parse_open_stt()

Could you sign the DCO commits via the instruction here - 
https://github.com/NVIDIA/NeMo/pull/2230/checks?check_run_id=2612023734

Also seems test failed due to style formatting. Could you run `python setup.py style --scope nemo --fix`

Could you rebase with the main branch? Seems incorrect branch was selected?

The asr parts was refactored in order to be modular, could you apply those changes in the conflicting files? Most of them just had path changes- but manifest and parsers now belong to nemo/collections/commons/parts instead of asr.

Closed due to inactivity. Please reach out in order to resume work on it.
How to inference using fp16 with a fp32 trained ASR model?
State: closed | Pull request: no


You can do it like this with autocast of PT:

```
  with torch.cuda.amp.autocast():
      with torch.no_grad():
          transcriptions = asr_model.transcribe(filepaths, batch_size=cfg.batch_size)

```

@VahidooX this answer is correct. You can use the pytorch cuda decorator for AMP as above to perform FP16 inference on GPU

Does not work for https://huggingface.co/nvidia/parakeet-rnnt-0.6b
I trained in FP32 on RTX titan, infer on fp16 give all `??` and is taking 5 times longer. FP32 works fine.

Noteworthy is that training with fp16 on this model also does not work.

@nithinraok, have we tried this model with fp16?

There are cases where a model trained in fp32 does not work with fp16 as some weights are out of the range or activations get too large in some layers.

Yeah these models are trained with bf16 and haven't tested or updated for fp16. Works out of the box with either fp32 or bf16.
About QA_Squad
State: closed | Pull request: no
Labels: name: bug

Hi,

I tested your tutorial for QA_squad and changed the tutorial code to apply Biomegatron.
But I got an error message during apply biomegatron.
So, would you help to solve this problem?
![Error_Message](https://user-images.githubusercontent.com/82014275/118435026-07ed7500-b719-11eb-8bf0-dab6b8b0e3a2.jpg)
![Pretrained_model](https://user-images.githubusercontent.com/82014275/118435029-091ea200-b719-11eb-8855-b06d9788d2c6.jpg)
![Pretrained_model2](https://user-images.githubusercontent.com/82014275/118435030-09b73880-b719-11eb-9626-3d908cc140cd.jpg)

Best Regards,
From Simon 

Blow I showed the code changed model.

1.Changed Pretrained_Bert_Model
# This is the model configuration file that we will download, do not change this
MODEL_CONFIG = "question_answering_squad_config.yaml"

# model parameters, play with these
BATCH_SIZE = 12
MAX_SEQ_LENGTH = 384
# specify BERT-like model, you want to use
PRETRAINED_BERT_MODEL = "**biomegatron-bert-345m-uncased**" -> changed
TOKENIZER_NAME = "**biomegatron-bert-345m-uncased**" -> changed


2.Changed Pretrained_model
# load pretained model
pretrained_model_name="**qa_squadv1.1_megatron_uncase**d" -> changed
model = nemo_nlp.models.QAModel.from_pretrained(model_name=pretrained_model_name)

3.Error Message
I got the these error message.

[NeMo I 2021-05-13 07:44:44 megatron_utils:269] Downloading from https://api.ngc.nvidia.com/v2/models/nvidia/biomegatron345muncased/versions/0/files/MegatronBERT.pt
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-18-762ee4f9b73a> in <module>()
      1 # initialize the model
      2 # dataset we'll be prepared for training and evaluation during
----> 3 model = nemo_nlp.models.QAModel(cfg=config.model, trainer=trainer)

7 frames
/usr/local/lib/python3.7/dist-packages/megatron/global_vars.py in _ensure_var_is_not_initialized(var, name)
    150 def _ensure_var_is_not_initialized(var, name):
    151     """Make sure the input variable is not None."""
--> 152     assert var is None, '{} is already initialized.'.format(name)
    153 
    154 

AssertionError: args is already initialized.



you can not initialize two models within same notebook for Megatron, i.e. initialize the model , change the type and initialize again. Could you reset your notebook before you initialize the model?

Thanks for your recommend.
According to your advice, I tried two times.

1st Trail: No Problem
  I Just switched the pretrained model.
   pretrained_model_name="qa_squadv1.1_bertbase"

2nd Trail: Occurred Error.
  Instead of bert-base-uncased, I used 'biomegatron-bert-345m-uncased."
  Please refer to the attached file.  

Could you give me any advice on using biomegatron for the QnA of biomedical document? 
[Megatron_Trial.pptx](https://github.com/NVIDIA/NeMo/files/6498847/Megatron_Trial.pptx)

Best Regards.

@blackbirt-5 please use for tokenizer for your biomegatron checkpoint bert-large-uncased, you should be also able to set it to null
FastSGT vs SGD-QA
State: closed | Pull request: no

I am doing some research related to dialogue state tracking and I have found this paper (https://arxiv.org/pdf/1805.01555.pdf).
While checking out the code I see that it has been removed from this repository as of version v1.0.0.b1

I see that for dialogue state tracking there is a new model called SGD-QA which is a multi-pass type model as opposed to FastSGD. 

My questions are the following:
- Why was FastSGT removed? 
- Where can I find some documentation regarding SGD-QA? 

+ Why was FastSGD removed?
When we moved to PTL, it got removed as we did not have enough time to import it into the new version of NeMo. We may be able to import that model in the near future if time permits.
However you can still use it with older version of NeMo here: 
https://github.com/NVIDIA/NeMo/blob/ner_0.11/examples/nlp/dialogue_state_tracking/dialogue_state_tracking_sgd.py
And you can find the documentation here:
http://nemo-master-docs.s3-website.us-east-2.amazonaws.com/nlp/dialogue_state_tracking_sgd.html

+ Where can I find some documentation regarding SGD-QA?
 I think here is the only place you may find some detail about SGD-QA:
 https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/dialogue_state_tracking/sgd_qa.py
 @yzhang123 may be able to provide more info on SGD-QA.


Thank you for the fast reply! I noticed I linked the wrong paper, my apologies, wanted to link (https://arxiv.org/pdf/2008.12335.pdf).

@KadarTibor I have uploaded the SGD-QA paper to Arxiv. Will be ready in 1-2 days.

That's amazing! Thank you 🙏 

https://arxiv.org/abs/2105.08049
Log audio file on loading errors
State: closed | Pull request: yes

Signed-off-by: Anas Abou Allaban <aabouallaban@pm.me>

It would be helpful to know offending audio files that fail to load if they are corrupt and/or not supported.

Seems theres some formatting errors, could you run `python setup.py style --scope nemo --fix`

This PR will be merged into another PR (https://github.com/NVIDIA/NeMo/pull/2240) due to inactivity. One difference is that it will not raise the error, but instead simply log that the file was not loaded properly.

Got it, sorry for the delay and thanks for the followup!

Thanks for the PR! Normally I'd wait, but this PR would have merge conflicts with that one so it would be easier to integrate into that one.
Support symlinked files
State: closed | Pull request: yes

To preserve disk space on my machine, I generally use symlinks to create different splits of my data.
The tarred audio dataset script doesn't follow symlinks, so simply pass `dereference=True` to the `tarfile.open()` method. 

See more info in the python docs: https://docs.python.org/3/library/tarfile.html#tarfile.TarFile

Just a note - The manifest file can (and generally it should) have absolute paths to the dataset files when on a local machine, which the tarfile would then use. 

> Just a note - The manifest file can (and generally it should) have absolute paths to the dataset files when on a local machine, which the tarfile would then use.

Yup, I noticed this. However, I was splitting based on specific audio/label heuristics unrelated to the train/val/test split.
I'll probably modify my code to make reading these splits better.
Can not run Relation-Extraction-BioMegatron.ipynb because of error ModuleNotFoundError: No module named 'apex.normalization'
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

A clear and concise description of what the bug is.

I am trying to use a pretrained model on the relation extraction task and chem-rot dataset. As suggested by the authors, I'm using the notebook provided: Relation-Extraction-BioMegatron.ipynb

**Steps/Code to reproduce bug**
When I run the codes in the notebook, I get different errors. The first one is inconsistency between torch, torchtext and coda version. I was able to solve this problem by using this command:

```
!pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 torchtext==0.8 -f https://download.pytorch.org/whl/torch_stable.html

```

Then I had problems with installing apex and I was able to solve by using this code:

```
%%writefile setup.sh


# git clone https://github.com/qywu/apex
# cd apex
# pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
!sh setup.sh
```


Now I have another issue, I can use this command:

```
import apex

```

but this command returns error:

```
from apex.normalization.fused_layer_norm import FusedLayerNorm

ModuleNotFoundError: No module named 'apex.normalization'

```

What do you suggest?


**Expected behavior**

I expect to be able to run the code without any errors.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used
I used everything mentioned in this notebook:
`Relation-Extraction-BioMegatron.ipynb`
**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model
I used GPU version of the Google Colab 

I couldn't reproduce any training issues with [this notebook](https://colab.research.google.com/github/NVIDIA/NeMo/blob/r1.0.0rc1/tutorials/nlp/Relation_Extraction-BioMegatron.ipynb) running on Colab. A few warnings do appear during installation but they can be disregarded. 
Could you please specify the exact cell you're having errors in? 

I am also facing the same issue of   No module named 'apex.normalization'.

 if you use NeMo container it has nvidia apex installed, otherwise you need use this link https://github.com/NVIDIA/apex/tree/master/apex
Remove memory leak from ASR notebook + update model notebook
State: closed | Pull request: yes

# Bugfix
- Fix a memory leak in the WER calculation loop in the notebook
- (NOTE: The notebook content has not changed, the diff is just large cause colab version updated)

# Update
- Update the model notebook to use the new artifact system + log the loss properly

Closes https://github.com/NVIDIA/NeMo/issues/2208
Problem installing NeMo from pip
State: closed | Pull request: no

Hi,

I have a problem installing NeMo from pip. I'm using the instructions [here](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/starthere/intro.html#pip)

Following them, I'm getting:
```
(venv)  ✘ mszolucha@szal-workstation  ~  sudo apt-get update && sudo apt-get install -y libsndfile1 ffmpeg
[...]
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
libsndfile1 is already the newest version (1.0.28-7).
ffmpeg is already the newest version (7:4.2.4-1ubuntu0.1).
[...]
0 upgraded, 0 newly installed, 0 to remove and 298 not upgraded.
(venv)  mszolucha@szal-workstation  ~  pip install Cython
Requirement already satisfied: Cython in ./virtualenv/venv/lib/python3.8/site-packages (0.29.23)
(venv)  mszolucha@szal-workstation  ~  pip install nemo_toolkit['all']==1.0.0rc1
zsh: no matches found: nemo_toolkit[all]==1.0.0rc1
(venv)  ✘ mszolucha@szal-workstation  ~  pip install nemo_toolkit[all]==1.0.0rc1
zsh: no matches found: nemo_toolkit[all]==1.0.0rc1
```

What should I do to install NeMo?

My environment:
```
(venv)  ✘ mszolucha@szal-workstation  ~  python --version
Python 3.8.5
(venv)  mszolucha@szal-workstation  ~  nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  TITAN RTX           Off  | 00000000:01:00.0 Off |                  N/A |
| 23%   44C    P0    33W / 280W |      0MiB / 24220MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
```
Ubuntu 20


which pip version do you have? Could be a problem with your environment. Are you able to pip install other packages? Like this: ```pip install pytorch-lightning==1.1.4``` ? I literally copy-pasted this: ```pip install nemo_toolkit[all]==1.0.0rc1``` and it worked.
Update models and notebook for 1.0
State: closed | Pull request: yes


This pull request **introduces 1 alert** when merging 85b00da1038709cc1b145f28c8e8dd043efb2932 into d1b37ed30a70bff00df4778b55b020d8bab2e6fb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-df3f0c730d38a856855a25d9a48542d309857460)

**new alerts:**

* 1 for Variable defined multiple times

This pull request **introduces 1 alert** when merging 76792ea5da58ca76b2b0aea6690b314041d2334b into 040b21fd72618113a2987406a6a4bc0b910b2b3b - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b9f1a282982ed548829360891e1dc363c99c0de5)

**new alerts:**

* 1 for Variable defined multiple times
fix the versioning name.
State: closed | Pull request: yes

Fix the versioning name to avoid creating the extra folder name "None"  when training is supposed to continue.

> 
> 
> Doesn't this just dump the files into the project dir? @blisc any thoughts if this is correct ?

Actually I made this from "" to None last time to hope it fixes the w&b but it made the issue that now we have a folder name "None" and then there is another folder under that named "checkpoints" where checkpoints are getting stored under. They would not get dumped into the project folder but under a folder named checkpoints.

Hmm I see, we need it to be None for future code but doing `Path(str(None))' returns "None" when we need it to be empty string

How about explicitly checking for "None" and then replacing it with ""

> 
> 
> How about explicitly checking for "None" and then replacing it with ""

Did this this way. Now the version property in the config would be set to None instead of "".

This overall looks like a good solution, @blisc to confirm the approach is correct and merge if its fine.
Memory leak in 01_ASR_with_NeMo.ipynb
State: closed | Pull request: no
Labels: name: bug

Running 01_ASR_with_NeMo.ipynb WER cells appears to be leaking data in CUDA. With batch_size 32, using print(torch.cuda.memory_allocated()) prints 91,033,600 prior to running, then 129,805,824 after. There's a hard delete in https://github.com/NVIDIA/NeMo/blob/8e239ecc7d7bc1206c72daa207b0f122e85551f1/examples/asr/speech_to_text_infer.py that's missing in the notebook. 

Additionally, a few more questions: 

1) Is there some other way to run WER? 

2) Why does batch size change the WER? Isn't it averaged across batches?

3) Is there any way to run this without Cuda, e.g., so that it can be used on Colab TPU's

Thanks! I have really appreciated getting to use this!


Thanks for finding this, I'll send a PR tomorrow patching this. If you only need inference, you can freeze the model with model.freeze() and this will prevent any leak. 

It's the notebook environment which has the potential for leaking cuda tensors, and scripts don't have this leakage.

The notebooks are meant to be tutorials, not full scale training scripts. For such scripts please visit the examples/asr/ directory. You might need to modify the scripts to suit finetuning purposes. 

The model uses batch norm, and is therefore affected by padding inside samples of uneven length. For academic work, we use batch size 1 inference on a single gpu. 

We do not support TPU training and it is not currently tested. 

Thanks! I really appreciate the quick response. To clarify, I got the prints before and after to match by adding:

del test_batch, log_probs, targets, targets_lengths, encoded_len, greedy_predictions

But maybe not all those deletes are needed? I hope that helps!


While I have you, I saw in some other comments that "iter_with_steps" can help with cuda memory issues. Where would I change that param? Thanks again!

That argument has been deprecated for quite some time, and it has no impact on memory. If you need to limit training, you can use the PTL trainer argument "max_steps"
ASR patches for v1.0.0
State: closed | Pull request: yes

# Bugfixes
- Correct WER calculation for RNNT by resetting the metric after each computation.
- Prevent numba from performing a sync across PT cuda stream simply to initialize a cost vector.
- Correctly update the trainer value using `set_trainer`
- Adds .nemo and .ckpt files to dockerignore

# Additions
- Adds RNNT global flags to enable detached gradient normalization of encoder, decoder and joint models
- Adds `weights_init_scale` and `hidden_hidden_bias_scale` values (default to original values) to attempt reduction in variance between runs
- Add `tds_uniform` and `tds_normal` initialization modes for Jasper based models.


@okuchaiev fyi regarding the dockerfile ignore and correctly updating the self.trainer inside set_trainer.
Support cloud-based storage for checkpoints
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**

I would like to store my checkpoints/models using AWS/GCP so that when I run training on ephemeral instances, I don't lose my progress. 

**Describe the solution you'd like**

Register a new callback on epoch end to upload the ckpt/Nemo file to AWS/GCP if the option is passed in the config file via the trainer (using the checkpoint callback field)

**Describe alternatives you've considered**

None, open to discussion though. 

**Additional context**

I have a working implementation for a local repo, but would like to gauge interest for upstream support in NeMo before submitting a PR.

Closing. re-open if issue is still relevant.
Fix bug in SpecCutout
State: closed | Pull request: yes

[Colab notebook demonstrating bug and fix](https://colab.research.google.com/drive/1Tksb-RXAeLSkBW72KDgm1DYUaPAnHXBx?usp=sharing)

There is a bug in the NeMo implementation of SpecCutout that causes the dimensions of masks to be transposed so that `rect_freq` is the size of the mask in the time dimension and `rect_time` is the size of the mask in the frequency dimension. This also causes the placement of the masks to be limited to the lowest frequency bins.

![image](https://user-images.githubusercontent.com/47190785/118002089-49df7b00-b315-11eb-9828-10436e84a674.png)

Output after fix:
![image](https://user-images.githubusercontent.com/47190785/118003342-7e076b80-b316-11eb-9ddc-17af39e6b9da.png)


Thanks Robert for catching the bug! It looks like it has been around for a long time.
Why do you believe that "it causes the placement of the masks to be limited to the lowest frequency bins"?
The starting point for the frequency masking looks correct to me and can be any frequency. The only problem is that maximum length in frequency dimension was set to 100 instead of 47 (these are the most common values used with this augmentation technique, F=47 and T=100).

This is an incredible find, thank you @rbracco ! I've verified that your colab (also, very nice way to show the correctness of your fix!) is indeed correct. 

PR approved. 

Thank you both for the quick review and feedback. 

> Thanks Robert for catching the bug! It looks like it has been around for a long time.
> Why do you believe that "it causes the placement of the masks to be limited to the lowest frequency bins"?
> The starting point for the frequency masking looks correct to me and can be any frequency. The only problem is that maximum length in frequency dimension was set to 100 instead of 47 (these are the most common values used with this augmentation technique, F=47 and T=100).

Sorry, that issue only occurs if you have a high F and low T and it's actually the upper frequency bins, but the change fixes the issue anyway.

Also, just a heads up `rect_x` here refers to the frequency mask but the frequency is across the y_axis. Since the mask is applied with `x[idx, rect_x : rect_x + w_x, rect_y : rect_y + w_y] = 0.0` and the 2nd dimension is actually the height/feature dimension, and 3rd is width/time so since x refers to the freq dimension in both places and y refers to the time dimension in both, it still works, but if you want to do a quick name change before merging it may make it more clear. Cheers.



A name change would be good, to reflect which axis they effect. Rather than x and y I'd rather just deference them to width and height too if possible.
Fine-tuning QuartNet15*5 with Persian long duration audio
State: closed | Pull request: no

Hi,
I used QuartNet15*5 and fine-tuned it with the Persian Common Voice dataset. the WER was 30%. 
Everything worked well and I saved it as a checkpoint. 
After a while, I restored the checkpoint. I fine-tuned it with another Persian data set that has a longer duration than 16.7 seconds (i.e., 2 minutes, 3minutes, and so on ). 
When I test the model with the long duration audio, the vocabulary disarranged and the model could not transcript any audio correctly.
What's your idea for fine-tuning QuartNet15*5 with a long duration?
Thanks

You generally should not train with such long sequences, since it would use far too much memory + enormous amounts of training data + compute to train a model on long segments. Generally durations of 15-20 second segments are used for training. 

It would be best to segment the audio in the second dataset. If that is not possible, try to combine both the shorter and longer duration datasets and train for a long time (at least a hundred or more epochs) to fit the long duration audio corpus properly. 

Also, note that you can use a QuartzNet model trained on 16.7 second audio duration, and evaluate on 2-5 minute long clips with no issue. There may be modest word error rate increase without finetuning it. 

> You generally should not train with such long sequences, since it would use far too much memory + enormous amounts of training data + compute to train a model on long segments. Generally durations of 15-20 second segments are used for training.
> 
> It would be best to segment the audio in the second dataset. If that is not possible, try to combine both the shorter and longer duration datasets and train for a long time (at least a hundred or more epochs) to fit the long duration audio corpus properly.
> 
> Also, note that you can use a QuartzNet model trained on 16.7 second audio duration, and evaluate on 2-5 minute long clips with no issue. There may be modest word error rate increase without finetuning it.

Thanks
Fix ipywidgets error in asr notebook
State: closed | Pull request: yes

Added `ipywidgets` to avoid `ImportError: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html` error.

Thanks for the PR, could you sign your commit according to the instructions on the dco bot ?(https://github.com/NVIDIA/NeMo/pull/2199/checks?check_run_id=2563046850)


@titu1994 done with the signoff

Thanks for the fix !
Transformer final norm preln
State: closed | Pull request: yes

Fix/Add final layer norm for pre_ln transformer encoder and decoder

wouldn't this break all the checkpoints trained with pre_ln=True?

This pull request **introduces 1 alert** when merging 06d1bb03d4144ea1d95b1656083db015711489c2 into d1b37ed30a70bff00df4778b55b020d8bab2e6fb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-72c6d188f1fb006ed4703c0de0401a0abd5eb9e6)

**new alerts:**

* 1 for Unused local variable
Add support to change the SE context window of ConvASREncoder
State: closed | Pull request: yes

# Changelog
- Adds the support to change the SqueezeExcite context window post instantiation inside a given ConvASREncoder module.
- Adds tests to ensure the model updates the configs and changes the modules SE behaviour

# Usage

```python
model = ASRModel.from_pretrained("stt_en_citrinet_256")
model.change_conv_asr_se_context_window(context_window=128, update_config=True)  # equivalent to 128 * 0.01s of context
```
Add TTS aligner and improved version of g2p for vocabs.Phonemes, small improvement in TalkNet
State: closed | Pull request: yes


This pull request **introduces 1 alert** when merging a5b2e9be03ccadfd608d829ab10809af788c0a2f into b44a1077df4579ecdba9a5386fbfe2c0bcd20667 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2d1e71128b7ca228053fb661556cd8f6fede1c7f)

**new alerts:**

* 1 for Unused import
Proper data (size & form) for fine-tunining speaker verification model
State: closed | Pull request: no

Hey there,

I was trying to fine-tune the pre-trained speaker recognition model [`Speakernet`](https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerrecognition_speakernet) in order for it to recognize our registered speakers. I noticed in the tutorials the dataset `an4` was only used for illustration purposes, I wonder in real cases, how much data would be considered reasonable for fine-tuning? For example, in our system, we have about 15 registered users, whose voice would need to be recognized through about 4 seconds of speech for inference. Would it be enough to fine-tune with 20 recordings of about 5 seconds of each user's speech? Or might it be a better idea to try to gather the long speech of the users (10 recordings of 30 seconds speech for instance)? Thank you for your help in advance!

Cheers,
Xu

It's better to have higher number_of_samples/speaker with shorter durations (of about 4 sec) than vice versa. 

Hi, @nithinraok  Thank you!
requirements.txt should limit pytorch-lightning to 1.2.10 (max), avoiding 1.3.* for now
State: closed | Pull request: no
Labels: name: bug

Some interface changes in 1.3.0 (at least around pl.ModelCheckpoint), breaks some expectation in the current code base.  

`requirements/requirements.txt`  has `pytorch-lightning>=1.2.8` which could lead the installation of pytorch-lightning 1.3.0.  

Or aybe you are already planning on supporting 1.3.0?  

This should be fixed via https://github.com/NVIDIA/NeMo/pull/2178
Acc tn
State: closed | Pull request: yes

improved cardinal in textnormalization in terms of perf

This pull request **introduces 1 alert** when merging bc2cdec3c9e9767af3ef755a8160db979f572d59 into d9aab777c4954d13ae95d8b71ef08de65c947f5c - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-30686a7b465ee52e9ed5f4017a818b2b8d5eb153)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 02b4f9f07c51d75a4296fb11fc0cec6df0dd1686 into b7ca1ea0c72752b37351623a1a5ee4d5d1b447e0 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1f075d3f2d7f3339d882aff37138fa5b7f3fab21)

**new alerts:**

* 1 for Unused import
Update ptl to 1.3 on main branch
State: closed | Pull request: yes

# Changelog
- Update PTL to 1.3
- Fix related broken tests in NeMo

This pull request **introduces 1 alert** when merging 1c2ead52bc3b9c67f46f46636b31a0fc01a39935 into b44a1077df4579ecdba9a5386fbfe2c0bcd20667 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-49f72b46b780b693b22b9ea54e2714adcf6aa8e4)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging af4f0ae39bbcb6b1c3a169c400ddb294bc21e6ad into b44a1077df4579ecdba9a5386fbfe2c0bcd20667 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-fa66ff601ca8f005bd838a12936ec73ea14b545d)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging ac3ddf6a8c5b22e732056754a4f3d6a5c1681c2f into b44a1077df4579ecdba9a5386fbfe2c0bcd20667 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-04bad2d21a3d210cc6265712240fb61299f28970)

**new alerts:**

* 1 for Unused import

The naming convention for the latest checkpoint in lightning changed in 1.3, so I updated it.

This pull request **introduces 1 alert** when merging b4b5854e2db3a1fe15e57405ca8b95ae2640c937 into b44a1077df4579ecdba9a5386fbfe2c0bcd20667 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-700a8bce21ca405f90a531a19b4793ce0b311f18)

**new alerts:**

* 1 for Unused import

Thanks @blisc !
BeamSearchDecoderWithLM Memory Requirement
State: closed | Pull request: no

Hello, I am using an n-gram KenLM (my language model is only 67 MBs) with BeamSearchDecoderWithLM. Scorer class inside BeamSearchDecoderWithLM which is imported from swig_decoders is using 2.3 gb of ram. I was wondering if there is anything I can do to reduce this memory requirement? I am trying to run this on a device with low memory and I am running out of memory during the initialization of BeamSearchDecoderWithLM.

Is there any alternatives to Scorer that I can swap? I'm okay with it performing worse as long as memory requirements are reduced. 

Thanks.

If it's the KenLM binary causing the memory issue, then a smaller N-gram can be used. However we don't have much control over the c++ swig decoders so from our side there isn't a sufficient solution to reduce memory. Maybe @vsl9 has some ideas?

If the memory issue is from the beam search decoder, then you may try to call the beam search decoder on each sample separately. So divide your batch into smaller batches or even single samples before calling the decoder. Beam search decoding is mostly done on CPU and batch computation does not increase the speed of the decoder, so you can get the same speed with single samples. 
Other thing you can try is to lower the number for num_cpus, not sure how much it helps.

> If the memory issue is from the beam search decoder, then you may try to call the beam search decoder on each sample separately. So divide your batch into smaller batches or even single samples before calling the decoder. Beam search decoding is mostly done on CPU and batch computation does not increase the speed of the decoder, so you can get the same speed with single samples.
> Other thing you can try is to lower the number for num_cpus, not sure how much it helps.

Thanks for the suggestions. I am trying to use it on on single instance so my batch size is 1.  Ram usage increases about 2.5 GBs when I am instantiating BeamSearchDecoderWithLM. So this is before even feeding anything to the network.

I see. In that case, memory consumption is mainly due to the ngram model. You may follow Som's suggestion as ngram models need significant memory. That is one of the main reasons they are not preferable for devices with limited resources. Using a model with smaller vocabulary size can also help to reduce the size of the ngram model but not sure if it solves your issue completely.

Is this the Cuda storage? Or just normal RAM? Thanks!

> Is this the Cuda storage? Or just normal RAM? Thanks!

I am only using normal RAM. My system doesn't have GPU.

Yes, it is CPU RAM usage for n-gram LM. Scorer object is instantiated during BeamSearchDecoderWithLM construction. And it loads n-gram LM into RAM. Reducing size of LM (number of tokens/words, LM's order) should help.
Enabled passing kwargs to export()
State: closed | Pull request: yes

Also, removed previous (non–functional) extension for dicts in export - this schema matches what onnx.export expects.
 
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

> exortable changes look good to me. @blisc what about tts changes?

tts changes are to conform to new extended syntax accepted by onnx.export (optional Dict as the last input tuple element, representing kwargs) instead of former non-standard extended syntax I briefly introduced in previous revision (accepting a Dict instead of a tuple as inputs, not really used except in this test example).
ITN fails to normalize telephone numbers with zero in them
State: closed | Pull request: no
Labels: name: bug

name: TN/ITN

ITN fails to normalize telephone numbers like this `five four two one two nine seven zero six five`
It works fine after replacing `zero` with `o`

I think this is because `zero` is in `zero.tsv` not `digit.tsv`, and you are not including `zero.tsv` in the digit FST.
https://github.com/NVIDIA/NeMo/blob/74f1264e049c10b012a1ea483fcd468f4d89ecf8/nemo_text_processing/inverse_text_normalization/taggers/telephone.py#L38

Also, it's quite common for ASR to output `oh` instead of `o`, like this `five oh seven` (507).
So do you think we can add two more rules like this?
```python
digit = ( pynini.invert(pynini.string_file(get_abs_path("data/numbers/digit.tsv"))).optimize() 
             | pynini.cross("0", "o") 
             | pynini.cross("0", "zero")
             | pynini.cross("0", "oh")
) 
```

@yzhang123 

thanks for feedback, will add.
Make Hifigan jittable
State: closed | Pull request: yes

This PR makes changes to the HifiGan module that should make it jittable/torchscriptable. nemo export should work as well.

Verified compatible with checkpoint on NGC.

```python
from nemo.collections.tts.models import HifiGanModel
model = HifiGanModel.from_pretrained(model_name="tts_hifigan")
model.export("test.onnx")
```

Signed-off-by: Ryan Leary <rleary@nvidia.com>
Minor patch for translate_ddp
State: closed | Pull request: yes

Variable name change

This pull request **introduces 1 alert** when merging ed015efee22a8d532065f533db44694352a8023a into 7a62084946e93c4ce4fc885c1af8634e8e160937 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-17e3c52503f2d4b8a1bb0c8816a7aa08885956f1)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging ed015efee22a8d532065f533db44694352a8023a into fa04e1248cafae15add6a080a210a961e3409702 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d2f3b4c5e5c12389136df0bfa5fb4bff536cbe1e)

**new alerts:**

* 1 for Unused import
Failed to build Tacotron2 in Jarvis
State: closed | Pull request: no

I trained Tacotron2 in Thai language using NeMo and it failed to build jmir format.
I use command for build jmir like in https://docs.nvidia.com/deeplearning/jarvis/user-guide/docs/service-tts.html#pipeline-configuration .

![image](https://user-images.githubusercontent.com/16872939/117111657-231fc280-adb2-11eb-97ea-0657c420ec11.png)

![image](https://user-images.githubusercontent.com/16872939/117111684-2ca92a80-adb2-11eb-90bb-ced61e68346f.png)



I tried to use Tacotron2 that provides in https://ngc.nvidia.com/catalog/models/nvidia:tlt-jarvis:speechsynthesis_english_tacotron2/files and its works.
What is different between these two nemo files?

Hello,
It looks like your nemo file does not have the model weights checkpoint that should be saved. Can you untar the nemo file and make sure it has a model_weights.ckpt and model_config.yaml? If it doesn't, please let me know what the contents are.

> Hello,
> It looks like your nemo file does not have the model weights checkpoint that should be saved. Can you untar the nemo file and make sure it has a model_weights.ckpt and model_config.yaml? If it doesn't, please let me know what the contents are.

I untar my nemo file and it has a model_weights.ckpt and model_config.yaml.
This my nemo file -> https://drive.google.com/file/d/1NAJoB_jRWofkTKT8aox8CBTm4M41yul5/view?usp=sharing

Hi @azahed98, Do you have any suggestions?

Hello,
Sorry for the delay. After some testing, I confirmed that this is a bug with Jarvis. I'll see if I can find a workaround and update you.

I found a workaround for you. The expected contents is is
```
model_config.yaml
model_weights.ckpt
```

However, the contents of your file has a preceding directory
```
./
./model_config.yaml
./model_weights.ckpt
```

You should untar your file into a directory, and retar it but without a preceding directory.

Now, It works. 
Thanks.
Text processing refactor
State: closed | Pull request: yes


This pull request **introduces 6 alerts** when merging 1677f4448ac8536a58f2d6a533e7234cabdfa369 into 855ce265b80c0dc40f4f06ece76d2c9d6ca1be8d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-796aa3ea4cf0df26414f426444817cfc0c976ee7)

**new alerts:**

* 5 for Unused import
* 1 for Missing call to \`\_\_init\_\_\` during object initialization

This pull request **introduces 1 alert** when merging 729688e836b196692c092821acb6001f802b82f5 into 26e5d3a9b3cc934db0004b1dd9263fb7bce9a964 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b210828c62ea08c7715ec6659ec679dc68d32416)

**new alerts:**

* 1 for Missing call to \`\_\_init\_\_\` during object initialization

@yzhang123 how about creating a subfolder for util scripts to have them separated from the executable scripts?
Outputs of same batch produce different results with deterministic settings on
State: closed | Pull request: no

I'm working on implementing something new but I'm having trouble getting the model to output the same thing each time. The output from the featurizer is consistent, but the output from Quartznet 15x5 (encoder + decoder) is slightly different each time. I am aware of #1030 and have taken the following steps. 

- Set pytorch lightning seed with `pl.utilities.seed.seed_everything(8)` 
- Set num_workers = 0 in the dataloader (This is the default NeMo value anyway, right?)
- Setting shuffle=False in the train dataloader
- Pass the flag `deterministic=True` to the lightning trainer.
- Turn off dithering (this shouldn't really matter since I'm setting a seed)
- Setting the environment variable `CUBLAS_WORKSPACE_CONFIG=:16:8`

Any idea what could be causing this? If I load the model weights in to a quartznet like model in pytorch lightning and set the seed, the output is the same every time. Thank you!

Could you try quartzner with batch size 1?

Issue solved, SpecAugment and SpecCutout weren't getting the seed, setting the seed manually on their rng with `self._rng.seed(8)` in the `__init__()` of each solved the issue. Thank you.
Removing graphsurgeon optional dependency, improving import error rep…
State: closed | Pull request: yes

…orting
I did not remove graph surgeon install from Dockerfile - as it's nice to have anyway and maybe some users already use it.
But I did remove the extra GS step on export.

This pull request **introduces 1 alert** when merging 3afd577d38d6304614bfc59b57beae7f58e2679f into b6bdccd4effef234b7cb44917d57f1368b9b23d9 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-cc87bcbdf4417483b27b2f378f45917a0f3961e2)

**new alerts:**

* 1 for Unused import
Itn add classes
State: closed | Pull request: yes


This pull request **introduces 1 alert** when merging fe433b5014740ebc502c7562933de52402e2a7a5 into c5426c871d7ae7a006f496e33d2977d1791ebe51 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-f9718579a2247addeb8826e193a17d91647924cb)

**new alerts:**

* 1 for Unused import
Update bert
State: closed | Pull request: yes


This pull request **introduces 2 alerts** when merging 786c4a83e0d6a48bf4732bb400e31c3b0b0dff21 into 6ce670b7ab3ab5c84ce388fa1968748cd859b8f8 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-82f3d3ec49f1c660c34799fa45e15ff9271c68ef)

**new alerts:**

* 2 for Unused local variable
Update TTS notebook with TalkNet inference
State: closed | Pull request: yes

Signed-off-by: Stanislav Beliaev <stasbelyaev96@gmail.com>

Can you send this to v1.0.0 instead?

@blisc ready for review

link is fixed
branch choice is yet to be commented by @blisc 

> link is fixed
> branch choice is yet to be commented by @blisc

Please use `v1.0.0` for all files and branches

Okay, now it's ready to be merged.
Few fixes to enable backtranslation again
State: closed | Pull request: yes


This pull request **fixes 3 alerts** when merging a0d7a2c00d22791313e5404ac1654023f4eb0ec0 into e59dd39f6f847155a82d40ebe2264a0b7f2b7a13 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-fca57bc50d3f751d5d455b5490d473ee1719d621)

**fixed alerts:**

* 3 for Unused import

This pull request **fixes 3 alerts** when merging 1c1c37403c2dbd93004959ed5ac8c87f2c6bd0f8 into dddf87e22a6e87a0e2698d4aa0653f963564d755 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-bde2e8d3ff9b822f6dea1badb045846bac6112c3)

**fixed alerts:**

* 3 for Unused import
Text norm
State: closed | Pull request: yes

* text normalization in WFST
* merge electronic (Evelina)
* updated docs for ITN and TN
* updated Tutorials
* unified deployment route
* unit tests for both

This pull request **introduces 21 alerts** when merging 98bdf7a16e27cf4157f0ec0fcf78e6b0d6373c02 into e59dd39f6f847155a82d40ebe2264a0b7f2b7a13 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-c158b8211dc388233d44ba88f0a83f5f8e4bd8e6)

**new alerts:**

* 18 for Unused import
* 3 for Unused local variable

This pull request **introduces 23 alerts** when merging 14b2fc2e260bdac70614dcd45840b51d11ed0cdc into e59dd39f6f847155a82d40ebe2264a0b7f2b7a13 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-f527cc40b39f9cf581abdedb89b37733f257c5ff)

**new alerts:**

* 20 for Unused import
* 3 for Unused local variable

This pull request **introduces 1 alert** when merging 51f164b270d649ec786149d06d2fad7e6b6219b3 into e59dd39f6f847155a82d40ebe2264a0b7f2b7a13 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-61e2d9e7b2f18635d94e5b0217567d559a223ac4)

**new alerts:**

* 1 for Unused import