renwei2024/nemo-github-issues huggingface dataset:
Speed up training of Citrinet models by more efficient masking.
State: closed | Pull request: yes

The proposed change includes three optimization steps that I found were beneficial for model training efficiency.

### 1. Mask caching
Currently `arange` for mask is computed at each step from scratch. While it might be cached by GPU, there is no guarantee as it is not needed after forward function execution. 
By caching it dynamically, it will be easier to strike balance between numerous short sequences that will benefit from caching and arbitrarily long sequences during inference.
 

### 2. Multiplication by mask instead of `masked_fill`
`masked_fill `supports arbitrary values to fill into the tensor as, thus, implemented as an iterator over Tensor memory.
In our models masks are binary, so we can leverage vectorized multiplication operation to speedup mask application.

### 3. Shortcutting new sequence length computation
Given that Citrinet uses same padding everywhere and that striding only happens at certain levels. We can avoid doing arithmetic with sequence lengths when not needed.
Additionally, NeMo DatasetLoaders guarantee that lengths are represented as longs, so we can remove redundant `to` conversion.


## Benchmarks

### NGC
Time per epoch, 384 filters model, 8 GPUs, average across 10 epochs:

| baseline  |  +Mask caching  | +Multiplication   | +Shortcuts  |
|---|---|---|---|
|50 m   | 46.5 m   | 45 m   | 44.5m   |

 
Time per epoch, 384 filters model, 4 GPUs, aggregate grads from 2 batches, average across 34 epochs:

| baseline  |  +Mask caching  | +Multiplication   | +Shortcuts  |
|---|---|---|---|
|78 m   | 71 m   | 69 m   | 70m   |


**Accuracy** for all improvement runs **is in bounds of statistical noise** compared to the baseline.

I am still waiting for the full run to complete in order to confirm that there is no degradation in quality.  For now, keeping in draft state. 

Thanks a lot everybody for the review!
[Question] How to adapt knowledge-distillation to the ASR classes?
State: closed | Pull request: no

Thanks for this great toolkit!

I'm seriously looking at implementing some knowledge distillation techniques for the ASR classes.  I found this very simple and elegant pytorch-lightning approach here: https://github.com/vrvlive/knowlege-distillation/blob/master/training_module.py  and I wonder how I could adapt it to `EncDecCTCModel` or `EncDecCTCBPEModel` classes (as well as for the rnn-t or conformer cases).

I would like a pointer or two, if possible, since the NeMo classes aren't trivial `pl.LightningModule`.  What "worries" me the most is how the audio preprocessing pipeline would impact how we can simply call "forward()" for example.

Where should I start?

Actually EncDecCTCModel  and EncDecCTCBPEModel are LightningModule, and you can call the forward like the example you provided. They just have some extra methods to support more capabilities. You should be able to use the same trick in that example for these two models. If you follow the approach used in the example, I think you do not need to worry about the data processing pipeline. 

You may start by creating a new class inheriting EncDecCTCModel  or EncDecCTCBPEModel and overriding init and training_step methods. For Conformer or Citrinet, BPE-based models should be better in terms of accuracy.

Thanks a lot!  I'll give it a try!

I would suggest also reading through the forward step + train/validation steps. RNNT in particular is not sufficient to just call forward - there's a lot of extra steps done after that inside the train and validation steps.
Prepare RNNT to switch to Numba loss for compatibility
State: closed | Pull request: yes

# Changelog
- Add rnnt loss switching capability.
- Enable passing arbitrary kwargs to the inner loss function.
- Add jenkins tests (commented out until 21.04 container is released which supports minimum numba version 0.53 for CUDA 11.2)
- Enable all RNNT model tests using the numba default loss

Signed-off-by: smajumdar <titu1994@gmail.com>

This pull request **fixes 2 alerts** when merging 4848e4c0e5059d4a31262dcd7c5ba54c077f2691 into 094b58bfcefd069aad930a375f7a93fd26af45a2 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5012310c004b9db9ea2d601621c420f3c4619363)

**fixed alerts:**

* 2 for Unused import

This pull request **fixes 2 alerts** when merging 0d2c04146e472b4714e2c727ec0f201afc252940 into 094b58bfcefd069aad930a375f7a93fd26af45a2 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-07a31dca89da3bde05f3506bc01ef5ea88fd6c7d)

**fixed alerts:**

* 2 for Unused import

This pull request **fixes 2 alerts** when merging b281d1f1f6e1fac7c6dfe71f6bd43d7712da6fe6 into 682e90d5ab1085ba07e520dec42cc60d362d4aca - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-8b570105ba7c3be4838c95b6386f78d53201d8e4)

**fixed alerts:**

* 2 for Unused import
Make numba dynamically checked and remove hard dependency in requirements.txt
State: closed | Pull request: yes

# Changelog
- Remove hard numba version requirements
- Dynamically check for version at runtime when it is required
- Update tests to dynamically switch off for cuda if version is incompatible (either due to numba version itself, or due to mismatch between supported CUDA version and actual CUDA version installed).
- Update docker file to forcibly update numba temporarily (to be removed at 20.04 release)
- Update docker file to preemptively import the domains to download whatever is necessary.
- Patch a bug in dynamic resolution of numba losses

Signed-off-by: smajumdar <titu1994@gmail.com>
Fix TalkNet and HiFiGAN and related files
State: closed | Pull request: yes

* Updated HiFiGAN training blocks and config (main version doesn't work)
* Updated TalkNet and config (main version doesn't work)
* Small changes in audio_to_text.py (load_audio flag)
* Small changes in vocabs.Phonemes/vocabs.Chars (add_blank_at flag)

Can you also update latest TalkNet weights in pretrained link?

LGTM
i will update ckpts links in following days, as well as TTS notebook for inference

@stasbel @blisc I found additional bug in HiFiGAN tuning. I will fix it and after that we will merge it

@blisc let's merge it after this PR: https://github.com/NVIDIA/NeMo/pull/1994
Fix formula to compute same padding for Jasper-based models.
State: closed | Pull request: yes

Notation: 
k - kernel size
p - padding
d - dilation

The old formula is only correct when d <= 3. 

This can be verified by choosing d = 4, k = 3 and computing the value of padding with an old formula _p_old = (d * k) // 2 - 1 = 5._ Whereas the correct answer is _(d * (k - 1)) // 2 = 4_.
This result can be verified in [convolution visualizer](https://ezyang.github.io/convolution-visualizer/index.html).

The change **does not affect** a case when **d=1 and k is odd**. In that case, (k - 1) // 2 = k // 2. Also **d=2** and **d=3** are unaffected due to coincidental correct answers that old formula gave.

The behaviour **changes for even kernels** but there's no correct symmetrical padding in that case, so we don't use it in practice.
update to unittesting
State: closed | Pull request: yes

This PR adds "--with_downloads" argument for unittests.
```
# regular, GPU-based unit tests, without downloading pre-trained checkpoints from the clould
pytest
```
```
# regular, GPU-based unit tests including the ones which download pre-trained checkpoints from the clould
pytest --with_downloads
```

I also marked some tests which download checkpoints from the cloud with corresponding ```@pytest.mark.with_downloads()``` fixture.


> get test is working, export is skipped (not working)


- No, locally I get: tests/collections/nlp/test_megatron.py::TestMegatron::test_get_model FAILED  
E           RuntimeError: The following operation failed in the TorchScript interpreter.
E           Traceback of TorchScript (most recent call last):
E           RuntimeError: The following operation failed in the TorchScript interpreter.
E           Traceback of TorchScript (most recent call last):
E           RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)
Adding the feature of prelaoding weights a checkpoint.
State: closed | Pull request: yes

It enables user to start training by loading weights from another checkpoint ('.ckpt') or '.nemo' file. 

It can be done by setting the 'model.init_weights_from_model' parameter in the config or by:
model.load_weights_from_checkpoint(init_model_path='/path/to/model.nemo', strict=True)

It is a feature which enables fine-tuning out of the box for all nemo model/scripts/examples by just using the config.

This pull request **introduces 1 alert** and **fixes 1** when merging 7bc984d05a1cb0684428995fdc4135ef3b9a8c36 into 8f7c18973bf92f30679c0f4416412d629bfc7bff - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-0ebacc9e5c231176025a17986eb8897d98fa1097)

**new alerts:**

* 1 for Unnecessary delete statement in function

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging 3fba4784f3f42e6202037bcbdb7c7a06dc8a4a99 into 8f7c18973bf92f30679c0f4416412d629bfc7bff - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-983adf1d405bed3dd4227058e8a8c9047c3fbe26)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging 92d986b0082037ea9361fae554bbac85242cfb9c into 8f7c18973bf92f30679c0f4416412d629bfc7bff - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-551286c4266b331d1384c6590727e7dd136d54eb)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging 1b7abb9f53e57c7996beea694942c80c55e6f282 into 8f7c18973bf92f30679c0f4416412d629bfc7bff - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-7e37fc8f694b12466dd09daf59e11e66e4c7fc95)

**fixed alerts:**

* 1 for Unused import

Can we get this merged?

I'm not sure what we're trying to accomplish with this PR. As @titu1994 has said, we can already restore weights and finetune in NeMo with .nemo files. Or to resume training, we use PTL.

Can you please make it more clear in the README what feature is being added and how to use it?

Also, this PR is adding a new feature and should not be going into RC1. RC1 PRs are only for bug fixes and documentation.

Please rebase to main.
multi core, gpu training fails
State: closed | Pull request: no
Labels: name: bug

I am doing the nemo asr training on multi gpus and workers and it hangs on the step:

`trainer = pl.Trainer(gpus=4, num_nodes=8, accelerator='ddp',max_epochs=200,amp_level='O1', precision=16,`

At the console 

![image](https://user-images.githubusercontent.com/52277510/112593013-dacfc500-8e41-11eb-87e0-878f0eab423b.png)


The same code works fine when using a single GPU and 1 core.

Is there any fix for it, my dataset is very large and thus training will take very long if using single core and GPU.

THanks

What resource manager are you using? So far we tested multi-node in a SLURM environment only

I am using google cloud vm without any resource manager.

Doesn't GCP have at maximum just 8 GPUs? I'm not familiar with any one click solution to launching a 32 gpu (8 nodes, 4 GPUs per node) cluster on GCP - please point to such a config.

If you want to use just 8 GPUs, your trainer config would be 

trainer = pl.Trainer(gpus=8, num_nodes=1, accelerator='ddp',max_epochs=200,amp_level='O1', precision=16, ...)



in GCP you can spin up a vm with 8 nodes, and 4 GPUs 

I am using this command 
> `trainer = pl.Trainer(gpus=4, num_nodes=8, accelerator='ddp',max_epochs=200,amp_level='O1', precision=16,..)`

I dont think this is a GCP issue but rather pytorch ligtning 1.7.1 issue where ddp hangs when initialising 

[https://github.com/PyTorchLightning/pytorch-lightning/issues/4612](url)

Is there a way to update nemo to work with an older or newer version of pytorch?

@sciai-ai just like workaround to fix your problem: 
```
pip uninstall -y torch pytorch-lightning torchtext torchaudio
pip install torchtext==0.8.0 torch==1.7.1 torchaudio==0.7.2 pytorch-lightning==1.2.5
```

I suppose problem related to https://github.com/PyTorchLightning/pytorch-lightning/issues/5865, https://github.com/PyTorchLightning/pytorch-lightning/issues/4612 and https://github.com/PyTorchLightning/pytorch-lightning/issues/6569

@sciai-ai can you please try 1.0.0 to see if this issue still persists?

@ericharper I noticed you re-opened this issue. We're still seeing this behavior in identical fashion, but also on CLI. Double checking, shall we re-open this one?  

Feel free to use this issue. Please update with the code that was run and the log files.

Great. Will do.

@ericharper We have it resolved! By submitting the batch job in Slurm, we were able to get multi-GPU training working. We confirmed that the trainer was configured in the YAML file.
cannot import name 'LabelStudioMLBase' from 'label_studio.ml'
State: closed | Pull request: no

I tried to run LabelStudio notebook but gave me this error 

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-ddf1c99aa95c> in <module>
      1 import nemo
      2 import nemo.collections.asr as nemo_asr
----> 3 from label_studio.ml import LabelStudioMLBase
      4 
      5 

ImportError: cannot import name 'LabelStudioMLBase' from 'label_studio.ml' (/home/SAM/anaconda3/envs/nemo/lib/python3.7/site-packages/label_studio/ml/__init__.py)


what could be the problem? 

Thanks for this issue @Omarnabk! We moved the Label Studio ML backend to a new repo, so the import path needs to be updated, and you'll also need to install the ML backend in addition to Label Studio.

a PR to fix this in the Jupyter notebook is coming soon, but in the meantime you can do the following:
1. After you install label studio, install the ML backend: `pip install git+git://github.com/heartexlabs/label-studio-ml-backend.git#egg=label-studio-ml` (soon there will be a proper label-studio-ml pip package but you can use this command in the meantime).
2. update the import statement to import `from label_studio_ml` instead of `from label_studio.ml`

let us know if you have any more questions!


https://github.com/NVIDIA/NeMo/pull/1989 should fix it.
run asr_webapp failed because version of pyramid has confuse?
State: closed | Pull request: no
Labels: name: bug

hi,
I try to run demo asr using asr_webapp but it's show error:
`ImportError: cannot import name 'UnencryptedCookieSessionFactoryConfig' from 'pyramid.session' (unknown location)`
and then i check  pyramid version: 2.0 and in pyramid github, they say UnencryptedCookieSessionFactoryConfig has remove. I try to another version pyramid for can running demo but it's failed.
So,  Can anyone check this demo again?
Thanks.
installing nemo_toolkit[all]==1.0.0b3 installs torch=1.8.0
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I created a fresh conda environment and installed PyTorch==1.7.1 and nemo_toolkit[all]==1.0.0b3.
Although it says that NeMo is not supporting PyTorch 1.8, installing NeMo uninstalled PyTorch 1.7.1 and installed 1.8 version. I am guessing this is a bug. I even got the following message:

```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchaudio 0.7.2 requires torch==1.7.1, but you have torch 1.8.0 which is incompatible.
```

**Steps/Code to reproduce bug**

- Create a new conda environment with Python 3.8.2
- Install PyTorch 1.7.1 with pip
- Install NeMo 1.0.0b3 with pip

**Expected behavior**

Should install PyTorch 1.7.1 instead of 1.8

**Environment overview (please complete the following information)**

 - Environment location: Windows 10 
 - Method of NeMo install: pip install nemo_toolkit[all]==1.0.0b3

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Windows 10
- PyTorch version: 1.7.1 - 1.8
- Python version: 3.8.2


Please start using rc1 version, we are going to release it soon
```
 python -m pip install git+https://github.com/NVIDIA/NeMo.git@r1.0.0rc1#egg=nemo_toolkit[all]
```
Update tarred dataset preprocessing script to use joblib
State: closed | Pull request: yes

# Changelog
- Update ASR tarred dataset preprocessing script to enable more than 1 workers via joblib
- Maintaining backward compatibility with exact same results even when `workers` > 1 as compared to `workers` == 1.

# Caveats
- The new manifest itself will temporarily require double the memory in ram now until all of the tarfiles are processed as we need to flatten the intermediate representation of each create tarfile output manifest
[Question] How to convert a .nemo checkpoint to onnx 
State: closed | Pull request: no

**Describe your question**

How to convert .nemo ckpt to onnx? 
I've tried to convert a .nemo model trained on 1.0.0b1 version of nemo to onnx. 
I want to use the onnx for deployment using Triton later on. 

**Environment overview **

 - Environment location: Conda 
 - Method of NeMo install: pip install== 1.0.0b3 


**Environment details**

- OS version: ubuntu 18.04 
- Torch 1.7.1
- Python 3.6.13

**Additional context**
 The model was trained on Arabic langauge. 
 I've tried to use this file: https://github.com/NVIDIA/NeMo/blob/v1.0.0b3/scripts/convasr_to_single_onnx.py
 and got this error: 

![image](https://user-images.githubusercontent.com/71659848/112324884-81d22680-8cb3-11eb-8843-ced825715353.png)

Thank you for your time! 

Can you try installing from main branch the most recent version?

Thank you for your prompt response! 
I've tried to installed it like this: 
apt-get update && apt-get install -y libsndfile1 ffmpeg
pip install Cython
pip install nemo_toolkit[all]==1.0.0b3

and I got another error related to writing the onnx file. 

![image](https://user-images.githubusercontent.com/71659848/112643590-ce993700-8e44-11eb-99db-46573036144b.png)

I'm not sure if this is what you mean by installing from main but I've tried,also, installing from the main git branch and I got an error in the nemo import related to pyramid installation:    
 from pyramid.session import UnencryptedCookieSessionFactoryConfig                                                                                                                                                                        ImportError: cannot import name 'UnencryptedCookieSessionFactoryConfig' from 'pyramid.session' (unknown location) 
 
 Thank you again. 


Hello again,
I think I found the issue, it's related to the path. 
I did as u said and It worked fine! 
Thank you for your time.
Make VAD inference tutorial clearer
State: closed | Pull request: yes

Reorg notebook
Include diagram to explain overlapped prediction
Add/update instruction on 
1. offline streaming inference on audio files (offline VAD);
2. finetuning and use posterior;
3. threshold tuning;
4. online streaming inference;
5. online streaming inference from a microphone's stream.

Signed-off-by: fayejf <fayejf07@gmail.com>
[Question] (How) Can I use Nemo offline on Pycharm with Windows ?
State: closed | Pull request: no

**Describe your question**

Hi i'm new to NeMo and I wanted to get started with it to use it for some ProjectIdeas.
I want to use NeMo offline so I tried to get it going on Pycharm.
Now I installed everything from Cuda to Kaldi and it's still not working.
I'm getting a freeze.Support() Runtime Error from multiprocessing/spawn.py when I only try to import
nemo.collections.asr only on Pycharm.
When I tryed it on JupiterNotebook this Error was gone but if I try to run a model torch/nn/modules/module.py l.826
throws "AttributeError: can't set attribute"
I wanna know now if there is a easy way to get such Problems fixed or a way to uninstall everything and get one clean installation for windows that works on pycharm aswell.
Or is it the best Choice to get Linux in a virtual machine and run it there?
I would very much appreciate any help.
Greetings from Germany. 



**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model


Hi, unfortunately, at this moment we only support Linux (Ubuntu).
[Question] OFFSET in Voice Activity Detection
State: closed | Pull request: no

Can someone briefly explain to me the Offset in the manifest file of VAD?
How can I decide it and what is it exactly?

Thanks in advance.

Hi thanks for your question! The offset in manifest (for all audio related manifest not limited to VAD)  means the **starting point (second) of the audio snippet**. 
For example` "audio_a.wav, offset:0, duarion:10"` means "audio snippet of audio_a from 0 to 10 seconds"; SImilarly 
`"audio_a.wav, offset:5, duarion:5" `means "audio snippet of audio_a from 5 to 10 seconds";


Thank you so much! @fayejf 

I am closing the issue.

Regards.
Add ASR Evaluation Service
State: closed | Pull request: yes

# Changelog
- Adds Flask based ASR Service with support for gunicorn and docker container
- Enables inference via pretrained checkpoint to evaluate models on audio data that is formatted to 16 KHz and mono-channel wav files.
- Enables upload of trained checkpoints for evaluation of custom models (can be disabled inside `templates/main.html`)

# Usage
## Flask Server

```bash
$ python asr_service.py
```

## Gunicorn Server
By default, the gunicorn server config is loaded from `gunicorn.conf.py`. Command line values will override these arguments.

**NOTE**: Limit the number of workers since each worker will maintain the cache of models in memory.

```bash
# wsgi.py is a thin wrapper over `asr_service.py` to enable the service without debugging mode
$ gunicorn wsgi:app
```

## Docker Server
The app can be built and run easily using docker. 

Build the docker container using -
```bash
$ bash docker_container_build.sh
```

Run the docker container (in detached mode with all available GPUs) using - 
```bash
$ bash docker_container_run.sh
```

This pull request **introduces 2 alerts** when merging 82fa2be0844042c845af3cf43e7e5e2cf989eab0 into da48c997177f89a9e305e508ab27b15954044814 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-87eabc38d29520a4b3fc1bcfa066c69c4774062f)

**new alerts:**

* 1 for Unnecessary pass
* 1 for Unused import

> 
> 
> This pull request **introduces 2 alerts** when merging [82fa2be](https://github.com/NVIDIA/NeMo/commit/82fa2be0844042c845af3cf43e7e5e2cf989eab0) into [da48c99](https://github.com/NVIDIA/NeMo/commit/da48c997177f89a9e305e508ab27b15954044814) - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-87eabc38d29520a4b3fc1bcfa066c69c4774062f)
> 
> **new alerts:**
> 
>     * 1 for Unnecessary pass
> 
>     * 1 for Unused import

Some LGTM warnings to address.

This pull request **introduces 2 alerts** when merging 70625bf33217750fe1931d091ca44f4c8d305487 into 532c71b8c632bc9c0fc1096d7630a02da3bee00e - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5af65e7126afc589c92131f898452b6562e14acf)

**new alerts:**

* 1 for Unnecessary pass
* 1 for Unused import
during training of NeMo ASR , model not able to save checkpoint at the end of every epoch
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

when i try to train the NeMo ASR model with the help of pre-train model quartznet(QuartzNet15x5Base-En) . model not saving checkpoint at the end of every epoch. 

**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

model save chechpoint at end of every epoch.



**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- Ubuntu 18.04
- PyTorch 1.1.4
- Python 3.6

**Additional context**

Add any other context about the problem here.
Example: GPU model


To save checkpoints at every epoch  pass `+exp_manager.checkpoint_callback_params.save_top_k=num_epochs` while calling script or add through yaml file 

nemo saves the last checkpoint, not the ones at the end of each epoch. It can also saves the top K best ones. You need to set K along with what metric to monitor and if best value is min or max. You may add it to your config file:

exp_manager:
  checkpoint_callback_params:
    # in case of multiple validation sets, first one is used
    monitor: "val_wer"
    mode: "min"
    save_top_k: 3
fixed branch in IR tutorial
State: closed | Pull request: yes

Signed-off-by: AlexGrinch <grinchuk.alexey@gmail.com>

wrong branch. I applied this fix to rc

This pull request **introduces 10 alerts** and **fixes 3** when merging 398409b424148af7b95a84fce097d7f053b6bca7 into 6594dd6dcd268cc7eeb1caf08a7b9ed18cd40556 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-13346d8c5f9ee999c6b43b23568223cccbc61a20)

**new alerts:**

* 6 for Unused import
* 2 for Modification of dictionary returned by locals\(\)
* 1 for Signature mismatch in overriding method
* 1 for Mismatch in multiple assignment

**fixed alerts:**

* 2 for Unused import
* 1 for Explicit export is not defined
RuntimeError: fft: ATen not compiled with MKL support [Question]
State: closed | Pull request: no

**Describe your question**
I am getting the error on title on arm processor when trying to do inference with QuartzNet model. This is expected since MKL is not supported on arm. How can do inference on arm devices? Are there any alternatives? 

I am using PyTorch 1.9 but I think this is not the reason for this error.


**Environment overview (please complete the following information)**

 - Arm emulated Docker Container, running on Windows 10
 - From source. Followed exact instructions explained on readme of 1.0.0b3:
```
apt-get update && apt-get install -y libsndfile1 ffmpeg
git clone https://github.com/NVIDIA/NeMo
cd NeMo
./reinstall.sh
```

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- Ubuntu 20.04 version
- PyTorch version 1.9 
- Python version 3.8.5

At the moment we do not test/support Arm architectures. Do you know which module need MKL? Are you able to install Pytorch on Arm without MKL?
Fix mcv dataset preparation
State: closed | Pull request: yes

# Changelog
- Update MCV dataset preparation script to utilize v6.1 dataset 

This pull request **introduces 1 alert** when merging c2593f8fe561b87c84cc95e892d8995a09a40592 into 29b1cbb91a96d97c4935739ea53bd3aa57214563 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d327932951605936d384c4f20833f2afebe27fb5)

**new alerts:**

* 1 for Implicit string concatenation in a list

This pull request **introduces 1 alert** when merging 389dda9092255c01f0aede60bad55deeb37cdba5 into 3436c5c01c3a1c52906fe51a6edc82a0a0d9bb50 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-eabc2cd68861cdcc6ba5c7c4b153c7035141e5ce)

**new alerts:**

* 1 for Implicit string concatenation in a list

This pull request **introduces 1 alert** when merging 7b3225c7d5142906c072a7a1a6285997e4e46cea into aa0b2df346ad6976284cb9e5e74eb311004f24d3 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-c89c964b7a6f99f2d4b3881a49c1a01ac5b4a462)

**new alerts:**

* 1 for Implicit string concatenation in a list

This pull request **introduces 1 alert** when merging c677d1ce0c70de3b145c92f618d02a2625031165 into da48c997177f89a9e305e508ab27b15954044814 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-3e83d49170526fc7a92ffb270d22cf87931e9a9c)

**new alerts:**

* 1 for Implicit string concatenation in a list
Error while multi-gpu training of megatron model
State: closed | Pull request: no
Labels: name: bug

**bug description**

Hello,
This error report might seem little rough around the edges mainly because I've been trying out different language models in different nemo branches and within different environments too.
Starting with multi-gpu training of QAModel with _"megatron-bert-345m-uncased"_ lm on squadv2 dataset, gives me this error while its doing evaluation.

_Environment location - GCP instance (with 2 x Nvidia Tesla T4)
Nemo branch - main (1.0.0rc1)_
![image](https://user-images.githubusercontent.com/72965135/111428452-84fe6d00-871d-11eb-8ed9-b1c87ce9f675.png)

I am also getting this error when using DGX machine
_Environment location - NVIDIA DGX Station Version 4.7.0 (using 2 x Tesla V100)
Nemo branch - main (1.0.0rc1)_
![image](https://user-images.githubusercontent.com/72965135/111428864-1ec61a00-871e-11eb-9703-a98f1b8ba774.png)

Changing language model from megatron to "bert-base-uncased" does not give keyerror but gets stuck after running evaluation:
_Environment location - NVIDIA DGX Station Version 4.7.0 (using 2 x Tesla V100)
Nemo branch - main (1.0.0rc1)_
![image](https://user-images.githubusercontent.com/72965135/111434612-c430bc00-8725-11eb-9545-2c1a570c2ff9.png)

However I don't get any issue training on single gpu in this branch and also when training on multi-gpu in nemo's r1.0.0b3 branch, but I am not able to use megatron there. I found a similar bug report in huggingface transformers gitrepo ([#1472](https://github.com/huggingface/transformers/issues/1472)) where they mention workaround to not do evaluation while training, can this be done for QAModel too by its config file?


Hi, you can remove validation by either removing it from the YAML file or from CLI you can use the `~` operator.

```
~model.validation_ds
```

Could you give me some more information on your environment? Are you using an NVIDIA container?

Also could you clarify `Nemo branch - main (1.0.0rc1)`? main is a branch and so is r1.0.0rc1. Are you using the r1.0.0rc1 branch? How are you installing NeMo?

Thanks.

Hi, @ericharper , Thanks for the reply, I'll try by removing validation,
I am using NVIDIA's PyTorch container version 20.12-py3 and installed nemo using `python -m pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[all]`. I mentioned 1.0.0rc1 because on doing `import nemo, nemo.__version__` I get this output. I also noticed now that there is an updated pytorch container, I'll try using that as well.

Okay, if you wish to use the `main` branch, then please use the latest PyTorch container, 21.02. 

Closing due to inactivity. Please reopen if still having the issue.
Remove redundant subclasses for ASR
State: closed | Pull request: yes

# Changelog
- Remove redundant subclasses for ASR which resolve to same base class

This pull request **introduces 5 alerts** when merging 9e7ed5e0720eb7b5f9498f3bd09ad4470488b8da into 3517e5686e2178e14dc4066e391784ccb9a8aba4 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-c0ff736e4f47dcf8fd899a21d2ce471404b7cbef)

**new alerts:**

* 5 for Explicit export is not defined

This pull request **introduces 5 alerts** when merging 54c5b9fd2b496bd241a48ae12d521287c27c217c into f8c2e6387de1f384abc73154c9825095ce9b8c00 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a412c91982805f690099020ce3da841e6326cddb)

**new alerts:**

* 5 for Explicit export is not defined
SentencePiece support for NMT
State: closed | Pull request: yes

This PR adds support for training (as well as providing pre-trained) SentencePiece tokenizer to the **MTEncDecModel**.

~~Minor changes in SentencePiece wrapper to retrieve _unk_id_ and remove _special_tokens_ at the time of "ID to Text" conversion.~~ Changes to the SentencePiece wrapper have been moved to a new, more generic PR #1973.


### SentencePiece for NMT Usage ### 
To use sentencepiece with MTEncDecModel, just set:

`model.encoder_tokenizer.library=sentencepiece` and
  `model.decoder_tokenizer.library=sentencepiece `

A pre-trained SentencePiece model can be provided using the:

`model.encoder_tokenizer.tokenizer_model` and `model.decoder_tokenizer.tokenizer_model` tags.

In case no pre-trained SentencePiece model is provided, it will be trained based on the configurations provided. 

The SentencePiece tokenizer can also be used for only the encoder while using a different tokenizer for the decoder, and vice versa. Similar mix-n-matches can also be done for training a tokenizer for encoder/decoder only (while providing a pre-trained version of the other) etc.

This pull request **introduces 2 alerts** when merging 8d829d152d8410a49d235910aef490433478d3a9 into 39e2e4a86ec76fb96ac79e47b523e89b6bd3e7a0 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b4d06bfdcdc04bba5c53e0dd570d5e155687daa4)

**new alerts:**

* 2 for Testing equality to None

Also please add example usage to the README. 

After discussion, default can be to drop special tokens, and asr can just use a constructor level flag to default to old behaviour as needed. 
Unblocked, address other reviewers and then ping me
Do we need to know the number of speakers before performing speaker_diarization?
State: closed | Pull request: no

From the[ speaker_diarization.yaml](https://raw.githubusercontent.com/NVIDIA/NeMo/main/examples/speaker_recognition/conf/speaker_diarization.yaml) file
There is `num_speakers: 2 # for each recording` value added in the yaml file.
Hence, do we need to know the number of speakers present in a audio recording to perform inference of speaker_diarization or can it be done unsupervised ?

Technically no, but it is always good to provide maximum number of speakers for higher accuracy. Support for dynamic computation for number of speakers is not yet fully tested. If you would like to try change 
min_clusters=1 and max_clusters=max_num_speakers (which you are sure it won't exceed that number) 
https://github.com/NVIDIA/NeMo/blob/27e5f2dce512495d93577c39618409d3403b210a/nemo/collections/asr/parts/speaker_utils.py#L256
Update RC1 to have joint installation of PT and PTL
State: closed | Pull request: yes

Signed-off-by: smajumdar <titu1994@gmail.com>

@DEKHTIARJonathan The code here is installed via reinstall.sh - which does pip install .[all]. 

It's still following all wheel semantics. The issue is that the pytorch container 21.02 is masquerading as a Pytorch 1.8.0a* - but it is in fact NOT pytorch 1.8. 

This confuses Pytorch Lightning which has code switching for deprecated torchtext functionality. 

The pip release is explicitly handled by @okuchaiev - he can remove this patch. But for the moment, none of the RC1 branch code will run or can be tested on pytorch 1.8 since 21.03 container has not been released yet. 

I still oppose this PR. Code relevant to your testing environment should not pollute the core of the package. This approach is just bad practices.
Speedup librispeech dataset processing by using multiple processes.
State: closed | Pull request: yes

It is 2021, we should use parallel computations for slow scripts when possible.

As each directory contains independent data, we can easily add more processes to get_librispeech_data script to decrease waiting time.

On my machine the speedup when using 4 processes was 7-8 fold.
The whole download process went from >2h waiting time to 15 minutes.

The downloaded data is identical, so there should not be any problems introduced by parallelizing the script.

PTAL. I addressed the ordering issue.

LGTM will merge after tests pass
ASR Doc update v2
State: closed | Pull request: yes

# Changelog
- Update documentation with new links and new models
- Add documentation for newer models
- Add docstrings for EncDecCTCModel and EncDecRNNTModel for training, validation and test set data loader setup
- Add documentation with links for datasets themselves
- Add documentation for useful misc classes

Thanks @VahidooX ! Let's merge once @redoctopus gives go ahead.
Updates to conformer.
State: closed | Pull request: yes

+ Made positional embedding more efficient.
+ Updated the rel_shift() method to avoid flipping which needs significant memory.
+ Fixed the onnx issue.
+ Added att_context_size feature to limit the context of attention with masking.

@borisfom or @drnikolaev would you please checkout the asr_model.py? 
I needed to make some changes to the export part. We did not pass the length to the encoder and it was fine for convolutional modules which do not use that input. But Conformer would use that input and export would not work on this model without passing lengths.
Clean up ASR augmentor config docs
State: closed | Pull request: yes

Removed redundant parameter table
Signed-off-by: Jocelyn Huang <jocelynh@nvidia.com>

It's up to you if you want to merge it or not. My concern is that we are lacking good docstrings and are trying to make up for it by adding it in the documentation

LGTM merge when ready 

Ideally the API documentation for this module should incorporate the above information. So just adding it to the docstring will add it to the API

> It's up to you if you want to merge it or not. My concern is that we are lacking good docstrings and are trying to make up for it by adding it in the documentation

I think this one is fine to remove from the docs; everything here is already covered in the `SpectrogramAugmentation` docstring so we don't lose anything by deleting it from this page.
(https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/api.html#nemo.collections.asr.modules.SpectrogramAugmentation)
HifiGAN bug fixes
State: closed | Pull request: yes

Minor bug fixes for HifiGAN

This pull request **introduces 1 alert** when merging 51b18761108fd3e3bb010ecdfc7de5dd9f58f1e7 into 646aff11aa788b7d874dae930ce6ffcd658c1f16 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-4f18e74fbecbcd7aa309fdb87b7e7f7dd90258ee)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 3f56fd44fed80502be09f8641d239930cb5add41 into 646aff11aa788b7d874dae930ce6ffcd658c1f16 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d30e4dfd1c67b1be0980fa203da66149d80ad625)

**new alerts:**

* 1 for Unused import
Added HifiGAN to TTS_Inference notebook
State: closed | Pull request: yes

This PR adds HifiGAN to the TTS inference notebook, and adds an NGC checkpoint trained on ground-truth LJSpeech mels to list of available models.

This pull request **introduces 1 alert** when merging 021e3b10e8a6a6b27cf059ed6a6425bb2c5ffda8 into 05eb2587a95e512b5e95705d9f77582634abbd95 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-30f680fe1081bb12260d2adaace8bd00410cb403)

**new alerts:**

* 1 for Unused import
removing loading of the trained model from the example to prevent mul…
State: closed | Pull request: yes

…ti gpu bug

Signed-off-by: Vlad Getselevich <vgetselevich@nvidia.com>

This pull request **introduces 1 alert** when merging 4a0ca9b202c7c01f9d827cf392f0cd7cbb3af566 into 5d80627ca696384dc16e462f1e17efd33917a9dd - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-119caa9229f75b6c349ae1b681724d504f3e2940)

**new alerts:**

* 1 for Unused local variable
[Question] Exporting custom NeMo models to Jarvis (.ejrvs)
State: closed | Pull request: no

Dear NeMo team, 

I'm currently looking into Jarvis and haven't found a pre-built way to export my custom NeMo models to .ejrvs format yet, only for TLT. Several sources (including https://developer.nvidia.com/nvidia-nemo) state that it's possible with a single command. If anyone knows the magic words, please let me know. 👍 

Kind regards!

PS: I'm using NeMo 1.0.0b3 via the official docker.

@xaiguy Can you please clarify what do you mean by "custom NeMo model"? 
Every TLT model can be exported to Jarvis. But NeMo contains more models than TLT, including many "research" ones which can't be deployed to Jarvis yet.
 
We are constantly increasing the number of models supported by Jarvis by adding them to TLT.


@okuchaiev Oh, pardon the ambiguity. With "custom" I just meant models fine-tuned with NeMo, in particular Jasper. I just mentioned TLT because I saw that it's very easy to save and export models to .ejvrs format from there and I was hoping to find something similar for NeMo.

I saw a new passage was added to the docs, thanks! I'll close this issue now, as the answer can be found following the link below:

https://docs.nvidia.com/deeplearning/jarvis/user-guide/docs/model-overview.html?highlight=servicemaker#nemo-to-jarvis
Add Stan's Durations and Gaussian Embedding
State: closed | Pull request: yes


This pull request **introduces 10 alerts** and **fixes 6** when merging f44c492473364106bcc445a9b723ed3c961f2ac3 into cf23c451ecdc8bd8b0d2bd48103717121780e989 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-0671fc7ef505fcdfab6011715546b767428230d0)

**new alerts:**

* 9 for Unused import
* 1 for Unnecessary pass

**fixed alerts:**

* 6 for Unused import
conformer to onnx not working
State: closed | Pull request: no
Labels: name: bug

Conformer model export to onnx not working as intended. Onnx export worked ok and did not report anything and onnx.checker.check_model worked without errors too. Loading the onnx file and running the sample audio file causes onnxruntime to die quietly (jupyter just reloads the kernel, and running script just says "segmentation fault")

Steps to reproduce:
load .nemo model for conformer 
`model = nemo_asr.models.EncDecCTCModelBPE.restore_from('conformer.nemo')`
export conformer encoder to onnx 
`model.encoder.export('conformer_encoder.onnx')`
load onnx in onnxruntime and run through a sample file
```
sess = onnxruntime.InferenceSession(conformer_encoder)
input_name = sess.get_inputs()[0].name
output_name = sess.get_outputs()[0].name
result = sess.run([output_name], {input_name: features})
```
After running these steps I got a segmentation fault error. For feature extraction and onnx export/load I used the same pipeline that I used for jasper models (and it worked perfectly for jasper). Conformer works okay during training and evaluation within nemo and ptl frameworks. 
In which case onnxruntime just fails completely without even reporting for something like incorrect dimensions or unsupported operations (like it usually does)?
Maybe I am missing something obvious here?

environment: 
DGX-1, Ubuntu 18.04
nemo - up to date
torch - 1.8.0 
onnxruntime - 1.6.0
onnxruntime-gpu - 1.4.0 

I tried different combination of versions of the libraries but most of them cause the effect as mentioned above and the others fail for other reasons - mostly during the export step.

I think we haven't properly implemented export for Conformer. @VahidooX to comment further. If this isn't supported yet, maybe should raise "NotImplemented" exception? 

We have not tested the export for Conformer yet. I would raise NotImplemented for now. 
Update: I fixed the issue with the export function in this PR https://github.com/NVIDIA/NeMo/pull/1898. It is just a part of that PR, and not merged yet. Now it creates the onnx file but there is no guarantee that it works. yet

closing since PR is merged
I often find conflicts in the environment when using NeMo.
State: closed | Pull request: no
Labels: name: feature

I often find/see conflicts in the environment between - torch, torchtext, torchaudio, torchvision, pytorch-lightning and nemo and this causes many problems for me. Also, it seems like many of the issues posted here are related to those conflicts. This is to be expected with the fast pace these libraries are developing, but things could improve for the user when it comes to theses issues. 

A solution would be to post fixed and verified library/environment versions and install readme:s for the environment supporting the release, e.g. an INSTALL.md with fixed versions per release. 

Another solution would be to be more strict with the versions required in the requirements-files under the requirements folder. That would not support anaconda environments, so please also consider to include how to achieve a stable anaconda environment. 

Further more, a nice example is the [pytorch local install page](https://pytorch.org/get-started/locally/), where you could iteratively get instructions how to install.

(OR a novel approach would be to simply tar the environment and offer it as an s3-artefact along with the release, e.g. arm, x86-64 etc?)

Br, PvR

I also had to fix torch to 1.7.1. I also opened a pull request for that, but is not accepted to stick the version of torch. You can read more here https://github.com/NVIDIA/NeMo/issues/1868  and follow until you see my pull request, which is a temporary workaround for me. Maybe you should apply to yourself the same practice.

Closing due to inactivity. Please reopen if still having the issue.
After upgrading P.Lightening to 1.2.3, remains other incompatibilities with torch 1.8.0. A workaround until is sorted out all, is to stick with torch 1.7.1.
State: closed | Pull request: yes

… pull down 1.8.0. https://github.com/PyTorchLightning/pytorch-lightning/pull/6168

This diff temporarily solves the imported incompatibility reported in https://github.com/NVIDIA/NeMo/issues/1868


thanks again @tiborkiss  but we'll close this one for now because we can't pin torch version to the exact number like 1.7.1 We could add something like this right before upload to pypi though.
Running NeMo on Linux or Collab and not working
State: closed | Pull request: no

Hello everyone,

I have some problems trying NeMo. I want to setup an environment on a Linux machine but after the installation of nemo_toolkit, the import (_import nemo.collections.nlp as nemo_nlp_) keeps rejecting an error: 
**ImportError: cannot import name 'Batch' from 'torchtext.data' (/usr/local/lib/python3.7/dist-packages/torchtext/data/__init__.py)
**

I also tried the projects from a NeMo example (https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/nlp/Punctuation_and_Capitalization.ipynb) but I got the same issue...

Can anyone tell me if I miss something? Thank you all in advance!

Yours.

This is a pytorch lightning issue, please downgrade to 1.2.1 for the time being until they are able to support pytorch 1.8 as well as it's dependency of torchtext

> This is a pytorch lightning issue, please downgrade to 1.2.1 for the time being until they are able to support pytorch 1.8 as well as it's dependency of torchtext

Me neither :/ 

I described already here https://github.com/NVIDIA/NeMo/issues/1868

Duplicate of #1868
EnJa tokenize output format fix
State: closed | Pull request: yes

~preprocessor `tokenize(txt)` methods should probably return a list of tokens, not a string. these can then be converted into ids without the intermediate steps of producing a string and then immediately splitting it for the tokens again, as was being done in `translate`~

it turns out that `text_to_ids(txt)` acts differently than `tokens_to_ids(txt.split(' '))`, so i am changing the output of the `processsor.tokenize` to be `str` again.

buncha tests are failing but i dont think theyre related to this pr