renwei2024/nemo-github-issues huggingface dataset:
wav2letter
State: closed | Pull request: no

**Describe your question**

I don't manage to find wav2letter in Nemo ASR models wheareas it existed in OpenSeq2Seq.

What's the point ?


**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model
Add preliminary support for ASR Model dataclasses
State: closed | Pull request: yes

# Changelog
- Adds nearly concrete implementations of dataclasses for QuartzNet 15x5, Jasper 10x5-DR and MatchboxNet-3x1x64 and VAD modes.
- Adds *ConfigBuilder class supporting the construction of arbitrarily complex interdependencies between sub-configs
- Adds various dataclasses for ASR datasets + ASR Classification datasets
- Adds core support for class-dataclass testing

This pull request **introduces 1 alert** when merging cdb20c1742dca80fd3519991029d356e95573f98 into 9ea071605034f9f56db9b677eb1fe54fdee2b820 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-cc6090175340df7c3bed280609bfe623e3253748)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 6ad69e3b240fe8ffedcaae7d4260dcbd05828c1b into 9ea071605034f9f56db9b677eb1fe54fdee2b820 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2bbd5a3297d9a000e28efc1a91aba9366c42046e)

**new alerts:**

* 1 for Unused import
Issue in loading model from checkpoint
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I am trying to run the Relation Extraction - BioMegatron notebook where once the model is trained, I am trying to load the best checkpoint.

**Steps/Code to reproduce bug**

- Train the model as usual

`checkpoint_path = trainer.checkpoint_callback.best_model_path`
`eval_model = nemo_nlp.models.TextClassificationModel.load_from_checkpoint(checkpoint_path=checkpoint_path)`



**Expected behavior**

<img width="1048" alt="Screen Shot 2021-01-25 at 8 01 09 PM" src="https://user-images.githubusercontent.com/7240751/105785078-13d5e100-5f48-11eb-991d-3560fa24a1e3.png">


**Environment overview (please complete the following information)**

 - Environment Location: Collab
 - Method of NeMo install:  pip
Text normalization
State: closed | Pull request: yes


This pull request **introduces 6 alerts** and **fixes 1** when merging af4278cd307544343b466d80f681f939f41d7a3e into df360cd3c856c645d23505305ee8b689636b2dd1 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-976deade5b08d9a975c92bc6433f92edeaf9f4ab)

**new alerts:**

* 3 for Except block handles &#39;BaseException&#39;
* 3 for Unused import

**fixed alerts:**

* 1 for Except block handles &#39;BaseException&#39;

This pull request **introduces 1 alert** when merging 9ed7a43b4919facffeee454d1fe3ff26bc182d17 into 9ea071605034f9f56db9b677eb1fe54fdee2b820 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-798486a6838f30a2b5a1dd7ef2a11ee73916b0c5)

**new alerts:**

* 1 for Unused import
Handle shape mismatch when restoring pretrained weights
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**
Loading pretrained weights onto a model with a slightly different architecture results in a shape mismatch error. This makes it unwieldy to use pretrained weights for fine-tuning a model with a different decoder shape (for example, ASR with a different vocab size). modelPT.restore_from has a strict=False option but will error if a layer has a similar name but different shape.

**Describe the solution you'd like**
My preferred approach is for restore_from to warn of shape mismatch and ignore loading weights in that circumstance. Something like this perhaps:
```
instance = cls.from_config_dict(config=conf)
instance = instance.to(map_location)

instance_state_dict = instance.state_dict()
state_dict = torch.load(model_weights, map_location=map_location)
if not strict:
    mismatched_layers = set()
    for k, v in state_dict.items():
        if k in instance_state_dict and instance_state_dict[k].shape != v.shape:
            mismatched_layers.add(k)
            logging.warning(f"Size mismatch for {k}: ignoring weights")
    [state_dict.pop(name) for name in mismatched_layers]

instance.load_state_dict(state_dict=state_dict, strict=strict)

logging.info(f'Model {cls.__name__} was successfully restored from {restore_path}.')
```

This is not a good approach imo, because it allows the chance that model weights are skipped with a warning only and training progresses as is. When developing on scheduled dispatchers like Slurm, we rarely every take constant look at slurm logs so this can silently introduce bugs.

The official way to change the decoder is using the provided method - it contructs a new randomly initialized decoder for you. If you only wish to load encoder weights, that is possibly as below

```python
old_ckpt = torch.load(...)
new_model = EncDecCTCModel(...)

# change decoder vocabular
new_model.change_vocabulary(...)

# do things manually (NOT ADVISED)
new_model.decoder = torch.nn.Sequential(...)
new_model.encoder.load_state_dict(old_ckpt['state_dict']['encoder'])
...
```

Thank you for the response! Yeah, that's a fair point. I'll just stick with the provided method then.
Licence of pretrained models
State: closed | Pull request: no

Hi, do the pretrained models from https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels have the same Apache2 licence as this repository?

1. NeMo software is released under [Apache 2.0 license](https://github.com/NVIDIA/NeMo/blob/main/LICENSE)
2. Pre-trained checkpoints which you can find on NVIDIA NGC are under [NGC TERMS OF USE](https://ngc.nvidia.com/legal/terms) unless another License/Terms Of Use/EULA is clearly specified.

Specifically, all models from:

- https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels/files?version=1.0.0a5
- https://ngc.nvidia.com/catalog/models/nvidia:nemonlpmodels/files?version=1.0.0a5
- https://ngc.nvidia.com/catalog/models/nvidia:nemottsmodels/files?version=1.0.0a5

(note ``?version=1.0.0a5`` part) are under [NGC TERMS OF USE](https://ngc.nvidia.com/legal/terms).

I think going forward we should explicitly specify License/EULA for pre-trained models' weights. Will try doing that in the next releases.

We are also open to community's feedback on this question.
Fix configs in ASR tutorials
State: closed | Pull request: yes

Update and fix the **outside of the model constructor configs** in ASR tutorials. Converting outside Hydra 0.x configs to Hydra 1.x configs.
Tiny bug fix for notebook 03 speech command

Related bug: [1649](https://github.com/NVIDIA/NeMo/issues/1649)

Hmm wait, the function exists in main but the error exists in b3? If the error exists only in main, then for now comment out these patches with a note that it needs to be uncommented. Or use a try catch. 

Also, tests are failing on a notebook only pr?

> Hmm wait, the function exists in main but the error exists in b3? If the error exists only in main, then for now comment out these patches with a note that it needs to be uncommented. Or use a try catch.
> 
> Also, tests are failing on a notebook only pr?

@titu1994 the error exists in b3 well the patch function exists in main. Is it safe to change branch in some notebooks to main now?  If it's not, I can revert it back to b3 and fix it by manually accessing params or writing the patch function explicitly in notebooks. 

Right not sure why the tests fails for notebook changes 

We will be changing branch later, main is not suitable for the notebooks cause it's not stable and tested as much as the exact branches.

For now, It might be best to do manual fixes by back porting the config updater to the notebooks.

> We will be changing branch later, main is not suitable for the notebooks cause it's not stable and tested as much as the exact branches.
> 
> For now, It might be best to do manual fixes by back porting the config updater to the notebooks.

Sounds great. will update it.
[Question]Can I integrate it with my college website to transcribe the lecture videos online?
State: closed | Pull request: no

I am new to ASR (Automatic Speech Recognition), I am working on a college website where professors can upload their lectures. I want to integrate an automatic speech transcript generator so that users can turn on the captions while streaming. While searching for open source options, I came across Nemo. Could anyone please tell me whether it is possible to do that and if Yes, could you please guide me?

>>"Can I integrate it with my college website"
- Yes.

>> "could you please guide me?"
- Have a look at this notebook for example. 
https://colab.research.google.com/github/NVIDIA/NeMo/blob/r1.0.0b3/tutorials/NeMo_voice_swap_app.ipynb
fix log dir
State: closed | Pull request: yes

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

This pull request **fixes 1 alert** when merging bdac5e9dff261cc3eb92ec5ce82746e0dab2a369 into a42987279f0226aea7d776840ff06a56c4bfb259 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-7fa88b8e67e895c005a893c335113b38bc7711b1)

**fixed alerts:**

* 1 for Except block handles &#39;BaseException&#39;
Fix for log_dir access in trainer
State: closed | Pull request: yes

Signed-off-by: Oleksii Kuchaiev <okuchaiev@nvidia.com>

This pull request **introduces 1 alert** when merging be2e4ab037277ddc4832ac2dc36155fc3ddccc63 into db60a786c71767478a49dfcad772718f9d4878d2 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-97c11b343a84e9d6d8a330403161e07698220e3c)

**new alerts:**

* 1 for Except block handles &#39;BaseException&#39;
Caching datasets for ASR training
State: closed | Pull request: no

**Caching datasets for ASR training**

The current data loaders for ASR seem to load the file from disk for every iteration with AudioSegment. This becomes a bottleneck when the disk is not very fast (i.e. is not a SSD). Is there a way to cache the audio files in RAM memory? Should we add the option to cache the datasets? We think that it may accelerate training when it fits.

Kind regards




currently we don't support such feature. Also, it would put high requirements on available RAM
[Question]Google Colab Error: MisconfigurationException: You requested GPUs: [1] But your machine only has: [0] 
State: closed | Pull request: no

Hi, I am a beginner in ASR technology and I am currently studying more about this field using your tutorial from this link: 
https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/asr/01_ASR_with_NeMo.ipynb

I was running the codes in your notebook, everything was fine until it reaches the "**Transfer Learning**" section, on the part where it asks to fine-tune the model for 2 epochs on AN4 dataset, I have stumbled on an error from this part of the code(bolded):

_#And now we can create a PyTorch Lightning trainer and call `fit` again.
**trainer = pl.Trainer(gpus=[1], max_epochs=2)
trainer.fit(quartznet)**_

The error was:

MisconfigurationException: 
                You requested GPUs: [1]
                But your machine only has: [0]

I am running the notebook on Colab and not locally, and there were no changes made on any part of the code. It was just me running cells to study. The runtime type has already been changed to GPU and it was a success on the "**Training with PyTorch Lightning**" section but not for **Transfer Learning**.

I would like to know what may be the reason for this error? 

For your information, I am using Chrome to run the notebook

If there is any inquires regarding my side, kindly do comment and reply, I will do my best to provide them.

Thank you in advance.


Could you try by replacing gpus=[1] with gpus=1? This seems to be trying to index GPU [1] which would be the second gpu, but colab has just 1 gpu

Hi Good Sir,

Yes, it seems that this is the case. It runs smoothly now once I have
changed to "gpus=1".

Although would just like to inform you that looks like the an4 server the
notebook is fetching data from is currently down. I have to use my own data
to validify your suggested solution.

Either way, thank you for your kind help.

Regards

On Thu, 21 Jan 2021 at 12:00, Somshubra Majumdar <notifications@github.com>
wrote:

> Could you try by replacing gpus=[1] with gpus=1? This seems to be trying
> to index GPU [1] which would be the second gpu, but colab has just 1 gpu
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/NVIDIA/NeMo/issues/1653#issuecomment-764221827>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ASQ73YTFZPOJPPHYQF6RTYDS26REPANCNFSM4WMBQN4A>
> .
>
Fix hydra runner
State: closed | Pull request: yes

This PR fixes hydra runner, enabling to use schemas with manual passing of configs files using command line.

- [x] Modified hydra_runner so it injects schema into the right "place" in ConfigStore.
- [x] Covers runner with integration tests

Usage: please take a look at tests and the used "example"  (`tests/hydra/my_app.py`)

Additionally cleans up some hydra settings in NeMo:
- [x] Sets hydra logging outputs only to stdout (no empty log file in  . dir!!)
- [x] Disables the creation of .hydra subdir 
- [x] Removes the old set_config wrapper


This pull request **introduces 2 alerts** when merging 1548c44f5dc90b95d7fc3d8fd1f1e83347753c74 into eb927fe6aa69f8d73705dcb785a2afb95e4c279d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2b1f14de5396282d23465f680fcf532753f62246)

**new alerts:**

* 1 for Non\-callable called
* 1 for Unused import
Clarify comments and neural types in Model Primer Notebook 
State: closed | Pull request: yes

# Changelog
- Address Neural Type comments from https://github.com/NVIDIA/NeMo/issues/1634 regarding LSTM
- Address clarification of "GPT Encoder neural module" vs "Transformer Decoder network" issue raised in https://github.com/NVIDIA/NeMo/issues/1638

Closes https://github.com/NVIDIA/NeMo/issues/1634 and https://github.com/NVIDIA/NeMo/issues/1638

Signed-off-by: smajumdar <titu1994@gmail.com>
Key 'window_stride' is not in struct
State: closed | Pull request: no
Labels: name: bug

I am trying to run the[ VAD notebook](https://github.com/NVIDIA/NeMo/blob/9aa4ada92b2e6ddd418704dc392e8576d44e5f41/tutorials/asr/07_Online_Offline_Microphone_VAD_Demo.ipynb). 

I followed every step but got this error:
![image](https://user-images.githubusercontent.com/39994105/105074869-f4a20400-5aae-11eb-85f0-6512c0f56d2a.png)



Thanks for reporting this. We will fix it asap and let you know.

@shfaizan  [Fixed](https://github.com/NVIDIA/NeMo/pull/1659)! Please pull and update. Thanks!
Quartznet RIR augmentation - tutorial 05
State: closed | Pull request: no
Labels: name: bug

Hello,
im tryin to implement RIR aug, just like suggested in the [tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/05_Online_Noise_Augmentation.ipynb
)
But when i run the cell:
```
rir_data_path = '.'
!python ../../scripts/get_openslr_rir.py --data_root {rir_data_path}
rir_manifest_path = os.path.join(rir_data_path, 'processed', 'rir.json')
!head -n 3 {rir_manifest_path}
```
It returns:

```
python3: can't open file '../../scripts/get_openslr_rir.py': [Errno 2] No such file or directory
head: cannot open './processed/rir.json' for reading: No such file or directory

```
So, i downloaded get_openslr_rir.py script manually from [here](https://raw.githubusercontent.com/NVIDIA/NeMo/main/scripts/get_openslr_rir.py)

And the cell to run became:
```
rir_data_path = '.'
!python get_openslr_rir.py --data_root {rir_data_path}
rir_manifest_path = os.path.join(rir_data_path, 'processed', 'rir.json')
!head -n 3 {rir_manifest_path}
```

But still get an error after downloading and extracting the file:

```
INFO:root:

Working on: slr28
INFO:root:Getting slr28
INFO:root:./slr28.zip does not exist. Downloading ...
INFO:root:Downloaded ./slr28.zip.
INFO:root:Extracting slr28
INFO:root:Processing slr28
Traceback (most recent call last):
  File "get_openslr_rir.py", line 139, in <module>
    main()
  File "get_openslr_rir.py", line 133, in main
    os.path.join(os.path.join(data_root, "processed", "rir.json")),
  File "get_openslr_rir.py", line 105, in __process_data
    with open(manifest_file, 'w') as man_f:
FileNotFoundError: [Errno 2] No such file or directory: './processed/rir.json'
head: cannot open './processed/rir.json' for reading: No such file or directory

```

Is this a known issue? Or i'm just bad af and i don't know what to do? I can't even look at how rir manifest are structured. And this is realy important for me to add RiRAug in my ASR pipeline.

Appreciate some help
Thanks in advance


@lodm94 The issue here is that the script get_openslr_rir.py only works with absolute paths. 
Please change the first line in the cell to:
`rir_data_path = os.getcwd()`
I will fix this in the main branch too. Thanks for letting us know of this issue.

Holy moly it finally works! Cheers man! Thank you
When fixing the main branch, consider that
`!python ../../scripts/get_openslr_rir.py --data_root {rir_data_path}`
returns error. I had to wget the script and change this path.
Key 'params' is not in struct
State: closed | Pull request: no
Labels: name: bug

I try to run the [offline colab notebook](https://colab.research.google.com/github/NVIDIA/NeMo/blob/r1.0.0b3/tutorials/asr/Offline_ASR.ipynb#scrollTo=8qgXHl78WneK).

Without changing anything I run cells and get error while running this line `labels = list(asr_model.cfg.decoder.params.vocabulary) + ['blank']:` 
Error:
![image](https://user-images.githubusercontent.com/76702393/104902439-c2a47b00-59a4-11eb-99c4-9e54e1e93b14.png)



You don't need the decoder.params.vocabulary anymore. Directly use decoder.vocabulary

Yes this solves the issue thanks
kaldi wav.scp and text to manifest support?
State: closed | Pull request: no
Labels: name: feature

Hi, guys. I am new to NeMo.
And I still reusing the data augment in Kaldi, since there have a lot data prepare script.
is it possible to read pipeline data in NeMo, I already wrote a script generate Manifest .json file from wav.scp and text file in kaldi stype dataset. But it only work for .wav not pipeline data.
And I found (https://github.com/NVIDIA/NeMo/issues/743)  KaldiFeatureDataLayer already removed. Do you guys have any plan for these?

Adding a `me too` comment here.  I'm actually interested in loading features archives computed by kaldi (like fbank).  This is a feature very useful we find in other toolkits like ESPnet for example.  Is there some documentation or some guidelines how one would implement that?

@jprobichaud I already fixed here
https://github.com/NVIDIA/NeMo/pull/1644

@qmpzzpmq Thanks for your reply.  This is useful for loading audio that can come from transformation, but that assumes that the entire audio will be one "segment".  The kaldi feature data layer allowed if I read properly to read in extracted features for each segments (one audio file could be split in multiple shorter segments.

It's already nice to have the pipe reading available.  Thanks for that!  I still wonder if reading audio + segment or raw features will be possible again in the future or if that was deem not useful.


Hi @jprobichaud and @qmpzzpmq Thanks for your requests and contributions.

Yes we think the external features (like fbank) is quite useful. Due to some maintenance reason, we don't have KaldiFeatureLayer now. But we are working on ExternalFeatureDataloader that reads the stored feature (in pickle) and we will provide the script to extract and store it to pickle files.  Hope this would give more flexibility. We might support reading from feature.ark as well later. 

>  extracted features for each segments (one audio file could be split in multiple shorter segments.

Like [feat1, feat2, feat3] corresponding to [utt1, utt2, utt3]? @jprobichaud  could you elaboration more so we can understand your idea clearly? 

@jprobichaud 
using "utils/data/extract_wav_segments_data_dir.sh"
Confusing GPT Terminology in NeMo Models tutorial
State: closed | Pull request: no

In the NeMo Models tutorial, in the refactored minGPT we have `GPTEmbedding`, `GPTTransformerEncoder`, `GPTDecoder`. I think `GPTTransformerEncoder` and `GPTDecoder` are confusing names both with respect to the original GPT architecture and the Transformer architecture. In the original Tranformer paper, namely the left stack of identical layers is called encoder and the right stack of identical layers is called decoder and that's why in general both Transformer encoder and decoder are used to describe two different and specific sub-architectures in the NLP community.
![](https://miro.medium.com/max/600/1*tkC_NqylGH4hv2rTLPOf5A.png). 

Based on the original Transformer architecture we have all the subsequent Transformer architecture descriptions using the encoder and decoder terminology. For example, authors of [BERT](https://arxiv.org/pdf/1810.04805.pdf) describe it as 
> BERT’s model architecture is a multi-layer **bidirectional Transformer encoder** based on the original implementation described in Vaswani et al. (2017).

Similarly, the original [GPT paper](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf) describes the architecture of GPT as a 
> **12-layer decoder-only transformer**

So there is no "encoder" in GPT's architecture and that's why to a beginner the tutorial's  `GPTTransformerEncoder` would come across as confusing. Especially because the work it is doing is that of the decoder as defined in the GPT paper. Similarly, the tutorial calls the final layer (LayerNorm + Linear) in the architecture `GPTDecoder`. Again, with the respect to the GPT paper this is a confusing name because the work this module is doing is not actually that of the GPT decoder. Including the GPT architecture from the original paper for reference: 
![image](https://user-images.githubusercontent.com/76540498/104793116-f50c6900-5755-11eb-8aa7-6e6d4c988f67.png)


While not consistent with the paper, this is a tutorial to explain how to refactor a monolithic concrete model into an abstract NeMo model - comprising of neural modules that represent the abstraction with neural types, and then instantiate them with actual concrete modules (which can be anything that fits the neural types) as a Nemo Model. 

The concept of a NeMo models components don't need to be 1:1 with a papers notation of terms - they are represented as "encoder" that takes certain neural type and returns some neural type, and a "decoder" that takes some neural type from the encoder and returns some other neural type. They don't refer to the encoder and decoder blocks of the gpt blocks only. 

See ASR models who also present encoder decoder architecture. 

The encoder neural module can be a cnn based QuartzNet or a self attention based transformer decoder block or a Conformer sandwich. 

Similarly, the decoder neural module can be a basic linear layer, a RNN based Autoregressive decoder, or even an RNNT Transducer based decoding scheme. 

For now, to be clear I will add a note there
Make onnx-graphsurgery an optional deferred import
State: closed | Pull request: yes

# Changelog
- onnx-graphsurgery is not available on all platforms, and breaks pip install (currently tested on darwin and some versions of ubuntu).
- Make it an optional import that performs runtime check instead.

Hmmm, I think the line that was breaking NeMo installation on my MacBook was in `setup.py`:
https://github.com/NVIDIA/NeMo/blob/45f60f03e5d15a4b6e7db145b5d731d73f3a891d/setup.py#L235

@borisfom ideas?

> Hmmm, I think the line that was breaking NeMo installation on my MacBook was in `setup.py`:
> https://github.com/NVIDIA/NeMo/blob/45f60f03e5d15a4b6e7db145b5d731d73f3a891d/setup.py#L235
> 
> @borisfom ideas?

@tkornuta-nvidia Doesn't seem like this PR breaks MacOS compatibility. It looks like it was broken before? This PR change reduces ```setup_requires=['pytest-runner', 'nvidia-pyindex'],``` to ```setup_requires=['pytest-runner']```

Awaiting @borisfom review before merge

Merged as main branch is broken without this, preventing other prs 

Sorry to be late for the game - I am on vacation right now ...
Too bad onnx-graphsurgeon is not available on all platforms - we would need to think how to handle those cases properly in general. It makes it rather difficult  to have seamless setup.py, without hairy scripts around. As my original change was primarily to enable a workaround for TRT issue that will hopefiully be fixed, it's probably fine in this case. 
The way it's left after this PR merged is not right though: 
1. constant_folding argument is defunct (always overridden). We should revert changes to it and only have onnx-graphsurgeon invocation under the GRAPGSURGEION_AVAILABLE conditional.
2. default setup.py would not install onnx-graphsurgeon on any platform. That's probably the best we can do at the moment and have developers who need to use this WAR, to install it manually - but we should think of general way of handling optional components to have best experience on all platforms. It sees that, besides WAR for TRT 'double' issue, onnx-graphsurgeon pass makes most of the graphs more compact (potentially faster, one needs to check), so it would probably be nice to keep it enabled for the platforms that have it, even when TRT issue is fixed.
Incorrect usage of nn.LSTM in NeMo Models Tutorial
State: closed | Pull request: no
Labels: name: bug

Under the Neural Types section, for Case 2 LSTM [ERROR CELL], the tutorial gives the wrong reason for the `TypeError: Number of output arguments provided (2) is not as expected (1)`. Specifically, it states:
> What exactly is going on here? Well, inside our `LSTMModule` class, we declare the output types to be a single NeuralType - an `EncodedRepresentation` of shape [B, T, C].

> But the output of an LSTM layer is a tuple of two state values - the hidden state `h` and the cell state `c`!

But actually from the nn.LSTM [docs](https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html) we see that ![image](https://user-images.githubusercontent.com/76540498/104661588-adb2a980-567d-11eb-9a18-6f845b6c92a0.png) it outputs a tuple `output, (h_n, c_n)` whose first element `output` contains the output features from the last layer of the LSTM for all timesteps. With `batch_first=True` this should be of the shape `B x T x C`. The second element of the tuple is again itself a tuple of the final hidden state (`h_n`) and final cell state (`c_n`). So the axes in `CorrectLSTMModule` is also wrong. Particularly, even with `batch_first=True` in the LSTM `h_n` and `c_n` are of shape `(num_layers * num_directions, batch, hidden_size)`. Since `h_n` and `c_n` are the final states there should be no Time (T) or seq_len dim in their shape. Here we are using unidirectional LSTM and representing batch with axis B and hidden_size with axis C so shape becomes `(num_layers, B, C)` not `(B, T, C)` as implemented in `CorrectLSTMModule`
![image](https://user-images.githubusercontent.com/76540498/104663013-e902a780-5680-11eb-8a9c-57f41d19d9a0.png).

Also wondering, is there an AxisKind to represent `num_layers` or `num_directions`? Couldn't find a suitable AxisKind in https://github.com/NVIDIA/NeMo/blob/main/nemo/core/neural_types/axes.py. Maybe "Any" should be used?


To give a more concrete example of the LSTM output. Let `(h_n, c_n)` be the tuple of final hidden and cell state and `y_t` be the outputs for all timestamps then

``` 
lstm = torch.nn.LSTM(1, 30, batch_first=True)
x3 = torch.rand(2, 5, 1)
# x3 shape is [batch_size x time stamps x dimension]
y_t, (h_n, c_n) = lstm(x3)
```
`y_t` is of shape ([2, 5, 30]) i.e `batch_size x time stamps x dimension`
`h_n` is of shape ([1, 2, 30]) i.e. `num_layers x batch_size x dimension`
`c_n` is of shape ([1, 2, 30]) i.e. `num_layers x batch_size x dimension`



This is a good find, I keep forgetting TF and PT have different semantics for their RNNs. 

NeMo currently ignores nested structures (h, c) (though we plan to add this), so that will be a todo for a subsequent patch. In the diagram above, "h" corresponds to "y_t" and "c" corresponds to both (h_n, c_n), so a bit of renaming will be necessary as well.

As to the axes, we cant detach the num_directions * num_layers since it flattens out into a single axis. The axiskind most appropriate to that would probably be Channel/Dimension rather than Any. Any is a bit too generic for an RNN state axis.
[Question] MatchboxNet and MarbleNet
State: closed | Pull request: no

Hi there,
@fayejf @titu1994 @borisgin

In the task of Voice Activity Detection, the following questions arise:
- After training the Voice Activity Detection model, How to deploy it for inferencing on arm based devices such as Raspberry Pi?
- How much memory is required to inference MatchboxNet
- How does MatchboxNet perform against MarbleNet

> After training the Voice Activity Detection model, How to deploy it for inferencing on arm based devices such as Raspberry Pi?

We dont have deployment scripts for Raspberry Pi. 

> How much memory is required to inference MatchboxNet

Depends on deployment framework, but it should be small considering the size of the model. 

> How does MatchboxNet perform against MarbleNet

They perform two different (but similar tasks) so they cant be compared directly. Matchboxnet is meant for speech command recognition - multiclass classification problem and Marblenet is meant for voice activity detection - binary classification problem. 

- For the Voice Activity Detection task, the deployment to ARM-based devices is vital and should be considered.
What difficulties need to be solved to be able to deploy the VAD model for inferencing on an Raspberry Pi?
- isn't Matchboxnet currently used for VAD?
Loss metric
State: closed | Pull request: yes

Add a PTL metric for averaging loss across processes in ddp mode.

Motivation.
If ddp is used, the validation loss for only one gpu is logged. This behavior impairs "Reduce on plateau" and "Early stopping" scheduling strategies. If many GPUs are used and the validation dataset is small, logged validation loss can differ from real validation loss a lot.

Overall, the PR looks good to merge. Please fix the tests and final review from @MaximumEntropy please.
[Question]
State: closed | Pull request: no

**How to extract embeddings from a specific portion of audio?**

I have audio files which contain speech from two different individuals, I want to use the pre-trained SpeakerNet_verification model to extract speaker embeddings of each audio file in a sliding window fashion (an embedding for every 1 second with 0.1 seconds overlap). Since the manifest file only accepts the absolute path to the audio is there a way to specify which chunk of audio the model should embed? 

What would be the easiest way to implement this? Alternatively, if I wish to add such a functionality myself, where can I start?



It is implemented for a WIP project now, It could be done using collate func to slice the batches and combine back them by backtracking the values. 

For reference see here:
https://github.com/NVIDIA/NeMo/blob/df360cd3c856c645d23505305ee8b689636b2dd1/nemo/collections/asr/data/audio_to_label.py#L161

And we capture outputs here:
https://github.com/NVIDIA/NeMo/blob/df360cd3c856c645d23505305ee8b689636b2dd1/nemo/collections/asr/models/label_models.py#L407
Update to Lightning 1.1.4
State: closed | Pull request: yes


This pull request **introduces 1 alert** when merging 73e9a8a210002ac77a8166a176651abbbea1a390 into d8150b41bbba0c2ee2e291b309651cef2ed248bf - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-81dddf30b77cce1659e91a2d505dce40fca10a67)

**new alerts:**

* 1 for Unused import
Add enemo export for Quartznet
State: closed | Pull request: yes

For direct usage within ServiceMaker (Nemo to Jarvis) without TLT. (Experimental)

This pull request **introduces 2 alerts** when merging cb3abd555688d923e20c31164a600388aa9edd37 into c6b90b3dd1321b3e95df8f91b89f55326d52a81c - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-ee19b48f4fc82cf01b0ea9b682c3c6e76bef760b)

**new alerts:**

* 1 for Unused local variable
* 1 for Unused import

@purn3ndu, thanks for the PR. Please fix code formatting, to unblock the CI. Here is how https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md

This pull request **introduces 2 alerts** when merging dbcb4a90037c1aecc465ecfcf07e8596e105eaf1 into f58ab531376a854e2e94b0fedc05705bc3711086 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e3be39f303d7b0988cf60b491a1b4eef7881eded)

**new alerts:**

* 1 for Unused local variable
* 1 for Unused import

This pull request **introduces 1 alert** when merging 874ba9580f22ebc42ff0ccbe8aace718192c30b6 into 4f46bd84d162da80a310c69f72980de935c1e99a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-ea5def0ed5805635111f6760041ee8e26e6132a3)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging b00ab452dd7290884149bf0703306dbb5578396a into 84f69c582a76b66b208dc40ef18499b3448f2142 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-31d742aa9480834f1a27c4c6f3b218ad737f8e90)

**new alerts:**

* 1 for Unused import
support other audio format
State: closed | Pull request: yes

Might solve #1549 

Currently I let pydub handle file that is not .wav format because there is something I'm not understand with current implementation.
1. "samples.transpose()" before init with cls
2. User also need to install [ffmpeg](https://github.com/jiaaro/pydub#getting-ffmpeg-set-up) to be able to load non wav file format, I don't know where to put the warning or write a script when user pip install
3. I'm not sure if NeMo dataloader will cache the loaded samples because it might convert using ffmpeg everytime it try to load non wav file.

So far I only tested inference with mp3 audio from Common Voice with model trained on wav file format, there is no problem with the prediction result.

This MR, make training so much slower. Pydub takes too much of CPU to convert media files. Need to make sure pydub is only used for unsupported media formats, right now we using FLAC, and it goes through pydub, but libsnd supports flac.

![image](https://user-images.githubusercontent.com/3962002/104477831-9f5b8380-55d2-11eb-83b2-1937140d843c.png)

Part in the middle is with MR, and parts before and after are without it. Training done on DGX A100.

Ok that's a problem. I'll go ahead and change the logic a bit

Hi @blisc,
I create [hotfix](https://github.com/NVIDIA/NeMo/pull/1630) for this bug 

@yoks, can you try #1630 and see if performance returns?

Will test as soon as we will have free video card available on DGX (tomorrow most likely)

Yea looks like it works as before (maybe a little bit slower, but can be related to task/node load).

Is there a reason this isn't using torchaudio? It supports multiple formats in a single library. 

I don’t have experience using torchaudio, after take a look at their repo I think torchaudio will be better for this, support multiple audio format including kaldi format. Maybe someone know how to use torchaudio better can contribute to this

> I don’t have experience using torchaudio, after take a look at their repo I think torchaudio will be better for this, support multiple audio format including kaldi format. Maybe someone know how to use torchaudio better can contribute to this

Hi

torchaudio maintainer here. I commented in #1549 but the current release of torchaudio does not support file-like object. I am adding that support right now and if will be released in the next release. (Early March). 
Is there any other requirements that NeMo has?

Hmmm, I'm not related to NeMo actually, I also don't know the requirements, I think @blisc can answer this question.

Theres a [very detailed and helpful response](https://github.com/NVIDIA/NeMo/issues/1549#issuecomment-767058227) from @titu1994 on why torchaudio isn't used as a dependency.
Multiple optimizers drops logs
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

With multiple optimizers in the `training_step`, only the final optimizer's logs get printed/logged on the progress bar and in tfevents file. No logged values from the previous optimizers appear on the progress bar or in the tfevents.

**Steps/Code to reproduce bug**

Running examples/tts/melgan.py will show the issue. Only `loss_generator` is logged,  despite in the first optimizer's code `self.log("loss_discriminator", sum_loss_dis, prog_bar=True, sync_dist=True)`


**Expected behavior**

All values logged in the earlier optimizers should persist and be logged.

**Environment overview (please complete the following information)**

 - Environment location: Bare-metal
 - Method of NeMo install: source, current head (1.0.0b4)

**Environment details**

- PyTorch version 1.7
- Python version 3.7


I updated pytorch lightning and that seems to have fixed it. Must have been a specific combination of PTL + NeMo that didn't work. Closing,
[Question] Porting speaker recognition from 0.11.0 to 1.0.0b
State: closed | Pull request: no

Hi, thanks for the great library!


I am currently porting a 0.11 speaker recognition checkpoint to 1.0.0b, I ran into some difficulties reproducing the outputs of the 0.11 checkpoint in 1.0.0b. In the 0.11 model, the decoder is `JasperDecoderForSpkrClass`, to my understanding, the corresponding class in 1.0.0b is `SpeakerDecoder`.

 In [JasperDecoderForSpkrClass.forward](https://github.com/NVIDIA/NeMo/blob/ac09a4cc2ae3fdfedaf6f6832bd2dfca94de19ec/nemo/collections/asr/jasper.py#L459), the returned embedding is the first transformation of the pooled encoder output, in the current release, [SpeakerDecoder.forward](https://github.com/NVIDIA/NeMo/blob/ede204df0d5adc54714981511b83cf6d2fa905b2/nemo/collections/asr/modules/conv_asr.py#L421) returns the last transformation of the pool. Is there a reason for this change?



Hi @TobiasPuetzKenbun ,

Did you check out [this script ](https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_checkpoint_port.py)to convert your 0.11 checkpoints to 1.0? 

And to your second question yes we did that on purpose, initially when we started out speaker recognition it was first trained end to end using cross entropy loss and hence last but one linear transformed output gave good results, but when we moved loss to angular softmax, the final transformed linear output showed better results on EER (though slightly), not a big change but decided to do it. 

Let me know if you have any more questions : ) 

Thank you for the quick answer!

> Did you check out this script to convert your 0.11 checkpoints to 1.0?

Yes, I've been using that script. It wasn't directly compatible since some module names were changed from 0.11 to 1.0, the mid1 & mid2 layers are now wrapped in a `Sequential`, so it was necessary to rename these keys in the decoder state dict to `emb_layers.0` / `emb_layers.1` before the `asr_checkpoint_port` script would work.

> And to your second question yes we did that on purpose, initially when we started out speaker recognition it was first trained end to end using cross entropy loss and hence last but one linear transformed output gave good results, but when we moved loss to angular softmax, the final transformed linear output showed better results on EER (though slightly), not a big change but decided to do it.

Thanks for the explanation, so for backward compatibility, I'll add my own legacy `SpeakerDecoder`.
NMT combine embedding plus encoder/decoder into one neural module.
State: closed | Pull request: yes

This PR create encoder and decoder neural modules for `TransformerEmbedding` and `TransfomerEncoder` / `TransformerDecoder`.

This is making it so that the neural types will be the same across NeMo Transformer, HF BERT, and Megatron BERT encoders.

The idea is to have `EncoderModule` which accepts `input_ids`, ` encoder_mask`, and `num_tokentypes` and outputs `last_hidden_states`.

Then  `DecoderModule` will accept `input_ids`, `decoder_mask`, `encoder_mask` and `encoder_states` and outputs `last_hidden_states`.

This PR also cleans up the relevant configs as the configuration needed to be updated to accept the new model structure.

This pull request **introduces 9 alerts** when merging 5a92be0f55a1185bf37c846b9367ab1f897fda2c into 7cb30843427fdf77a55a0cdd67cbbde3c067c850 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-07bdc86a1172f21038bff622c9c4381f3d7e1e83)

**new alerts:**

* 9 for Unused import

This pull request **introduces 9 alerts** when merging 783901cac81afff9c79d2c5596bdb734ff48a107 into 7cb30843427fdf77a55a0cdd67cbbde3c067c850 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-02c819094dccfeaa3330e727ebbb8dff941d4a3d)

**new alerts:**

* 9 for Unused import

This pull request **introduces 9 alerts** when merging 124c3e990ab085752e6220b638a7d274246b11ec into 36c4a8fcca4e6e74360886fd78818055114a15ec - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5bfa7d35da4a8db9cdd13767f81688f1bdbb98f4)

**new alerts:**

* 9 for Unused import

This pull request **introduces 9 alerts** when merging 29f44bf0427545bb81a686035d7e40f858d68cbf into 36c4a8fcca4e6e74360886fd78818055114a15ec - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-9c0738081d80d722cfaf578afc5b3f7c3dcc669f)

**new alerts:**

* 9 for Unused import

This pull request **introduces 9 alerts** when merging add99cda5189594483a95950dc984fe89621d4d0 into 36c4a8fcca4e6e74360886fd78818055114a15ec - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-481382bf66d61d91fa345a3c996222082a1feeee)

**new alerts:**

* 9 for Unused import

This pull request **introduces 8 alerts** when merging 78a3bcf9bc0da9e3c69c96192b8d274715f6bd4a into 36c4a8fcca4e6e74360886fd78818055114a15ec - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-f991a030d4546452dae3f7e6370e14f6fe6d723e)

**new alerts:**

* 8 for Unused import
Workaround for TRT lack of support for DOUBLE
State: closed | Pull request: yes

Background: 
https://nvbugswb.nvidia.com/NvBugs5/SWBug.aspx?bugid=3221866&cmtNo=


@borisfom please fix formatting for CI to start:
```
python setup.py style --fix
```

This pull request **introduces 1 alert** when merging f261f1a3a5ceaa203dfe605be371dcdc20e4d95f into 4f46bd84d162da80a310c69f72980de935c1e99a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-abaf66c15dc6db2b910655263f4b9046ab0386e2)

**new alerts:**

* 1 for Syntax error

oh, I see @okuchaiev  merged this... :]
For Megatron models, local vocab files are not supported.
State: closed | Pull request: no

https://github.com/NVIDIA/NeMo/blob/51092ef7a9810ab096148cdf2d44ddfe6c72d028/nemo/collections/nlp/modules/common/megatron/megatron_utils.py#L135

It seems from the above code snippet that for Megatron models, it always tries to download the vocab files from the cloud. However, currently for bio-Megatron models the [link](https://api.ngc.nvidia.com/v2/models/nvidia/biomegatron345mcased/versions/0/files/vocab.txt) doesn't work. This has been already raised as a separate [issue](https://github.com/NVIDIA/NeMo/issues/1582). 

It should accept the vocab file passed to `config.model.tokenizer.vocab_file` and should only check for vocab file online, if the user doesn't provide a vocab file.


ERROR while running Token Classification - BioMegatron notebook

<img width="1080" alt="Screen Shot 2021-01-05 at 8 24 51 PM" src="https://user-images.githubusercontent.com/7240751/103717828-14580900-4f94-11eb-925b-25ce3650b002.png">


I was able to run 
```
vocab = get_megatron_vocab_file('biomegatron-bert-345m-uncased')
```
without any issues.

I was wondering if you could help clarify what you're trying to do.

Are you looking to use 'biomegatron-bert-345m-uncased', or are you pretraining a megatron language model on your own vocab and data? Or are you wanting to use 'biomegatron-bert-345m-uncased' configuration but pretrained on your own vocab and data?

Also, there is a known bug with the notebook and config interpolation which we are working on: https://github.com/NVIDIA/NeMo/issues/1582

I am just trying to run the notebook to understand NeMo framework better.

Initially, I tried to just pass "biomegatron-bert-345m-uncased" to `config.model.language_model.pretrained_model_name = PRETRAINED_BERT_MODEL` and it didn't work as mentioned here #1582 .

So I downloaded the model, config, and vocab file for Megatron models from the links given in this [notebook](https://colab.research.google.com/github/NVIDIA/NeMo/blob/r1.0.0b3/tutorials/nlp/01_Pretrained_Language_Models_for_Downstream_Tasks.ipynb#scrollTo=mla-BZexMM2u). However, even if I passed the vocab file manually, the line

`vocab = get_megatron_vocab_file('biomegatron-bert-345m-uncased') 
`

doesn't check for it and always tries to download it from the cloud given the pre-trained model name. I hope this makes it clear.

This should be fixed here: https://github.com/NVIDIA/NeMo/pull/1803
Please feel free to reopen if there are still any problems.
Add entire dataset caching via a preproc script, bpe dropout and deto…
State: closed | Pull request: yes

Adding the following:

1. A way to cache the entire translation dataset object rather than just the IDs, to avoid recomputing the bucketing and padding steps. 
2. BPE dropout to YTTM.
3. Sacremoses for detokenization.

This pull request **introduces 2 alerts** when merging 9ccc0dbd61d9c6b0e034e406f90b3a5795fac574 into 8ed6b966a843286c715484b0fadc266253f2f272 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-162bbcfbe9077b4f7d1d3dca2ab7d0f7aec26a49)

**new alerts:**

* 2 for Unused import

This pull request **introduces 2 alerts** when merging 4eefe78e7a5b4f9282ca7913646d65005bf595f9 into 8ed6b966a843286c715484b0fadc266253f2f272 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-894362736755457c26db38e4564709e8e5d65f2f)

**new alerts:**

* 2 for Unused import
Patch ASR to support < PT 1.7
State: closed | Pull request: yes

# Changelog
- Wrap torch.stft to switch signature between  PT< 1.7 and >= 1.7

Can we add the following?
1) A reminder to remove this patch once we require pytorch >= 1.7
2) A unittest that fails or gives a warning if a PR contains "torch.stft" while we still allow pytorch < 1.7
[Question] NeMo public roadmap
State: closed | Pull request: no

Hello, awesome people!

First of all, thank you very much for the job that you do with this toolkit.

Secondly, do you have a public roadmap for NeMo toolkit? I think will be useful for everyone that uses NeMo to know when new version/features will come.

In my case, I'm interested to know when will be available support for words timestamps and confidence with language model.

Regards,
Adrian

Hi @adiIspas,

This is a good question. First, let me mention that:
* NeMo is being developed in the open. E.g. what you see in the "main" branch on GH is what our team members use in their daily work. In this sense, things which are now WIP in the main branch will be appearing as features in our next releases.
* We accept external contributions, so if you'd like to send a PR to us, please have a look here: https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md 

Here is what we can share right now:

* we plan to release "1.0.0rc" version soon (this winter) which will have some configuration improvements, ASR models in few more languages (using Mozilla's Common Voice), machine translation models and other new features and bugfixes.
* in about a month after that we'll release "1.0.0" which shouldn't differ much from "1.0.0rc" in terms of features and usage but should provide more stability and overall performance.

We'll think about sharing a full roadmap (or a subset of it) publicly.