renwei2024/nemo-github-issues huggingface dataset:
Fine-tune QuartzNet15x5NR-En to another language [Question]
State: closed | Pull request: no

Hello!
Thank you for this opensource product!
I want to try fine-tune model **QuartzNet15x5NR-En** to Russian language. This article [Transfer learning from English to Russian](https://developer.nvidia.com/blog/jump-start-training-for-speech-recognition-models-with-nemo/) is good practice for my experiment, but i have some questions.

1. In `config.yaml` in this model `RIR_and_Noise` i find this sample for augmentation:
  > rir_manifest_path: /data/datasets/freesound_20s/rir_noises_tarred/rir_tarred/tarred/tarred_audio_manifest.json
  > rir_tar_filepaths: /data/datasets/freesound_20s/rir_noises_tarred/rir_tarred/tarred/audio_{0..63}.tar
  
   How i download this files and create this manifest?
   I find this script - `download_resample_freesound.sh` - what parameters should be specified for download?

2. How many `hrs` should dataset have? What type of audio data should it contain? (phone calls, audio book...)
3. What `learning rate` to use? What parametrs set to `SpecAug`?

Thank you!

@okuchaiev Maybe there is some information on this issue? Thanks!

There are several notebooks for transfer learning char and subword models in Nemo now, please have a look and reopen the issue if they are insufficient.
[Load token classification model from checkpoint get No such file or directory: 'megatron-bert-cased_encoder_config.json']
State: closed | Pull request: no

**Describe your question**

After training token classification model, several checkpoints .ckpt were generated along with .nemo.  Load .nemo is ok. But I tried to load checkpoints via:

```
nemo_nlp.models.TokenClassificationModel.load_from_checkpoint(my_chk_path)
```

but give an Error:
```
No such file or directory: 'megatron-bert-cased_encoder_config.json'
```

I unpacked the .nemo and found 5 files:
```
label_ids.csv
megatron-bert-cased_encoder_config.json
model_config.yaml
model_weights.ckpt
vocab.txt
```

I did not know where did the megatron-bert-cased_encoder_config.json come from?


**Environment overview (please complete the following information)**

using nemo docker from ngc and at BRANCH = 'r1.0.0rc1'


Hi, restoring NLP models can be complicated as there are potentially many different artifacts needed for restore. This is why we've created the .nemo file and .restore_from. We mainly use the PyTorch Lightning Checkpoints to auto-resume from training. They can be used in other workflows outside of NeMo, but then the user would need to add custom implementation for restore.

And to answer your question "I did not know where did the megatron-bert-cased_encoder_config.json come from?"
Megatron models need certain artifacts to be instantiated, mainly the config and vocab. We generate both of these and automatically save them in the .nemo file when using Megatron models in NeMo NLP.

Also, we will be updating the NeMo NLP documentation with this information in the near future. Sorry for the current inconvenience.
ASR with Speaker Diarization noteboook
State: closed | Pull request: yes

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>
* Adds ASR with Speaker Diarization tutorial notebook that outputs transcripts with speaker label along with time stamps
Sample output:
`speaker_0 [0.40 - 2.62 sec] eleven twenty seven fifty seven`
`speaker_1 [3.20 - 5.02 sec] october twenty fourth nineteen seventy`
Patch SPE tokenizer not being available in older ASR Checkpoints
State: closed | Pull request: yes

# Bugfix
- Older checkpoints might not have the `tokenizer.vocab` file when using SentencePiece tokenizers. This will cause model restoration to fail.
- Wrap in a try catch that catches ModelPT raised FileNotFoundError for this particular case to enable backward compatibility

Signed-off-by: smajumdar <titu1994@gmail.com>
fix to avoid 'DATA_DIR not found'
State: closed | Pull request: yes

Signed-off-by: Hoo Chang Shin <hshin@nvidia.com>

Thanks @ekmb , just did :) 

yes please, this looks like bugfix for r1.0.0rc1

Thanks @ekmb and @okuchaiev - rebasing to r1.0.0rc1 resulted in many file changes, but notebook hasn't changed at all.

This pull request **introduces 4 alerts** when merging f6e7fc182f103e0b58c47ccb39645995ac178324 into bc38019ffa144dcf0bfc735a9838bb0e6fba71ab - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2d6425e234837de89ff7012c4a2fb38315c3ad09)

**new alerts:**

* 4 for Unused import
Fix enja decoding (round 2)
State: closed | Pull request: yes

after trying hard to make yttm and spm play nicely together, i am giving up on the latter. the en<>ja pipeline is now moses -> yttm. from what i can see, this fixes tokenization & detokenization issues we were seeing (eg, `&apos;` was being split apart incorrectly during tokenization, preventing it from being detokenized correctly).

we also turn off moses normalization for japanese because it wasn't handling punctuation correctly
Bug in the Text_Classification_Sentiment_Analysis notebook when using wietsedv/bert-base-dutch-cased config
State: closed | Pull request: no

Hello,

When im trying to run the sentiment notebook in the main branch I can perfectly use the bert-base-uncased and bert-base-multilingual-uncased models and finetune on our data. However, when we run the wietsedv/bert-base-dutch-cased model we get the following error: Index out of range in self. To me it seems like there is something going wrong in the embedding layers. When we ran this model in NeMo version 0.11 it worked fine.

![image](https://user-images.githubusercontent.com/19664770/109788048-27dbc500-7c0f-11eb-956a-5adb849d7587.png)

Is there anything in the configs that changed?
bug removed : onnx file was not getting added to the tarfile (.enemo) 
State: closed | Pull request: yes

**What was wrong**  : renaming and adding onnx to tar  was not working
**How solved**  :  make a temp copy of the file rename and add to tar(.enemo). cleanup extra file
**Nvidia internal bug** : 200706409

This pull request **fixes 1 alert** when merging aebab0e12600a6fa4be346c88112a88716f7dc42 into 79bef5681e9023287c8a839b3eb5d0bad25dd3eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-427bb1d02f76b9b8378adc5f5b98f401e4e73e54)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging 5c91f70297421bb1ded56458e495982d792fe642 into 79bef5681e9023287c8a839b3eb5d0bad25dd3eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-f81a394a8cc0e26c328d44e7ef1ff6720d0d5c78)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging 71382baea3f1b4ad5dca5c110ce2e9a2e8c640eb into 91774a74a38335d6b0090174861fecde54c764ff - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-26525b782c9a8caa3f1ebac9212adc62a61c6849)

**fixed alerts:**

* 1 for Unused import

@purn3ndu Please have a look

This pull request **fixes 1 alert** when merging 789e805879470681bea97140154030977073e61d into 91774a74a38335d6b0090174861fecde54c764ff - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-0bbe084f8116a5f9e7f31fee1e1b69ca85e9b896)

**fixed alerts:**

* 1 for Unused import

Approved, pending CI.

This pull request **fixes 1 alert** when merging 57c696a633a6d8d1907655ad26c8ab5ae7579cec into 91774a74a38335d6b0090174861fecde54c764ff - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-566dec3fc8194ef92f30111e8d9cafdc42767199)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging 8b58cb55fd62896c8d6536509b3a5a1bb0f90560 into 37f65336d0fe8fb0360258da98756384f2f11196 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-3946a2ce11d9dda043a870035889dafb0688c2a2)

**fixed alerts:**

* 1 for Unused import
Fixing NMT DDP override that no longer works with PTL 1.2 
State: closed | Pull request: yes

Also, need to update/add tests that would have caught this problem.

thanks @ericharper !

This pull request **introduces 3 alerts** when merging 9dcc1f64b1c17437a1403e44e44ef03cfce3e122 into 79bef5681e9023287c8a839b3eb5d0bad25dd3eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-70210a020f64159d504b1a88930512edba534cf8)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging dbfa7b3854023ef0910227de56ce26457f036edb into 79bef5681e9023287c8a839b3eb5d0bad25dd3eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-6cca4e6f4f304898526462dbd12742b412bde2d5)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging 0f411448a03fc5d4515edaf1af3b2f282adef0b8 into 79bef5681e9023287c8a839b3eb5d0bad25dd3eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1d886c4d0c8f69a070dbfb8071a77b1c22dfc067)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging 9631de965b199109a65c7dc491ca0ddbce21a40c into 79bef5681e9023287c8a839b3eb5d0bad25dd3eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-de2cf13324bb7e30b1e222a508c63c509f1104bc)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging 182eb34c53839e8f01983b14cdbaec77bb53c4e1 into 79bef5681e9023287c8a839b3eb5d0bad25dd3eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-61260e6612c74a9efc3f854317ef448854a8be1c)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging 86bc8dea2aa33a4025b62284105dfc2b74b44faa into 79bef5681e9023287c8a839b3eb5d0bad25dd3eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1571b280ef8b396e3c8be708ab133621c586a039)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging 8644e66856d98be42e51d2cd70dc50593b464037 into 79bef5681e9023287c8a839b3eb5d0bad25dd3eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-34316b2eaedd7c8bf182799535c6745c7ae97d0f)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging c0099e9d464493ad3ccdd68aeb4fd13f8c163426 into 79bef5681e9023287c8a839b3eb5d0bad25dd3eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d3bcf405a97cc818651b03e0a12bd33dc2d6901c)

**new alerts:**

* 3 for Unused import

This pull request **introduces 3 alerts** when merging 6c488a4ade1d451ff513ab046d2cc53759c5dae2 into 79bef5681e9023287c8a839b3eb5d0bad25dd3eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-39a83a00139f45b4f32a32c18c6d5ee3c6bd0a5e)

**new alerts:**

* 3 for Unused import
PTL 1.2.0 upgrade breaks draco training
State: closed | Pull request: no
Labels: name: bug

traceback error on draco:
```
Traceback (most recent call last):
  File "/home/dcg-adlr-mchrzanowski-source/NeMo/examples/nlp/machine_translation/enc_dec_nmt.py", line 127, in main
    trainer.fit(mt_model)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 471, in fit
    self.call_setup_hook(model)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1072, in call_setup_hook
    model.setup(stage_name)
  File "/home/dcg-adlr-mchrzanowski-source/NeMo/nemo/collections/nlp/models/machine_translation/mt_enc_dec_model.py", line 453, in setup
    self._trainer.accelerator_backend.ddp_plugin.configure_ddp = self.configure_ddp
AttributeError: 'GPUAccelerator' object has no attribute 'ddp_plugin'
```

reverting the changes from [PR 1774](https://github.com/NVIDIA/NeMo/pull/1774) fixes things.

This should be fixed on main now: https://github.com/NVIDIA/NeMo/pull/1829

Please let use know if there are still issues.

Closing, reopen if needed
[Question] When is the latest beta version going to be on PyPI?
State: closed | Pull request: no

Hello,

I would like to experiment with the latest version of NeMo, v1.0.0b4, however due to some firewall issues at my workplace I can only install through PyPI and not from Github.

Do you have any plan to release that version on PyPI in the near future?

Thanks!

You can use pip install --upgrade nemo_toolkit[all] to get the latest version now.
Tried to restore a nemo model after 3 months, but can't do it just like I used to.
State: closed | Pull request: no
Labels: name: bug

I trained **EncDecCTCModel** model with Mozilla Voice Project dataset like 3 months ago and save it by `.save_to` and I used to restore it by `nemo.collections.asr.models.EncDecCTCModel.restore_from(restore_path=_path_to_nemo_model_)`. I didn't have any problem at that moment and I stopped working on that project. 

Today I tried to restore just like I did it before and got an error that says: `AttributeError: can't set attribute`.  I followed the traceback and found a comment which says  `# models are handled differently for now` that makes me thinks something has changed and probably I need to restore it in other way. If that's correct, how should I restore the model at the moment ? Do I need to retrain a new model because something has change about the training and the way which a model is stored ? I would appreciate any answer or some advice.

Brad,

Thanks in advance.

```
- Environment location:  Google Colab
- Method of NeMo install: !python -m pip install git+https://github.com/NVIDIA/NeMo.git@$v1.0.0b2#egg=nemo_toolkit[asr]
```


can you try ``pip install nemo_toolkit['all']==1.0.0rc1`` ?

I just tried it and it works. Thank you very much, I was waiting literally like 2 months for this helpful answer. Can you please explain what's the difference between `pip install nemo_toolkit['all']==1.0.0rc1`, `!pip install nemo_toolkit[all]==1.0.0b3` and `!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[asr]` PLEASE ?
Fix enja decoding
State: closed | Pull request: yes

the current en<>ja preprocessing pipeline is being shifted to using integers for intermediate steps. this will avoid tokenization issues between yttm, sentencepiece, and sacremoses due to parsing incomptabilities. this PR changes the detokenization to deal with integer inputs and output tokens only at the end.

@mchrzanowski can you please send these changes to ``batch_translate`` branch?

Will do at some point soon. don’t let me block you for now and merge away;
I’ll fix things in a subsequent pr later

On Mon, Mar 1, 2021 at 16:53 Oleksii Kuchaiev <notifications@github.com>
wrote:

> @mchrzanowski <https://github.com/mchrzanowski> can you please send these
> changes to batch_translate branch?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/NVIDIA/NeMo/pull/1820#issuecomment-788461636>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKER5ZWEGVZUMZOCBII5W3TBQZJHANCNFSM4YM72J6A>
> .
>


i think this PR hopefully wont have to be merged. i think the path of least resistance for en<>ja will be to have the preprocessing pipeline produce integer ids as opposed to tokens. that way, there's no ambiguity between sequences of chars or tokens. detokenization will i think at that point just work with the default pathway and we wont have to special case en<>ja like this draft does.
[Question] how to improve QuartzNet for production
State: closed | Pull request: no

**Describe your question**
seeking guidelines from the community for improving QuartzNet for production:

1. different accents, for example, US/UK/AU English
2. filler words, for example, um; ah
3. domain words, for example, company names, finance terms




Please look into Nvidia Jarvis - https://developer.nvidia.com/nvidia-jarvis
Update EncDecClassificationDatasetConfig Dataclass
State: closed | Pull request: yes


This pull request **introduces 4 alerts** when merging f0e045e8eee83caf8d2db6464752d36f76871aa2 into 0f9a772feea56397479d06354ee4fcb9db8c6b04 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-23db6c861086ff7b5c8149283645de15e29f3d75)

**new alerts:**

* 4 for Unused import
Handle class balancing in a way to support TLT.
State: closed | Pull request: yes


This pull request **fixes 3 alerts** when merging 2817d5e5b74e67436fcec1ff3adef8f9b138c265 into efaab9e67346d633e3efd49c44a5c2b83661d1fc - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-f1f11b89c657df4cd1e55fc67022c52b4bf50e66)

**fixed alerts:**

* 2 for Unused local variable
* 1 for Unused import

This pull request **fixes 3 alerts** when merging fc7a45b0719139f60f0a62510c1f19b266940522 into 4d6cad6a4e59e9d10fcc82984c1ddb3a1f40067f - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-64d97a7b249c433648647052ef331e51826914f1)

**fixed alerts:**

* 2 for Unused local variable
* 1 for Unused import

> 
> 
> Please rebase to latest main - export() and attach_onnx_to_onnx should disappear.

Dropped them.

This pull request **fixes 2 alerts** when merging 2c32926718d5e73878a294594a449e316b6ed9bf into f4ffb4300d6ef1227e0a2cec9c692a38606be299 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a65e4ee17c4a44d729f0bcb33450c2768f21faec)

**fixed alerts:**

* 2 for Unused local variable

This pull request **fixes 2 alerts** when merging 270225370b979757b56fc83aed66a5ec611e5ba7 into 4b74402feeeb672f8982bb9fa51861f6c1d5f94b - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-4cb7bb67a2c3b70709410a3d4600a565f7756d97)

**fixed alerts:**

* 2 for Unused local variable

This pull request **introduces 4 alerts** and **fixes 3** when merging e960719173952bde00afb73f501fdf59ea89d049 into 89a8ea037a121001e105e95b49d71906a8de29aa - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-dff537058ff2f9f5e2402227e7a70b7901933ea1)

**new alerts:**

* 4 for Unused import

**fixed alerts:**

* 2 for Unused local variable
* 1 for Unused import

This pull request **fixes 2 alerts** when merging e960719173952bde00afb73f501fdf59ea89d049 into e816905e286df545ce344e865ea6cf8e4b0c85ca - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-00578344bfd1f5b5828305d31be724d2517c5881)

**fixed alerts:**

* 2 for Unused local variable

This pull request **fixes 2 alerts** when merging d975797c7625d62d8a94c808c3107c7d58cec3e7 into 89a8ea037a121001e105e95b49d71906a8de29aa - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-75901b1980139ecbd71de6d9fa50d2aa4d594d5b)

**fixed alerts:**

* 2 for Unused local variable
Applying PR 1794 to  r1.0.0rc1
State: closed | Pull request: yes

PR https://github.com/NVIDIA/NeMo/pull/1794 is important for nemo saving but was applied to the main branch only; for most of us, the working branch is r1.0.0rc1. This PR applies the same change to this branch.

Signed-off-by: mchrzanowski mike.chrzanowski0@gmail.com
[Question] FileNotFoundError in Relation Extraction w/ BioMegatron
State: closed | Pull request: no

Executing the example notebook 'Relation_Extraction-BioMegatron.ipynb' leads to an error

> FileNotFoundError: Could not find tokenizer.vocab_file

after running cell `model = nemo_nlp.models.TextClassificationModel(cfg=config.model, trainer=trainer)`. This happens both when I run the notebook on colab or on my local setup. 

How can I fix this error?

Thanks!

Hi there is a branch change, I had the same issue. 

changeing the variable at the start of the notebook from
BRANCH = 'main'
to
BRANCH = 'r1.0.0rc1'
Add basic grpc MT server
State: closed | Pull request: yes

Add readme, server updates

Signed-off-by: Ryan Leary <rleary@nvidia.com>

This pull request **introduces 5 alerts** when merging 619df560e621887e9e39036327916f33ed1f2884 into efaab9e67346d633e3efd49c44a5c2b83661d1fc - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5d117bfdd38ba6fec4b6fc1cdf243cb5d2c7c0fd)

**new alerts:**

* 5 for Unused import

This pull request **introduces 5 alerts** when merging f044219620649fd7ab955162679af4ae186c6f9e into 4d6cad6a4e59e9d10fcc82984c1ddb3a1f40067f - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b8718369403138f57b007315c303021993359be2)

**new alerts:**

* 5 for Unused import

This pull request **introduces 5 alerts** when merging 79d9ee7515229a969ccc82e4332279254bd66454 into 91774a74a38335d6b0090174861fecde54c764ff - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-4f2048c187405a163f5a4ec692ab2d47163efd3f)

**new alerts:**

* 5 for Unused import

This pull request **introduces 5 alerts** when merging 645d84e8c12c338ff5f0d93957f2f65229aeccd8 into 7ecf04ffd7ee05ad557a7d1aab5b7c6218e256ba - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-69c60a8069fcba6422b44ca370300ddddee693b2)

**new alerts:**

* 5 for Unused import

This pull request **introduces 5 alerts** when merging 40640081543c22b2980b0e5a762da467f53b834f into 52c20a17d6b2ccc761f284663ab629b4b5502747 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5dc138d588dac762b8c829858a0efd736924c94d)

**new alerts:**

* 5 for Unused import

This pull request **introduces 6 alerts** when merging 6b474ed61ba0d7d05669e88dc788d9e938407741 into 241855cc2a41af737b01565ab472f0ef84f74584 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-99a06654b2a178850e483b666af1d8ea0b8bc9e0)

**new alerts:**

* 4 for Unused import
* 2 for Modification of parameter with default

This pull request **introduces 2 alerts** when merging 323d938a4aa9f38665ef8ccd9eb9cc1c0b791c79 into 241855cc2a41af737b01565ab472f0ef84f74584 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a38f6e01cdee5910e2aa9d628ba0bd3b9cbf2ad3)

**new alerts:**

* 2 for Modification of parameter with default

This pull request **introduces 2 alerts** when merging af51a3d11b8468ad185563d81e8d7f64b2327c87 into 241855cc2a41af737b01565ab472f0ef84f74584 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b5f66ad9ba7a3b5f9b7d99baed89d06af35d77a2)

**new alerts:**

* 2 for Modification of parameter with default

This pull request **introduces 2 alerts** when merging e3f7898d85fa1a78fd07989d15c74a3d015050ce into 7e6197d33da24465fe89a3e2bd9d2a35395643d2 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-30324401431086c680e63b3a6391e0e3d52c96d3)

**new alerts:**

* 2 for Modification of parameter with default
[WIP] Refactoring translation routines
State: closed | Pull request: yes

This PR does the following:
* Makes sure that all pipeline pieces are create inside the constructor: src_normalizer, src_pre_tokenizer, encoder/decoder_tokenizers, tgt_post_tokenizer. This is necessary for their artifacts to be handled correctly and be included in .nemo file
* Refactors "batch_translate", "translate" and "eval_step" so that:
  - eval_step relies on batch_translate
  - translate and batch_translate call pipeline pieces, hopefully, in a correct order 

TODOs:
translate method should really just prepare input batch and call batch_translate so that core pipeline logic is implemented in a single place (batch_translate) 

@ericharper I suspect if we the conversion of raw text training data to tarred files might be missing corrsponding src_normalizer and src_pre_tokenizer calls.

This pull request **introduces 1 alert** when merging 363f32e68832590ee89335af22f4a878a29b598d into efaab9e67346d633e3efd49c44a5c2b83661d1fc - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-979f06969d7ee8670113e8c823984cfdd8315809)

**new alerts:**

* 1 for Unused local variable

This pull request **introduces 1 alert** when merging 8886b806f38c1fa1d3ecccb41f83d0106a3dffcc into efaab9e67346d633e3efd49c44a5c2b83661d1fc - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-4793ce9452b56c71730cafd6c9d3064bf02cd5c6)

**new alerts:**

* 1 for Unused local variable

This pull request **introduces 1 alert** when merging 71b99105d635686ef8a3d2d3650308518de1212e into 4d6cad6a4e59e9d10fcc82984c1ddb3a1f40067f - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-3c21db23a3add8743d7dc315fea42502f6e33cb2)

**new alerts:**

* 1 for Unused local variable

This pull request **introduces 1 alert** when merging 9bb8a42e1031673add45830cc49c195954c29078 into f4ffb4300d6ef1227e0a2cec9c692a38606be299 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-96e68b4597c7b3938ef8f60957bb746ed7745c02)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 82503548d2fff72044e4d5d2c5c6bd9b6a943974 into f4ffb4300d6ef1227e0a2cec9c692a38606be299 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5c342ec512de2a5cf460061466b77dff394519c3)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging f58fe8e6dddee6438fe6796073015e889551f759 into 7d4ad85dc38d41eae4369f2d1b6ec8f3de5cf1b7 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-90f900630c6b454b2d25a26e0035dfa03a0e0ac8)

**new alerts:**

* 1 for Unused import
Hotfix for en/ ja preprocessing
State: closed | Pull request: yes

1) hotfix for https://github.com/NVIDIA/NeMo/pull/1801. one call to `get_detokenizer` wasn't updated.
2) @MaximumEntropy noticed that the previous logic would apply the en/ja preprocessing in the case that the user did src == tgt (for whatever reason they wanted to do this). The current logic fixes this.
Speaker Diarization tutorial
State: closed | Pull request: yes

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>
* Add speaker diarization tutorial

This pull request **introduces 1 alert** when merging fc4e65d2e9ab237e6a8f967ee78ec699b6f2ff41 into 4efb9055494a623b2dd4b0366fb51022633662da - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-cb69f4b17826be237efbbf1894cf25f80a6f8e89)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging b01ffab694408453586234f56e0a5dffc5feb5b1 into efaab9e67346d633e3efd49c44a5c2b83661d1fc - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-19343706cb7a1972e452539930d35fa8e76a4b7a)

**new alerts:**

* 1 for Unused import

> 1. Should we merge it to RC instead of main?
I think main should be fine since this PR is not for any bug fix. @titu1994 what do you think?
share the same pre and post processing pipelines for Ja & En
State: closed | Pull request: yes

The current japanese and english pre and post processing pipelines are shared. However, that wasnt reflected in the nemo code. This PR rectifies this discrepancy and makes sure english is processed the same way as japanese text is whenever the source or target language is the latter.

This pull request **introduces 2 alerts** when merging 2a7441a7ca2cd2d9914cd41b68fbec872047e9ff into b467d6f20a13965e65522b3fbaf833b7cbf9f3ac - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-3cdfd9c5b9dcc8e2bbebd0e0aa3b259abcb75ea0)

**new alerts:**

* 2 for Variable defined multiple times
Fastspeech2
State: closed | Pull request: yes


This pull request **introduces 5 alerts** when merging 23a6b0e5cb4cb53703400b9bd7bcec44906c6afa into f2378dd28b1c1a0bac985a979bb79059143d8026 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a6b19a6b94f0b89c2397eb614e1f75249088c340)

**new alerts:**

* 4 for Unused import
* 1 for Unused local variable

This pull request **introduces 5 alerts** when merging 619be165a7a12708ed649ce6133dba2cba605fb2 into f2378dd28b1c1a0bac985a979bb79059143d8026 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-bbbd15bbaaa2720bf8eb5fa61fe56270f657ed19)

**new alerts:**

* 4 for Unused import
* 1 for Unused local variable

This pull request **introduces 5 alerts** when merging 29a1dd910673c372909fd05c997f6c25e88e8d75 into f2378dd28b1c1a0bac985a979bb79059143d8026 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-30dd77b19347fcb2e6d5cf4236e5b0930bf80547)

**new alerts:**

* 4 for Unused import
* 1 for Unused local variable

This pull request **introduces 5 alerts** when merging 952bd13864fe73a3de78d3c982abeb19bd80980c into f2378dd28b1c1a0bac985a979bb79059143d8026 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-fc4657d6769d9851fff2c718f7a7e5f8190ae756)

**new alerts:**

* 4 for Unused import
* 1 for Unused local variable

This pull request **introduces 5 alerts** when merging 1b1a2daed825a0c689908d264c2ea50b05878202 into f2378dd28b1c1a0bac985a979bb79059143d8026 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-7c3e5b3aa97e5ced36a0d24d9f8e08f86b0e0721)

**new alerts:**

* 4 for Unused import
* 1 for Unused local variable

This pull request **introduces 5 alerts** when merging b3a43bed5737f3eb78f99e9a5ce5d8b6ab3adfbe into f2378dd28b1c1a0bac985a979bb79059143d8026 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-7cb2de0d78daa4692edeb1a8b4318fa6299bc1f0)

**new alerts:**

* 4 for Unused import
* 1 for Unused local variable
Notebook fix and modified some scripts
State: closed | Pull request: yes

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

* Pytorch Lightening links fix
* renamed get_spkr_emb.py to extract_speaker_embeddings.py 
* Added helper speaker_reco_infer.py example script to get labels from trained speaker recognition model
* Added get_hi-mia.py script. (use to be part of old nemo -> used in tutorial) 

This pull request **introduces 1 alert** when merging 23389db2bfbb66f34af4fc4bfc3c70d58a37b791 into 1b82b571c2f32e1b109677a67c5820222cb696fe - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-73b2bd0cd56408059f13b25d589e438f9dcb3fba)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging fa085934cf1e249a38988d8be3420bec67f461c8 into 1b82b571c2f32e1b109677a67c5820222cb696fe - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a484d0a7cf244b188356a2d7da6ce0cb8bec3807)

**new alerts:**

* 1 for Unused import
Fix language filtering
State: closed | Pull request: yes

Signed-off-by: PeganovAnton <peganoff2@mail.ru>

The script `filter_langs_nmt.py` did not work on monolingual data. This pull request fixes the bug.

I cannot understand how to fix
```
==================================== ERRORS ====================================

_______________ ERROR collecting examples/tts/test_tts_infer.py ________________

ImportError while importing test module '/home/mrjenkins/workspace/NeMo_PR-1791/examples/tts/test_tts_infer.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/opt/conda/lib/python3.6/importlib/__init__.py:126: in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

examples/tts/test_tts_infer.py:27: in <module>

    from nemo.collections.asr.metrics.wer import word_error_rate

nemo/collections/asr/__init__.py:15: in <module>

    from nemo.collections.asr import data, losses, models, modules

nemo/collections/asr/models/__init__.py:16: in <module>

    from nemo.collections.asr.models.classification_models import EncDecClassificationModel

nemo/collections/asr/models/classification_models.py:38: in <module>

    from nemo.utils.export_utils import attach_onnx_to_onnx

E   ImportError: cannot import name 'attach_onnx_to_onnx'

------------------------------- Captured stderr --------------------------------

[NeMo W 2021-02-24 15:37:06 nemo_logging:349 rank:0] /opt/conda/lib/python3.6/site-packages/pydub/utils.py:170: RuntimeWarning: Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work

      warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning)
```
Removing attach_onnx_to_onnx
State: closed | Pull request: yes

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>

This pull request **introduces 1 alert** when merging 647a5bc614dcf01757efb363fabd1958b233e663 into bd6e1ff7a16e2cf0ec5c52df53d975df459cfe4e - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2bdcfa9f3a865f559588bd10e67a3d2798937dcd)

**new alerts:**

* 1 for Unused import
Refactor of tokenization and detokenization within the NMT model
State: closed | Pull request: yes

This PR refactors some of the fetching of tokenizers, normalizers, and detokenizers within the model and creates Chinese/Japanese tokenizer and detokenizer classes to match Moses semantics.

This pull request **introduces 4 alerts** and **fixes 1** when merging ae81427bba749fe8dd90229c958aef151bd8aa33 into cc4bfc06993326fcd8bb978460754b89026341ff - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-257f9409c9404fe2fe6c2d87d34dc968e99b8e1c)

**new alerts:**

* 4 for Unused import

**fixed alerts:**

* 1 for Unused import

thank you for doing this. lgtm.

This pull request **introduces 4 alerts** and **fixes 1** when merging 9193ed5c9dd5a6002c2c366556783ee68978a825 into 1b82b571c2f32e1b109677a67c5820222cb696fe - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-80bd0bb0142abf15d8bcf3404d2ff85ea5ab25cf)

**new alerts:**

* 4 for Unused import

**fixed alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** and **fixes 1** when merging ed75070f3ba732944f4ae09ac6c913ba5da3470d into 1b82b571c2f32e1b109677a67c5820222cb696fe - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d3199dbf754c721c131a5e33985cad4ea3fb9ef1)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging b761630ea0b35facb77865e3b126cb958b374740 into 1b82b571c2f32e1b109677a67c5820222cb696fe - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-53e715d1d4bfc01e223032c2575b07bbda2109b5)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging 843d255e3f9d52e4cc09ad220c9986c80adb0342 into 1666139b6b3334dc8f699f69d5a97aec4fe53419 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5fda55ea2dcf9960b6efde9acea83ddbabad54fa)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging e3ec54b607ed63a44cd55b867876eecb310ec34c into 2e9e5b174ed3b766abbd6bb89b9d2c96e5d8f66c - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5e15a729fe2d136a97e9b121d1cb6b589edb1016)

**fixed alerts:**

* 1 for Unused import

Lgtm. Thanks again!
how to compute log filter bank energy in audio_preprocessing (asr collection in nemo) compare with python_speech_feature?
State: closed | Pull request: no

I want re-procedure result like when i use compute log-filterbank energy of lib: python_speech_feature by using torchaudio.
this is my code:

```
# load audio data by librosa
path_audio = "audio_a.wav"
y, sr = librosa.load(path_audio, sr=16000, offset=0.5, duration=0.4)

# load audio data by torch audio
audio_ft, sr = torchaudio.load(path_audio)
audio_ft = audio_ft.squeeze(0)
y_torch = audio_ft[int(0.5*16000):int(0.9*16000)]

# the result is the same then i compute log filterbank energy
# log filter bank energy compute by python_speech_feature lib
ft_f_bank = python_speech_features.logfbank(y, samplerate=16000, winlen=0.025, winstep=0.01, nfilt=64,nfft=512)
print(ft_f_bank.shape) # result: (39, 64)

#  log filter bank energy compute by FilterbankFeatures module in audio_preprocessing (asr collection in nemo)
 self.featurizer = FilterbankFeatures(sample_rate=16000, n_window_size=int(0.025*16000), n_window_stride=(0.01* 16000), n_fft=64, log=True)
ft_by_f_bank_nemo = self.fearturizer(y_torch) #result shape: (41, 64)

# log filter bank energy compute by torch audio kaldi compliance
ft_f_bank_by_torch = torchaudio.compliance.kaldi.fbank(y_torch, sample_frequency=16000.0, frame_length=25.0, frame_shift=10.0, use_log_fbank=True, use_energy=True, num_mel_bins=64)
print(ft_f_bank_by_torch.shape) # result: (38, 65)
```

How can i make result return by module filterbankfeature in nemo or torchaudio is the same with python speech feature. I'm not have deep understand more about speech feature, so question can so weird, sorry.
Thankyou
my torchaudio: 0.6.0,
pytorch:1.6.0

In FilterbankFeatures class we extend time length features to multiples of `pad_to` value here 16(default) for faster processing. Try changing that to -1 and try again.
Ja Source Language Preprocessing
State: closed | Pull request: yes

This PR adds preprocessing for Japanese as a source language to nemo. Previously, support only included Japanese when it was a target language.

This pull request **introduces 1 alert** when merging c5907d9e737d91da1be9f0dd3a1bde351feb6325 into bb84595e2811f374c93866ee50d65518fc33faed - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d35a3ab6e82c4b3a3cba216f1d3e26e53f98351e)

**new alerts:**

* 1 for Wrong name for an argument in a class instantiation

This pull request **introduces 1 alert** when merging 09b8ef0bbc2e3af89faa8e6e7aad02e67b5f7c21 into bb84595e2811f374c93866ee50d65518fc33faed - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1bd20499f4ea178421f0b93b73dd15db5d80181d)

**new alerts:**

* 1 for Wrong name for an argument in a class instantiation

This pull request **introduces 1 alert** when merging 5b8958dabb847c937c10cd3a8ecdc9a014f005cc into bb84595e2811f374c93866ee50d65518fc33faed - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-67a157eeb9c41fbd5329479d54ac1a2e1c27325e)

**new alerts:**

* 1 for Wrong name for an argument in a class instantiation
HifiGAN finetuning on synthetic mels
State: closed | Pull request: yes

This PR introduces fine-tuning of HifiGAN on synthetic mels generated by a spectrogram generator model (for example, Tacotron2).

This pull request **introduces 2 alerts** when merging eeeebf02faa79e72f4921a91517cba26173669a1 into 16b87d1f7d22126b8eb471ca0538f75ade9620cf - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-aa22cf3bd77bdd09b203bef1f11fdcc603a8ffeb)

**new alerts:**

* 1 for Unused local variable
* 1 for Unused import
Jasper model not loading on CPU
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Loading `Jasper10x5Dr-En` model fails on a system without a GPU.

**Steps/Code to reproduce bug**

Notebook with example:
https://colab.research.google.com/drive/1NDzwQl0KyOOYPheW60colRD2Sm8Ng_vD?usp=sharing


**Expected behavior**

NeMo should detect that GPU is not available and load the model to CPU.

**Environment overview (please complete the following information)**

 - Environment location: Google Colab
 - Method of NeMo install: `!pip install nemo_toolkit[all]==1.0.0b2`

**Environment details**

- OS version:
```
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
```
- PyTorch version `1.7.0+cu101`
- Python version `Python 3.6.9`


You can use from_pretrained(..., map_location="cpu")

closing, please re-open is @titu1994 's suggestion did not help.
Update/Fix for classification models
State: closed | Pull request: yes

1. Change self.log(tensorboard_logs) and remove the 'log' part in the return  requested in  PR #[1707](https://github.com/NVIDIA/NeMo/pull/1707) 
2. Fix for logging for top-k acc metrics
3. Fix **tarred** dataloader for speech regression support


just to confirm that this is not a bugfix for rc1? 

@okuchaiev Part of this (Point 2)  should be fixed in RC but this might raise conflict and not very essential (just logging), so we decide to merge it to main.  I could split it and merge to RC if it's better. 

Split this PR into two please. Bugfix to RC, logging to main. You can revert the RC patch from this PR and do it in another or vice-versa.

@titu1994 
Moved the following to RC in https://github.com/NVIDIA/NeMo/pull/1878. Could you please review it? 
1. Change self.log(tensorboard_logs) and remove the 'log' part in the return requested in PR **before regresion**
2. Fix for logging for top-k acc metrics

Let wait for RC to merge to main and I will update self.log and bug fix for **regression** to avoid any conflict.


Close this since first part has been merged in https://github.com/NVIDIA/NeMo/pull/1878 and second part https://github.com/NVIDIA/NeMo/pull/1907  is ready for review
extract speaker embeddings with `spkr_get_emb.py`
State: closed | Pull request: no

Hi!

I am just getting started with `NeMo`, and got a little confused when experimenting with `spkr_get_emb.py`, it appears one will need to use `SpeakerNet_verification_3x2x512.yaml` config and set up `cfg.model.test_ds`, however in this configuration, there is no `test_ds`. Was this an error or does this mean one needs to write `test_ds` to the verification model config? Any help on this would be appreciated!

Cheers,
Xu

Yes you can either add it to config and call the script or while calling the script you may give test_ds options as command line arguments prepended `+`. For reference look at hydra config https://hydra.cc/docs/tutorials/basic/your_first_app/config_groups 

Thanks for your answer, Nithin! I did an experiment with speaker verification without setting up test_ds in the model training stage and set up `test_ds` the way it was done in the tutorial
```python 
test_config = OmegaConf.create(dict(
    manifest_filepath='/embeddings_manifest.json',
    sample_rate=16000,
    labels=None,
    batch_size=1,
    shuffle=False,
    time_length=8,
    embedding_dir='./'
))
verification_model.setup_test_data(test_config)
```
Should this be one way to do it? Thanks again for your help!

yes, pass this to setup_test_data with model instantiated and extract embeddings. 

Many thanks again for your help!
Add tutorial on using Label Studio with NeMo ASR model
State: closed | Pull request: yes


Can you please sign your committed commits? See DCO's bot output how to sign already committed commits  https://github.com/NVIDIA/NeMo/pull/1767/checks?check_run_id=1925053254 

also, please re-base on the latest main, because currently CI does not pass due to a bug (which was fixed) in older version of main 

Can you allow us write access to this PR? We can't merge unless your branch is up to date with main and we update main many times a day
B4 leftovers
State: closed | Pull request: yes

Integrating last couple of PRs that went into b4.

This pull request **introduces 1 alert** and **fixes 2** when merging d285219e23c779109783e6db98f856ccf20730c6 into d7ad10435e97024c7c9fdbdd4228e0011dc9e432 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b278125132b6fe7b7e4bf372a40fa8d12c683374)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for Unused import

This pull request **introduces 1 alert** and **fixes 2** when merging 331c9d4e2b693d48c8ec024576ee34aab65672a4 into eff003635f6d7c4fbb1be4f46b747db63b58cd34 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d3c05e91e5cd8e8107cc690ddd36c5c7a8aac545)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for Unused import
Add FS2S
State: closed | Pull request: yes

- [x] Merge with main
- [x] Use HiFiGAN modules
- [x] Use our FS2 modules

This pull request **introduces 2 alerts** when merging 900c52a2fc5e01c04dddb29d51b3894c504e837b into 45aaaf4aad712d53b58e6b14db6343c4e9940f25 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-3758cb3dc44cd572ab179a17194a1cafa9166522)

**new alerts:**

* 2 for Unused import

This pull request **introduces 2 alerts** when merging ed9d82b7a6032b7304e436434c0c6e78eff6e983 into 2b0711778f3463188ea87bdfd1b0b4ecc48ec11a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-f13ab895beee0599ab6d003ce727ebb54e1dc13c)

**new alerts:**

* 2 for Unused import

This pull request **introduces 6 alerts** and **fixes 3** when merging fd12af9df4dacabd9125780096e4894df72d3fd3 into 2b0711778f3463188ea87bdfd1b0b4ecc48ec11a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a88f3b11a7402e4cda247156a26cc184b2d2cd9f)

**new alerts:**

* 6 for Unused import

**fixed alerts:**

* 3 for Unused import

This pull request **fixes 5 alerts** when merging c0c4b099004111aad1369cb2d8dc0279262f223a into 2b0711778f3463188ea87bdfd1b0b4ecc48ec11a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-53f33ff6e93e839ad537f3d533f1298f69698046)

**fixed alerts:**

* 5 for Unused import

This pull request **fixes 5 alerts** when merging 6d082c04ce7cf0265b4e3d6f50f5581b1967e8b0 into 2b0711778f3463188ea87bdfd1b0b4ecc48ec11a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5bc4526daea285700462f767ea9bb14afb250381)

**fixed alerts:**

* 5 for Unused import
Moving calculating the class weights to the data loader creation method.
State: closed | Pull request: yes

It is necessary to support class_balancing in TTL.

This pull request **fixes 3 alerts** when merging 0bc9d88cc28dbdb604348502888931556acde702 into 7d8b7d679a2a4c9d3a6b2d8def8888e53c804ded - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1f5d9f48dd6cb74ec4d5390dc1d8290bf990ed9d)

**fixed alerts:**

* 2 for Unused local variable
* 1 for Unused import

This pull request **fixes 2 alerts** when merging 4715c44a944ff344f6fb967393df078aea321216 into c5cd85f4202a94a44aba3103628fc39d73c48379 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-c0f6315b050920f0072cc5321f3a29a7b3a0b65d)

**fixed alerts:**

* 2 for Unused local variable

Can we still make changes to b4 branch? @tkornuta-nvidia @okuchaiev ?

> 
> 
> Can we still make changes to b4 branch? @tkornuta-nvidia @okuchaiev ?

Which branch should we use for bug fixes needed for TLT?

This pull request **fixes 2 alerts** when merging af60927630e6ae145748eb9ef73ba7e82172a714 into c5cd85f4202a94a44aba3103628fc39d73c48379 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-ba6fbbbe145ae3615601b252ce75d1d24357b625)

**fixed alerts:**

* 2 for Unused local variable
Update webdataset call to support v0.1.48
State: closed | Pull request: yes

# Changelog
- Update API for webdataset calls
- Add lower bound for webdataset requirements (may want to pin it for release, @okuchaiev)

@ericharper Can you pull this branch and verify NMT models have no issue running their WebDataset training on some toy / small dataset?

@fayejf Same for ASR Classification models

I've verified that ASR CTC/RNNT work 1:1 as before.