renwei2024/nemo-github-issues huggingface dataset:
Running examples `enc_dec_nmt.py` script and getting a TypeError: expected str, bytes or os.PathLike object, not NoneType
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I am running `enc_dec_nmt.py` as is from https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/machine_translation/enc_dec_nmt.py

Here's the error:

```
Traceback (most recent call last):
  File "/home/jpilault/projects/nemo_examples/enc_dec_nmt.py", line 124, in main
    mt_model = MTEncDecModel(cfg.model, trainer=trainer)
  File "/home/jpilault/anaconda3/envs/myenv/lib/python3.8/site-packages/nemo/collections/nlp/models/machine_translation/mt_enc_dec_model.py", line 84, in __init__
    self.setup_enc_dec_tokenizers(
  File "/home/jpilault/anaconda3/envs/myenv/lib/python3.8/site-packages/nemo/collections/nlp/models/machine_translation/mt_enc_dec_model.py", line 442, in setup_enc_dec_tokenizers
    self.encoder_tokenizer = get_nmt_tokenizer(
  File "/home/jpilault/anaconda3/envs/myenv/lib/python3.8/site-packages/nemo/collections/nlp/modules/common/tokenizer_utils.py", line 164, in get_nmt_tokenizer
    return YouTokenToMeTokenizer(model_path=tokenizer_model, bpe_dropout=bpe_dropout, r2l=r2l)
  File "/home/jpilault/anaconda3/envs/myenv/lib/python3.8/site-packages/nemo/collections/common/tokenizers/youtokentome_tokenizer.py", line 26, in __init__
    model_path = Path(model_path).expanduser()
  File "/home/jpilault/anaconda3/envs/myenv/lib/python3.8/pathlib.py", line 1042, in __new__
    self = cls._from_parts(args, init=False)
  File "/home/jpilault/anaconda3/envs/myenv/lib/python3.8/pathlib.py", line 683, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/home/jpilault/anaconda3/envs/myenv/lib/python3.8/pathlib.py", line 667, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

**Steps/Code to reproduce bug**

Run the code

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

It should not give an error.

**Environment overview (please complete the following information)**

used pip install

**Environment details**

- OS version:  Ubuntu 20.04.2 LTS
- PyTorch version: 1.10.0+cu113
- Python version: 3.8

**Additional context**

Add any other context about the problem here.
Example: GPU model
Citrinet 256 and 512 for WSJ - limited resources
State: closed | Pull request: no

Hello!  I wanna get baselines for citrinet-256 and citrinet-512 using WSJ, so that I can perform some experiments after. I will use speed perturbation and specAugment.

For now, I am training citrinet-256  with 4 GPUs, and batch 60. I also created a tokenizer with 250 for vocab and using WPE (not sure if this is the best for low resources).  I also decided to use dropout equal to 0.2 since there was a huge gap between validation and train in the first epochs. It is also relevant to note that I moved from learning rate of 0.01 to 0.001 since the loss was going to NaN after epoch 8. Also, as a note, I experimented to set the flag detect anomaly to True in the Trainer and it got an error in the first steps for getting NaN results in the backward pass - not sure if that is okay or not to happen...

Below are some configurations that I am using.
<img width="1010" alt="Captura de ecrã 2021-12-13, às 20 36 52" src="https://user-images.githubusercontent.com/15928244/145884925-dcf112e1-d427-4df4-8931-564001d534e0.png">
<img width="1015" alt="Captura de ecrã 2021-12-13, às 20 37 23" src="https://user-images.githubusercontent.com/15928244/145884979-f9a57682-d06c-493e-b43c-91b7ae6e5f6b.png">


It would be great to know if I am in the right direction! Thanks a lot
Inverse Text Normalization. Make matching between normalized and non-normalized tokens
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**

Let we have the next text (in Russian): "куку девятьсот шестьдесят шесть миллиардов восемьсот сорок четыре миллиона семьсот пять тысяч четыреста двадцать пять девятьсот сорока пяти купить"

`inverse_normalizer.find_tags` and. `inverse_normalizer.select_tag` give us the next tokens representation:
_' tokens { name: "куку"  }  tokens {  cardinal { integer: "966844705425"  }  }  tokens {  cardinal { integer: "945"  }  }  tokens { name: "купить"  } '_

But by this output, it's complicated to make matching between "966844705425"  and corresponding words in the input text.

**Describe the solution you'd like**

`inverse_normalizer.find_tags` and `inverse_normalizer.select_tag` give us next tokens representation in another way:
_' tokens { name: "куку" , **(0, 4)** }  tokens {  cardinal { integer: "966844705425",  **(5, 107)**}  }  tokens {  cardinal { integer: "945", **(108,  128)** }  }  tokens { name: "купить", **(129,  135)**  } '_ (span values are approximately, just for demonstrating goal)

So we will have corresponding tokens spans in input text, and it will be easy to solve the matching problem between entities.

Thank you very much for your attention!
calculate WER
State: closed | Pull request: no
Labels: name: feature

Hi,
Thanks for your great repo. I have 2 questions:
1- In the Nemo repo is there any implemented code for calculating the WER (word error rate) for a data set & model's transcription? (_i.e._, I mean for example I want to calculate the WER of a trained model on a test set, which I have its ground truth in a text file. So, is there any script in Nemo that I can use it? or I must use other codes such as: [jiwer](https://pypi.org/project/jiwer/))
2- In the [`ASR_CTC_Language_Finetuning`](https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/asr/ASR_CTC_Language_Finetuning.ipynb#scrollTo=iK-RQXEZfq1V) tutorial, if one enables `use_cer` (character error rate) parameter, then the tensor board results will be shown the CER instead of WER? (In other words, I have used the `use_cer = True`, and for example the tensor board showed me the **WER** of the trained model on epoch=31 on validation set is ~ 0.1028. Now I want to know that it is really WER or it is CER? and if it is CER, how I can calculate the WER of the model on my validation or test set?) 
![val_WER](https://user-images.githubusercontent.com/28620609/145693294-daf62515-e6a7-4ff7-a4f8-defdf0eb629b.png)

Best
Fix MixerTTS generate_spectrogram types and dimensions
State: closed | Pull request: yes

Fixes the typecheck decorator for generate_spectrogram in MixerTTS and the swapping melspec and time dimension (as also done here https://github.com/NVIDIA/NeMo/blob/3687f2eaae36de6dd686caa99d2fe3100a5f4431/nemo/collections/tts/models/mixer_tts.py#L649)

You're probably in the middle of a refactor so sorry if you already got that, @Oktai15  (?)

(just FYI, I just trained MixerTTS from LJ with the current main head and everything worked fine, except this transpose missing when synthesizing and the mismatch of the decorator with the parameters)
Question[multi node training]
State: closed | Pull request: no

How to train a  model using multi-node(eg. 2 node, each node with 4gpus, just set the node numbers in the config doesn't work, In the pytorch-lighting tutorial, It needs the parameters like port and rank, etc) ?  Do you have any tutorial or examples like in the speech_to_text_bpe.py?

# Training the model
```sh
python speech_to_text_bpe.py \
    # (Optional: --config-path=<path to dir of configs> --config-name=<name of config without .yaml>) \
    model.train_ds.manifest_filepath=<path to train manifest> \
    model.validation_ds.manifest_filepath=<path to val/test manifest> \
    model.tokenizer.dir=<path to directory of tokenizer (not full path to the vocab file!)> \
    model.tokenizer.type=<either bpe or wpe> \
    trainer.gpus=-1 \
    trainer.accelerator="ddp" \
    trainer.max_epochs=100 \
    model.optim.name="adamw" \
    model.optim.lr=0.001 \
    model.optim.betas=[0.9,0.999] \
    model.optim.weight_decay=0.0001 \
    model.optim.sched.warmup_steps=2000
    exp_manager.create_wandb_logger=True \
    exp_manager.wandb_logger_kwargs.name="<Name of experiment>" \
    exp_manager.wandb_logger_kwargs.project="<Name of project>"
```
Fix for Stalled ASR training/eval on Pytorch 1.10+ (multigpu/multinode)
State: closed | Pull request: yes

# Bugfix
- Training / inference stalls on multi gpu / multi node on Pytorch 1.10+
- This happens because the ASR models attempt to adaptively increase the masking window size or positional encoding (conformers) during training.
- If some ranks update their mask/pe buffer and others don't, then all reduce cannot properly perform a sync and code stalls.
- Solution is to perform 1 all reduce (max) sync per acoustic model forward pass, gathering the max(length) across all nodes, and adaptively updating the seq length in the modules if required. 

Signed-off-by: smajumdar <titu1994@gmail.com>
trouble with dedicating multiple GPU speech_to_text_bpe
State: closed | Pull request: no
Labels: name: bug

Hi, I've trouble with dedicating multiple GPU for asr/speech_to_text_bpe.py.
In config file of conformer, I tried gpu=-1 and also gpu=2 but when I run the code, it just use the first GPU and it doesn't use the second one.
Could you please help me to solve it?
As you see when I run the code, it use just the first GPU and just use 11Mb of the second one.

![Screenshot from 2021-12-08 16-58-13](https://user-images.githubusercontent.com/95332871/145216745-c1413158-3836-489b-a3c6-58f32df45470.png)
Fix specaugment time start for numba kernel
State: closed | Pull request: yes

# Bugfix
- Correct the variable for time_starts when executing the numba kernel for adaptive spec augment

# Note:
- This bugfix does not affect training runs used with the NeMo container or the pytorch container
- The CUDA version in the containers is always higher than the max supported CUDA version in Numba, therefore by default the numba kernel for spec aug is disabled.
- Only way to bypass this is to pass the environment flag `STRICT_NUMBA_COMPAT_CHECK=False` when running the training scripts.

Closes #3298 

Signed-off-by: smajumdar <titu1994@gmail.com>
Spec augmentation time masking
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I was wondering if this line of [code](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/parts/numba/spec_augment/spec_aug_numba.py#L286) for time masking of spec augmentation is correct.
Unless intended, it looks more reasonable with `time_starts = torch.cat(time_starts, 0)` rather than `time_starts = torch.cat(time_lengths, 0)` as this line of code makes `time_starts` and `time_lengths` equivalent to each other.

**Steps/Code to reproduce bug**

Any scripts useing spec augmentation, e.g., `python speech_to_text_bpe.py --config-path conf/conformer --config-name conformer_ctc_bpe`

**Expected behavior**

It currently generates time masks only at the beginning of the sequence.
![Screenshot from 2021-12-07 22-47-11](https://user-images.githubusercontent.com/50283958/145161693-8069705e-3599-494c-89b4-70eec6140545.png)

**Environment overview (please complete the following information)**

`pip install -e .`

**Environment details**

N/A

**Additional context**

N/A
Is "offset" supported in audio reader ?
State: closed | Pull request: no

Hi, I am totally new to NeMo and I would just like to ask if "offset" is already in use for loading audio? Because I read this warning in the document : 
![image](https://user-images.githubusercontent.com/41187612/145059151-91b5e489-fe8a-43e4-a6bf-71170a92f462.png)
Megatron AMP fix for scheduler step counter
State: closed | Pull request: yes

# Bugfix
- Attempt to prevent a step counter update of the scheduler when skipping a step in the optimizer
- Add training steps when nan/inf gradients are detected
- Properly support LR scheduler with "constant lr" at end of training. Refactor CosineAnnealing model for such support.

# Details

Since there are no pre-existing hooks in PTL to modify the scheduler step before the optimizer step, ive had to splice the support into the current GradScaler and then use the on_train_batch_end hook and manually monitor when the optimizer step occurs via the flag  optimizer_update_skipped - None means optimizer didnt step this time (due to accumulated gradient), and True means grads were accumulated but optimizer did not perform step() - skipped step. In such case, perform a no-op for the scheduler too.

Signed-off-by: smajumdar <titu1994@gmail.com>
NLP text augmentation
State: closed | Pull request: yes

This PR adds a text augmentation script. The script can be used to generate human-like typos when testing model robustness.

# Summary

The script supports the following text augmentations:

1. Dropping a letter in a word
2. Swapping two letters in a word
3. Human-like typos (based on keyboard structure, and common misspelling) using https://github.com/facebookresearch/AugLy package.
GPU memory is used although cpu() is called
State: closed | Pull request: no
Labels: name: bug

I run the following commands: 

```
import nemo
import nemo.collections.asr as nemo_asr
import nemo.collections.nlp as nemo_nlp
import nemo.collections.tts as nemo_tts

quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="stt_en_quartznet15x5").cpu()
English_text = quartznet.transcribe(['~/audio1.wav'])

```

In the output of `nvidia-smi`, I see that `800 Mb` of GPU is used when running the above commands. Why? My expectation is that no GPU memory should be used when I use `cpu()`.
Conformer training from scratch not achieving the reported WER
State: closed | Pull request: no

Hello! Thanks for providing such a nice toolkit.

I am currently training conformer-small on librispeech-960hr from scratch, following the training recipe and instructions in `examples/asr/conf/conformer/conformer_ctc_bpe.yaml`.
However, I'm not able to reproduce the WER reported in this [webpage](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_conformer_ctc_small_ls): reported ones are 3.4/8.8, but our WER is only around 4.5/11 on the clean/other datasets with ~300 epochs of training and doesn't seem to decrease anymore.

Here is the list of hyperparameters I'm using:
* d_model, n_heads, n_layers: 176, 16, 16 
* tokenizer: WPE with vocab_size 128 (generated using the script in NeMo)
* spec_augment: freq_masks 2, time_masks 5, freq_width:27, time_width: 0.05
* lr: 5.0, betas: [0.9, 0.98], weight_decay: 1e-3 (adamw)
* warmup_steps: 10000 (noam annealing)
* Batch size: 2048 (using 8 machines, local batch size 16, and accumulate_grad_batches 16, multiplying up to 2048)
All the other hyperparameters are the same as the default.


Here is one of the learning curves (validation WER on dev-clean) from our runs:
<img width="1665" alt="Screen Shot 2021-12-05 at 1 18 53 PM" src="https://user-images.githubusercontent.com/50283958/144764357-9cc2609c-cf8e-4158-8693-5e6a7b7c5bd8.png">

I'm wondering if we should change the hyperparameters to achieve the reported WER values, or if the 300-epoch of training is not enough for the model convergence, or if there's something that we are missing.
Any clues regarding the hyperparameters, the number of training epochs, etc. would be really appreciated!

Thank you.
Multiscale diarization and update on offline diarization with ASR script
State: closed | Pull request: yes

This pull request includes the following features:
- Multiscale diarization: segmentation and clustering parts are updated (experimental).
- Optional pyctcdecode feature to apply beam search decoder for ASR. 
- Language model based realigning feature for diarization (experimental).  
- Added CTM based evaluation (WDER and WER evaluation)
- Miscellaneous updates and bug fix for diarization with ASR.
GPU utilization is zero
State: closed | Pull request: no
Labels: name: bug

I run the following commands: 

```
import nemo
import nemo.collections.asr as nemo_asr
import nemo.collections.nlp as nemo_nlp
import nemo.collections.tts as nemo_tts

quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="stt_en_quartznet15x5").cuda()
English_text = quartznet.transcribe(['~/audio1.wav'])
```

But GPU utilization is zero as it can be seen from the output of `nvidia-smi
`:
![nv](https://user-images.githubusercontent.com/89482819/144546023-d8df81f2-591b-4d74-83a8-973f3d9a0e5b.jpg)
kenlm for conformer-ctc
State: closed | Pull request: no
Labels: name: bug

Hi. i need a LM for ctc-conformer and i trained a kenlm moel by this command:

python train_kenlm.py --train_file /home/mount/librispeech-lm-norm.txt --nemo_model_file stt_en_conformer_ctc_large.nemo --kenlm_model_file ../nemo_lm_6_new --ngram_length 6 --kenlm_bin_path ../kenlm/build/bin/ --do_lowercase

but when i try to use this LM for beam search decode, the output was something like this :(wowliadfbjdslfsofymnyq)
i also played with arameters alpha and beta. but not useful.
could you please help me?
AttributeError: type object 'EncDecCTCModel' has no attribute 'list_available_models'
State: closed | Pull request: no
Labels: name: bug

```
import nemo
import nemo.collections.asr as nemo_asr

asr_model = nemo_asr.models.EncDecCTCModel.list_available_models()
print("done")
```

Whenever I run this code, it gives me this error:
`AttributeError: type object 'EncDecCTCModel' has no attribute 'list_available_models'
`
I have all the dependencies installed.

When I hover over the line of code in PyCharm, it shows me the definition of the function and everything, which means it recognizes it. But for some reason it doesn't work when I run it.

<img width="1014" alt="image" src="https://user-images.githubusercontent.com/88787251/144137632-b741ba3a-c8c6-491b-bee7-01daeee7b3fa.png">
Normalizer to TTS models, TTS tokenizer updates, AxisKind updates
State: closed | Pull request: yes

* Add normalizer for FastPitch and Mixer-TTS
* Update types for FastPitch and Mixer-TTS via new proposed signature for time
* Fix small bugs and refactor code in Mixer-TTS, FastPitch and TTS tokenizers
* Add German tokenizer
* Add CMUDict

Signed-off-by: Oktai Tatanov <oktai.tatanov@gmail.com>
Fine-tuning Conformer
State: closed | Pull request: no

Hi,
I want to fine-tune **stt_en_conformer_ctc_large** with Persian dataset. I used **conformer_ctc_char.yaml** from your example folder in repository. I changed the labels, too. Now when I want to run **trainer.fit(asr_model)** I have the following error:
`Interpolation key 'model.encoder.d_model' not found
    full_key: optim.sched.d_model
    object_type=dict`
Would you please help me how to resolve this error?
Thanks

_Environment overview_

> Environment location: Google Colab
> Method of NeMo install: !pip install nemo_toolkit[asr]
> NeMo version: r1.2.0
> Learning Rate: 1e-4

_Environment details_

> OS version : "Ubuntu20.04.3 LTS"
> PyTorch version : "1.10.0"
ECAPA TDNN pretrained fails to export ONNX - missing 'input_example'
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I cannot export the pertained ECAPA TDNN model for embeddings to .onnx 
Exporting  requires the model to have a `input_example` which is not available on the ecapa tdnn model.

**Steps/Code to reproduce bug**

```
import nemo.collections.asr as nemo_asr
speaker_model = nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained(model_name='ecapa_tdnn')
speaker_model.export(output='mymodel.onnx')
```

Output:
```
TypeError                                 Traceback (most recent call last)
<ipython-input-51-d55b2004ba1e> in <module>()
      3 import nemo.collections.asr as nemo_asr
      4 speaker_model = nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained(model_name='ecapa_tdnn')
----> 5 speaker_model.export(output='mymodel.onnx')

1 frames
/usr/local/lib/python3.7/dist-packages/nemo/core/classes/exportable.py in _setup_input_example(self, input_example)
    389 
    390     def _setup_input_example(self, input_example):
--> 391         input_list = list(input_example)
    392         input_dict = {}
    393         # process possible kwargs

TypeError: 'NoneType' object is not iterable

```

**Expected behavior**

I get a mymodel.onnx

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

Nemo version: r1.5.0

**Additional context**

GPU: Tesla T4
The results cannot be reproduced using the pre-training model
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I tested the model nmt_en_zh_transformer24x6 and nmt_en_zh_transformer6x6  on WMT18 test set and found that the blue score was a little low.

**Steps/Code to reproduce bug**

`model = MTEncDecModel.from_pretrained("nmt_en_zh_transformer24x6")`

`import copy`
`cfg = copy.deepcopy(model.cfg)`

`from omegaconf import OmegaConf, open_dict`

`with open_dict(cfg):`
 ` cfg.test_ds.use_tarred_dataset = False`
 ` cfg.test_ds.src_file_name = "/datasets/wmt18/clean.en"`
  `cfg.test_ds.tgt_file_name = "/datasets/wmt18/clean.zh"`

`with open_dict(model.cfg):`
  `model.cfg.tokens_in_batch=512`
  `model.cfg.clean=False`
  `model.cfg.max_seq_length = 512`
  `model.cfg.cache_ids=False`
  `model.cfg.cache_data_per_node=False`
  `model.cfg.use_cache=False`
  `model.cfg.shuffle=False`
  `model.cfg.num_samples=-1`
  `model.cfg.drop_last=False`
  `model.cfg.pin_memory=False`
  `model.cfg.load_from_cached_dataset=False`
  `model.cfg.reverse_lang_direction=False`
  `model.cfg.load_from_tarred_dataset=False`
  `model.cfg.tar_shuffle_n=100`

`import pytorch_lightning as ptl`

`trainer = ptl.Trainer(gpus=1,`
                      `num_nodes=1,`
                      `max_steps=100000,`
                      `accumulate_grad_batches=1,`
                      `checkpoint_callback=False,`
                      `val_check_interval=578,`
                      `sync_batchnorm=False)`
`trainer.test(model)`
![ac2dc0c25892480a3e690e2df9fab50](https://user-images.githubusercontent.com/56379157/144012971-2dbd226e-8b34-4e2a-9d23-e2ac319d8783.png)

24x6 blue score is 29.38408660888672 ，10 points lower than 39.9 in the introduction
6x6 blue score is 2.8，but in the introduction it is 35.0

How can I set the parameters to get the correct results?
Quartznet replicate some results
State: closed | Pull request: no

Hello! 

I am trying to replicate Quartznet 5x5 results to get around 11/12% WER on the test set.
So far my last experiment was using 4 gpus, with batch 64, using speed perturbed data, specaugmentation, learning rate 0.01, dropout 0.2. I got around 23.18% on the validation and 17.11% on test set. Part of my config file is below:

<img width="702" alt="Captura de ecrã 2021-11-29, às 12 55 10" src="https://user-images.githubusercontent.com/15928244/143871857-85e7f9e4-f1f9-40f2-b9f7-0dff50e08d20.png">


Here I show my tensorboard results for training loss and validation loss: 
<img width="392" alt="Captura de ecrã 2021-11-29, às 12 49 48" src="https://user-images.githubusercontent.com/15928244/143871967-ffe62a03-7f10-448e-aaab-610f52291a79.png">
<img width="383" alt="Captura de ecrã 2021-11-29, às 12 49 52" src="https://user-images.githubusercontent.com/15928244/143871974-70da4f61-735c-46d2-94bf-05261a9f10f9.png">
<img width="376" alt="Captura de ecrã 2021-11-29, às 12 49 59" src="https://user-images.githubusercontent.com/15928244/143871990-e897e64e-77e2-44cb-94d5-3d47eeeee979.png">

<img width="428" alt="Captura de ecrã 2021-11-29, às 12 49 55" src="https://user-images.githubusercontent.com/15928244/143871984-cb3e0b91-130d-4f9a-8d1d-b073deb3bc27.png">


Do you guys have any guess what I should go for to reach a better WER for this architecture? Thank you, guys. Also, If I wanna to replicate results for architecture 5x3, using 4 gpus what hyperparameters should I use? 

Thanks a lot!
Using G2P (Grapheme2Phoneme) for ASR
State: closed | Pull request: no
Labels: name: feature

Hi,
Thank you for your great repo. I have 2 questions:
1- Is it better to use G2P for English ASR (or just use the raw text for it)? if yes, what model (G2P) do you suggest for this work? 
2- Is it better to use G2P for ASR of other languages or not? (for example Arabic ASR)
Best
Fine tune “nmt_en_zh_transformer24x6” using a single GPU, dict has not been initialized
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

`/NeMo/nemo/collections/nlp/models/machine_translation/mt_enc_dec_model.py in eval_epoch_end(self, outputs, mode)`
`356             # we also need to drop pairs where ground truth is an empty string`
`357             dist.all_gather_object(`
`--> 358                 tr_and_gt, [(t, g) for (t, g) in zip(translations, ground_truths) if g.strip() != '']`
`359             )`
`/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py`
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

**Steps/Code to reproduce bug**

`import os`
`from nemo.collections.nlp.models import MTEncDecModel`
`os.environ["CUDA_VISIBLE_DEVICES"] = "0"`
`model = MTEncDecModel.from_pretrained("nmt_en_zh_transformer24x6")`
`trainer = ptl.Trainer(gpus=1,
                      num_nodes=1,
                      max_epochs = EPOCHS,
                      accumulate_grad_batches=1,
                      checkpoint_callback=False,
                      logger=False,
                      log_every_n_steps=5,
                      check_val_every_n_epoch=10,
                      sync_batchnorm=False)`
`model.set_trainer(trainer)`
`trainer.fit(model)`

**Expected behavior**

I only have one machine with two GPUs. so I can't use the torch.distributed. What should I do to ignore this?

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version  1.10.0
- Python version 3.7
How to train NeMo TTS Tacotron2 model from pretrained model?
State: closed | Pull request: no

**Describe the bug**

Do Nemo TTS models train from pre-trained model?

Command is blow

python tacotron2.py train_dataset=filelists/transcript_train.json validation_datasets=filelists/transcript_val.json trainer.accelerator=null trainer.check_val_every_n_epoch=1 --pretrained_model=tts_en_tacotron2.nemo
[NeMo W 2021-11-26 11:34:27 optimizers:47] Apex was not found. Using the lamb optimizer will error out.
[NeMo W 2021-11-26 11:34:32 nmse_clustering:54] Using eigen decomposition from scipy, upgrade torch to 1.9 or higher for faster clustering
[NeMo W 2021-11-26 11:34:34 experimental:28] Module <class 'nemo.collections.asr.data.audio_to_text_dali._AudioTextDALIDataset'> is experimental, not ready for production and is not fully supported. Use at your own risk.
usage: tacotron2.py [--help] [--hydra-help] [--version]
                    [--cfg {job,hydra,all}] [--resolve] [--package PACKAGE]
                    [--run] [--multirun] [--shell-completion]
                    [--config-path CONFIG_PATH] [--config-name CONFIG_NAME]
                    [--config-dir CONFIG_DIR]
                    [--info [{all,config,defaults,defaults-tree,plugins,searchpath}]]
                    [overrides [overrides ...]]
tacotron2.py: error: unrecognized arguments: --pretrained_model=tts_en_tacotron2.nemo
Colab throws an error
State: closed | Pull request: no
Labels: name: bug

Colab notebook: [lab.research.google.com/github/NVIDIA/NeMo/blob/r1.0.0rc1/tutorials/tts/2_TTS_Tacotron2_Training.ipynb](https://colab.research.google.com/github/NVIDIA/NeMo/blob/r1.0.0rc1/tutorials/tts/2_TTS_Tacotron2_Training.ipynb#scrollTo=HEvdSU5WYZbj)

Error:
No module named 'nemo'
Diarization Inference Parameters
State: closed | Pull request: no
Labels: name: feature

Hey guys,

I'm using "Speaker_Diarization_Inference.ipynb", I'm having a problem with Vad Diarization parameters
```
output_dir = os.path.join(ROOT,'outputs')
config.diarizer.paths2audio_files = paths2audio_files
config.diarizer.path2groundtruth_rttm_files = path2groundtruth_rttm_files
config.diarizer.out_dir = output_dir # Directory to store intermediate files and prediction outputs
config.diarizer.speaker_embeddings.model_path = pretrained_speaker_model

#Here we use our inhouse pretrained NeMo VAD 
config.diarizer.vad.model_path = pretrained_vad
config.diarizer.vad.window_length_in_sec = 0.15
config.diarizer.vad.shift_length_in_sec = 0.01
# config.diarizer.vad.threshold = 0.8 threshold would be deprecated in release 1.5
config.diarizer.vad.postprocessing_params.onset = 0.8 
config.diarizer.vad.postprocessing_params.offset = 0.7
config.diarizer.vad.postprocessing_params.min_duration_on = 0.1
config.diarizer.vad.postprocessing_params.min_duration_off = 0.3
```
Testing with different audios, I have to constantly change the parameters to get a better result with each audio, or avoid the error
`IndexError: list index out of range`

This happens to me mainly in version 1.4.0 of Nemo, in version 1.5.0 I noticed an improvement but still occurs the cases.

I'm not using the "vad_marblenet" model because it wasn't meeting my needs, I'm using a model that I'm training with "Voice_Activity_Detection.ipynb"

My question is if there is any metric for setting the values ​​of these parameters?
If you have an ideal value for these parameters, which has better results with a greater number of audios that go through the script?
Error "Can't pickle local object" when using multiprocessing on GPU
State: closed | Pull request: no
Labels: name: bug

I'm looking to develop an API that will perform multiple simultaneous transcripts every time an external request arrives. To perform these simultaneous transcriptions I thought of using python multiprocessing to help me.
Performing CPU transcripts in multiprocessing works fine, however when using GPU I get the error "`Can't pickle local object FilterbankFeatures.__init__.<locals>.<lambda>"
`
Here is an example of the two scenarios on GPU and CPU:
https://colab.research.google.com/drive/1O8fEkC4SKeILJeotYGU54dMkX15go1nK?usp=sharing

I have little experience with this type of processing, I have been trying to resolve this for a few weeks. If anyone has a solution, thank you in advance.
How to use model after transfer learning on new dataset
State: closed | Pull request: no

Hi,
Used `quartznet_15x5` model for the transfer learning on the open-source dataset (AN4 dataset) for ASR. Once the pre-trained model gets trained with the newer dataset. I tried saving the trained model. Once I tested with the saved model using `nemo_asr.models.<MODEL_BASE_CLASS>.restore_from(restore_path="<path/to/checkpoint/file.nemo>")`. It shows the empty result `[' ']` even on the trained data.

Kindly provide us with the steps to follow after the transfer learning progress.

Code: 
```
config_path = './quartznet_15x5.yaml'

yaml = YAML(typ='safe')
with open(config_path) as f:
    params = yaml.load(f)

trainer = pl.Trainer(gpus=1, max_epochs=50)
quartznet_model = nemo_asr.models.EncDecCTCModel(cfg=DictConfig(params['model']), trainer=trainer)

trainer.fit(quartznet_model)
quartznet_model.save_to("./tl-osd.nemo")

```
Speaker Diarization Notebook has bug
State: closed | Pull request: no
Labels: name: bug

Hey!

I am working with this [notebook](https://github.com/NVIDIA/NeMo/blob/offline_asr_diar_vad/tutorials/speaker_tasks/Speaker_Diarization_Inference.ipynb) for diarizing the audio files. This notebook seems to work perfectly with the given "an4_diarize_test.wav" test file but when I try it on my own audio files then model gives runtime error as shown below. I wonder why it is not processing the audio properly and giving the empty tensor. My audio files are also in .wav format, therefore, there is not any inconsistency as far as audio file format is concerned.

![image](https://user-images.githubusercontent.com/32700434/143193719-e0ef20ee-2711-48df-82a9-3dc6c655c36b.png)

I also went through the source code of [audio_to_label.py](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/data/audio_to_label.py) and I found that audio_signal tensor has 0 elements. Moreover, the audio_lengths is tensor(2) which means it is only processing two samples of audio which does not make sense. Any help would be much appreciated!


torch version == 1.10
torchaudio ==  0.10.0+cpu 
torchvision  == 0.11.1+cpu
nemo-toolkit  == 1.3.0
Port Variational Inference with Adversarial Learning (VITS) to NeMo TTS
State: closed | Pull request: yes

# Overview

This change adds [Variational Inference with adversarial learning (VITS)](https://github.com/jaywalnut310/vits) to the collection of TTS models.

VITS uses the `monotonic_align` module, which requires Cython to be installed prior to building (i.e. run `pip install cython` before running `./reinstall.sh`).

# Tasks

- [x] Fix errors in audio and spectrogram dimensions
- [ ] Fix logging for validation loss
- [ ] Verify training performance
- [ ] Verify quality of TTS output
- [ ] Refactor code and cleanup to be more consistent with other models
Support TPU Training
State: closed | Pull request: no
Labels: name: feature

Hi,
Thanks for your great repo. I think it would be really great if Nemo support TPU training (due to the fact that training on some large datasets on GPU is slow, and TPU will probably help in these scenarios. In addition, other speech frameworks, to the best of my knowledge, such as SpeechBrain, ESPNet, etc., currently do not support TPU training, so in this viewpoint, it would be a great benefit for Nemo).
Best
Add Beta-binomial Interpolator to TTSDataset
State: closed | Pull request: yes

Beta-Binomial Interpolator is taken from [DeepLearningExamples](https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/SpeechSynthesis/FastPitch/fastpitch/data_function.py#L45) repository.

Signed-off-by: Oktai Tatanov <oktai.tatanov@gmail.com>
Inconsistent model inference
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Outputs different predictions for the same input depending batch. I suspect the problem is related to pad tokens not being masked properly.

**Steps/Code to reproduce bug**

```
import nemo.collections.asr as nemo_asr

model = nemo_asr.models.ASRModel.from_pretrained(model_name="stt_en_conformer_ctc_large")
model.transcribe(audio_paths[:2], batch_size=1)

# output
# ['take care of over excitement and endeavor to keep a quiet mind even for your health it is the best advice that can be given to you # # your moral and spiritual improvement will then keep pace with the culture of your intellectual powers',
# 'ab crowd']

model.transcribe(audio_paths[:2], batch_size=2)
# output
# ['take care of over excitement and endeavor to keep a quiet mind even for your health it is the best advice that can be given to you # # your moral and spiritual improvement will then keep pace with the culture of your intellectual powers',
# 'crow']

# the provided example occurred with audio files from the libritts100 dataset namely (3575_170457_000032_000001.wav and 6829_68771_000042_000001.wav), but I am quite sure it can be reproduced with other audio files as well. 
```

**Expected behavior**

Outputs should be identical independent of batch size or items in batch.

**Environment overview**

Linux Ubuntu 20.04
Python version: 3.8
PyTorch version: 1.10.0+cu113
Nemo version: 1.4.0
CUDA/cuDNN version: 11
GPU model and memory: RTX 3070, Tesla V100-SXM2-16GB
Fine-tune optimal hyperparameters for small datasets?
State: closed | Pull request: no

I'm trying to improve the accuracy of the ASR model for a specific domain.

The base model it's a fine-tuned model from English (QuartzNet) to my language, and now I'm trying to improve this base model on a specific dataset.

The dataset has around 12h, and I used speed perturbation to generate around 37h.

I use these hyperparameters by the loss seems to stay the same after a significant number of epochs.

```
new_optimizer = {
        'name': 'novograd',
        'lr': 0.0001,
        'betas': (0.95, 0.25),
        'weight_decay': 0.001,
        'sched': {
            'last_epoch': -1,
            'min_lr': 0.0,
            'monitor': 'val_loss',
            'name': 'CosineAnnealing',
            'reduce_on_plateau': False,
            'warmup_ratio': 0.12,
            'warmup_steps': None,
        },
    }
```

I use 500 epochs for this training.

Do you have any advice on how I can optimally improve the accuracy of this small dataset, or how is a minimum number of hours that can generate a good improvement?
Adding adaptive bucketing for tarred datasets.
State: closed | Pull request: yes

This PR adds the support of adaptive batch size for bucketing.
+It currently just supports tarred datasets
+train_ds.bucketing_batch_size is added which user may use to enable the adaptive batching and specify the batch sizes for the buckets
+when bucketing_batch_size is not set, train_ds.batch_size is going to be used for all buckets (fixed-size bucketing)
+bucketing_batch_size can be set as a list to explicitly specify the batch size for each bucket 
+When bucketing_batch_size is set, traind_ds.batch_size need to be set to 1
+if bucketing_batch_size is set to be an integer, then linear scaling is being used to scale-up the batch sizes for batches with shorted audio size. For example, setting train_ds.bucketing_batch_size=8 for 4 buckets would use these sizes [32,24,16,8] for different buckets.
+Adaptive batching may improve the training speed per epoch more than 2x but may affect the final accuracy of the model
KenLM with ASR
State: closed | Pull request: no
Labels: name: bug

Hi, I'm fine tuning Quartznet (character model) and CTC-Conformer (BPE model) on new language and the results are good. Then I use KenLM with both. I have some question:
1. When I download and see "3-gram.pruned.1e-7.arpa" file in [this tutorial](https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/asr/Offline_ASR.ipynb#scrollTo=-0Sk0C9-LmAR), I recognize that this file for word level, but Quartznet is character level, but It work well. Can you explain how it work ?
2. I train KenLM for CTC-Conformer (BPE model) according to [asr language model tutorial](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/asr_language_modeling.html), and the training was successful. But when I apply it with beamsearch to CTC-Conformer, it gives me irrelevant results(wer upto 40%), while using greedy search wer ~ 9%. I don't know what makes it not work. Can you give me some advice?
Thank you!
Machine translation shared embeddings
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

If my understanding is correct, an [MTEncDecModel](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/models/machine_translation/mt_enc_dec_model.py#L54) exposes no options for sharing embeddings between the encoder and decoder, but this seems to be the de facto standard. E.g. from the AAYN white paper:

> ...we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation

So MT models with shared vocab will learn separate embeddings for each token on both sides.

**Expected behavior**

An option like `cfg.share_embeddings = True` to tie the encoder/decoder embedding weights.
Fix Masked SE for Citrinets + export Limited Context Citrinet
State: closed | Pull request: yes

# Changelog
- Fix masking in SqueezeExcite block where padding causes incorrect calculation of global average. Slight improvements in WER found after retraining / fine-tuning. Masking should no longer be affected by batch size.
- Add sequence length tracking to Jasper block.

# NOTE:
- This is a breaking change. Models trained without this fix might show slightly increased WER in any batch size compared to before (other than batch size 1). 
- A minimal amount of fine-tuning of a previous checkpoint will fix this issue (and may obtain slightly better scores than before).

Signed-off-by: smajumdar <titu1994@gmail.com>
How to do "Riva Custom Voice" which is presented in 2021 GTC
State: closed | Pull request: no
Labels: name: feature

I would like to train and deploy a branded voice in Nvidia Riva. This is presented in NVIDIA GTC Nov. 2021 by the CEO Jensen Huang. He states that you can clone your voice with 30 mins of speech data and fine tune Riva model. 

Here is a link to a paper for the Custom Voice. https://blogs.nvidia.com/blog/2021/11/09/riva-enterprise-custom-voice-speech-ai/

In its current state, NVIDIA Riva has the option of two pipelines for Text to Speech : 

1. FastPitch and HiFi-GAN Pipeline 
2. Tacotron2 and Waveglow

I suspect that we should train Spectrogram generators (either FastPitch or Tacotron2) with selected voice recordings to be cloned via Nemo and then deploy using Riva. 

I have checked Nvidia NGC Catalog and there seems to be nothing about it. Could you please help me here?
Add support to modify nemo cache directory
State: closed | Pull request: yes

# Changelog
- Adds support to modify the nemo cache directory using an environment variable

# Usage

```sh
NEMO_CACHE_DIR="~/nemo_cache_dir/" python nemo_example_script.py
```

Closes https://github.com/NVIDIA/NeMo/issues/3185, https://github.com/NVIDIA/NeMo/issues/2025

Signed-off-by: smajumdar <titu1994@gmail.com>
Error while trying to use the NeMo's pretrained model based on TalkNet in r1.2.0 branch 
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

A clear and concise description of what the bug is.

**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model
Update TTS Dataset, FastPitch with TTS dataset and small improvements in HiFiGAN
State: closed | Pull request: yes

* Update TTSDataset to use speaker id
* Update FastPitch to use new TTS Dataset 
* Update HiFiGAN (more flexible `configure_optimizers`) and rename generator config for 44100 khz (it is just extended v1, but for 44100, it shouldn't be v4, IMO)


Signed-off-by: Oktai Tatanov <oktai.tatanov@gmail.com>
Fix RNNT ALSD Decoding Indexing bug
State: closed | Pull request: yes

# Bugfix
- Fix an off-by-one indexing issue with ALSD decoding that causes it to skip the 0th timestep of the acoustic time dimension (T)

Closes https://github.com/NVIDIA/NeMo/issues/3199

Signed-off-by: smajumdar <titu1994@gmail.com>
Speaker Recognition Verification tutorial fails
State: closed | Pull request: no
Labels: name: bug

Hi,
It's in the Speaker Recognition Verification tutorial.

While opening the jupyter notebook in collab, fails to import nemo.collections.asr



error message : "OSError: libtorch_hip.so: cannot open shared object file: No such file or directory"

it seems to be a pytorch or torchaudio wrong version


Thanks ;)
Index bug in RNN-T beam search algorithm ALSD
State: closed | Pull request: no
Labels: name: bug

Please refer to https://github.com/espnet/espnet/issues/3781 for details.

Codes locate at
https://github.com/NVIDIA/NeMo/blob/807b001f2e20ac6ab2930419c32076d242128c5d/nemo/collections/asr/parts/submodules/rnnt_beam_decoding.py#L785-L787
issue with setting checkpoints
State: closed | Pull request: no
Labels: name: bug

Hi, I have trouble with checkpoints. I wanna set checkpoints at end of each epoch. but it give me error.
It's error message is "pytorch_lightning.utilities.exceptions.MisconfigurationException: Invalid type provided for checkpoint_callback: Expected bool but received <class 'list'>."
Could you please help me to set checkpoints after end of each epoch and save them?
NMT validation examples with inputs
State: closed | Pull request: yes

This PR extends the log of NMT models.
It adds input to the translations and ground truths of the logged samples.

Example of updated log (notice `Input: `):

```
[NeMo I 2021-11-15 14:11:28 mt_enc_dec_model:389] Dataset name: Validation, Dataloader index: 0, Translation Examples:
[NeMo I 2021-11-15 14:11:28 mt_enc_dec_model:394]     E̲x̲a̲m̲p̲l̲e̲ ̲0̲:
[NeMo I 2021-11-15 14:11:28 mt_enc_dec_model:395]     Input:        Salto al azul
[NeMo I 2021-11-15 14:11:28 mt_enc_dec_model:396]     Prediction:   iereiereiereieres
 [NeMo I 2021-11-15 14:11:28 mt_enc_dec_model:397]    Ground Truth: Leap into the water
[NeMo I 2021-11-15 14:11:28 mt_enc_dec_model:394]     E̲x̲a̲m̲p̲l̲e̲ ̲1̲:
[NeMo I 2021-11-15 14:11:28 mt_enc_dec_model:395]     Input:        Reacciones
[NeMo I 2021-11-15 14:11:28 mt_enc_dec_model:396]     Prediction:   ristris
[NeMo I 2021-11-15 14:11:28 mt_enc_dec_model:397]     Ground Truth: Reactions
[NeMo I 2021-11-15 14:11:28 mt_enc_dec_model:394]     E̲x̲a̲m̲p̲l̲e̲ ̲2̲:
[NeMo I 2021-11-15 14:11:28 mt_enc_dec_model:395]     Input:        Informalidad
[NeMo I 2021-11-15 14:11:28 mt_enc_dec_model:396]     Prediction:   iereiereiereiere 
[NeMo I 2021-11-15 14:11:28 mt_enc_dec_model:397]     Ground Truth: Informality
```
"use_cer" in ASR training with pytorch lighnting.
State: closed | Pull request: no
Labels: name: bug

The CER value during the validation of the training process does not align with the inference process. And during the inference process, the execution is like the following part.
`wer_value = word_error_rate(hypotheses=hypotheses, references=references, use_cer=args.use_cer)`.
Meanwhile, the training codes are like:

```
@hydra_runner(config_path="conf_decay", config_name="config")
def main(cfg):
    profiler = PyTorchProfiler(emit_nvtx=True)
    trainer = pl.Trainer(**cfg.trainer,profiler=profiler)
    exp_manager(trainer, cfg.get("exp_manager", None))
    asr_model = EncDecCTCModel(cfg=cfg.model, trainer=trainer)
    pretrain_model = cfg.get("pretrain_model", None)
    if pretrain_model:
        if pretrain_model.endswith(".nemo"):
            asr_model = EncDecCTCModel.restore_from(pretrain_model)
        elif pretrain_model.endswith(".ckpt"):
            asr_model = EncDecCTCModel.load_from_checkpoint(pretrain_model)
        asr_model.setup_optimization(optim_config=cfg.model.optim)
        asr_model.setup_training_data(train_data_config=cfg.model.train_ds)
        asr_model.setup_validation_data(val_data_config=cfg.model.validation_ds)
    trainer.fit(asr_model)
    nvtx.range_pop()
    save_model = cfg.get("save_model", "quartznet.nemo")
    asr_model.save_to(save_model)
```

And the script for the execution like the following:
`python /ws/nemo/speech_to_text.py \
...
+use_cer=True
...`
The usage of parameter 'use_cer' is right? I didn't find any deep explanation for this parameter in Nemo reference documents.