renwei2024/nemo-github-issues huggingface dataset:
Speaker Verification
State: closed | Pull request: no
Labels: name: feature

Hello everyone, how are you? I've been messing with Speaker_Recogniton_Verification.ipynb together with DiegoJota's issue: https://github.com/NVIDIA/NeMo/issues/2719. I got good results with the following script for extracting embs

```
verification_model = nemo_asr.models.ExtractSpeakerEmbeddingsModel.restore_from('Train_Verification.nemo')
emb = verification_model.get_embedding(Person1.wav).cpu()[0]
emb2 = verification_model.get_embedding(Person2.wav).cpu()[0]
score = np.dot(emb, emb2) / ((np.dot(emb, emb) * np.dot(emb2, emb2)) ** 0.5)
score = (score + 1) / 2
```


As I said I got good results managing to impose a limit for scores higher than 0.7 being considered good within the recognition between two different audios from the same speaker, examples:

AudioCaio1.wav x AudioCaio2.wav score: 0.74455544
AudioCaio1.wav x AudioCaio3.wav score: 0.73555545
AudioCaio1.wav x AudioCaio4.wav score: 0.78555545
......

But in cases of being different speakers the scores are very close to 0.69, 0.63, 0.67.... examples:

AudioCaio1.wav x AudioDiego1.wav score: 0.695478
AudioCaio1.wav x AudioDiego2.wav score: 0.635495
AudioCaio1.wav x AudioAntonio1.wav score: 0.627564
AudioCaio1.wav x AudioAntonio2.wav score: 0.687899
AudioCaio1.wav x AudioJose1.wav score: 0.656455
AudioCaio1.wav x AudioJose2.wav score: 0.679564
.....

We arrived at this result through the training present in Speaker_Recogniton_Verification which we generate the "Train_Verification.nemo" used in the verification_model = nemo_asr.models.ExtractSpeakerEmbeddingsModel.restore_from() script above.

As the results saw that the error margin is very close, we want to increase this error margin. Would it be possible to improve the margin of error through a more diversified training? Or with audios of a certain size? Would you have any recommended amount of training base?
Should the people I'm evaluating be present within the training set?

We didn't find many recommendations or documentation talking about how to be trained in recognition matters.

For information - We use the script only for voices in Portuguese, we do not intend to use it for voices in English or other languages.
add zero shot intent model
State: closed | Pull request: yes

- New model class ZeroShotIntentModel is a subclass of TextClassificationModel
- Allows training/inference with either three classes ("entailment", "contradiction", and "neutral") or two classes ("entailment" and "not_entailment")
- ZeroShotIntentDataset and ZeroShotIntentInferenceDataset are subclasses of GLUEDataset
- For inference, users can specify multi_class = True or False and can supply their own hypothesis template
- `list_available_models` is not yet implemented. Tutorial notebook also needs to be updated with the name(s) of pretrained models users can get from NGC. These do not exist yet.
Fix window_stride calculation in DALI pipeline & Fix dither generation
State: closed | Pull request: yes

Fixes a bug in the calculation of the window stride uses for spectrogram calculation in the DALI implementation.
Fixes a bug in the generation of gaussian noise (dither). Generate noise with the shape of the audio, instead of a single sample (bug)

Signed-off-by: Joaquin Anton <janton@nvidia.com>
[BUG] DALI result is worse than normal pytorch data loader
State: closed | Pull request: no

Hello,

When running the same training or even the same validation script without DALI, we get the expected WER.
However, when we enable DALI using the same configuration, we get much worse WER.

I have created a Google Colab example that replicates this issue, which can be found here:
https://colab.research.google.com/drive/1Rz42EeQVDHhTso3kWvBp1wQc6x1FxXgY?usp=sharing

To make the debugging more straightforward, I have disabled the training on the example.
So simply, it performs testing with DALI and without DALI.

This is related to issue #2817 , which was fixed by @titu1994 .
Question in get_plurals and get_singulars in text normalization of en.
State: closed | Pull request: no
Labels: name: bug

Should both functions be something like `fst @ SINGULAR_TO_PLURAL` instead of `SINGULAR_TO_PLURAL @ fst`? Besides, it seems that these functions are not used actually.

https://github.com/NVIDIA/NeMo/blob/main/nemo_text_processing/text_normalization/en/graph_utils.py#L129

https://github.com/NVIDIA/NeMo/blob/main/nemo_text_processing/text_normalization/en/graph_utils.py#L129
nmt_webapp cant start?
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
python nmt_service.py  occcur that

![image](https://user-images.githubusercontent.com/49515380/133885574-5b293afd-6883-468b-aa74-bdf9adcbf648.png)

config.json

en-zh:"NGC/nmt_en_zh_transformer6x6"
zh-en:"NGC/nmt_zh_en_transformer6x6"
Joint Intent notebook error
State: closed | Pull request: no
Labels: name: bug

Hello everybody, I trying to run 'Joint_Intent_and_Slot_Classification.ipynb' and looks okay until the train part, when starts the evaluation part, return this error.

![MicrosoftTeams-image (4)](https://user-images.githubusercontent.com/47003942/133801257-44c743fc-e91a-4d7f-b856-08a1c9d3f070.png)
![MicrosoftTeams-image (3)](https://user-images.githubusercontent.com/47003942/133801198-c6d4c462-26f1-482a-8e43-1ad77af61207.png)

This script is on 'r1.3.0' version, and everything is okay when I run on 'r1.2.0'. I just run the original script in both cases. Could be a bug? Am I doing something wrong?
speaker_reco_infer.py - pytorch version issue?
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

**speaker_reco_infer.py loads the model and manifestfiles and then breaks,
I guess its again a pytorch issue? wanted to use the model from yesterday:**

[NeMo W 2021-09-17 12:18:42 patch_utils:49] torch.stft() signature has been updated for PyTorch 1.7+
    Please update PyTorch to remain compatible with later versions of NeMo.

RuntimeError: Argument #4: Padding size should be less than the corresponding input dimension, but got: padding (256, 256) at dimension 2 of input [1, 1, 2]

**Steps/Code to reproduce bug**

Container:
nvcr.io/nvidia/nemo:1.2.0

but installed:
python -m pip install git+https://github.com/NVIDIA/NeMo.git@'main'
python -m pip install pytorch_lightning==1.4.2

(like used/working in the goolge colab, 
I also tried nemo 1.2 and pytorch-lightning 1.3.8 and nemo 1.3 and recent 1.4.7 later on)

run:
https://github.com/NVIDIA/NeMo/blob/48fe9e69feba7651694fd6ae0a096a0655ed601c/examples/speaker_tasks/recognition/speaker_reco_infer.py

with:
model, train.json from:

https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb

added:
test.json and bonian.wav
{"audio_filepath": "bonian.wav", "offset": 0, "duration": 11.370666666666667, "label": ""}

**Expected behavior**

working :-)

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version nemo 1.2 container
- PyTorch version tryed 1.3.8, 1.4.2, 1.4.7
- Python version 3.8.10

**Additional context**
https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/speaker_recognition/results.html

A little bit more explanation to the inference part would be nice, 
like a link to the script that i was using here, and also how to use the embedding that is created at the end of the jupyter notebook
Speaker Diarization Finetune
State: closed | Pull request: no
Labels: name: feature

I ran **Speaker_Diarization_Inference.ipynb** notebook for getting diarization results on my audio data using the pretrained model. But as my data is quite noisy, the diarization results are not at as expected. Is there any way I can finetune the diarization module for my data? 
If you can provide a notebook for finetuning the model, it would be really helpful.
Update collection of pretrained models for RNNT
State: closed | Pull request: yes

# Changelog
- Add pretrained checkpoints of RNNT models
  - ContextNet RNNT trained on ASR SET
  - ContextNet RNNT trained on MLS
  - Conformer RNNT trained on ASR SET
  - Conformer RNNT trained on MLS
- Add pretrained model for ECAPA TDNN (Speaker model)

Signed-off-by: smajumdar <titu1994@gmail.com>
Add support for `LightningDataModule`
State: closed | Pull request: no
Labels: name: feature

One of the [recommended](https://pytorch-lightning.readthedocs.io/en/latest/guides/data.html) ways of dealing with data in pytorch-lightning is using the `LigtningDataModule`. This allows decoupling the model code from the data code. Then the training code would look something like this:

```python
trainer = Trainer(...)
model = MyModel(...)
data = MyDataModule(...)

trainer.fit(model, data)
```

The biggest blocker with using this at the moment is that the Nemo ModelPT abstract class requires you to implement all the data setup methods within the model itself, if you remove them an error is thrown. 

I have removed the inheritance for the `MTEncDecModel` and moved all of the data preprocessing and loading to a LightningDataModule class and already observed a number of benefits:
- Previously all of the data partitions would be preprocessed and loaded into memory every time the Model class gets initialized. DataModule seems to be smarter where it only prepares the data right before it's needed. So for example validation sanity check runs and only then, right before training, does training data get loaded.
- As another addition to the point above, the data loading and preprocessing only happens after all of the DDP processes have been initialized allowing each process do to its own thing in parallel. Whereas before it seemed a lot more sequential and took a long time.
- The setup function allows adding functionality from `MTDataPreproc` module so that all of the data tasks live within a single place.
- It incentivizes us to not use trainer-specific parameters like `world_size` for data preprocessing as all of that should be handled by the DataLoader.

**Describe the solution you'd like**

Essentially there should at least be an option to use LigtningDataModule, and ideally, the models should use it by default to be more aligned with PyTorch recommendation.

**Describe alternatives you've considered**

You could also possibly solve the above error by implementing functions required by the abstract class without returning anything, but I am not sure how it would play with LightningDataModule, and it's not a very clean solution. (Came up with it while writing this up, will give it a go)
Bugs in `CharTokenizer` class
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
Apparently class `CharTokenizer` needs to be rewritten.
1. The tokenizer does not add special tokens such as `<EOS>` and `<CLS>` automatically, so it is not consistent with tokenizers like `yttm`.
2. To add special tokens, it is not enough to insert something like `"<EOS>"` into input text because substring `"<EOS>"` will be interpreted not like special token but as 5 tokens `['<', 'E', 'O', 'S', '>']`. So a class user has to identify a character (e.g. `'@'`) which does not occur in a tokenized text, insert it into the text in required positions, and pass this character as a special token parameter to the class constructor. It is inconvenient. 
3. A vocabulary file cannot contain space character. If space character is present in `self.vocab` it will be replaced with an empty string. So space character has to be passed as a special token. This feature is not documented.
4. Special token attributes like `self.unk_token` are not defined but used in properties like `unk_id`.
AttributeError: 'NoneType' object has no attribute 'get'
State: closed | Pull request: no

I am facing issue when I train a tansformer LM, its giving attribute error  'NoneType' object has no attribute 'get. Please help me to resolve this
I have update the tansformer config file as given below

train_ds = "dummydata.txt"
vocab="vocab.txt"
config.model.train_ds.file_name =  train_ds
config.model.validation_ds.file_name =  train_ds
config.model.test_ds.file_name=train_ds
config.model.tokenizer.vocab_file=  vocab 
config.model.tokenizer.tokenizer_model= "tokenizer_spe_v2000"


AttributeError                            Traceback (most recent call last)
<ipython-input-43-cb4d43dca025> in <module>
      2 
      3 
----> 4 model = nemo_nlp.models.language_modeling.TransformerLMModel(cfg=config.model, trainer=trainer)
      5 

/opt/conda/lib/python3.6/site-packages/nemo/collections/nlp/models/language_modeling/transformer_lm_model.py in __init__(self, cfg, trainer)
     51         self.dataset_cfg = cfg.dataset
     52 
---> 53         vocab_file = cfg.language_model.get("vocab_file", None)
     54 
     55         if vocab_file is not None:

AttributeError: 'NoneType' object has no attribute 'get
Validation not running when using `TarredTranslationDataset`
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

The validation step is not running when training a [mt_enc_dec](https://github.com/NVIDIA/NeMo/blob/r1.2.0/nemo/collections/nlp/models/machine_translation/mt_enc_dec_model.py) model using `TarredTranslationDataset`. As a result, the checkpoint call back is not invoked and no validation metrics are reported. The initial validation sanity check runs fine and then training proceeds reporting a decrease in train_loss over time, but the validation is never invoked again.

**Steps/Code to reproduce bug**
I am using a cleaned wmt dataset with 3.6M rows and training using ddp.

Here are the relevant bits from the config:
```yaml
model:
  ...
  train_ds:
    src_file_name: ???
    tgt_file_name: ???
    use_tarred_dataset: true # if true tar_file_name and meta_file_name will be used (or created automatically)
    # config for preprocessing training data and creating a tarred datset automatically
    tar_file_prefix: parallel # prefix for tar file names
    tar_files: null # if data has already been preprocessed (rest of config ignored)
    metadata_file: null # metadata for tarred dataset
    lines_per_dataset_fragment: 1000000 # Number of lines to consider for bucketing and padding
    num_batches_per_tarfile: 100 # Number of batches (pickle files) within each tarfile
    tar_shuffle_n: 100 # How many samples to look ahead and load to be shuffled
    shard_strategy: scatter # tarred dataset shard distribution strategy
    n_preproc_jobs: -2 # number of processes to use for data preprocessing (-2 means all but 2)
    tokens_in_batch: 8000
    clean: true
    max_seq_length: 512
    shuffle: true
    num_samples: -1
    drop_last: false
    pin_memory: false
    num_workers: 0

  validation_ds:
    src_file_name: ???
    tgt_file_name: ???
    tokens_in_batch: 1024
    clean: false
    max_seq_length: 512
    shuffle: false
    num_samples: -1
    drop_last: false
    pin_memory: false
    num_workers: 8

trainer:
  gpus: 8
  num_nodes: 1
  max_steps: 500000
  amp_level: O2 # O1/O2 for mixed precision
  precision: 16 # Should be set to 16 for O1 and O2, default is 16 as PT ignores it when am_level is O0
  accelerator: ddp
  checkpoint_callback: false
  logger: false
  log_every_n_steps: 50  # Interval of logging.
  check_val_every_n_epoch: 1
```

Here is the screenshot of the logs:
![Screenshot 2021-09-13 at 16 51 47](https://user-images.githubusercontent.com/22484212/133116307-027ccbc0-978d-4437-bea7-d20414d2d4d6.png)

**Environment overview **

- Environment Location: Docker on Kubernetes
- Method of NeMo install: using helm chart using nvcr.io/nvidia/nemo:1.2.0

**Additional context**

Training on 8 A100 GPUs using DGX box
[bug] Cannot resolve schedulerDictConfig Error in ASR_with_NeMo.ipynb
State: closed | Pull request: no

I was trying to train the ASR Model using Nemo on Colab but I was facing this issue on this row

trainer.fit(first_asr_model)

ValueError: Cannot resolve schedulerDictConfig'. Available optimizers are : dict_keys(['WarmupPolicy', 'WarmupHoldPolicy', 'SquareAnnealing', 'CosineAnnealing', 'NoamAnnealing', 'WarmupAnnealing', 'InverseSquareRootAnnealing', 'SquareRootAnnealing', 'PolynomialDecayAnnealing', 'PolynomialHoldDecayAnnealing', 'StepLR', 'ExponentialLR', 'ReduceLROnPlateau', 'CyclicLR'])
Take the exact tense of each word
State: closed | Pull request: no

Hi
I need to take the exact time each word was said to build a transcription "timeline".
I saw that this script exemplifies how to accomplish this task
https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/asr/Offline_ASR.ipynb#scrollTo=a-LSg9dSL_O1

However, as it was done, I only have the information about the final time of each word and the initial time is determined by the final time of the previous word.
This way I'm not getting good results, because if there is a silence interval between one word and another, the "timeline" will inform you that the next word has already started to be spoken.
Is there an alternative to recover the exact time of each word?
CTC segmentation fails with dict' object has no attribute 'get_node flag' when using other pretrained model
State: closed | Pull request: no
Labels: name: bug

**Description of bug**: 

The cut-audio.py script fails if using models other than QuartzNet15x5Base-En, e.g. stt_en_quartznet15x5 or stt_en_jasper10x5dr. 

Traceback (most recent call last):
  File "tools/ctc_segmentation/scripts/cut_audio.py", line 299, in <module>
    stats = process_alignment(alignment_file, args)
  File "tools/ctc_segmentation/scripts/cut_audio.py", line 205, in process_alignment
    add_transcript_to_manifest(
  File "tools/ctc_segmentation/scripts/cut_audio.py", line 59, in add_transcript_to_manifest
    transcripts = get_transcript(manifest_original, asr_model, batch_size)
  File "tools/ctc_segmentation/scripts/cut_audio.py", line 93, in get_transcript
    asr_model.setup_test_data(
  File "/home/markk/miniconda3/envs/dlenv/lib/python3.8/site-packages/nemo/collections/asr/models/ctc_models.py", line 482, in setup_test_data
    self._test_dl = self._setup_dataloader_from_config(config=test_data_config)
  File "/home/markk/miniconda3/envs/dlenv/lib/python3.8/site-packages/nemo/collections/asr/models/ctc_models.py", line 348, in _setup_dataloader_from_config
    audio_to_text_dataset.inject_dataloader_value_from_model_config(self.cfg, config, key='sample_rate')
  File "/home/markk/miniconda3/envs/dlenv/lib/python3.8/site-packages/nemo/collections/asr/data/audio_to_text_dataset.py", line 55, in inject_dataloader_value_from_model_config
    with open_dict(dataloader_cfg):
  File "/home/markk/miniconda3/envs/dlenv/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/markk/miniconda3/envs/dlenv/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 943, in open_dict
    prev_state = config._get_node_flag("struct")
AttributeError: 'dict' object has no attribute '_get_node_flag'

**Steps/Code to reproduce bug**

The script I am using as follows:
MODEL=stt_en_quartznet15x5
OUTPUT_DIR=output
TOOLS_DIR=tools/ctc_segmentation/scripts
DATA_DIR=data
WINDOW=16000
OFFSET=0
THRESHOLD=-5

python $TOOLS_DIR/prepare_data.py \
--audio_format '.wav'  \
--in_text $DATA_DIR  \
--output_dir $OUTPUT_DIR/processed  \
--language 'eng' \
--model $MODEL \
--audio_dir $DATA_DIR

python $TOOLS_DIR/run_ctc_segmentation.py \
--output_dir $OUTPUT_DIR \
--data $OUTPUT_DIR/processed \
--model $MODEL \
--window_len $WINDOW \
--no_parallel

python $TOOLS_DIR/cut_audio.py \
--output_dir $OUTPUT_DIR \
--model $MODEL \
--alignment $OUTPUT_DIR/segments/ \
--threshold $THRESHOLD \
--offset $OFFSET

**Environment **

nemo v1.3.0 
python 3.8.11
Fix maes and lr_scheduler division by zero
State: closed | Pull request: yes

# Bugfix
- Fix maes num steps < 2, force to be >= 2 for now until auto tuning is ready
- Fix lr scheduler division by 0 error if num_workers is 0
- Fix dataloader injection when provided base dictionary and not DictConfig
- Fix DALI on CPU

Signed-off-by: smajumdar <titu1994@gmail.com>
NMT input ID validity test for negative values
State: closed | Pull request: yes

This PR extends validity checks of src/tgt input IDs:

1. Added a test for special tokens being valid (i.e., non-negative) for Sentencepiece tokenizer.
2. Extended MTEncDecModel id validity test to include non-negativity.
3. Added `model.validate_input_ids` to `MTEncDecModel` config to allow disabling id validity (`true` by default).
AttributeError: 'dict' object has no attribute '_get_node_flag'
State: closed | Pull request: no
Labels: name: bug

Hi, I installed the nemo 1.3.0 latest version recently and find there is an attribute error as shown below. Although I have reinstall my environment for a few times, it still have this error. Has anyone encounter this recently with NeMo 1.3.0?
my code：
quartznet = nemo_asr.models.EncDecCTCModel.restore_from("./model/QuartzNet15x5-Zh-NR.nemo",strict=False)

error：
 AttributeError: 'dict' object has no attribute '_get_node_flag'
Cannot reproduce environment 
State: closed | Pull request: no
Labels: name: bug

I can't reproduce the environment to run the code in this tutorial https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/zh/latest/nlp/bert_pretraining.html , I have followed pip installation instructions but I keep getting errors when I import things 



AttributeError: module 'nemo.collections.nlp.data' has no attribute 'NemoBertTokenizer'
>>> ckpt_callback = nemo.core.CheckpointCallback(folder=nf.checkpoint_dir,
...                     load_from_folder=args.load_dir)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'nemo' is not defined
>>> import nemo
>>> ckpt_callback = nemo.core.CheckpointCallback(folder=nf.checkpoint_dir,
...                     load_from_folder=args.load_dir)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'nemo.core' has no attribute 'CheckpointCallback'
New TTSDataset, tts tokenizers and g2ps
State: closed | Pull request: yes

This PR adds TTSDataset, english tokenizers (char and phoneme) and g2ps, base tokenizer and g2p classes for any other language.

I have fixed many bugs in [nemo/collections/asr/data/vocabs.py](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/data/vocabs.py) and decided to put result in TTS code, because I want to add new classes to [nemo/collections/tts/torch/data.py](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/torch/data.py) and suggest to use it as default for TTS pipelines.
Exporting feature extractor (Preprocessor) as a .onnx model. 
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**

I have follow your [example](https://github.com/NVIDIA/NeMo/blob/v1.0.0b3/scripts/convasr_to_onnx.py) to convert a single asr model (.nemo) into encoder and decoder. But, it does not allow to export the Preprocessor model as .onnx. When i checked the functions that attached to Preprocessor, the .export() function is not implemented. Is there a reason why don't you allow to export the Preprocessor part out from asr model?   
  
**Describe the solution you'd like**

I would like to find out how to export the Preprocessor to .onnx format trough a .export() function as you already implemented for encoder and decoder as follows (Referred from [example](https://github.com/NVIDIA/NeMo/blob/v1.0.0b3/scripts/convasr_to_onnx.py)).

![Screenshot from 2021-09-08 09-07-43](https://user-images.githubusercontent.com/15320876/132434732-ac1de0ed-d1d4-432a-a8b0-7ed048676199.png)
         
**Describe alternatives you've considered**

There's a way to split the asr model (.nemo) to sub modules using [.extract_state_dict_from()](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/v1.0.0b4/api-docs/nemo.html#nemo.core.ModelPT.extract_state_dict_from)  and it saves as a ckpt. Is there a way to convert the preprocessor.ckpt that can load as a triton model ? 

![Screenshot from 2021-09-08 09-24-52](https://user-images.githubusercontent.com/15320876/132436238-37b29291-421a-451d-bf4c-db6d27b2810b.png)

![Screenshot from 2021-09-08 09-24-15](https://user-images.githubusercontent.com/15320876/132436195-591dad94-94d8-440a-9ca0-ee5bd7fa15e4.png)
Machine Translation SacreBLEU is different when running `trainer.test()` vs directly calling `corpus_bleu()`
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**


The SacreBLEU score in [mt_enc_dec_model](https://github.com/NVIDIA/NeMo/blob/r1.3.0/nemo/collections/nlp/models/machine_translation/mt_enc_dec_model.py) is computed using detokenized target text:
```
286:        ground_truths = [self.decoder_tokenizer.ids_to_text(tgt) for tgt in np_tgt]
287:        ground_truths = [self.target_processor.detokenize(tgt.split(' ')) for tgt in ground_truths]
```
which is the likely reason for the difference in the BLEU score I get when computing directly using the `corpus_bleu` method from sacreblue. 

As an example, on the newstest2014 en -> de dataset, using `trainer.test()` I get `30.14` and running `model.translate()` followed by `corpus_bleu()` results in `30.42`. The difference is even larger on my own dataset (1-2 units).

Ideally, the BLEU score should be computed directly on the raw target text so it is easy to reproduce the metric externally. 

A potential solution could be to add raw_tgt as an additional output of the TranlationDataset.

**Expected behavior**

SacreBLEU score should be identical when computed within the model and externally.

**Environment overview (please complete the following information)**

 - Environment Location: Docker on Kubernetes
 - Method of NeMo install: using helm chart using `nvcr.io/nvidia/nemo:1.2.0`
Tune frame length while training Tacotron?
State: closed | Pull request: no
Labels: name: bug

Thank you for building a great open source project.

I'm trying to train Tarcotron using Korean datasets.

While training I just noticed that depending on `max_duration`, frame padding length will be changed, below images show `val_target` Mel spectrogram where `max_duration` is over 3.5sec and below 3.5sec. 

![image](https://user-images.githubusercontent.com/32811094/132269756-fceb2fc5-fab8-4b18-9a82-b8f22e8f8a82.png)

and I think it makes `valid_prediction` unstable. like the below images, Tacotron generates some noise at the end of a frame.

![image](https://user-images.githubusercontent.com/32811094/132270032-5b52f505-9cb9-4cdc-bc53-9cda5e29a83c.png)

Is there any way to tune this problem? or is it normal behavior for Tacotron?
ValueError: Error instantiating 'nemo.collections.asr.modules.conv_asr.SpeakerDecoder' : invalid literal for int() with base 10: ','
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Following "NGC Pretrained Checkpoints"

from

https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/speaker_recognition/results.html

**Steps/Code to reproduce bug**

Running: 

nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained(model_name="speakerrecognition_speakernet")


**Expected behavior**

no error when loading pretrained model 

**Environment overview (please complete the following information)**

- google colab:
https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/speaker_recognition/Speaker_Recognition_Verification.ipynb

same on dgx with working riva-client container that has been used for other applications

**Environment details**

- google colab
- riva-client 1.4 with nemo "main"
torch.stft() signature has been updated for PyTorch 1.7+     Please update PyTorch to remain compatible with later versions of NeMo.  
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

[NeMo W 2021-09-06 11:58:47 patch_utils:50] torch.stft() signature has been updated for PyTorch 1.7+
    Please update PyTorch to remain compatible with later versions of NeMo.

and followed by 

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in _pad(input, pad, mode, value)
   4157             assert len(pad) == 2, "3D tensors expect 2 values for padding"
   4158             if mode == "reflect":
-> 4159                 return torch._C._nn.reflection_pad1d(input, pad)
   4160             elif mode == "replicate":
   4161                 return torch._C._nn.replication_pad1d(input, pad)

RuntimeError: Argument #4: Padding size should be less than the corresponding input dimension, but got: padding (256, 256) at dimension 2 of input [1, 2, 2]

also in this notebook, next to the AN4 Source not available problem:

Original Cell: restored_model.setup_finetune_model(config.model)

TypeError                                 Traceback (most recent call last)

<ipython-input-68-be381c0bac3a> in <module>()
----> 1 restored_model.setup_finetune_model(config.model)

if i change to 
Cell: restored_model.setup_finetune_model(model_config = config.model)

TypeError: setup_finetune_model() missing 1 required positional argument: 'model_config'
NameError                                 Traceback (most recent call last)
<ipython-input-49-fec8ad3e9b0e> in <module>()
----> 1 restored_model.setup_finetune_model(self, model_config=config.model)

NameError: name 'self' is not defined

same with this 
cell: restored_model.set_trainer(trainer_finetune)

TypeError                                 Traceback (most recent call last)
<ipython-input-67-22f1286fa285> in <module>()
----> 1 restored_model.set_trainer(trainer_finetune)
      2 log_dir_finetune = exp_manager(trainer_finetune, config.get("exp_manager", None))
      3 print(log_dir_finetune)

TypeError: set_trainer() missing 1 required positional argument: 'trainer'

**Steps/Code to reproduce bug**

Cell: trainer.fit(speaker_model)

in 

https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/speaker_recognition/Speaker_Recognition_Verification.ipynb


**Expected behavior**
(as expected by the ppl that made this notebook.... Colab training should work without bugfixing :-))

Torch 1.9 is installed, no updates possible as it seems...

**Environment overview (please complete the following information)**

torch @ https://download.pytorch.org/whl/cu102/torch-1.9.0%2Bcu102-cp37-cp37m-linux_x86_64.whl
torch-stft==0.1.4
torchaudio==0.9.0
torchmetrics==0.5.1
torchsummary==1.5.1
torchtext==0.10.0
torchvision @ https://download.pytorch.org/whl/cu102/torchvision-0.10.0%2Bcu102-cp37-cp37m-linux_x86_64.whl

**Environment details**

GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
AttributeError: type object 'EncDecCTCModel' has no attribute 'from_pretrained'
State: closed | Pull request: no

Hi, I installed the nemo 1.3.0 latest version recently and find there is an attribute error as shown below. Although I have reinstall my environment for a few times, it still have this error. Has anyone encounter this recently with NeMo 1.3.0? 
=================================================================================
my code: 
char_model =  nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="stt_zh_citrinet_1024_gamma_0_25")

error:
AttributeError: type object 'EncDecCTCModel' has no attribute 'from_pretrained'
=================================================================================

Thanks!
Error: module 'nemo.collections.nlp.data' has no attribute 'BERTPretrainingDataDesc'
State: closed | Pull request: no

I have raw text that I want to preprocess for Pretraining BERT. I followed all the steps mentioned in https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/zh/latest/nlp/bert_pretraining.html . I got stuck while creating the tokenizer where I get an error saying that there is no attribute named BERTPretrainingDataDesc. Any suggestion is appreciated. 

OS version: Ubuntu, 20.04.2 LTS
Python version: 3.6.13
PyTorch version: 1.9.0
Memory allocation when transcribing
State: closed | Pull request: no
Labels: name: bug

Hi

I'm building a communication API for transcription with NeMo and I ran into a critical infinite memory allocation problem.
When using a lib that monitors memory allocation, I noticed that the "transcribe" method used on top of a restored model is allocating memory every time when transcribing a new file.
My API basically works like this:
When starting, I load the .nemo model creating the asr_model, then open a listener and keep receiving hundreds of files on the network, carrying out their individual transcription on top of that same nemo_asr object.
At this point the problem comes up, because every time I call the transcribe method it keeps allocating more and more memory... I left it running for a night and it allocated 50GB of ram (considering a small stream of audios to transcribe)
Is there an alternative so I can work around this problem? I don't think it's perfect to have to create a new nemo_asr object every time I transcribe, so I keep it in this format.
Here are two examples below:
![Ev1](https://user-images.githubusercontent.com/49459897/131869414-b711961f-0640-4f73-8078-1e15d6832787.PNG)
![Ev2](https://user-images.githubusercontent.com/49459897/131869432-3e259a63-a8c2-4165-bdb2-88ada3ffc860.PNG)

SO: Ubuntu 18.04
Nemo: 1.2.0
Documentation for usage of tarred dataset created by`create_tarred_parallel_dataset.py` is wrong
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

According to documentation on page https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/v1.3.0/nlp/machine_translation.html#tarred-datasets-for-large-corpora the second way to use tarred dataset is

**Quote**
2. Using a separate script without Hydra.

Tarred datasets for parallel corpora can also be created with a script that doesn’t require specifying a configs via Hydra and just uses Python argparse.

For example:
```
python examples/nlp/machine_translation/create_tarred_parallel_dataset.py \
  --shared_tokenizer \
  --clean \
  --bpe_dropout 0.1 \
  --src_fname train.tokenized.en \
  --tgt_fname train.tokenized.es \
  --out_dir /path/to/preproc_dir \
  --vocab_size 32000 \
  --max_seq_length 512 \
  --min_seq_length 1 \
  --tokens_in_batch 8192 \
  --lines_per_dataset_fragment 1000000 \
 --num_batches_per_tarfile 200
```
*You can then set `model.preproc_out_dir=/path/to/preproc_dir` and `model.train_ds.use_tarred_dataset=true` to train with this data.*
**End quote**

If I set parameter `model.preproc_out_dir` to directory with tar files and train dataset config is
```
  train_ds:
    src_file_name: null
    tgt_file_name: null
    use_tarred_dataset: True
    tar_file_prefix: parallel
```
calling `python enc_dec_nmt.py` results in error
```
Traceback (most recent call last):
  File "enc_dec_nmt.py", line 121, in main
    MTDataPreproc(cfg=cfg.model, trainer=trainer)
  File "/home/lab/anaconda3/envs/nemo/lib/python3.8/site-packages/nemo/collections/nlp/data/machine_translation/preproc_mt_data.py", line 181, in __init__
    raise ValueError(
ValueError: src_file_name and tgt_file_name needed to create tarred dataset but could not be found.
```

**Environment overview (please complete the following information)**

 - Environment location: Bare-metal
 - Method of NeMo install: pip

**Environment details**

- OS version: Ubuntu 20.04.2 LTS
- PyTorch version: 1.8.1
- Python version: 3.8
Training performance in GPU Tesla vs GPU RTX
State: closed | Pull request: no

Hi,

I've been working on a dataset for a long time and slowly I see that my simple GPU isn't playing such a good role.
When searching I see several people using RTX and Tesla. Does anyone know what is the advantage of a Tesla GPU over an RTX GPU? Going into the question of price, I see that an RTX it's much more affordable...
For example, an RTX 3090 versus a Tesla V100, on comparison sites, I see that RTX stands out in a number of ways. Someone who has experience and can share, thank you in advance.
⁇ in predicted text
State: closed | Pull request: no

The conformer network sometimes predicts "⁇" symbols. After greedy decoding, all the outputs have zero index, which corresponds to <"unk"> token. And finally, after applying sentencepiece tokenization I am getting  ⁇ symbols. 


Btw, all my transcripts successfully tokenized without producing any <"unk"> symbols. 

What can cause "⁇"  symbol?
Get feature map after encoder block in EncDecCTCModel
State: closed | Pull request: no

I have trained a **EncDecCTCModel** using quartznet as the base architecture. While doing inference, I would like to get feature maps after the encoder and the decoder block separately. Using the transcribe function, I am able to get the audio transcripts but I am not able to check the feature maps in the intermediate layers. Is there any way to check out this?
Error converting nemo file to riva
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When I attempt to convert a nemo file (fine tuned from the conformer model), I get the following error

```
nemo2riva --out moneypenny-confirmer-v1.riva /data/models/moneypenny-confirmer-v1.nemo
[NeMo W 2021-08-28 08:49:32 optimizers:47] Apex was not found. Using the lamb optimizer will error out.
INFO: Logging level set to 20
INFO: Restoring NeMo model from '/data/models/moneypenny-confirmer-v1.nemo'
ERROR: Nemo2Jarvis: Failed to restore model from NeMo file : /data/models/moneypenny-confirmer-v1.nemo. Please make sure you have the latest NeMo package installed with [all] dependencies.
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/riva/bin/nemo2riva", line 8, in <module>
    sys.exit(nemo2riva())
  File "/home/ubuntu/anaconda3/envs/riva/lib/python3.8/site-packages/nemo2riva/cli/nemo2riva.py", line 50, in nemo2riva
    Nemo2Riva(args)
  File "/home/ubuntu/anaconda3/envs/riva/lib/python3.8/site-packages/nemo2riva/convert.py", line 42, in Nemo2Riva
    raise e
  File "/home/ubuntu/anaconda3/envs/riva/lib/python3.8/site-packages/nemo2riva/convert.py", line 35, in Nemo2Riva
    model = ModelPT.restore_from(restore_path=nemo_in)
  File "/home/ubuntu/anaconda3/envs/riva/lib/python3.8/site-packages/nemo/core/classes/modelPT.py", line 270, in restore_from
    instance = cls._save_restore_connector.restore_from(
  File "/home/ubuntu/anaconda3/envs/riva/lib/python3.8/site-packages/nemo/core/connectors/save_restore_connector.py", line 136, in restore_from
    instance = calling_cls.from_config_dict(config=conf)
  File "/home/ubuntu/anaconda3/envs/riva/lib/python3.8/site-packages/nemo/core/classes/common.py", line 473, in from_config_dict
    instance = cls(cfg=config)
TypeError: Can't instantiate abstract class ModelPT with abstract methods list_available_models, setup_training_data, setup_validation_data

```

**Steps/Code to reproduce bug**

ngc registry resource download-version nvidia/riva/riva_quickstart
conda create --name riva python=3.8
conda activate riva
cd riva_quickstart_v1.5.0-beta
pip install nemo-toolkit[all]
pip3 install riva_api-1.5.0b0-py3-none-any.whl
pip3 install nemo2riva-1.5.0b0-py3-none-any.whl
pip3 install nvidia-pyindex
pip3 install nemo2riva
nemo2riva --out moneypenny-confirmer-v1.riva /data/models/moneypenny-confirmer-v1.nemo

-- Running nemo2riva then throws the error outlined above --

**Expected behavior**

The nemo file should be converted into a riva file

**Environment overview (please complete the following information)**

 - AWS EC2 instance based on the Nvidia image, T4 GPU

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version : Ubuntu 18.04.5 LTS
- PyTorch version : 1.9.0+cu102
- Python version : Python 3.8.10
Does we have any docs of how to create new grammar/rules for inverse text normalization for a new language?
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**

I want to create ITN grammar for the Ukrainian language. As we already have one for Russian I think it would be easy to create a new one for Ukrainian (as it has almost the same rules a Russian).

**Describe the solution you'd like**

I'm looking for some docs or guides on how to create a new grammar for Inver Text Normalization for a new language?

**Describe alternatives you've considered**


**Additional context**
Initial support for retrieval models
State: closed | Pull request: yes

This pull request provides intial support for incorporating retrieval models by 

- creating a new `RetrievalTranslationDataset` 
- adding support for training with nearest neighbors by providing a file with indices to them and relevant src and tgt files from where those sentences are to be retrieved
- adding a new inference script `nmt_transformer_infer_retrieval.py` which adds nearest neighbor to the sentences at test time
Issue in saving the checkpoint after epoch
State: closed | Pull request: no
Labels: name: bug

**Issue in saving the checkpoint after epoch**
After the epoch runs successfully it tries to save the model checkpoint file in storage and throws exception

**MisconfigurationException: ModelCheckpoint(monitor='val_loss') not found in the returned metrics: ['lr', 'train_loss']**


**Steps/Code to reproduce bug**
Following the IMDB dataset structures put up custom dataset for classification model training and run the training epochs. While it tries to save the checkpoint exception is generated.

**Expected behavior**
Model should be saved to the storage after each epoch or number of epochs and next iteration should be running.

[Screenshot from 2021-08-27 12-57-25](https://user-images.githubusercontent.com/41611198/131109161-0252e098-3d4c-46d1-be31-b62a61d00e90.png)

**Environment overview (please complete the following information)**

 - Environment location: Kubernetes pod
 - Method of NeMo install: NeMo image as base

**Additional context**
NeMo version 1.0.1 and Python version 3.8
Model used is bert base uncased
View NVidia/Riva advertised pipeline in nemo
State: closed | Pull request: no
Labels: name: feature

The configuration of nvidia/Riva advertised  benchmarks, **pipeline tab** is unclear, 
https://developer.nvidia.com/deep-learning-performance-training-inference 

Goal: replicate the models in Nemo and see the config.
I.e, have a clear config file per Nemo model.

Examples:
- which model is used for TTS?
- what is the specific Jasper config used there?
Fine-tune stt_zh_quartznet15x5 model  with custom dataset
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

1. Output reference is not correct on chinese, could it affect the model?
![image](https://user-images.githubusercontent.com/25794106/130772485-45c4e6c0-470c-431f-8230-1ef2a296dc23.png)

  I follow the  link below:
  https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/asr/10_ASR_CTC_Language_Finetuning.ipynb#scrollTo=x0i8hvt688hc
2. I have four gpus, But  just use one; 





**Steps/Code to reproduce bug**

 step 1: I  prepare the dataset ,  
          **manifest files:**
          train_manifest="dataset/manifest_train.json"
          test_manifest = "dataset/manifest_test.json"
          dev_manifest = "dataset/manifest_dev.json"
         **charset:**
          train_charset = get_charset(train_manifest_data)
          dev_charset = get_charset(dev_manifest_data)
          test_charset = get_charset(test_manifest_data)

      
step 2: load model config , update the config 
            freeze_encoder = True
            cfg.train_ds.manifest_filepath  = "train_manifest,dev_manifest"

            use_cer = True
            log_prediction = True
            model._wer.use_cer = use_cer
            model._wer.log_prediction = log_prediction

step 3: train 
          
![image](https://user-images.githubusercontent.com/25794106/130769707-8ac3b2d4-dc08-4878-9d1c-8c4a907f34ba.png)

         
**Expected behavior**

the reference part  should be the correct sentence；
as you see ,the output is out-of-order ， I expect the out  is  1 2 3 4 ,but the result is 2 1 3 4 ;
**Environment overview (please complete the following information)**

 - Environment location: local server  DGX V100
 - Method of NeMo install:  pip install 


**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version :ubuntu 18.04
- PyTorch version:pytorch 1.9.0
- Python version :python3.7

**Additional context**

This is my train script:
[nemo_train.txt](https://github.com/NVIDIA/NeMo/files/7046204/nemo_train.txt)
Deploy speaker diarize model?
State: closed | Pull request: no
Labels: name: feature

Hello,

I trained VAD model and Speaker Net model to get timestamp of who is speaker in audio.
After speaker diarize, i want to pass the signal of speech to ASR model to get both who is speaker and text of speech.
Some thing like that:
![1](https://user-images.githubusercontent.com/49829097/130759168-29b6ea7b-a9f6-4cd4-b251-e6f00b6e747c.PNG)

So, what is the solution to deploy speaker diarize model in production?
Thanks!
Pass manifest to ClusteringDiarizer
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**

Clustering diarizer class accepts paths to audiofiles instead of a manifest as all other models, however it still converts it to a NeMo-like manifest inside path2audio_files_to_manifest function. 

**Describe the solution you'd like**

An option to provide a manifest instead of a list of paths, e.g.
```python
assert paths2audio_files is not None or manifest_path is not None
if manifest_path is None:
   manifest = path2audio_files_to_manifest(paths2audio_files)    
```
Speaker Verification
State: closed | Pull request: no

Hi,

I'm working with NeMo to find out if a particular voice belongs to a specific person, to use as authentication. I saw several issues about and tested some scripts but I don't know if I'm on the right path.
From what I've read, the best script to follow would be this https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_recognition/voxceleb_eval.py where a score is calculated on two audio embeddings and on this score I could define if both are the same person.
For simplicity's sake leave it that way where I easily retrieve the embeddings of two audios and do the calculation:

```
verification_model = nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained(model_name="speakerverification_speakernet")
emb = verification_model.get_embedding('Person1.wav').cpu()[0]
emb2 = verification_model.get_embedding('Person2.wav').cpu()[0]
score = np.dot(emb, emb2) / ((np.dot(emb, emb) * np.dot(emb2, emb2)) ** 0.5)
score = (score + 1) / 2
```

When testing this script I realized that people of the same sex have a hard time getting it right... I have this doubt because the script seems too "simple" for a very complicated task that involves this voice analysis. This way what I'm doing is right? Or should I go the other way?