renwei2024/nemo-github-issues huggingface dataset:
Out of memory error
State: closed | Pull request: no
Labels: name: bug

I am training Conformer-medium on Multilingual Librispeech Dataset (~45000 hours).
During the training epoch, RAM is constantly increasing until no free space is left. Is there anything i can do to handle this?

Params:
RAM  = 256G
Batch_size = 32
Num_workers = 8

Also when I experimented with 21000 hours, at the end of the epoch, the memory has released.


Here is the trainer params:

trainer:
  gpus: -1
  num_nodes: 1
  max_epochs: 1000
  max_steps: null
  val_check_interval: 0.025 
  accelerator: ddp
  accumulate_grad_batches: 20
  gradient_clip_val: 0.0
  amp_level: O0 
  precision: 32 
  log_every_n_steps: 1  
  progress_bar_refresh_rate: 1
  resume_from_checkpoint: null
  num_sanity_val_steps: 0
  check_val_every_n_epoch: 1 
  sync_batchnorm: true
  checkpoint_callback: false
  logger: false
Train and Embedding from Speaker Recognition and Verification
State: closed | Pull request: no

Hello guys, I have two question.
First, I had a "small" dataset, around 20 hours for some tests.  I successfully trained this dataset but I had some doubts about this train, if I train correctly or not. So, do you give an advice about the train in general? Like how much hours I need to train, an order to do the train, any recommendation if possible.

Second, I tested some audios with voxceleb_eval.py in two moments, with a unique embedding and with two different embedding (that sounds a good ideia for a test),  and I got good results, but I need to do it if audios that won't be in train or test dataset. Like a test in transcritor, I trained a large dataset but when I will test the train, I put an audio that won't be in the train. Is that possible? What I need to do?
Minimim requirments to run a training on hi-mia dataset?
State: closed | Pull request: no
Labels: name: bug

It is not a bug but a request in case some one know what the are the hardware (minimum) requirements to run a speaker verification on hi-mia dataset. I mean how much memory, gpu etc?

Indeed, did you have any time expectation to finish the training on hi-mia? one day, two days, etc
Packages not installed after setup.py install, error importing mpu from megatron
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

After installing using setup.py inside NVIDIA Pytorch container the following were not installed and had to be manually installed: 

pip install braceexpand
pip install webdataset
pip install transformers
pip install pytorch_lightning
pip install rapidfuzz
pip install megatron

**Steps/Code to reproduce bug**

Download NVIDIA Pytorch container
Mount the NeMo directory
run python setup.py install in the NeMo dir
try to run data/import_datasets.py from the examples dir

In addition I am getting the following after installing the above:

 from megatron import mpu
ImportError: cannot import name 'mpu' from 'megatron' (/opt/conda/lib/python3.8/site-packages/megatron/__init__.py)

**Expected behavior**

All required packages should be installed during install

**Environment overview (please complete the following information)**

 - Docker
 - python setup.py install
 - sudo docker run --gpus all -it --rm -v HOMEDIR/NeMo:/NeMo --shm-size=8g -p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit stack=67108864 --device=/dev/snd nvcr.io/nvidia/pytorch:21.05-py3

**Environment details**

NA

**Additional context**

NVIDIA GTX 1050ti
Add mAES beam search for Transducer models
State: closed | Pull request: yes

# Changelog
- Add mAES Transducer algorithm for further improved WER at the cost of speed of execution. 
- Update configs with new decoding scheme
- Begin adding hooks for external LM rescoring

# References
- [Accelerating RNN Transducer Inference via Adaptive Expansion Search](https://ieeexplore.ieee.org/document/9250505)

Ported from ESPNet v1
vad_infer.py. Change the type of an argument "num_workers" from float to int.
State: closed | Pull request: yes

num_workers' type must be int, not float.
It induces TypeError even when using additional input as `--num_workers=20`

```
>>> python vad_infer.py  --dataset=manifest.json --num_workers=20

Traceback (most recent call last):
  File "vad_infer.py", line 171, in <module>
    main()
  File "vad_infer.py", line 106, in main
    manifest_vad_input = prepare_manifest(config)
  File "/workspace/NeMo/nemo/collections/asr/parts/utils/vad_utils.py", line 48, in prepare_manifest
    p = Pool(processes=config['num_workers'])
  File "/opt/conda/lib/python3.8/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 318, in _repopulate_pool_static
    for i in range(processes - len(pool)):
TypeError: 'float' object cannot be interpreted as an integer
```
Rnnt greedy_batch decoding fix
State: closed | Pull request: yes

# Bugfix
- Fixes a bug with an extreme edge case thereby bringing 1:1 parity with sample wise greedy decoding.

# Case
- Timestep T = 0, Label ID U = 0
- Batch size > 1 and greedy_batch decoding is used
- Some samples of the first batch emit a label but other samples do not. Therefore for some samples T=0, U=1 and other samples T=1, U=0 step should occur.
- To accommodate T=0, U=1 case, the model steps forward along label axes, but forgets to reset the state of the samples who should have followed the T=1, U=0 case. 
- As such, the state of the T=1, U=0 samples degrades to an incorrect value, and ruins the first subword predicted by the samples of that batch.
- This has significant impact on WER but minute effect on CER. 

# Fix notes
- Cover the edge case where T= 0, U= 0 explicitly
- Check if there is a mix of label forward and time forward actions and explicitly reset the states (to 0) of the samples where time forward action should take place.
ASR model to TensorRT
State: closed | Pull request: no

Recently, I want to convert my NeMo ASR model to TensorRT. My model was trained with NeMo v0.10.2 and when I tried to quantify it to int8, it came up with an error:

> [TensorRT] WARNING: Int8 calibration in TensorRT won't be used on networks containing boolean operators or loops. Please use quantization-aware training to generate a network with Quantize/Dequantize nodes.

Then, I transform my model into NeMo v1.2.0 following [the code here](https://github.com/NVIDIA/NeMo/discussions/1680#discussioncomment-321962). It works well without any quantification. But there are still some errors when converting to int8 following [sample code](https://github.com/NVIDIA/NeMo/tree/main/examples/asr/quantization)  without any changes except model weight parameters.

> [TensorRT] ERROR: ModelImporter.cpp:720: While parsing node number 39 [QuantizeLinear -> "687"]:
[TensorRT] ERROR: ModelImporter.cpp:721: --- Begin node ---
[TensorRT] ERROR: ModelImporter.cpp:722: input: "encoder.encoder.1.mconv.1.conv.weight"
input: "684"
input: "2973"
output: "687"
name: "QuantizeLinear_39"
op_type: "QuantizeLinear"
attribute {
  name: "axis"
  i: 0
  type: INT
}
[TensorRT] ERROR: ModelImporter.cpp:723: --- End node ---
[TensorRT] ERROR: ModelImporter.cpp:726: ERROR: builtin_op_importers.cpp:1039 In function QuantDequantLinearHelper:
[6] Assertion failed: scaleAllPositive && "Scale coefficients must all be positive"
[TensorRT] ERROR: 4: [network.cpp::validate::2411] Error Code 4: Internal Error (Network must have at least one output)
[TensorRT] ERROR: 2: [builder.cpp::buildSerializedNetwork::417] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed.)

And there are also many warnings like

> I0819 14:32:00.546462 140236154607360 _utils.py:72] Input is fake quantized to 8 bits in QuantConv1d with axis None!
I0819 14:32:00.546510 140236154607360 _utils.py:75] Weight is fake quantized to 8 bits in QuantConv1d with axis 0!
I0819 14:32:00.546570 140236154607360 tensor_quantizer.py:105] Creating Max calibrator
I0819 14:32:00.546631 140236154607360 tensor_quantizer.py:105] Creating Max calibrator
I0819 14:32:00.546823 140236154607360 tensor_quantizer.py:105] Creating Max calibrator

Is there anything wrong? I would appreciate it if any help is offered!
Delay resetting IntentSlotClassificationModel restore state until after the bert model is initialized
State: closed | Pull request: yes

Fixes #2686 

When a trained IntentSlotClassificationModel is exported via `save_to` the resulting archive includes BERT configuration artifacts. During `IntentSlotClassificationModel.restore_from` using the saved archived the BERT model should be initialized using these configuration artifacts. However, because the model restore state is reset before initializing the BERT model these artifacts cannot be found.
Delaying the resetting of the restore state until after the BERT model is initialized resolves the problem.
Cut long dataset
State: closed | Pull request: no

Hi,
Assuming I have a large volume of data, around 10,000 hours, doing this on a relatively weak gpu would take a long time, more than months...
If I want, can I split my base into several parts and do a workout for each of them, always restoring the model from the last workout I did? Or is it better if I do the full base training? For this training in parts, do I have to worry about a parameter?
I'm building a dataSet in Portuguese and I'm facing this problem...
GPU memory leak with non-fixed audio buffer length
State: closed | Pull request: no
Labels: name: bug

Hello
I am using online recognition as in the example:
https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/Online_ASR_Microphone_Demo.ipynb
But when a large buffer size is set hard (example, 40 sec), the quality of recognition of small files suffers. This is due to the fact that the missing buffer is filled with zeros in the transcribe() function.

def _decode(self, frame):
assert len(frame)==self.n_frame_len
self.buffer[:len(frame)] = frame
logits = infer_signal(asr_model, self.buffer).cpu().numpy()[0]
decoded = self._greedy_decoder(logits, self.vocab)
return decoded

@torch.no_grad()
def transcribe(self, frame=None, merge=True):
if frame is None:
frame = np.zeros(shape=self.n_frame_len, dtype=np.float32)
if len(frame) < self.n_frame_len:
frame = np.pad(frame, [0, self.n_frame_len - len(frame)], 'constant')
unmerged = self._decode(frame)
if not merge:
return unmerged
return self.greedy_merge(unmerged)

![2](https://user-images.githubusercontent.com/68752467/129671213-922425c1-c831-4930-87f0-65232b9329c2.PNG)

This issue is resolved by switching to a resizable buffer, but results in GPU memory leaks.

def _decode(self, frame):
logits = infer_signal(asr_model, frame).cpu().numpy()[0]
decoded = self._greedy_decoder(logits, self.vocab)
return decoded

@torch.no_grad()
def transcribe(self, frame=None, merge=True):
if frame is None:
frame = np.zeros(shape=self.n_frame_len, dtype=np.float32)
unmerged = self._decode(frame)
if not merge:
return unmerged
return self.greedy_merge(unmerged)

![1](https://user-images.githubusercontent.com/68752467/129671208-b2799a29-7887-4e33-9a5e-bbea0699cc9c.PNG)

What is the cause of this memory leak and what are the solutions other than a fixed buffer size?

**Environment details**
Ubuntu 18.04
Driver Version: 465.31 
CUDA Version: 11.3 
Docker image nvcr.io/nvidia/nemo:1.0.1

**Additional context**
RTX3090
Pretrained weights being downloaded from huggingface even if downloaded earlier
State: closed | Pull request: no

**Describe the bug**
I have been using the language model for bert pre training and also bert classifier and every time when I restart the container it goes to download the models and updates the existing ones.
I think it should first check for the directory which is present in the code but still it goes for downloading the pre trained weights from the hugging face link when training starts

**Steps/Code to reproduce bug**
Just download the weights for the first time and trying running the same container second time without net access. If we check the directory model weights are there still it tries to download.

**Environment overview (please complete the following information)**

 - Image pulled from NGC cloud. version 1.0.1
dump_all() got an unexpected keyword argument 'sort_keys' when save .nemo file
State: closed | Pull request: no

Hi,
I fine-tuned the QuartzNet15x5Base-En model with 235 hours dataset of Persian Mozilla Common Voice 3 months ago
 and saved the check point file. Now I resorted the CKPT file and start fine-tuning it with another Persian dataset. When I want to save the .nemo file after fine-tuning it has error **dump_all() got an unexpected keyword argument 'sort_keys'** . How caan I solve this error? 
Thanks


**Environment overview**

Environment location: Google Colab
Method of NeMo install: 

BRANCH = 'r1.0.0rc1'
!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]

Learning Rate: 0.002

**Environment details**

OS version : "Ubuntu20.04.3 LTS"
PyTorch version : "1.7.1"
Dataset changes to allow mixed representation training for TTS
State: closed | Pull request: yes

This PR adds a parameter `phoneme_prob` to the `vocab` section of TTS datasets. That parameter sets the probability of using phonemes vs. graphemes during training. When using graphemes, the whole G2P pipeline is skipped and the model is presented with graphemes only. Functionality when using phonemes is not affected, and the default value of `phoneme_prob` is 1 so older config files will not be affected by this change.
Finetuning Citrinet using speech_to_text_bpe poor WER
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I'm trying to use speech_to_text_bpe.py to fine-train Citrinet. I've managed to fine-train citrinet with reasonable success using my own code getting a WER of 16 on 8Khz audio upsampled to 16Khz. However, I would rather use the scripts if possible as it will be easier to maintain longer term. When I use the speech_to_text_xxx.py scripts, the results are poor with my WER sitting at around 95, so clearly I'm doing something wrong. 

**Steps/Code to reproduce bug**

I first create by subword tokenizer, this was using spe but i've also tried wpe. To be honest I'm not sure about the correct setting for vocab_size and if 'spe' is the correct tokenizer to use.

!python /root/asr16000/NeMo/scripts/tokenizers/process_asr_text_tokenizer.py \
  --manifest="/root/asr16000/training/training_samples.json" \
  --data_root="/root/asr16000/tokenizer" \
  --vocab_size=64 \
  --tokenizer='spe' \
  --no_lower_case \
  --spe_type='unigram' \
  --spe_character_coverage=1.0 \
  --log

Then I train the model, I assume I should use the '_bpe' script version here.

!python /root/asr16000/NeMo/examples/asr/speech_to_text_bpe.py \
    --config-path='/root/asr16000/NeMo/examples/asr/conf/citrinet/' \
    --config-name='citrinet_512' \
    model.train_ds.manifest_filepath='/root/asr16000/training/training_samples.json' \
    model.train_ds.batch_size=16 \
    model.validation_ds.manifest_filepath='/root/asr16000/training/eval_samples.json' \
    model.validation_ds.batch_size=16 \
    model.test_ds.manifest_filepath='/root/asr16000/training/eval_samples.json' \
    model.test_ds.batch_size=16 \
    model.tokenizer.dir='/root/asr16000/tokenizer/tokenizer_spe_unigram_v64/' \
    model.tokenizer.type='bpe' \
    model.optim.name="adamw" \
    model.optim.lr=0.001 \
    model.optim.betas=[0.9,0.999] \
    model.optim.weight_decay=0.0001 \
    model.optim.sched.warmup_steps=2000 \
    trainer.gpus=1 \
    trainer.accelerator="ddp" \
    trainer.max_epochs=100 \
    exp_manager.create_wandb_logger=True \
    exp_manager.wandb_logger_kwargs.name="nemo-asr-script-citrinet" \
    exp_manager.wandb_logger_kwargs.project="speech_to_text" 

The /root/asr16000/NeMo directory is a clone of the 1.2 nemo repo. I have about 350 hours of audio for training.

**Expected behavior**

I would expect the WER to be better than 95, I was hoping for something lower than 16. Any pointers would be much appreciated.

**Environment overview (please complete the following information)**

 - Environment location: AWS Sage Maker
 - Method of NeMo install: %pip install nemo_toolkit['all']==1.2.0

**Environment details**

- OS version : Debian GNU/Linux 10 (buster)
- PyTorch version: 1.9.0+cu102
- Python version: 3.6.13

**Additional context**
GPU model A100
Word timestamps
State: closed | Pull request: no
Labels: name: feature

Hi!
You have a nice [CTC_Segmentation_Tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tools/CTC_Segmentation_Tutorial.ipynb) where you split audio and its transcript into small chunks. Is it possible to return a timestamp for each word in the transcript? Or maybe NeMo has another model for that?
FileNotFound loading fine tuned stt_en_citrinet_512 model
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

If I attempt to fine tune stt_en_citrinet_512, the training runs as expected but when I subsequently try to load a checkpoint I get the following error

FileNotFoundError: src path does not exist or it is not a path in nemo file. src value I got was: /home/smajumdar/PycharmProjects/nemo-eval/nemo_beta_eval/asrset/manifests/asrset_1.4/tokenizers/no_appen/tokenizer_spe_unigram_v1024/tokenizer.model. Absolute: /home/smajumdar/PycharmProjects/nemo-eval/nemo_beta_eval/asrset/manifests/asrset_1.4/tokenizers/no_appen/tokenizer_spe_unigram_v1024/tokenizer.model

I can see that the path is the tokenizer path from the original stt_en_citrinet_512 model, however, I don't understand why it's trying to load the file when I load the checkpoint.

**Expected behavior**

The checkpoint file should load without throwing a FileNotFound error

**Environment overview (please complete the following information)**

AWS Sagemaker
ml.g4dn.xlarge
%pip install nemo_toolkit['all']==1.2.0

Otherwise, please provide:
- OS version: Debian GNU/Linux 10 (buster)
- PyTorch version : 1.9.0+cu102
- Python version : 3 .6.13

**Additional context**

Add any other context about the problem here.
Example: GPU model
Fix incorrect masking that always included all samples.
State: closed | Pull request: yes

The current nemo version of the tdnn masking performs comparison as follows:
```
    lens_mat = torch.arange(max_len).to(device)
    lens = lens * max_len
    mask = lens_mat[:max_len].unsqueeze(0) < lens.unsqueeze(1)
```

If we expand the comparison and simplify device and unsqueeze calls , we get

```
torch.arange(max_len)[:max_len] < lens * max_len
```

This will be always True as long as each element of `lens` is larger than 1.  The multiplication by `max_len` is not needed. 

The bug can also be confirmed by running the test present in the PR after undoing the fix. 

Signed-off-by: Aleksei Kalinov <alekseia@nvidia.com>
custom changes on nemo models are not taking place in riva
State: closed | Pull request: no
Labels: name: bug

The problem is that the changes that I introduced at a model level in nemo are not taking place in riva.

For example, for FastPitch, I added custom text parser. Also, for HiFiGAN, I added bias denoiser during inference. These changes are taking place at nemo. However, when I convert the .nemo models to riva for deployment, these changes are not taking place at all.
I am using riva version 1.4.0b0 and NeMo version of 1.2.0.
How one can fix it?
failed to inference on single input
State: closed | Pull request: no

https://github.com/NVIDIA/NeMo/blob/c9d04851e8a9c1382326862126788fadd77663ac/nemo/collections/common/losses/spanning_loss.py#L64

if we inference on single input data, the squeeze function will convert a list to a scalar.

take question_answering for example, it will fail when inferring on a single input.
VAD postprocessing - binarization, filtering
State: closed | Pull request: yes

Add postprocessing for Voice Activity Detection 


Binarization:
- `onset` and `offset` threshold for detecting the beginning and end of a speech.
- padding durations `pad_onset` before and padding duarations `pad_offset` after each speech segment;

Filtering:
- `min_duration_on` threshold for short speech segment deletion,
- `min_duration_on` threshold for small silence deletion,
- `filter_speech_first` to control whether to perform short speech segment deletion first

Note ⚠️ We will deprecate **threshold** in diarization in release 1.5.

VAD postprocessing improved peroformance on CH-109 (109 conversations from the CALLHOME American English Speech (LDC97S42) corpus that have 2 speakers only) 

from  (with single threshold=0.7)
`FA 2.1%	MISS 3.7 %	Diarization ER: 8.6%	Confusion ER: 2.8% `
to (with params in speaker_diarization.yaml)
`FA:3.1%   MISS 1.8%         Diarization ER:7.0%      Confusion ER: 2.1% `

with collar=0.25 and skip overlap. Threshold is tuned on 11 multi-speaker sessions from CALLHOME


TODO
- [  ] update main to stable in yaml files in notebook _before release_
patch quantization
State: closed | Pull request: yes

Update the code to be compatible with latest nemo.
With batchsize=1, now the WER/CER is normal.

But still, the exported onnx model doesn't contain seq_length, which affects models with ```conv_mask:True``` and when batch size > 1 and the samples length is different in a batch, the WER/CER will be degraded if tested by generated trt engine.
bug in get_hi-mia_data.py
State: closed | Pull request: no
Labels: name: bug

In the script of downloading `get_hi-mia_data.py` there is a small bug. I think line  `l.output.write_wav(line, y, sr)` in `__process_data ` needs to be changed by `sf.write(line, data=y, samplerate=sr)`.
By the way when we run this script we will get three folders:

1. dev
> dev.jason including : 16464 samples, dev_all.json: 164640 samples and train.json: 148176 samples

2. test
> test_all.sjon uncluding 165120 samples. When you check the number of labels we will find 13620 labels. How that? How they are related to speakers, or more precisely to 44 speakers?
3. train
> dev with 99309 samples, train: 893774 and train_all:993083 samples

To the best of my knowledge, the number of speakers in dev is 42, in test 44 and in train in 254 (correct me if  I am wrong, please) and each speaker records 160 discours. So, for example, 42*160 != 164640. or you mean the sum of the 160 for each speaker are decomposed, and the sum of discours for all the speakers =  164640.

Indeed, which one of dev, test, train can we use for just train a model and generate the labels for speaker identification.? can I use only dev since it containes dev.json, train.json and the sum of their samples is 16464+148176=164640 samples?
how to properly checkpoint for incremental optimization of the ASR model
State: closed | Pull request: no

I am using [this code](https://github.com/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/01_ASR_with_NeMo.ipynb) and a VCTK corpus for training. Due to the large data-set size I split it up in chunks and use checkpoint to save the model, and then read the checkpoint model for training on further data records. The issue is that when I read in the checkpoint the inference does not deliver the expected results. So I suspect the way I handle the checkpoint is the culprit:

In [this notebook](https://github.com/joepareti54/joepareti54/blob/main/NB_3_chkpt.ipynb) I train the model, save a checkpoint and do some inference. The results appear reasonable.

In [this notebook ](https://github.com/joepareti54/joepareti54/blob/main/NB_3a_chkpt.ipynb) I read in the checkpoint from the step above, train the model on just a few lines and do inferences: the results are different, and it looks like the model was not properly saved or restored.

My overall assumption is that the NeMo model lends itself to transfer learning, so that I can use the public version and refine it incrementally with my own data: is this a valid assumption?
Fixed an error related to the task indicator of the tagger during inference.
State: closed | Pull request: yes

Changes:

- Fixed a bug where the task indicator for each tagging instance is always set to be TN _during inference_. This bug only affects the evaluation of ITN scores. I re-evaluated all the models. All the ITN scores changed only a tiny bit. For German duplex model though, the ITN score went from 79.89% to 87.15% after the bug is fixed.
- Removed data augmentation for the tagger. After the bug is fixed, I tried several times and found data augmentation for the tagger does not improve the performance. data augmentation for the decoder still improves performance.

@yzhang123
Evaluate the performance of the decoder for each semiotic class
State: closed | Pull request: yes

@yzhang123 

For the `english-joint` model, these are the evaluation results:
```
TN (Forward Direction)
DATE                  0.999% (2829/2832)
CARDINAL              0.994% (1031/1037)
LETTERS               0.996% (1345/1350)
PLAIN                 0.997% (352/353)
VERBATIM              1.0% (302/302)
MEASURE               0.993% (141/142)
DECIMAL               1.0% (92/92)
FRACTION              0.875% (14/16)
ORDINAL               0.99% (102/103)
TELEPHONE             0.919% (34/37)
MONEY                 0.973% (36/37)
ELECTRONIC            0.735% (36/49)
DIGIT                 0.886% (39/44)
TIME                  1.0% (8/8)
ADDRESS               1.0% (4/4)
```

I don't add a new script for evaluating the tagger because it can be simply evaluated as follows:
```
tagger_trainer, tagger_model = instantiate_model_and_trainer(cfg, TAGGER_MODEL, False)
tagger_model.setup_test_data(cfg.data.test_ds)
tagger_trainer.test(model=tagger_model , verbose=False)
```
nemo2jarvis/riva conversion error
State: closed | Pull request: no
Labels: name: bug

I am getting the following error while trying to conver from .nemo to .ejrvs format via
`nemo2jarvis --out HFG.ejrvs HFG.nemo`. Using NeMo version of 1.2.0 and Jarvis version of 1.3.0b0. Any idea on how to fix it? Thanks a lot in advance.
```
INFO: Restoring NeMo model from 'HFG.nemo'
ERROR: Nemo2Jarvis: Failed to restore model from NeMo file : HFG.nemo. Please make sure you have the latest NeMo package installed with [all] dependencies.
Traceback (most recent call last):
  File "/opt/conda/bin/nemo2jarvis", line 8, in <module>
    sys.exit(nemo2jarvis())
  File "/opt/conda/lib/python3.8/site-packages/nemo2jarvis/cli/nemo2jarvis.py", line 48, in nemo2jarvis
    Nemo2Jarvis(args)
  File "/opt/conda/lib/python3.8/site-packages/nemo2jarvis/convert.py", line 44, in Nemo2Jarvis
    raise e
  File "/opt/conda/lib/python3.8/site-packages/nemo2jarvis/convert.py", line 37, in Nemo2Jarvis
    model = ModelPT.restore_from(restore_path=nemo_in)
  File "/workspace/nemo/nemo/core/classes/modelPT.py", line 481, in restore_from
    return cls._default_restore_from(restore_path, override_config_path, map_location, strict, return_config)
  File "/workspace/nemo/nemo/core/classes/modelPT.py", line 434, in _default_restore_from
    instance = cls.from_config_dict(config=conf)
  File "/workspace/nemo/nemo/core/classes/common.py", line 472, in from_config_dict
    instance = cls(cfg=config)
TypeError: Can't instantiate abstract class ModelPT with abstract methods list_available_models, setup_training_data, setup_validation_data

```
How to reproduce the same result as in the paper
State: closed | Pull request: no
Labels: name: bug

I tried to reproduce the result of speakerrecognition_speakernet results which:

` description="SpeakerNet_recognition model trained end-to-end for speaker recognition purposes with cross_entropy loss. It was trained on voxceleb 1, voxceleb 2 dev datasets and augmented with musan music and noise. Speaker Recognition model achieves 2.65% EER on voxceleb-O cleaned trial file",`

So, to infer label we need two files:
1- train_manifest=/path/to/train/manifest/file'
2- test_manifest=/path/to/train/manifest/file'

and then call the provided script: `python speaker_reco_infer.py --spkr_model='/path/to/.nemo/file' --train_manifest=/path/to/train/manifest/file --test_manifest=/path/to/train/manifest/file --batch_size=32`

So, correct me please. The test_manifest is `https://www.robots.ox.ac.uk/~vgg/data/voxceleb/meta/veri_test2.txt`. But Usualy we need it in json format where can I find this file?

Where can I find the `train_manifest`?

If I am wrong, how can reproduce the speaker recognition on voxceleb1 and voxceleb2 as in the paper? Indeed, you was using Voxceleb1 and Voxceleb2 for training. Can for example, we use Voxceleb1 for training and then Voxceleb2 for testing?
resolved MisconfigurationException
State: closed | Pull request: yes

end of epoch will throw execption:

pytorch_lightning.utilities.exceptions.MisconfigurationException: ModelCheckpoint(monitor='val_loss') not found in the returned metrics: ['loss', 'l_mle', 'l_length', 'logdet']. HINT: Did you call self.log('val_loss', value) in the LightningModule?


Signed-off-by: Qiliang <xiaoqlster@gmail.com>
[FastPitch Training]  FastPitchDataset AttributeError
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
I am getting the following error at `FastPitchDataset` while I want to train FP model.
```
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/data/audio_to_text.py", line 727, in __getitem__
    audio_path = self.collection[item].audio_file
AttributeError: 'FastPitchDataset' object has no attribute 'collection'
```
Can you guys fix it?
Thanks.
Restore LM when using PunctuationCapitalizationModel
State: closed | Pull request: no

Hello friends,

I am using the punctuation capitalization model, 'punctutation_en_bert' or 'distilbert_en' , but although I use restore_from the nemo file I still see it downloads lm from huggingface (from what i understand), i think it is the bert-uncased. 

1. Is there an option to restore the LM as well instead of downloading it? if so how? I want all models etc will be restored from and not downloaded
2. Is there an option to change the LM to different huggingface models.

Thank you very much
Resolved MisconfigurationException in glow_tts
State: closed | Pull request: yes

end of epoch will throw execption:

pytorch_lightning.utilities.exceptions.MisconfigurationException: ModelCheckpoint(monitor='val_loss') not found in the returned metrics: ['loss', 'l_mle', 'l_length', 'logdet']. HINT: Did you call self.log('val_loss', value) in the LightningModule?
Silent failure: no warning when user-supplied lm_checkpoint file does not exist
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
I'm using the intent/slot model and supplying my own language model checkpoint and config (for a 2-layer version of DistilBert), as follows:

```
  language_model:
    max_seq_length: 50
    pretrained_model_name: distilbert-base-uncased
    lm_checkpoint: /home/carola/Documents/models/distillation/atis_snips_pred_2L.ckpt
    config_file: /home/carola/Documents/configs/language_models/joc_bert_2l.json
    config: null
```

During normal training this is logged:

```
[NeMo I 2021-08-03 13:11:43 bert_module:54] Restoring weights from /home/carola/Documents/models/distillation/atis_snips_pred_2L.ckpt
[NeMo I 2021-08-03 13:11:43 bert_module:82] Weights for DistilBertEncoder restored from /home/carola/Documents/models/distillation/atis_snips_pred_2L.ckpt
```

However, if I mispell the checkpoint file or otherwise provide a file that doesn't exist, like this:

```
 language_model:
    max_seq_length: 50
    pretrained_model_name: distilbert-base-uncased
    lm_checkpoint: /home/carola/Documents/models/distillation/my_fake_file.ckpt  # file that does not exist
    config_file: /home/carola/Documents/configs/language_models/joc_bert_2l.json  # this one does exist
    config: null
```

Then the only sign that something is wrong is that the "Restoring weights" lines are missing from the log, which is hard to notice. The model also performs poorly. Otherwise, training proceeds apparently normally and the resulting saved models have the expected number of layers/parameters, as specified by the config I provided. I guess the weights must be getting initialized from the default 6-layer DistilBert checkpoint, which should not happen.


**Expected behavior**
An exception should be raised if a checkpoint file is provided in the config but does not exist.


**Environment details**
nemo-toolkit 1.2.0 (fresh install via reinstall.sh)
Ubuntu 18.04
python 3.7
PyTorch 1.9.0+cu102
Dataloader chokes when input text contains certain emojis/characters
State: closed | Pull request: no
Labels: name: bug

When using the NER and intent/slot models with a dataset containing certain characters (examples below), training errors out like this:

`RuntimeError: stack expects each tensor to be equal size, but got [75] at entry 0 and [76] at entry 28`

The underlying cause is that tokens made up solely of such characters, when tokenized, yield nothing -- no tokens or subtokens. Yet these lines still get executed during dataset formation:

```
loss_mask.append(1)
subtokens_mask.append(1)
```

As a result, the loss_mask and subtokens_mask are longer than the other input vectors and we get something like this:

```
input_ids:  torch.Size([1, 75])
segment_ids:  torch.Size([1, 75])
input_mask:  torch.Size([1, 75])
subtokens_mask:  torch.Size([1, 76])
loss_mask:  torch.Size([1, 76])
labels:  torch.Size([1, 75])
```

The workaround I'm using is to convert the problematic tokens into `[UNK]` in my own data, prior to attempting to train. But I think some logic should be added in the NeMo data processing to use the `[UNK]` token whenever tokenization yields an empty list of subtokens. 

I don't think the solution should be to skip these tokens, because they could be part of entities that users are trying to tag.

-------------------------------------------------------------------
Example tokens/characters causing this problem:
`\U000fe814\U000fe814\U000fe814`
`\U000feb7b`
`\U000fe814\U000feb0e\U000fe814`
`\U000fe334`

Example document:
`Baby update ; guess what , it's a boy ❤️ my lil baby boy \U000feb18 mommy &amp; daddy love you mi amor \U000fe823\U000fe32c`
import nemo.collections.nlp fails
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
Hi, exec `python examples/nlp/duplex_text_normalization/google_data_preprocessing.py --data_dir=/ssd5/exec/shenchengen/data/tn/ --output_dir=preprocessed/ --lang=en`, there has import error：
```
Traceback (most recent call last):
  File "google_data_preprocessing.py", line 42, in <module>
    from helpers import flatten
  File "/ssd5/exec/shenchengen/software/NeMo/examples/nlp/duplex_text_normalization/helpers.py", line 18, in <module>
    from nemo.collections.nlp.models import DuplexDecoderModel, DuplexTaggerModel
  File "/ssd5/exec/shenchengen/software/NeMo/nemo/collections/nlp/__init__.py", line 15, in <module>
    from nemo.collections.nlp import data, losses, models, modules
  File "/ssd5/exec/shenchengen/software/NeMo/nemo/collections/nlp/models/__init__.py", line 16, in <module>
    from nemo.collections.nlp.models.duplex_text_normalization import (
  File "/ssd5/exec/shenchengen/software/NeMo/nemo/collections/nlp/models/duplex_text_normalization/__init__.py", line 15, in <module>
    from nemo.collections.nlp.models.duplex_text_normalization.duplex_decoder import DuplexDecoderModel
  File "/ssd5/exec/shenchengen/software/NeMo/nemo/collections/nlp/models/duplex_text_normalization/duplex_decoder.py", line 25, in <module>
    import nemo.collections.nlp.data.text_normalization.constants as constants
  File "/ssd5/exec/shenchengen/software/NeMo/nemo/collections/nlp/data/text_normalization/__init__.py", line 14, in <module>
    from nemo.collections.nlp.data.text_normalization.decoder_dataset import TextNormalizationDecoderDataset
  File "/ssd5/exec/shenchengen/software/NeMo/nemo/collections/nlp/data/text_normalization/decoder_dataset.py", line 21, in <module>
    import nemo.collections.nlp.data.text_normalization.constants as constants
AttributeError: module 'nemo.collections' has no attribute 'nlp'
```

then I use ipython, also hase AttributeError: module 'nemo.collections' has no attribute 'nlp'

Installed by cloning the git repo and running ./reinstall.sh. I have also tried installing via` pip install nemo_toolkit['all']`, and `pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]`.  .These all produced the same result.

environment:
python 3.6.13 nemo 1.2.0
Question: How to speed up training?
State: closed | Pull request: no
Labels: name: feature

Hi
I have a question about fast training.

I am trying to train QuartzNet15x5 from scratch on 6000 hours of data with 2xA100 GPUs. It takes 10 hours per epoch.

How can I speed up the training or convergence speed? 

I am using
2xA100 with 40GB RAM
mixed precision
80 num workers in torch loader

what else can i do to speed up the training?
DataSet for 8K phone calls
State: closed | Pull request: no

Hi,

I have a question about how best to assemble a training dataset, who has experience and can share, thank you. I've been working with NeMo for a few months setting up trainings in Portuguese and so far I'm having good results in transcribing calls. But I still notice some simple errors during transcripts...
My question is, for the type of audio I'm transcribing, should I prefer them in my dataSet? For example, I'm building a dataSet thinking about transcribing phone calls (8K), in my dataSet should I include more phone calls or do I have nothing to do with it? Or would increasing the number of speakers and increasing the audio quality be the best solution?
Since building a good dataSet with hundreds of hours takes a lot of time, I want to avoid investing in one side, otherwise it would have better results...
Allow more then 1 NLP node
State: closed | Pull request: yes

Current __init__ to 1 will not allow expansions to over 1 node when using a https://pytorch-lightning.readthedocs.io/en/latest/clouds/cluster.html#general-purpose-cluster
Changing to None allows for replacement.

with num_nodes: int = 1 and 2 nodes requested 
Node 1 
[NeMo I 2021-07-29 00:29:42 nlp_overrides:56] 
     init_ddp_connection global_rank 0 world_size 1
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/1
Added key: store_based_barrier_key:1 to store for rank: 0
Rank 0: Completed store-based barrier for 1 nodes.
[NeMo I 2021-07-29 00:29:42 nlp_overrides:65] 
     init_ddp_connection global_rank 0 world_size 1

Node 2 
[NeMo I 2021-07-29 00:29:42 nlp_overrides:56] 
     init_ddp_connection global_rank 1 world_size 1
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/1
Added key: store_based_barrier_key:1 to store for rank: 1

with num_nodes: int = None,
Node 1 
[NeMo I 2021-07-29 00:54:36 nlp_overrides:56] 
     init_ddp_connection global_rank 0 world_size 2
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Added key: store_based_barrier_key:1 to store for rank: 0
Rank 0: Completed store-based barrier for 2 nodes.
[NeMo I 2021-07-29 00:54:37 nlp_overrides:65] 
     init_ddp_connection global_rank 0 world_size 2

Node 2 
[NeMo I 2021-07-29 00:54:37 nlp_overrides:56] 
     init_ddp_connection global_rank 1 world_size 2
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
Added key: store_based_barrier_key:1 to store for rank: 1
Rank 1: Completed store-based barrier for 2 nodes.
[NeMo I 2021-07-29 00:54:37 nlp_overrides:65] 
     init_ddp_connection global_rank 1 world_size 2
config.model.decoder.angular = True?
State: closed | Pull request: no
Labels: name: bug

It is a bug but more a clarification that I need to understand the speakerNet model.
I need to do speaker verification using speakerNet. So I follow the tutorial https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_recognition/Speaker_Recognition_Verification.ipynb .

In this tutorial you suggest to only change  `config.model.decoder.angular = True` when we want to do speaker verification. So, I need to confirm if that not other parameter need to be changed. 

My second concerns why in https://github.com/NVIDIA/NeMo/blob/v1.0.2/examples/speaker_recognition/speaker_reco.py you set the name of classes to 2. Is just an example or it must be always like that?
Using 64 cores instead of 32
State: closed | Pull request: no
Labels: name: feature


Hello
I've been training with "10_ASR_CTC_Language_Finetuning" on the CPU but I noticed that the training doesn't allocate all available cores, it's currently using 32 cores, I'd like to use the machine's 64

Is there any parameter which I must pass in the script to use the 64 cores?
Comparation of FastSpeech2_HifiGan_E2E and FastSpeech2+HiFiGAN
State: closed | Pull request: no

Hi,
I download the models of FastSpeech2_HifiGan_E2E and FastSpeech2+HiFiGAN, and generate audios by them.
I find out that the voice quality of FastSpeech2+HiFiGAN is better than FastSpeech2_HifiGan_E2E.
Does this results match your expectations? 
In fact, I I'm going to implement E2E TTS structure recentlty. So I hope to discuss the feasibility of this framework.
Thank you~
Error: Cannot resolve schedulerDictConfig when run trainer.fit(Punctuation_and_Capitalization_model)
State: closed | Pull request: no
Labels: name: feature

Hello, I'm training Punctuation_and_Capitalization on a french dataset. The first time everithing was fine, but when I tried to change epochs from 3 to 5, and execute again the : trainer.fit(model), I got this error:

ValueError: Cannot resolve schedulerDictConfig'. Available optimizers are : dict_keys(['WarmupPolicy', 'WarmupHoldPolicy', 'SquareAnnealing', 'CosineAnnealing', 'NoamAnnealing', 'WarmupAnnealing', 'InverseSquareRootAnnealing', 'SquareRootAnnealing', 'PolynomialDecayAnnealing', 'PolynomialHoldDecayAnnealing', 'StepLR', 'ExponentialLR', 'ReduceLROnPlateau', 'CyclicLR'])

It still giving this error even I changed the epochs and re installed the packages. 
Would you mind helping me solve this bug, please?
Punctuation_and_Capitalization model finetuning
State: closed | Pull request: no
Labels: name: bug

I'm following this tutorial to train a punctuation and capitalization model: https://github.com/NVIDIA/NeMo/blob/stable/tutorials/nlp/Punctuation_and_Capitalization.ipynb
and trying to finetune punctuation_en_distilbert model for Spanish in Google Colab.

When I use tatoeba Spanish dataset ~ 300K lines it works, but validation loss does not decrease and is about 0.2 - 0.25.
When I took a larger dataset, 3M lines, then there was not enough memory for gpu and runtime crashed on pretrained_model.setup_training_data() step.
When I took 1M lines, again val loss for 15 epochs was around 0.2 - 0.25, and does not decrease.

Is there any tips what could I do to get good result for finetuned model?
Integrate NVIDIA DALI 1.4 to NeMo ASR
State: closed | Pull request: yes

# Changelog
- Refactor out text processing to NeMo while using DALI for audio processing.
- Refactor DALI dataset to partial implementation, subclass for Char and Subword models
- Add unittests for DALI datasets (if installed)
- Add jenkins test for Citrinet based subword model dataset processing
- Version check that DALI 1.4 is used
- Comment out DALI tests for the time being until PT container 21.08 is released and used
VoiceSwapSample.ipynb [.numpy() bug]
State: closed | Pull request: no
Labels: name: bug

You need to call ```.detach()``` before calling ```.numpy()``` in a ```text_to_audio()``` function.

Edited:

```
def text_to_audio(text):
  parsed = spectrogram_generator.parse(text)
  spectrogram = spectrogram_generator.generate_spectrogram(tokens=parsed)
  audio = vocoder.convert_spectrogram_to_audio(spec=spectrogram)
  return audio.to('cpu').detach().numpy()
```
Machine Translation: Nemo to Jarvis 
State: closed | Pull request: no
Labels: name: bug

I have seen [Nemo-2-jarvis](https://docs.nvidia.com/deeplearning/jarvis/user-guide/docs/model-overview.html#nemo2jarvis-export-for-nemo) conversion approach.

.ejrvs file can be made from .nemo model
But , to build .jmir file for Machine Translation models(pretrained), what would be **pipeline** then,

jarvis-build **pipeline** /servicemaker-dev/<jmir_filename>:<encryption_key> /servicemaker-dev/<ejrvs_filename>:<encryption_key> <optional_args>

Thanks
@okuchaiev
Bug in `tutorials/speaker_recognition/ASR_with_SpeakerDiarization.ipynb`
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

A clear and concise description of what the bug is.

![e](https://user-images.githubusercontent.com/56601466/127006734-044c514e-b5da-45df-bb8c-480030da3b00.PNG)


**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version - Google Colab
- PyTorch version 
- Python version - 3.7

**Additional context**

Add any other context about the problem here.
Example: GPU model
Nvidia NEMO / ASR model to be trained on VCTK data
State: closed | Pull request: no
Labels: name: feature

I would like to use the [tutorial](https://github.com/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/01_ASR_with_NeMo.ipynb) to train a model on a VCTK data set. There are wav files and text files with identical descriptors. The text file contains the sentence spoken in the wav file.
`
joepareti54@MSI /cygdrive/f/x/finance-2020/AI/Listen_attend_spell/VCTK-Corpus
$  ls -l txt/p225 | head -5
total 231
-rw-r--r--+ 1 joepareti54 None  20 Aug 22  2012 p225_001.txt
-rw-r--r--+ 1 joepareti54 None  55 Aug 22  2012 p225_002.txt
-rw-r--r--+ 1 joepareti54 None 103 Aug 22  2012 p225_003.txt
-rw-r--r--+ 1 joepareti54 None  68 Aug 22  2012 p225_004.txt

joepareti54@MSI /cygdrive/f/x/finance-2020/AI/Listen_attend_spell/VCTK-Corpus
$ ls -l wav48/p225 | head -5
total 93464
-rw-r--r--+ 1 joepareti54 None  196990 Aug 23  2012 p225_001.wav
-rw-r--r--+ 1 joepareti54 None  389676 Aug 23  2012 p225_002.wav
-rw-r--r--+ 1 joepareti54 None  749754 Aug 23  2012 p225_003.wav
-rw-r--r--+ 1 joepareti54 None  423528 Aug 23  2012 p225_004.wav`
`
Is this feasible? Any guidelines to implement?
ASR tutorial based on Nemo: cannot import name 'SCMode' from 'omegaconf'
State: closed | Pull request: no
Labels: name: bug

This [notebook ](https://github.com/NVIDIA/NeMo/blob/v1.0.2/tutorials/asr/01_ASR_with_NeMo.ipynb)under Ubuntu fails to `import nemo.collections.asr as nemo_asr` 

Here is the exception report:
`ImportError: cannot import name 'SCMode' from 'omegaconf' (/home/joepareti54/anaconda3/lib/python3.8/site-packages/omegaconf/__init__.py)`