renwei2024/nemo-github-issues huggingface dataset:
Dropout value in QuartzNet 15*5
State: closed | Pull request: no

Hi
I fine-tuned the QuartzNet 15*5 model with 235 hours dataset of Persian Mozilla Common Voice. After that I restored the ckpt file and fine-tuned it with 25 hours of another persian dataset. Now Totally I have 50 epochs. From 18th epoch the loss start to increase. While before the 18th epoch the WER and loss both were decreased. So I deiced to change the dropout vale but in config file of QuartzNet 15*5 we have more than one dropout variable. Which of them  should I change to get the best result?

**Environment overview**

- Environment location: Google Colab
- 
- Method of NeMo install: !pip install nemo_toolkit[asr]
- 
- NeMo version: r1.2.0
- 
- Learning Rate: 2e-3

**Environment details**

- OS version : "Ubuntu20.04.3 LTS"
- PyTorch version : "1.9.1"
Adding parallel transcribe for ASR models -  suppports multi-gpu/multi-node
State: closed | Pull request: yes

+Added transcribe_speech_parallel.py which supports multi-gpu/multi-node inference for ASR models
+Supports both tarred and non-tarred datasets
+works for both CTC and Transducer models
+Added extra sample_id output to the samples returned by the ASR datasets. It enables us  to track down the samples. Would be useful for other tasks in future.
+Added the config description for ASR datasets/datalaoders

Needs to be done:
+Adding support for changing the decoder
+Adding fused_batch support?
+Retrieving some configs from the model itself instead of trusting the use to set them like use_start_end_token

Example for non-tarred datasets:

```
python transcribe_speech_parallel.py \
model=stt_en_conformer_ctc_small \
predict_ds.manifest_filepath=/dataset/manifest_file.json \
predict_ds.batch_size=16 \
output_path=/tmp/
```

Example for tarred datasets:

```
python transcribe_speech_parallel.py \
model=stt_en_conformer_ctc_small \
predict_ds.is_tarred=true \
predict_ds.manifest_filepath=/tarred_dataset/tarred_audio_manifest.json \
predict_ds.tarred_audio_filepaths=/tarred_dataset/audio__OP_0..127_CL_.tar \
predict_ds.batch_size=16 \
output_path=/tmp/
```

By default the trainer uses all GPUs available and precision is fp32. By setting the trainer config user may control these configs. For example to do the predictions with AMP on just two GPUs:

```
python transcribe_speech_parallel.py \
trainer.precision=16 \
trainer.gpus=2 \
model=stt_en_conformer_ctc_small \
predict_ds.manifest_filepath=/dataset/manifest_file.json \
predict_ds.batch_size=16 \
output_path=/tmp/
```

Or control the dataloader's config by setting the predict_ds:

```
python transcribe_speech_parallel.py \
predict_ds.num_workers=8 \
predict_ds.min_duration=2 \
predict_ds.sample_rate=16000 \
model=stt_en_conformer_ctc_small \
predict_ds.manifest_filepath=/dataset/manifest_file.json \
predict_ds.batch_size=16 \
output_path=/tmp/
```
Fixes in `GlobalAverageLossMetric` and `PerplexityMetric`
State: closed | Pull request: yes
Labels: name: bug

1. Fix assertions in `pl_utils.py`. `assert (condition, message)` is incorrect use of assertion.
2. Fix function `reference_loss_func` in file `pl_utils.py`. Previously the function implicitly modified its argument `loss_sum_or_avg`.
3. Fix method `update` in class `GlobalAverageLossMetric`. Previously it implicitly modified its argument `loss` if `self.take_avg_loss` was `True`. Also detach inputs argument `loss` from graph.
4. Detach arguments of `PerplexityMetric.update` from graph.
5. Several fixes in docstrings.
Bug running Voice_Activity_Detection
State: closed | Pull request: no
Labels: name: bug

Hello, how are you?
I ran the script "Voice_Activity_Detection.ipynb"
in colab, and it kills the execution when it starts the training, thus giving the message "Session failure, restarting automatically", I couldn't identify more information about this problem. I leave the error prints.
script is at version 1.4.0

![image](https://user-images.githubusercontent.com/49493322/137534954-edd39602-74c6-4763-8c8d-cefa0bfad6bb.png)
![image](https://user-images.githubusercontent.com/49493322/137535025-79b50ef2-41b1-4d7e-b12b-9db6a3e7a6e9.png)

I didn't change the script, I just took it from the repository and ran it in collab
cusolver error: speaker diarizer, nmse_clustering
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

```
<ipython-input-24-56e2b5b7ea8a> in perform_clustering_ori(self, embeddings, time_stamps, speakers, audio_rttm_map, out_rttm_dir, max_num_speakers)
    570 
    571             cluster_labels = COSclustering(
--> 572                 uniq_key, emb, oracle_num_speakers=NUM_speakers, max_num_speaker=max_num_speakers, cuda=cuda,
    573             )
    574             for i in np.unique(cluster_labels):

/opt/conda/lib/python3.7/site-packages/nemo/collections/asr/parts/utils/nmse_clustering.py in COSclustering(key, emb, oracle_num_speakers, max_num_speaker, min_samples_for_NMESC, enhanced_count_thres, fixed_thres, cuda)
    556 
    557     if emb.shape[0] > min_samples_for_NMESC:
--> 558         est_num_of_spk, p_hat_value, best_g_p_value = nmesc.NMEanalysis()
    559         affinity_mat = getAffinityGraphMat(mat, p_hat_value)
    560     else:

/opt/conda/lib/python3.7/site-packages/nemo/collections/asr/parts/utils/nmse_clustering.py in NMEanalysis(self)
    401         self.p_value_list = self.getPvalueList()
    402         for p_value in self.p_value_list:
--> 403             est_num_of_spk, g_p = self.getEigRatio(p_value)
    404             est_spk_n_dict[p_value] = est_num_of_spk
    405             eig_ratio_list.append(g_p)

/opt/conda/lib/python3.7/site-packages/nemo/collections/asr/parts/utils/nmse_clustering.py in getEigRatio(self, p_neighbors)
    465 
    466         affinity_mat = getAffinityGraphMat(self.mat, p_neighbors)
--> 467         est_num_of_spk, lambdas, lambda_gap_list = estimateNumofSpeakers(affinity_mat, self.max_num_speaker, self.cuda)
    468         arg_sorted_idx = np.argsort(lambda_gap_list[: self.max_num_speaker])[::-1]
    469         max_key = arg_sorted_idx[0]

/opt/conda/lib/python3.7/site-packages/nemo/collections/asr/parts/utils/nmse_clustering.py in estimateNumofSpeakers(affinity_mat, max_num_speaker, is_cuda)
    242     """
    243     laplacian = getLaplacian(affinity_mat)
--> 244     lambdas, _ = eigDecompose(laplacian, is_cuda)
    245     lambdas = np.sort(lambdas)
    246     lambda_gap_list = getLamdaGaplist(lambdas)

/opt/conda/lib/python3.7/site-packages/nemo/collections/asr/parts/utils/nmse_clustering.py in eigDecompose(laplacian, cuda, device)
    217         else:
    218             laplacian = torch.from_numpy(laplacian).float()
--> 219         lambdas, diffusion_map = eigh(laplacian)
    220         lambdas = lambdas.cpu().numpy()
    221         diffusion_map = diffusion_map.cpu().numpy()

RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling `cusolverDnSsyevd(handle, jobz, uplo, n, A, lda, W, work, lwork, info)`
```
**Steps/Code to reproduce bug**

```
        sd = ClusteringDiarizer(config)
        sd.diarize()
```

Hi Nemo team. 

I tried to perform a clustering diarizer but it returned me this error. May I know how can I solve this? 

Thanks!
Pass num_workers to transcribe method
State: closed | Pull request: no
Labels: name: feature

Currently transcribe method creates a dataloader with num_workers=0 ([here](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/models/rnnt_bpe_models.py#L356), for example). This leads to GPU being underused on large lists of files. There should be an argument that you can pass to transcribe method to increase number of workers, like this:
 ```
def transcribe(
        self, paths2audio_files: List[str], batch_size: int = 4, return_hypotheses: bool = False, num_workers: int = 0
    ) -> (List[str], Optional[List['Hypothesis']])
```
Add new features to ASR with diarization with modified tutorial and README.
State: closed | Pull request: yes

This is PR for newly added feature (VAD based word-timestamp compensation, ConformerCTC support).
This PR reflects @fayejf 's contribution on asr_with_diarization.py (VAD model input etc)
Also, tutorial file and README file are also revised to reflect the modified parts.
Still lots of issues are remaining, but for the sake of keeping an error-free version, this PR needs to be merged.
NMT timing and tokenizer stats utils
State: closed | Pull request: yes

This PR adds three features:
1. A utility to collect stats of tokenization of raw text given a tokenizer (i.e., regarding sentences length)
2. NMT inference script supports detailed timing of encoder/decoder/sampler (i.e., timing per sentence length).
3. A plotting script to visualize the detailed timing.

# Summary 

## NMT Detailed Timing

By adding the flag `--write_timing` (as shown below), output file with `.timing.json` suffix will be created with a dictionary of lists that contains `encoder` [sec], `sampler` [sec], `mean_src_length` [tokens/batch], and `mean_tgt_length` [tokens/batch] keys.

Command:

```
python3 -- examples/nlp/machine_translation/nmt_transformer_infer.py ... --write_timing --batch_size=1
```

**NOTE:** a `batch_size=1` is recommended when measuring timing.

Plots:

The command
```
python3 -- examples/nlp/machine_translation/plot_detailed_timing.py FILENAME.timing.json
```
Will create 3 `FILENAME.timing.*.png` files, shown below.

![newstest2013-es-en clean tok ref timing encoder-src_len](https://user-images.githubusercontent.com/7453913/138139749-1870236e-2414-4201-a5f2-abee6082f7fc.png)
![newstest2013-es-en clean tok ref timing sampler-src_len](https://user-images.githubusercontent.com/7453913/138139755-139bd640-8c96-4ce0-b5a8-1d53556e21e0.png)
![newstest2013-es-en clean tok ref timing sampler-tgt_len](https://user-images.githubusercontent.com/7453913/138139757-dc8e1e7b-a924-47a6-86b9-ebfff6ff7129.png)

## Tokenized Data Stats

See below example for collecting all stats in `stats/` directory:

```
python3 -- examples/nlp/machine_translation/collect_tokenizer_dataset_stats.py \
  --tokenizer_library byte-level \
  --tokenizer_model "" \
  --max_lines 10000000 \
  --out_dir stats \
  ./nmtset1.1-en-de/train.dedup.de 
```

terminal output:
```
stats = 
{'samples': 10000000, 'mean': 105.3137831, 'stdev': 112.8504785619722, 'min': 8.0, 'max': 1157.0, 'median': 69.0}
```

Histogram (per sentence length in byte-level tokens):
![lengths_hist](https://user-images.githubusercontent.com/7453913/137540936-e40dfe7a-bd5c-4380-b1aa-bee222519f84.png)
Adding ConformerCTC support and VAD based word-timestamp compensation feature
State: closed | Pull request: yes

- Added support for ConformerCTC model 
 The script can extract word timestamps from both QuartzNet and ConformerCTC.

- Added VAD based word-timestamp compensation feature
If a pre-trained VAD model is provided to asr_with_diarization.py script, the script compensates the word-timestamp using the frame level VAD output.

This PR reflects @fayejf's contribution on asr_with_diarization.py regarding pretrained VAD model input. In the next update, README.md file will reflect the change with yaml/hydra input.
Transcribing with existing Conformer RNNT (tokenizer missing)
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

When trying to transcribe (or even import) Conformer transducer model, the following error comes:
```bash
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/NeMo/nemo/core/classes/common.py", line 705, in from_pretrained
    instance = class_.restore_from(
  File "~/NeMo/nemo/core/classes/modelPT.py", line 270, in restore_from
    instance = cls._save_restore_connector.restore_from(
  File "~/NeMo/nemo/core/connectors/save_restore_connector.py", line 136, in restore_from
    instance = calling_cls.from_config_dict(config=conf)
  File "~/NeMo/nemo/core/classes/common.py", line 493, in from_config_dict
    raise e
  File "~/NeMo/nemo/core/classes/common.py", line 488, in from_config_dict
    instance = cls(cfg=config)
  File "~/NeMo/nemo/collections/asr/models/rnnt_bpe_models.py", line 124, in __init__
    self._setup_tokenizer(cfg.tokenizer)
  File "~/NeMo/nemo/collections/asr/parts/mixins/mixins.py", line 74, in _setup_tokenizer
    model_path = self.register_artifact('tokenizer.model_path', model_path)
  File "~/NeMo/nemo/core/classes/modelPT.py", line 202, in register_artifact
    return self._save_restore_connector.register_artifact(self, config_path, src, verify_src_exists)
  File "~/NeMo/nemo/core/connectors/save_restore_connector.py", line 280, in register_artifact
    raise FileNotFoundError(
FileNotFoundError: src path does not exist or it is not a path in nemo file. src value I got was: /tokenizers/nemo_asr_set/nemo_asr_set_1.4/no_appen/tokenizer_spe_unigram_v1024/tokenizer.model. Absolute: /tokenizers/nemo_asr_set/nemo_asr_set_1.4/no_appen/tokenizer_spe_unigram_v1024/tokenizer.model

```
**Steps/Code to reproduce bug**
Either directly in python:
```bash
import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.ASRModel.from_pretrained(model_name="stt_en_conformer_transducer_large")
```
or when using the transcribe script:
```bash
python examples/asr/transcribe_speech.py  pretrained_name="stt_en_conformer_transducer_large" audio_dir="some_data"
```

Also when untarring the .nemo file, there is no tokenizer folder available as opposed to what is there in the conformer-ctc models.

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**
Nemo installed using ./reinstall.sh at this commit:
commit a0e8018fd72ed8c0c9227bfe583f70eece7af744 (HEAD -> main, origin/main, origin/HEAD)
Author: Evelina <10428420+ekmb@users.noreply.github.com>
Date:   Mon Oct 11 15:56:25 2021 -0700

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model
Adding bucketing for ASR models with tarred datasets
State: closed | Pull request: yes

Adding bucketing support for training ASR models with tarred datasets. 

+ User need to enable --sort_in_shard and also use --buckets_num=NUM_OF_BUCKETS with scripts/speech_recognition/convert_to_tarred_audio_dataset.py script. The script would create multiple tarred dataset, one per bucket, based on the audio durations.


+ Then user need to pass the multiple tarred datasets as a list of lists in the data loader config section. User need to pass a list of lists to enable bucketing. If user passes just a list of strings, then the datasets would simply get concatenated which would be different from bucketing. Here is an example for 4 buckets and 512 shards.

```
python speech_to_text_bpe.py 
...
model.train_ds.manifest_filepath=[[PATH_TO_TARS/bucket1/tarred_audio_manifest.json], 
[PATH_TO_TARS/bucket2/tarred_audio_manifest.json],
[PATH_TO_TARS/bucket3/tarred_audio_manifest.json],
[PATH_TO_TARS/bucket4/tarred_audio_manifest.json]]
model.train_ds.tarred_audio_filepaths=[[PATH_TO_TARS/bucket1/audio__OP_0..511_CL_.tar],
[PATH_TO_TARS/bucket2/audio__OP_0..511_CL_.tar],
[PATH_TO_TARS/bucket3/audio__OP_0..511_CL_.tar],
[PATH_TO_TARS/bucket4/audio__OP_0..511_CL_.tar]]
```

+ When bucketing is enabled, in each epoch, first all GPUs would use the first bucket, then go to the second bucket, and so on. It guarantees that all GPUs are using the same bucket at the same time. It reduces the number of paddings in each batch and speedup the training significantly without hurting the accuracy significantly.

+ Also added the support for relative paths in the manifest file to the convert_to_tarred_audio_dataset.py script. When it cannot find a file in the path specified in the manifest, it looks into the folder of the manifest file.
Batchsize decreasing with each new GPU
State: closed | Pull request: no

Hi,

With every new GPU I add I find I'm forced to lower batchsize because of memory overflows. This is normal? 
For example with 1 GPU I can use 68 BatchSize, with 2 GPUs I can use 67 and with 5 I can use 62.

OS:
Ubuntu 18.04
NeMo 1.3.0
Python 3.8
Training never finished
State: closed | Pull request: no
Labels: name: bug

Hi,

My last workouts never end correctly, he gets stuck at this stage:
![Image](https://user-images.githubusercontent.com/49459897/136842870-f279377b-cb4e-40e4-938f-e44f5d6dd417.png)
If I force a ctrl+c it ends, but I'm afraid to kill some processing. I leave it for hours and this stage is never changed. Any suggestions for this issue? Or is it something normal?

Environment: 
JupyterLab
NeMo 1.3.0 (Previous also have the same problem)
GPUs: 3
Python: 3.8
OS: Ubuntu
Memory usage during training
State: closed | Pull request: no

Hi,

I would like to know if there is any documentation on memory usage during training. I have a base of approximately 1000 hours and there are times when memory usage reaches 160GB (which is my current limit). I'm afraid this will interfere with training, hence my doubt. I also notice that this increase is constant each time I grow my database and the amount of GPUs

NeMo Version: 1.3.0
GPUs: 5
CPU Cores: 64
RAM: 160 GB
Not able to covert stt_en_contextnet_1024.nemo into riva format
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

A clear and concise description of what the bug is.

**Steps/Code to reproduce bug**
nemo2riva --out /servicemaker-dev/stt_en_contextnet_1024.riva /servicemaker-dev/stt_en_contextnet_1024.nemo
nemo-toolkit                     1.4.0
nemo2riva                        1.6.0b0



Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 


**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model
nemo v1.4.0  : AttributeError: type object 'EncDecRNNTModel' has no attribute 'from_pretrained'
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

AttributeError: type object 'EncDecRNNTModel' has no attribute 'from_pretrained'


code:
import nemo.collections.asr as nemo_asr
conformer_rnnt = nemo_asr.models.EncDecRNNTModel.from_pretrained()

console:
'''
Traceback (most recent call last):
  File "exp1.py", line 19, in <module>
    conformer_rnnt = nemo_asr.models.EncDecRNNTModel.from_pretrained()
AttributeError: type object 'EncDecRNNTModel' has no attribute 'from_pretrained'
'''

**Environment details**
macOS Catalina 10.15.7
python version 3.8.10

**Additional context**

no GPU
MTEncDecModel has no attribute device
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I called
MTEncDecModel.restore_from('zh_en_24x6.nemo').cuda('cuda:0')
if I assign the parameter to .cuda() as above, it will raise an error 
'no attribute device'

**Steps/Code to reproduce bug**

as above

A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 

**Expected behavior**

as above

**Environment overview (please complete the following information)**

as instructed in the readme, pip install nemo_toolkit['all'], python 3.8 nemo 1.4 pytorch 1.9

**Environment details**

as above

**Additional context**

probably not
NMT evaluation timing
State: closed | Pull request: yes

This PR extends the timing measurement for NMT models, in addition to bug and performance fixes.

# Summary

* Added timing measurement of encoder, decoder, and sampling stages in eval_step.
* Added support in sliding mean - useful in cases when post processing of log is not available, and smoother timing is required.
* Fixed logging issue of log variance (missing reduction). This bug affected only MIN/VAE training.
* Improved evaluation performance by caching the loss values to be reused in translation (inside eval_step).

# Configuration

The following configuration can be used in config file to enable sliding mean over 10 steps:

```
exp_manager:
  step_timing_kwargs:
    # if positive, defines the size of a sliding window for computing mean
    buffer_size: Optional[int] = 10 # 
```

The following flag can be used in command line to enable sliding mean over 10 steps:

```
+exp_manager.step_timing_kwargs.buffer_size=10
```

# Logging

## NMT Timing Plots

The following keys will be logged for NMT models: `val_sampler_timing`, `val_decoder_timing`, `val_encoder_timing`.

![image](https://user-images.githubusercontent.com/7453913/136821312-ef4e3dc5-c67f-4faf-afc4-d820c08c667d.png)

## Sliding Mean Plots

Below is a comparison of two runs. Cyan shows timing averaged over a sliding window of 10 samples. Gold shows raw timing.

![image](https://user-images.githubusercontent.com/7453913/137145269-d36588c3-fe98-4ec7-9155-7811e70f9d81.png)
Unable to run inference with my trained VAD model.
State: closed | Pull request: no

Hi, I am facing the problem of running inference with my own trained VAD model in the diarization pipeline. I have found #2348 this solution and did the same what you have said. But now I am facing this issue:

```
[NeMo I 2021-10-05 08:44:52 speaker_utils:337] Performing Clustering

  0%|          | 0/1 [00:00<?, ?it/s]
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
/tmp/ipykernel_126575/2138038744.py in <module>
     45 
     46 sd_model = ClusteringDiarizer(cfg=config)
---> 47 sd_model.diarize()
     48 # IPython.display.clear_output(wait=False)
     49 

~/env/nemo/lib/python3.8/site-packages/nemo/collections/asr/models/clustering_diarizer.py in diarize(self, paths2audio_files, batch_size)
    352         os.makedirs(out_rttm_dir, exist_ok=True)
    353 
--> 354         perform_diarization(
    355             embeddings_file=self._embeddings_file,
    356             reco2num=self._num_speakers,

~/env/nemo/lib/python3.8/site-packages/nemo/collections/asr/parts/speaker_utils.py in perform_diarization(embeddings_file, reco2num, manifest_path, sample_rate, window, shift, audio_rttm_map, out_rttm_dir, max_num_speakers)
    336     )
    337     logging.info("Performing Clustering")
--> 338     all_reference, all_hypothesis = perform_clustering(
    339         embeddings, time_stamps, speakers, audio_rttm_map, out_rttm_dir, max_num_speakers
    340     )

~/env/nemo/lib/python3.8/site-packages/nemo/collections/asr/parts/speaker_utils.py in perform_clustering(embeddings, time_stamps, speakers, audio_rttm_map, out_rttm_dir, max_num_speakers)
    262 
    263         lines = time_stamps[uniq_key]
--> 264         assert len(cluster_labels) == len(lines)
    265         for idx, label in enumerate(cluster_labels):
    266             tag = 'speaker_' + str(label)

AssertionError: 
```
how to convert nemo model to torch_script format as i am getting TypeError: All arguments must be passed by kwargs only for typed methods
State: closed | Pull request: no

my_code
import nemo.collections.asr as nemo_asr
import torch.nn as nn

model1 = model.extract_state_dict_from('/root/.cache/torch/NeMo/NeMo_1.3.0/QuartzNet15x5Base-En/2b066be39e9294d7100fb176ec817722/QuartzNet15x5Base-En.nemo', save_dir='./pt_ckpt/QuartzNet15x5Base-En.pt')

model.load_state_dict(torch.load('./pt_ckpt/model_weights.ckpt'))

model.eval()

input_example = torch.randn(8, 1 , 64)



traced_model = torch.jit.trace(model, input_example)


TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_40865/3321835653.py in <module>
      6 
      7 
----> 8 traced_foo = torch.jit.trace(model, input_example)

~/api/lib/python3.7/site-packages/torch/jit/_trace.py in trace(func, example_inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)
    742             strict,
    743             _force_outplace,
--> 744             _module_class,
    745         )
    746 

~/api/lib/python3.7/site-packages/torch/jit/_trace.py in trace_module(mod, inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)
    957                 strict,
    958                 _force_outplace,
--> 959                 argument_names,
    960             )
    961             check_trace_method = module._c._get_method(method_name)

~/api/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

~/api/lib/python3.7/site-packages/torch/nn/modules/module.py in _slow_forward(self, *input, **kwargs)
   1037                 recording_scopes = False
   1038         try:
-> 1039             result = self.forward(*input, **kwargs)
   1040         finally:
   1041             if recording_scopes:

~/api/lib/python3.7/site-packages/nemo/core/classes/common.py in __call__(self, wrapped, instance, args, kwargs)
    790         # Check that all arguments are kwargs
    791         if input_types is not None and len(args) > 0:
--> 792             raise TypeError("All arguments must be passed by kwargs only for typed methods")
    793 
    794         # Perform rudimentary input checks here

TypeError: All arguments must be passed by kwargs only for typed methods

​please Help to debug the error
AssertionError: 80, 1: /home/user/Documents/LJSpeech-1.1/wavs/LJ012-0230.wav
State: closed | Pull request: no
Labels: name: bug

I've been trying to train fastpitch + hifigan e2e with `fastpitch_hifigan_e2e.py`

I have used the script at https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/SpeechSynthesis/FastPitch/prepare_dataset.py to create the durations and pitch .pt files. On starting the training, I get the assertion error. 

```sh
File "/home/user/Labs/NeMo/nemo/collections/asr/data/audio_to_text.py", line 758, in _collate_fn
    assert len(durs_i) == len(pitch_i), f"{len(durs_i)}, {len(pitch_i)}: {path_i}"
AssertionError: 80, 1: /home/user/Documents/LJSpeech-1.1/wavs/LJ012-0230.wav
```
It looks like there is a mismatch in the lengths of duration and pitch. It would be great if I could get some help with this.
Speaker_Diarization handle overlap speech
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Hi teams, may I know whether Speaker_Diarization models can separate speaker utterances from overlap speech? If not, will model directly ignore this overlap part? 
 https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Diarization_Inference.ipynb

Thanks in advance
Experiment manager step timing
State: closed | Pull request: yes

This PR adds the ability to log the timing of a step for train/validation/test modes during training.
This feature is added to all NeMo models.
Time is measured in seconds.

# Configuration

The following config allow to control logging of step timing:

```
exp_manager:
  log_step_timing: True # logging is enabled by default
  step_timing_kwargs:
    reduction: "mean" # reduction for multiple measurements
    sync_cuda: False # if True calls torch.cuda.synchronize() for start/stop of measurement. Disabled by default
```

By default the logging is turned on. In order to turn off the feature, the following config can be used:

```
exp_manager.log_step_timing=False
```

By default cuda sync is turned off. In order to turn it on, the following config can be used:
```
exp_manager.step_timing_kwargs.sync_cuda=True
```

# Logging

The following keys will be logged in the corresponding mode (when available): `train_step_timing`,   `train_backward_timing`, `validation_step_timing`,   `test_step_timing`.

## Weights & Biases Output Example

![image](https://user-images.githubusercontent.com/7453913/135684915-16900e7c-b079-4db9-898d-f73ddce12754.png)

## Pytorch Lightning Training Loop

Below is `pytorch lightning` training loop, in order to better understand what exactly is measured:

```
def train_loop():
    on_epoch_start()
    on_train_epoch_start()

    for batch in train_dataloader():
        on_train_batch_start()

        on_before_batch_transfer()
        transfer_batch_to_device()
        on_after_batch_transfer()

        training_step()

        on_before_zero_grad()
        optimizer_zero_grad()

        on_before_backward()
        backward()
        on_after_backward()

        on_before_optimizer_step()
        optimizer_step()

        on_train_batch_end()

        if should_check_val:
            val_loop()
    # end training epoch
    training_epoch_end()

    on_train_epoch_end()
    on_epoch_end()
```

See <https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html#hooks> for full details.
Adding complete method to MegatronGPTModel
State: closed | Pull request: yes

This PR adds the following method ```response=MegatronGPTModel.complete(request)``` and example how to use it.
Example usage:
```
python examples/nlp/language_modeling/megatron_gpt_eval.py --model_file megatron_gpt.nemo \
--prompt "How to fix gpu memory? A: " --tokens_to_generate 256
```
With a somewhat good converged model you should be getting completion like this:
```
How to fix gpu memory? A: 

You can use the gpu_mem_info() function to get the information about the memory.
```
How to train end2end tts
State: closed | Pull request: no
Labels: name: bug

I want to train an end2end tts model like fastpitch_hifigan_e2e but I don't find any tutorial for that. I tried to train it like Tacotron but I understood that fastpitch_hifigan_e2e needs "Phoneme durations" file and again I don't find any guide for that(exactly what should this file have and what is format of it. [this explanation](https://github.com/NVIDIA/NeMo/blob/main/docs/source/tts/datasets.rst) is not complete). 
thanks for any help.
I am getting undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE error. This I am getting when I am trying to "import torch from nemo.collections import nlp". I am trying to use pytorch ngc container 21.05. I tried to import torch before the nemo extension. Please suggest how I can resolve this.
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I am getting this issue when trying to import nlp models from nemo.

**Steps/Code to reproduce bug**
I did the following installation of packages.
!pip install wget
!apt-get install sox libsndfile1 ffmpeg
!pip install unidecode
!pip install braceexpand
!pip install webdataset

BRANCH = 'main'
!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]

!pip install torchaudio>=0.6.0 -f https://download.pytorch.org/whl/torch_stable.html

!mkdir configs

!pip install Cython
!pip install nemo_toolkit['all']

Then when I am doing the import, I am getting the error of undefined symbol.
**Expected behavior**

I intended to import nemo's nlp model succesfully to test out different models.

**Environment overview (please complete the following information)**

 - Environment location: Docker version 20.10.8
 - Method of NeMo install: !python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]
 - If method of install is [Docker],: docker run -it --restart=on-failure:1 -v $PWD/NeMo:/workspace/Nemo --shm-size=8g --net=host --ulimit memlock=-1 --ulimit stack=67108864 --device=/dev/snd nvcr.io/nvidia/pytorch:21.05-py32


**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model: Nvidia GeForce RTX 2060
Fix numba spec augment for cases where batch size > MAX_THREAD_BUFFER
State: closed | Pull request: yes

# Bugfix
- Fix an issue where numba spec augment will throw cryptic CUDA error if batch size > MAX_THREAD_BUFFER

Closes https://github.com/NVIDIA/NeMo/issues/2916

Signed-off-by: smajumdar <titu1994@gmail.com>
Adding simple, greedy-search based text generation to MegatronGPTModel
State: closed | Pull request: yes

This PR adds the following method ```response=MegatronGPTModel.complete(request)``` and example how to use it.
Example usage:
```
python examples/nlp/language_modeling/megatron_gpt_eval.py --model_file megatron_gpt.nemo \
--prompt "How to fix gpu memory? A: " --tokens_to_generate 256
```
With a somewhat good converged model you should be getting completion like this:
```
How to fix gpu memory? A: 

You can use the gpu_mem_info() function to get the information about the memory.
```
[wav2vec] mask fragmentation
State: closed | Pull request: no

Hi,

First of all, thanks for developing this incredibly useful toolkit.

I am experimenting with the wav2vec code in NeMo, and would like to ask a question regarding the computation of the masked indices for the context encoder network. More precisely, the `compute_mask_indices` function in `collections/asr/modules/wav2vec_modules.py` appears to implement a normalisation step at the end, ensuring that the total number of masked timesteps for each entry in the minibatch is identical.

```
    min_len = min([len(m) for m in mask_idcs])
    for i, mask_idc in enumerate(mask_idcs):
        if len(mask_idc) > min_len:
            mask_idc = np.random.choice(mask_idc, min_len, replace=False)
        mask[i, mask_idc] = True
```

At runtime, I am observing that the masks for the longer sequences in a minibatch end up being fragmented, especially when there is a considerable length ratio between the longer and the shortest utterance. In contrast, [the article of Baevski et al. ](https://arxiv.org/abs/2006.11477)mentions the masking of at least M consecutive timesteps, and Figure 2 appears to indicate no masks shorter than 10 timesteps in that example.

Would you please explain the role of the code snippet above in `compute_mask_indices` ? Do you consider this could have a significant impact upon the convergence speed of the learning algorithm? I am currently exploring an alternative base architecture and seeing relatively large loss values in the range of 1e+4 after 50k iterations, and was wondering if I should apply a different masking strategy.

Thanks for any help. @SeanNaren I hope you don't mind the ping.
Swapped inputs in decoder onnx file
State: closed | Pull request: no
Labels: name: bug

It appears the transformer decoder when exported to onnx has swapped the `encoder_mask` and `encoder_embeddings`:

```python
encoder_mask : ['encoder_mask_dynamic_axes_1', 'encoder_mask_dynamic_axes_2', 1024]
encoder_embeddings : ['encoder_embeddings_dynamic_axes_1', 'encoder_embeddings_dynamic_axes_2']
```

You can obtain the above output with the following code snippet (used MT as that is my use case, but assume it would be the case for any decoder.)

```python
from nemo.collections.nlp.models.machine_translation.mt_enc_dec_model import MTEncDecModel
import onnxruntime as rt

mt_model = MTEncDecModel.from_pretrained('nmt_en_de_transformer12x2')
mt_model.decoder.export('decoder.onnx')

decoder_session = onnxruntime.InferenceSession('decoder.onnx')
for i in decoder_session.get_inputs():
        print(f'{i.name} : {i.shape}')
```

** Environment overview (please complete the following information) **
- Environment Location: Docker on Kubernetes
- Method of NeMo install: helm chart using nvcr.io/nvidia/nemo:1.2.0
CUDA error: an illegal memory access was encountered while training MatchboxNet (Numba CUDA)
State: closed | Pull request: no
Labels: name: bug

Hello!
I train MatchboxNet model with custom labels and standard config file.
If I use batch_size (train, test, validation) = 512 I consume 3Gb of GPU memory (Tesla V100) and everything works fine. 
If I increase batch_size to 1024 I get ```RuntimeError: CUDA error: an illegal memory access was encountered``` at the first step. 
Running script with ```CUDA_LAUNCH_BLOCKING=1``` gives next output:
```numba.cuda.cudadrv.driver.CudaAPIError: [700] Call to cuLaunchKernel results in UNKNOWN_CUDA_ERROR```
Tried different torch versions - no results.

Any ideas how to fix this?

Numba version: 0.54.0
Python: 3.7.9
CUDA version: 11.1
Driver Version: 455.32.00
Model Conformer-CTC-Char import error from toolkit
State: closed | Pull request: no
Labels: name: bug

I am used nemo_toolkit==r1.4.0
However, I have load model  error model if I loaded it with

`import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.EncDecCTCModelBPE.from_pretrained(model_name="stt_en_conformer_transducer_medium")`.
Update Offline_ASR.ipynb
State: closed | Pull request: yes

without this change, the following error occurs :
max() received an invalid combination of arguments - got (out=NoneType, axis=NoneType, ), but expected one of:
 * ()
 * (name dim, bool keepdim)
      didn't match because some of the keywords were incorrect: out, axis
 * (Tensor other)
 * (int dim, bool keepdim)
      didn't match because some of the keywords were incorrect: out, axis
Error when running example training TTS on new data
State: closed | Pull request: no
Labels: name: bug

**Bug description**

I want to train tacotron2 on new data and at first I just try to run [2_TTS_Tacotron2_Training.ipynb](https://github.com/NVIDIA/NeMo/blob/r1.0.0rc1/tutorials/tts/2_TTS_Tacotron2_Training.ipynb) in colab without any changes! But when I run following code
```
!wget https://github.com/NVIDIA/NeMo/releases/download/v0.11.0/test_data.tar.gz && mkdir -p tests/data && tar xzf test_data.tar.gz -C tests/data

# Just like ASR, the Tacotron2 require .json files to define the training and validation data.
!cat tests/data/asr/an4_val.json

# Now that we have some sample data, we can try training Tacotron 2
# NOTE: The sample data is not enough data to properly train a Tacotron 2. This will not result in a trained Tacotron 2 and is used to illustrate how to train Tacotron 2 model
!python tacotron2.py sample_rate=16000 train_dataset=tests/data/asr/an4_train.json validation_datasets=tests/data/asr/an4_val.json trainer.max_epochs=3 trainer.accelerator=null trainer.check_val_every_n_epoch=1
```

I get this error:
```
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!
[nltk_data] Downloading package cmudict to /root/nltk_data...
[nltk_data]   Package cmudict is already up-to-date!
[NeMo W 2021-09-27 19:13:14 experimental:28] Module <class 'nemo.collections.asr.data.audio_to_text_dali.AudioToCharDALIDataset'> is experimental, not ready for production and is not fully supported. Use at your own risk.
################################################################################
### WARNING, path does not exist: KALDI_ROOT=/mnt/matylda5/iveselyk/Tools/kaldi-trunk
###          (please add 'export KALDI_ROOT=<your_path>' in your $HOME/.profile)
###          (or run as: KALDI_ROOT=<your_path> python <your_script>.py)
################################################################################

[NeMo W 2021-09-27 19:13:14 nemo_logging:349] /usr/local/lib/python3.7/dist-packages/torchaudio/backend/utils.py:54: UserWarning: "sox" backend is being deprecated. The default backend will be changed to "sox_io" backend in 0.8.0 and "sox" backend will be removed in 0.9.0. Please migrate to "sox_io" backend. Please refer to https://github.com/pytorch/audio/issues/903 for the detail.
      '"sox" backend is being deprecated. '
    
Traceback (most recent call last):
  File "tacotron2.py", line 44, in <module>
    main()  # noqa pylint: disable=no-value-for-parameter
  File "/usr/local/lib/python3.7/dist-packages/nemo/core/config/hydra_runner.py", line 103, in wrapper
    strict=None,
TypeError: _run_hydra() got an unexpected keyword argument 'strict'
```
I changed nothing in code and just ran cells of this code that is a tutorial for training of tacotron.
Thank you for any help
Fix several bugs in punctuation and capitalization inference and make minor improvements
State: closed | Pull request: yes
Labels: name: bug

1. A parameter `--input_text` of the script `punctuate_capitalize_infer.py` was not working because of wrong implementation of data reading. Fixed.
2. Add parameter `--save_labels_instead_of_text` to script `punctuate_capitalize_infer.py` and parameter `return_labels` to `PunctuationCapitalizationModel.add_punctuation_capitalization` method. Useful feature in my opinion.
3. Remove device choice from `PunctuationCapitalizationModel.add_punctuation_capitalization` method. Device choice should be done in `main` function.
4. Add parameter `--device` to the script `punctuate_capitalize_infer.py`.
Questions about training acoustic models - Lexicon 
State: closed | Pull request: no
Labels: name: feature

Hi, I noticed that in the training pipelines, lexicon mapping of word spelling is not part of the workflow, I noticed in some other pipelines such as flashlight, they use it to improve the accuracy of decoding.  For NeMo, is there some similar executions like lexicon mapping? or if not, is there any plan on including it in the future? Thank you.
batch size for asr model 
State: closed | Pull request: no
Labels: name: feature

I noticed that for ASR training batch size setup, I noticed that there is a trick to set the accumulate_grad_batches  value to cater for large batches, the performance is still quite different between set batch size = 32 vs. set batch size =16 with   accumulate_grad_batches = 2. Although in theory it should work, but when I do the training for  QuartzNet15x5-Zh-NR.nemo,  I found if using batch size = 32, I can get loss < 20 in about 10 epochs, whereas if I use  batch size =16 with   accumulate_grad_batches = 2, I train for 100 epochs, the loss level is at 67 (which means the training diverges). 

Does this mean, we would have to use high RAM GPUs to train the models? Does anyone observe the similar issue before?

below is a sample setup for batch size, this is because my server couldn't cater for batchsize =32, therefore, I use this trick, but the training metrics was very bad. 
 cfg = copy.deepcopy(char_model.cfg)
 with open_dict(cfg):    
   cfg.train_ds.manifest_filepath = f"{train_manifest_cleaned},{dev_manifest_cleaned}"
  cfg.train_ds.labels = list(train_dev_set)
  cfg.train_ds.normalize_transcripts = False
  cfg.train_ds.batch_size = 16
  cfg.train_ds.num_workers = 8
  cfg.train_ds.pin_memory = True
  cfg.train_ds.trim_silence = True 
trainer = ptl.Trainer(gpus=4, 
                      accelerator='ddp',
                      max_epochs=EPOCHS, 
                      accumulate_grad_batches=2,
                      checkpoint_callback=False,
                      logger=False,
                      log_every_n_steps=5,
                      check_val_every_n_epoch=10)
Variables `hidden_steps` (line 306) and `_SUPPORTED_ARCH` (line 337) in `transformer_bottleneck.py` are not defined
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Variables `hidden_steps` (line 306) and `_SUPPORTED_ARCH` (line 337) in https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/modules/common/transformer/transformer_bottleneck.py are not defined.
No module named 'nemo'
State: closed | Pull request: no
Labels: name: bug

Hello,

I am trying to run the script self_alignment_pretraining.py (location - https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/entity_linking/self_alignment_pretraining.py)

I keep getting the error: ModuleNotFoundError: No module named 'nemo'

I have tried installing the nemo toolkit using the following:

!pip install git+https://github.com/NVIDIA/NeMo.git

and

BRANCH = 'r1.3.0'
#!pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[nlp]

and

!pip install nemo
!pip install nemo_toolkit
!pip install tensorboard==1.15.0

But continue to get the same error.

**Environment details**

- OS version - I am running it on spyder on my local laptop (MacBook Pro - macOS Big Sur version 11.6), no GPU being used.
- PyTorch version - torch version 1.9.1
- Python version - Python 3.6.8 :: Anaconda, Inc.

Thank you.

Apologies if I have provided insufficient information, I am fairly new to programming.
MatchboxNet questions
State: closed | Pull request: no

Hello!
Need help with some moments:
1. How to perform GPU batched inference with trained model?
2. Is it possible to make MatchboxNet architecture more heavy? Encountered problem that https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/matchboxnet/matchboxnet_3x1x64_v2.yaml was stucked on ~50% accuracy on clean audio commands (38 labels, trained from scratch, 100 epochs, batch size 512).
Fix DALI log default floor and normalization formula (#2869)
State: closed | Pull request: yes

* Fix DALI log default floor and normalization formula

Signed-off-by: Joaquin Anton <janton@nvidia.com>

* Fix style

Signed-off-by: Joaquin Anton <janton@nvidia.com>
(cherry picked from commit ad75db34c816bff23f64de7a875d89c50c319c85)
Fix window_stride calculation in DALI pipeline & Fix dither generatio…
State: closed | Pull request: yes

…n (#2858)

Signed-off-by: Joaquin Anton <janton@nvidia.com>

Co-authored-by: Somshubra Majumdar <titu1994@gmail.com>
(cherry picked from commit 6987b6cfc9c9769f61116d0ac5a49088798266f8)