DALI Audio To Text Fails
State: closed | Pull request: no

Hello,

I am trying to train the conformer model with the DALI input pipeline, but it crash at the very early start.
Here is the logs:

```
[NeMo I 2021-09-14 16:32:36 mixins:147] Tokenizer SentencePieceTokenizer initialized with 128 tokens
[NeMo I 2021-09-14 16:32:36 ctc_bpe_models:172]
    Replacing placeholder number of classes (-1) with actual number of classes - 128
[NeMo W 2021-09-14 16:32:36 nemo_logging:349] /home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nvidia/dali/ops.py:627: DeprecationWarning: WARNING:
`normal_distribution` is now deprecated. Use `random.normal` instead.
      op_instances.append(_OperatorInstance(input_set, self, **kwargs))

Error executing job with overrides: ['+init_from_nemo_model=/home/ael/Scripts/nemo/training/pretrained_models/stt_en_conformer_ctc_small/stt_en_conformer_ctc_small.nemo']
Traceback (most recent call last):
  File "/home/ael/Scripts/nemo/training/speech_to_text_bpe.py", line 101, in <module>
    main()
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/core/config/hydra_runner.py", line 101, in wrapper
    _run_hydra(
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/hydra/_internal/utils.py", line 377, in _run_hydra
    run_and_report(
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
    raise ex
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/hydra/_internal/utils.py", line 378, in <lambda>
    lambda: hydra.run(
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 111, in run
    _ = ret.return_value                                                                                                                                             File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/hydra/core/utils.py", line 233, in return_value
    raise self._return_value                                                                                                                                         File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/hydra/core/utils.py", line 160, in run_job
    ret.return_value = task_function(task_cfg)                                                                                                                       File "/home/ael/Scripts/nemo/training/speech_to_text_bpe.py", line 81, in main
    asr_model = EncDecCTCModelBPE(cfg=cfg.model, trainer=trainer)
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/collections/asr/models/ctc_bpe_models.py", line 179, in __init__
    super().__init__(cfg=cfg, trainer=trainer)
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/collections/asr/models/ctc_models.py", line 153, in __init__
    super().__init__(cfg=cfg, trainer=trainer)
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/core/classes/modelPT.py", line 120, in __init__
    self.setup_training_data(self._cfg.train_ds)
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/collections/asr/models/ctc_models.py", line 423, in setup_training_data
    self._train_dl = self._setup_dataloader_from_config(config=train_data_config)
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/collections/asr/models/ctc_bpe_models.py", line 201, in _setup_dataloader_from_config
    dataset = audio_to_text_dataset.get_dali_bpe_dataset(
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/collections/asr/data/audio_to_text_dataset.py", line 275, in get_dali_bpe_dataset
    dataset = audio_to_text_dali.AudioToBPEDALIDataset(
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/collections/asr/data/audio_to_text_dali.py", line 645, in __init__
    super().__init__(
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/collections/asr/data/audio_to_text_dali.py", line 398, in __init__
    self._iter = DALIPytorchIterator(
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nvidia/dali/plugin/pytorch.py", line 190, in __init__
    self._first_batch = DALIGenericIterator.__next__(self)
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nvidia/dali/plugin/pytorch.py", line 201, in __next__
    outputs = self._get_outputs()
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nvidia/dali/plugin/base_iterator.py", line 254, in _get_outputs
    outputs.append(p.share_outputs())
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nvidia/dali/pipeline.py", line 906, in share_outputs
    return self._pipe.ShareOutputs()
RuntimeError: Critical error in pipeline:
Error when executing GPU operator Pad encountered:
[/opt/dali/dali/operators/generic/pad.h:99] Assert on "a > 0" failed: Values of `align` argument must be positive.
Stacktrace (11 entries):
[frame 0]: /home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nvidia/dali/libdali_operators.so(+0x55daee) [0x7ff58607aaee]
[frame 1]: /home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nvidia/dali/libdali_operators.so(+0x138c8ed) [0x7ff586ea98ed]
[frame 2]: /home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nvidia/dali/libdali_operators.so(+0x1360928) [0x7ff586e7d928]
[frame 3]: /home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nvidia/dali/libdali.so(void dali::Executor<dali::AOT_WS_Policy<dali::UniformQueuePolicy>, dali::UniformQueuePolicy>::RunHelper<dali::DeviceWorkspace>(dali::OpNode&, dali::DeviceWorkspace&)+0x845) [0x7ff584ff7745]
[frame 4]: /home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nvidia/dali/libdali.so(dali::Executor<dali::AOT_WS_Policy<dali::UniformQueuePolicy>, dali::UniformQueuePolicy>::RunGPUImpl()+0x41c) [0x7ff584ff852c]
[frame 5]: /home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nvidia/dali/libdali.so(dali::Executor<dali::AOT_WS_Policy<dali::UniformQueuePolicy>, dali::UniformQueuePolicy>::RunGPU()+0xe) [0x7ff584ff8f4e]
[frame 6]: /home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nvidia/dali/libdali.so(+0xa4ea8) [0x7ff584fa9ea8]
[frame 7]: /home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nvidia/dali/libdali.so(+0x123e8c) [0x7ff585028e8c]
[frame 8]: /home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nvidia/dali/libdali.so(+0x70a55f) [0x7ff58560f55f]
[frame 9]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7ff6876676db]
[frame 10]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7ff68739071f]
```

The same model doesn't crash at this point when I don't use DALI.
The only difference is that I enable "use_dali: true" at the yaml file.

Any idea what is the cause of this problem and how I can solve it ?