normalize_batch ValueError
State: closed | Pull request: no

Hello,

I am fine-tuning Conformer-small using a custom tarred dataset.
During training the following error occurs:

```
Error executing job with overrides: 
['+init_from_nemo_model=/home/ael/Scripts/nemo/training/pretrained_models/stt_en_conformer_ctc_small/stt_en_conformer_ctc_small.nemo']
Traceback (most recent call last):                                                                                                                                   File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1044, in _run_train                                self.fit_loop.run()                                                                                                                                              File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 111, in run                                             self.advance(*args, **kwargs)                                                                                                                                    File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 200, in advance                                     epoch_output = self.epoch_loop.run(train_dataloader)                                                                                                             File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 111, in run                                             self.advance(*args, **kwargs)                                                                                                                                    File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 130, in advance                    batch_output = self.batch_loop.run(batch, self.iteration_count, self._dataloader_idx)                                                                            File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 100, in run                        super().run(batch, batch_idx, dataloader_idx)                                                                                                                    File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 111, in run                                             self.advance(*args, **kwargs)                                                                                                                                    File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 147, in advance                    result = self._run_optimization(batch_idx, split_batch, opt_idx, optimizer)                                                                                      File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 193, in _run_optimization          closure()                                                                                                                                                        File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 235, in _training_step_and_backward_closure                                                                                                                                                           result = self.training_step_and_backward(split_batch, batch_idx, opt_idx, optimizer, hiddens)     
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 536, in training_step_and_backward
    result = self._training_step(split_batch, batch_idx, opt_idx, hiddens)
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 306, in _training_step             training_step_output = self.trainer.accelerator.training_step(step_kwargs)
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 193, in training_step
    return self.training_type_plugin.training_step(*step_kwargs.values())
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 384, in training_step
    return self.model(*args, **kwargs)                                                                                                                               File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 224, in forward
    return self.module(*inputs, **kwargs)
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl                                          return forward_call(*input, **kwargs)                                                                                                                            File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/pytorch_lightning/overrides/base.py", line 82, in forward
    output = self.module.training_step(*inputs, **kwargs)                                                                                                            File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/utils/model_utils.py", line 339, in wrap_training_step                                    output_dict = wrapped(*args, **kwargs)                                                                                                                           File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/collections/asr/models/ctc_models.py", line 559, in training_step                         log_probs, encoded_len, predictions = self.forward(input_signal=signal, input_signal_length=signal_len)                                                          File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/core/classes/common.py", line 798, in __call__                                            outputs = wrapped(*args, **kwargs)                                                                                                                               File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/collections/asr/models/ctc_models.py", line 538, in forward                               processed_signal, processed_signal_length = self.preprocessor(                                                                                                   File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl                                          return forward_call(*input, **kwargs)                                                                                                                            File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/core/classes/common.py", line 798, in __call__                                            outputs = wrapped(*args, **kwargs)                                                                                                                               File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context                                     return func(*args, **kwargs)                                                                                                                                     File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/collections/asr/modules/audio_preprocessing.py", line 83, in forward                      processed_signal, processed_length = self.get_features(input_signal, length)                                                                                     File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/collections/asr/modules/audio_preprocessing.py", line 259, in get_features                return self.featurizer(input_signal, length)                                                                                                                     File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl                                          return forward_call(*input, **kwargs)                                                                                                                            File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context                                     return func(*args, **kwargs)                                                                                                     
    return func(*args, **kwargs)                                                                                                                                     File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/collections/asr/parts/preprocessing/features.py", line 416, in forward
    x = normalize_batch(x, seq_len, normalize_type=self.normalize)
  File "/home/ael/anaconda3/envs/nemo-1.3.0/lib/python3.9/site-packages/nemo/collections/asr/parts/preprocessing/features.py", line 60, in normalize_batch             raise ValueError(
ValueError: normalize_batch with `per_feature` normalize_type received a tensor of length 1. This will result in torch.std() returning nan
```

If my understanding is correct, it seems there is a single sample that it is length is less than a single frame. However, I have double-checked and the smallest sample in the dataset has a duration of 0.1

For completeness, this is the preprocessor and spec_augment data on the YAML file:

```
  preprocessor:
    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
    sample_rate: ${model.sample_rate}
    normalize: "per_feature"
    window_size: 0.025
    window_stride: 0.01
    window: "hann"
    features: 80
    n_fft: 512
    log: true
    frame_splicing: 1
    dither: 0.00001
    pad_to: 0
    pad_value: 0.0

  spec_augment:
    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
    freq_masks: 2 # set to zero to disable it
    # you may use lower time_masks for smaller models to have a faster convergence
    time_masks: 2 # set to zero to disable it
    freq_width: 27
    time_width: 0.05
```

Any idea what could be the reason behind this error  and how to fix it ?