Tarred datasets for ASR results in ultra-slow training
State: closed | Pull request: no

### Discussed in https://github.com/NVIDIA/NeMo/discussions/4217

**Original post:**
<div type='discussions-op-text'>

<sup>Originally posted by **riqiang-dp** May 20, 2022</sup>
I'm trying to train ASR models on a pretty unbalanced data set (lengthwise, a lot short utterances but also long utterances (up to 1 min)). I have been able to train with smaller batch size and also with some pre-segmentation to shorten the max duration. I'm now just trying bucketing, but it seems like it only works with tarred data sets.

After tarring the original set and using some bucketing, I'm seeing that the estimated time for one epoch goes up to ~7h instead of 2h for training with the raw form audio files. If I don't do bucketing and still use the tarred version, I get ~10h estimated time for one epoch.

I'm setting something like 4 buckets, 128 shards. Using A100 GPUs on a Google Cloud VM, using NeMo 1.7.0

I'm just wondering, are these numbers normal? If so when does it make more sense to use tarred + bucketing strategy versus just train using the wav files?
</div>

**Updated info**
The info below doesn't 100% align with the description above but the gist is the same. For example I have used just shorter audio for further testing, instead of using an unbalanced dataset. But the slow down is still happening. For something that's usually 20-30 mins / epoch with wav files, it's hours for tar files. I've tested a lot of combinations still don't know why this is happening. Everything's stored in a disk attached to the GCP VM. Please see details below. Otherwise, I did notice in the logs that the train_step_timing looks normal (apart from some very high spikes) but when I observe the progress bar it seems to pause every `batch_size` steps.

Below are the stats of the tarred dataset and the training steps/speed

> bucket 1 with min_duration=2.5 and max_duration=6.875
> 172416 files
> bucket 2 with min_duration=6.875 and max_duration=11.25
> 107136 files
> bucket 3 with min_duration=11.25 and max_duration=15.625
> 105600 files
> bucket 4 with min_duration=15.625 and max_duration=20.00001
> 136192 files
> total: 521344
> 
> eval files: 2406
> eval batch size: 8 (these are long form audio)
> 
> num gpus: 8 (for all the cases)
> batch size: 1
> bucketing batch size: 30
> bucketing strategy: synced_randomized 
> step per epoch (bucketing): 8966
> training time per epoch: 1h15m
> 
> batch size: 30
> step per epoch (non bucketing): 17417
> training time per epoch: 1h45m

However, if I just train the original wav data set (same eval set)

> total files: 521543
> train batch size: 30
> step per epoch: 2212
> training time per epoch: 20m

code to create the tarred / bucketed dataset

```
python <scripts path>/convert_to_tarred_audio_dataset.py \
        --manifest_path=data/train.json \
        --target_dir=data/train \
        --num_shards=128 \
        --max_duration=20 \
        --min_duration=2.5 \
        --shuffle \
        --shuffle_seed=1 \
        --sort_in_shards \
        --workers=32 \
        --buckets_num=4
```
Extra args to make my normal training script work with tarred dataset
```
      model.train_ds.manifest_filepath=[[data/train/bucket1/tarred_audio_manifest.json],[data/train/bucket2/tarred_audio_manifest.json],[data/train/bucket3/tarred_audio_manifest.json],[data/train/bucket4/tarred_audio_manifest.json]] \
       +model.train_ds.tarred_audio_filepaths=[[data/train/bucket1/audio__OP_0..127_CL_.tar],[data/train/bucket2/audio__OP_0..127_CL_.tar],[data/train/bucket3/audio__OP_0..127_CL_.tar],[data/train/bucket4/audio__OP_0..127_CL_.tar]] \
       +model.train_ds.bucketing_strategy=synced_randomized \
       +model.train_ds.bucketing_batch_size=30 \
       model.train_ds.batch_size=1 \
       +model.train_ds.tarred_shard_strategy="scatter" \
       model.train_ds.shuffle_n=2048 \
       model.train_ds.is_tarred=True \
       model.train_ds.max_duration=20 \
       model.train_ds.min_duration=2.5 \
       model.train_ds.num_workers=32 \
       model.train_ds.pin_memory=True \
```

Environment: Using 8 A100 GPUs on a single Google Cloud VM, using NeMo 1.7.0

I have also tried setting num_workers to 8 as you guys suggested, doesn't seem to affect the speed much.