Cyclical behavior of tarred ASR training data sets
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Using tarred buckets with ASR data exhibits cyclical behavior that makes it difficult for ASR models to converge in some situations.
Not sure if I'm using it wrong, or if this is be design.

I believe the problem comes from here:

https://github.com/NVIDIA/NeMo/blob/d06d353ae3ce1224d95dc3cdeffb987f66d911ab/nemo/collections/asr/data/audio_to_text.py#L1286-L1287


For large ASR training sets, I use tarred buckets with the typical 8 buckets up to ~18 seconds max. This issue happens across many languages with a similar setup.


<details>
  <summary>Example config</summary>

  ```yaml

  train_ds:
    sample_rate: 16000
    batch_size: 1  # 1 because we're using tarred datasets
    shuffle: true
    num_workers: 8
    pin_memory: true
    use_start_end_token: false
    trim_silence: false
    max_duration: 20.0
    min_duration: 1.0
    shuffle_n: 512
    is_tarred: true
    manifest_filepath:
      - - /path/to/bucket1/tarred_audio_manifest.json
      - - /path/to/bucket2/tarred_audio_manifest.json
      - - /path/to/bucket3/tarred_audio_manifest.json
      - - /path/to/bucket4/tarred_audio_manifest.json
      - - /path/to/bucket5/tarred_audio_manifest.json
      - - /path/to/bucket6/tarred_audio_manifest.json
      - - /path/to/bucket7/tarred_audio_manifest.json
      - - /path/to/bucket8/tarred_audio_manifest.json
    tarred_audio_filepaths:
      - - /path/to/bucket1/audio_{0..511}.tar
      - - /path/to/bucket2/audio_{0..511}.tar
      - - /path/to/bucket3/audio_{0..511}.tar
      - - /path/to/bucket4/audio_{0..511}.tar
      - - /path/to/bucket5/audio_{0..511}.tar
      - - /path/to/bucket6/audio_{0..511}.tar
      - - /path/to/bucket7/audio_{0..511}.tar
      - - /path/to/bucket8/audio_{0..511}.tar
    bucketing_batch_size: [44, 26, 18, 14, 11, 9, 8, 7]
  ```
</details>


Fine-tuning a large Conformer, this is what I see when evaluating every 1/4 epoch (domain fine-tuning, expect a couple point drop in WER):

![image](https://user-images.githubusercontent.com/103518212/220742240-dd40c2c2-b254-4868-8124-15e5cedec192.png)

Since this is a fine-tuning, so a low LR is used (effective LR on the order of 1e-5), and LR is unlikely the problem w.r.t. convergence. 

Re-scaling shows that the two datasets are negatively correlated, as the `RandomizedChainDataset` tunes the model to different buckets:

![image](https://user-images.githubusercontent.com/103518212/220743868-6d321c96-31f6-4559-bb1d-5d593bf9e88d.png)

I suspect this is because `RandomizedChainDataset` exhausts each bucket in turn (as linked above) and does not re-shuffle at each epoch, and the training set duration are clustered as shown here (four common, large, publicly-available datasets):

![image](https://user-images.githubusercontent.com/103518212/220743372-dc42bea8-adb7-41db-99c6-abc87c14c3bc.png)

So different buckets are dominated by different datasets, and exhausting a bucket without training from the others fine-tunes the model to a particular dataset.

**Expected behavior**

* Sample from a random bucket each step, rather than exhausting each bucket in turn.
* Use different order each epoch.