Bucketing strategy not working properly for multiple GPUs? Cyclical training loss
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

It's a similar observation to #6084, but since that one is closed and it's slightly different (plus I have some additional information) I'd like to ask again if this is a bug.

I looked at my training loss / batch WER when using bucketing, and found that the loss and WER follow the same pattern for every epoch, even though I specified `fully_randomized` as the bucketing strategy. Later I discovered that because used `pl.seed_everything()`, all the random generators were initialized the same and therefore all the GPUs use the same buckets. I changed this (not seeding now), and printed the `shuffled order` and `dataset_idx` in `RandomizedChainDataset`'s iter method. I can see that now there's some randomization happening when the datasets are loaded, but the train_loss graph still looks exactly the same as before. If I track the loss in the terminal progress bar, I can also see that for each epoch the loss is about in the same range at a certain point, e.g. 25% into the epoch, the loss is always around 100 whereas 75% it's about 50.

My questions are:
- is this desired behavior? I would expect that for example for 2 buckets, instead of following a pattern every epoch, it would have cycle through a pattern every half epoch if it's fully randomized. And even if it's synced randomized, it should be different every epoch.
- does it have to do with the `global_rank` and how the datasets are loaded at each GPU? I'm not familiar with this and it would be good if someone can explain how this works. Because now I see that when I print the `shuffled_order` at the start of every epoch, and I have 8 GPUs, it is printed 8x8 = 64 times. Is this `num_gpus` x `num_workers`?
- I would have expected the randomization of bucketing to still work despite using a random seed for my experiments. Is that doable?

Some graphs:

For training with 2 buckets, data size about 1000 hrs.
Training loss of the same training, orange is when I used random seed, blue is when I stopped the first run, removed random seed and continued from checkpoint. I see random dataset orders printed out, but the the train loss is exactly the same patter every epoch despite the randomization.
<img width="827" alt="image" src="https://user-images.githubusercontent.com/43883260/226186308-3826aee6-e91c-45bf-bf55-6c7d91d651f5.png">

Training batch WER for the same run. It looks slightly `smoothed out` after I removed the random seed, but I would have expected 2 peaks per epoch for example.
<img width="709" alt="image" src="https://user-images.githubusercontent.com/43883260/226186561-56244d11-5b06-488e-8c98-662d36ecd59f.png">

For training with 4 buckets
<img width="428" alt="image" src="https://user-images.githubusercontent.com/43883260/226186732-45ac67fe-ccd6-4faa-b95e-ee1171cc6503.png">
Here I started the training run again without the random seed (blue line) and with a higher batch size. But still it seems like I can clearly see where one bucket ends and another bucket starts because they have different lengths and therefore different average loss. Even though in the terminal during one epoch I see different `dataset_idx` printed out at different times. This training also doesn't converge, I'm wondering if it only has to do with the longer audio or because of the lack of randomization as well.

**Steps/Code to reproduce bug**

used `[speech_to_text_ctc_bpe.py](https://github.com/NVIDIA/NeMo/blob/a0f584a4480165ff144601336280a2276b43d4bd/examples/asr/asr_ctc/speech_to_text_ctc_bpe.py)` to run the training,

Cfg is conformer_ctc_bpe_streaming.yaml.
4 buckets: batch size = 24, 2 buckets batch size = 60 (bucketing batch size)
8 gpus
8 workers
lr: 2
warmup 10000 / 20000

**Expected behavior**

- Training loss should not be uniformly cyclical per epoch training using multiple GPUs and fully randomized bucketing.

**Environment overview (please complete the following information)**

 - Environment location: GCP
 - Method of NeMo install: followed the instruction in the readme to install in a conda env. First using pip install, then pip install -e for printing out the dataset_idx.
 - 8 GPUs


**Environment details**

python: 3.8.0
pytorch: 1.24.1
pytorch-lightning: 1.8.6
nemo-toolkit: 1.15.0

**Additional context**

GPU: A100