Repository: pytorch
Issue: Issue with ShufflerIterDataPipe in torch 1.13.1
Labels: module: dataloader,triaged,module: data

### 🐛 Describe the bug


Hi, I am trying to fine-tune a model in PyTorch using multiple GPUs. For this purpose I load a base model from `transformers` and I use the library `accelerate` to handle the multi-gpu setting. I have also created a new dataset class which inherits from `IterableDataset`. In that class, I define a shuffle function which make use of `ShufflerIterDataPipe`, and I use it to shuffle my dataset. From this dataset, I define a dataloader (using `DataLoader` from pytorch) and I go trough it with the help of a for loop.

TL;DR : The code works fine with PyTorch `1.10.1` but fails with PyTorch `1.13.1` due to `ShufflerIterDataPipe`

Here is my code

```python
import torch
from torch.utils.data import IterableDataset
from torch.utils.data.dataloader import DataLoader
from torch.utils.data.datapipes.iter.combinatorics import ShufflerIterDataPipe
from accelerate import Accelerator
from transformers import AutoModelForCausalLM

class ConstantLengthDataset(IterableDataset):
    def __init__(
        self,
        dataset,
    ):
        self.dataset = dataset
    def __iter__(self):
        for element in self.dataset :
            yield element

    def shuffle(self, buffer_size=1000):
        return ShufflerIterDataPipe(self, buffer_size=buffer_size)

model = AutoModelForCausalLM.from_pretrained("codeparrot/codeparrot-small")

train_data = [torch.randint(0, 20000, (1024,)) for _ in range(10000)]
train_dataset = ConstantLengthDataset(train_data,).shuffle(buffer_size=100)
train_dataloader = DataLoader(train_dataset, batch_size=4)

accelerator = Accelerator()
model, train_dataloader = accelerator.prepare(model, train_dataloader)

for step, batch in enumerate(train_dataloader, start=1):
    print(batch.shape)
    print("step = "+str(step+1))
```

However when I use PyTorch `1.13.1`, I have an issue :

- When I do not shuffle my dataset, which implies not using `ShuffleIterDataPipe`, my code works perfectly whether I use 1 GPU or 4 GPUs.

- When I do shuffle my dataset, which implies using `ShufflerIterDataPipe`, my code works only when I use 1 GPU. And it does not work when I use more GPUs (typically 4 or 8). The error is

`RuntimeError: Trying to create tensor with negative dimension -327818479245320786: [-327818479245320786]`

On the other hand, when I use PyTorch `1.10.1` there is no issue when using `ShuffleIterDataPipe`, whether I run my code with 1 or 4 GPUs.


Can you give me some information about why it happens? Maybe there is a problem with `ShufflerIterDataPipe` when I use multiple GPUs. The update from torch `1.10.1` to torch `1.13.1` may have changed something.

The whole trace in case of error

```
  File "/fsx/armel/miscellaneous/minimal.py", line 35, in <module>
    for step, batch in enumerate(train_dataloader, start=1):
  File "/fsx/armel/miniconda3/envs/env/lib/python3.10/site-packages/accelerate/data_loader.py", line 528, in __iter__
    next_batch, next_batch_info = self._fetch_batches(main_iterator)
  File "/fsx/armel/miniconda3/envs/env/lib/python3.10/site-packages/accelerate/data_loader.py", line 506, in _fetch_batches
    broadcast_object_list(batch_info)
  File "/fsx/armel/miniconda3/envs/env/lib/python3.10/site-packages/accelerate/utils/operations.py", line 339, in broadcast_object_list
    torch.distributed.broadcast_object_list(object_list, src=from_process)
  File "/fsx/armel/miniconda3/envs/env/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2090, in broadcast_object_list
    object_tensor = torch.empty(  # type: ignore[call-overload]
RuntimeError: Trying to create tensor with negative dimension -327818479245320786: [-327818479245320786]
```
The expected output should be like 

```
torch.Size([4, 1024])
step = 2
torch.Size([4, 1024])
step = 3
torch.Size([4, 1024])
step = 4
torch.Size([4, 1024])
step = 5
torch.Size([4, 1024])
step = 6
torch.Size([4, 1024])
step = 7
torch.Size([4, 1024])
step = 8
torch.Size([4, 1024])
step = 9
torch.Size([4, 1024])
step = 10
torch.Size([4, 1024])
...
```

### Versions

Environment with torch `1.13.1`

```
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.24.3
Libc version: glibc-2.31

Python version: 3.10.11 (main, Apr 20 2023, 19:02:41) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-1023-aws-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
GPU 2: NVIDIA A100-SXM4-80GB
GPU 3: NVIDIA A100-SXM4-80GB
GPU 4: NVIDIA A100-SXM4-80GB
GPU 5: NVIDIA A100-SXM4-80GB
GPU 6: NVIDIA A100-SXM4-80GB
GPU 7: NVIDIA A100-SXM4-80GB

Nvidia driver version: 510.73.08
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          96
On-line CPU(s) list:             0-95
Thread(s) per core:              2
Core(s) per socket:              24
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
Stepping:                        7
CPU MHz:                         2999.998
BogoMIPS:                        5999.99
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       1.5 MiB
L1i cache:                       1.5 MiB
L2 cache:                        48 MiB
L3 cache:                        71.5 MiB
NUMA node0 CPU(s):               0-23,48-71
NUMA node1 CPU(s):               24-47,72-95
Vulnerability Itlb multihit:     KVM: Mitigation: VMX unsupported
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Mmio stale data:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:          Vulnerable
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==1.13.1
[pip3] torch-struct==0.5
[pip3] torchaudio==0.13.1
[pip3] torchvision==0.14.1
[conda] blas                      1.0                         mkl  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0           py310h7f8727e_0  
[conda] mkl_fft                   1.3.1           py310hd6ae3a3_0  
[conda] mkl_random                1.2.2           py310h00e6091_0  
[conda] numpy                     1.24.3          py310hd5efca6_0  
[conda] numpy-base                1.24.3          py310h8e6c178_0  
[conda] pytorch                   1.13.1          py3.10_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-cuda              11.6                 h867d48c_1    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch-struct              0.5                      pypi_0    pypi
[conda] torchaudio                0.13.1              py310_cu116    pytorch
[conda] torchvision               0.14.1              py310_cu116    pytorch
```


Environment with torch `1.10.1`

```
PyTorch version: 1.10.1
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.24.3
Libc version: glibc-2.31

Python version: 3.9.16 (main, Mar  8 2023, 14:00:05)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-1023-aws-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: 
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
GPU 2: NVIDIA A100-SXM4-80GB
GPU 3: NVIDIA A100-SXM4-80GB
GPU 4: NVIDIA A100-SXM4-80GB
GPU 5: NVIDIA A100-SXM4-80GB
GPU 6: NVIDIA A100-SXM4-80GB
GPU 7: NVIDIA A100-SXM4-80GB

Nvidia driver version: 510.73.08
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          96
On-line CPU(s) list:             0-95
Thread(s) per core:              2
Core(s) per socket:              24
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
Stepping:                        7
CPU MHz:                         2999.998
BogoMIPS:                        5999.99
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       1.5 MiB
L1i cache:                       1.5 MiB
L2 cache:                        48 MiB
L3 cache:                        71.5 MiB
NUMA node0 CPU(s):               0-23,48-71
NUMA node1 CPU(s):               24-47,72-95
Vulnerability Itlb multihit:     KVM: Mitigation: VMX unsupported
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Mmio stale data:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:          Vulnerable
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==1.10.1
[pip3] torchaudio==0.10.1
[pip3] torchvision==0.11.2
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2023.1.0         h6d00ec8_46342  
[conda] mkl-service               2.4.0            py39h5eee18b_1  
[conda] mkl_fft                   1.3.6            py39h417a72b_1  
[conda] mkl_random                1.2.2            py39h417a72b_1  
[conda] numpy                     1.24.3           py39hf6e8229_1  
[conda] numpy-base                1.24.3           py39h060ed82_1  
[conda] pytorch                   1.10.1          py3.9_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.10.1               py39_cu113    pytorch
[conda] torchvision               0.11.2               py39_cu113    pytorch
```


cc @SsnL @VitalyFedyunin @ejguan @NivekT @dzhulgakov