Repository: pytorch
Issue: dataloader bug need help
Labels: module: dataloader,triaged

Hello. I use python3+torch1.5_cu101, I want to use multi gpu for training, so I want to speed up the dataloading speed ,and then I set dataloader(num_worker=0),  and after a 1 day the following bug emits

'7264850 /mnt/mnt1/wav/G0825/session01/DTM2_100_ahead_000_G0825_S5786_00.wav
7264851 (81920,)
7264852 (81920,)
7264853 (81920,)
7264854 (81920,)
7264855 (81920,)
7264856 (81920,)/pytorch/aten/src/ATen/native/BinaryOps.cpp:81: UserWarning: Integer         division of tensors using div or / is deprecated, and in a future release d        iv will perform true division as in Python 3. Use true_divide or floor_divid        e (// in Python) instead.
7264857 Traceback (most recent call last):
7264858   File "train.py", line 91, in <module>
7264859     trainer.train()
7264860   File "/root/panxin_m/src/trainer.py", line 25, in train
7264861     train_loss = self.iter_one_epoch()
7264862   File "/root/panxin_m/src/trainer.py", line 48, in iter_one_epoch
7264863     train_data = next(loader_iter)
7264864   File "/root/.local/lib/python3.7/site-packages/torch/utils/data/dataloader        .py", line 345, in __next__
7264865     data = self._next_data()
7264866   File "/root/.local/lib/python3.7/site-packages/torch/utils/data/dataloader        .py", line 856, in _next_data
7264867     return self._process_data(data)
7264868   File "/root/.local/lib/python3.7/site-packages/torch/utils/data/dataloader        .py", line 881, in _process_data
7264869     data.reraise()
7264870   File "/root/.local/lib/python3.7/site-packages/torch/_utils.py", line 395,         in reraise
7264871     raise self.exc_type(msg)
7264872 ValueError: Caught ValueError in DataLoader worker process 6.
7264873 Original Traceback (most recent call last):
7264874   File "/root/.local/lib/python3.7/site-packages/torch/utils/data/_utils/wor        ker.py", line 178, in _worker_loop
7264875     data = fetcher.fetch(index)
7264876   File "/root/.local/lib/python3.7/site-packages/torch/utils/data/_utils/fet        ch.py", line 47, in fetch
7264877     return self.collate_fn(data)
7264878   File "/root/panxin_m/src/data.py", line 139, in __call__
7264879     padded_waveforms, lengths = load_batch_waves(paths, self.channels)
7264880   File "/root/panxin_m/src/data.py", line 99, in load_batch_waves
7264881     sample_rate, waveform = utils.load_wave(path, channels)
7264882   File "/root/panxin_m/src/utils.py", line 33, in load_wave
7264883     datas = np.append(datas, data[np.newaxis,:], axis=0)
7264884   File "<__array_function__ internals>", line 6, in append
7264885   File "/root/anaconda3/lib/python3.7/site-packages/numpy/lib/function_base.        py", line 4693, in append
7264886     return concatenate((arr, values), axis=axis)
7264887   File "<__array_function__ internals>", line 6, in concatenate
7264888 ValueError: all the input array dimensions for the concatenation axis must m        atch exactly, but along dimension 1, the array at index 0 has size 69632 and         the array at index 1 has size 36677
7264889'

In the experiment I read wave audio files, each file has 16 input channels. And I can 100% sure that the length of each channels are the same. but I dont know why in one of the workers, this error occurs. seems the dataloader did not read different channels and concat them as I expected. can you help me if I made something wrong ,and thank you.

cc @SsnL @VitalyFedyunin @ejguan