Repository: pytorch
Issue: DataLoader performance drop on 4-channel images?
Labels: needs reproduction,module: dataloader,triaged

## 🐛 Bug


I have a dataset with RGBA images and have noticed that iteration over this data (with a DataLoader) is considerably slower, than when I get rid of the alpha channel. I use TensorDataset (so it's not an I/O issue) and provide code to reproduce it in a Jupyter Notebook, with TQDM measuring iteration speed.

## To Reproduce

Steps to reproduce the behavior:

```
import torch
from tqdm import tqdm_notebook as tqdm
from torch.utils.data import TensorDataset, DataLoader


ds_rgb = TensorDataset(
    torch.rand(10000, 3, 164, 64), # X
    torch.rand(10000), # y
)

ds_rgba = TensorDataset(
    torch.rand(10000, 4, 164, 64), #X
    torch.rand(10000), #y

)

for _ in tqdm(ds_rgb, "d3"):
    pass


for _ in tqdm(ds_rgba, "d4"):
    pass

dl_rgb_regular = DataLoader(ds_rgb, batch_size=64)
dl_rgba_regular = DataLoader(ds_rgba, batch_size=64)



for _ in tqdm(dl_rgb_regular, "l3r"):
    pass


for _ in tqdm(dl_rgba_regular, "l4r"):
    pass

```

which results in:

```
d3
100% 10000/10000 [00:00<00:00, 82409.47it/s]

d4
100% 10000/10000 [00:00<00:00, 75360.41it/s]

l3r
100% 157/157 [00:01<00:00, 133.63it/s]

l4r
100% 157/157 [00:28<00:00, 5.54it/s]
```

## Expected behavior

I expect DataLoader to perform equally well on RGB and RGBA images

## Environment

```
Collecting environment information...
PyTorch version: 1.7.1+cu101
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.2 LTS (x86_64)
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: GeForce RTX 2080 Ti
Nvidia driver version: 418.67
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] efficientnet-pytorch==0.6.3
[pip3] numpy==1.16.4
[pip3] numpy-indexed==0.3.5
[pip3] torch==1.7.1+cu101
[pip3] torchaudio==0.7.2
[pip3] torchvision==0.8.2+cu101
[conda] Could not collect
```

## Additional context

I believe the issue may be in the collate function. I implemented my own collate FN, which speeds up iteration over RGBA a bit, but slows down iteration over RGB:


```
# continued from reproduce snippet
def col_fn(batch):
    rgb = torch.stack([b[0][:3] for b in batch], 0)
    a = torch.stack([b[0][3:] for b in batch], 0)
    y = torch.stack([b[1] for b in batch], 0)
    rgba = torch.cat([rgb, a], dim=1)
    return rgba, y

dl_rgb_custom = DataLoader(ds_rgb, batch_size=64, collate_fn=col_fn)
dl_rgba_custom = DataLoader(ds_rgba, batch_size=64, collate_fn=col_fn)


for _ in tqdm(dl_rgb_custom, "l3c"):
    pass


for _ in tqdm(dl_rgba_custom, "l4c"):
    pass
```

resulting in:

```
l3c
100% 157/157 [00:03<00:00, 47.18it/s]

l4c
100% 157/157 [00:06<00:00, 23.24it/s]
```



cc @SsnL @VitalyFedyunin @ejguan