Repository: pytorch
Issue: DataLoader is slow in spawned processes
Labels: module: performance,module: multiprocessing,module: dataloader,triaged

## 🐛 Bug

Iterating a dataloader in a spawned process is much slower compared to the main process.
In my case, dataloading is even slower than a complete forward + backward pass.

## To Reproduce

Steps to reproduce the behavior:

1. Use `torch.multiprocessing` to spawn a process
2. Iterate a `DataLoader` in the spawned process

Here's a code snippet measuring the iteration times, both in the main process and in a spawn process:
```python
from time import time

from torch import multiprocessing as mp
from torch.utils.data import DataLoader
from torchvision.datasets import FakeData
from torchvision.transforms import ToTensor


def measure_dataloader_times(dataset):
    dataloader = DataLoader(dataset, batch_size=32, pin_memory=True, num_workers=2, shuffle=False)

    i = 0
    times = []
    before = time()
    for _ in dataloader:
        after = time()
        time_taken = after - before
        print(f'iter {i}: time taken = {time_taken}')
        times.append(time_taken)
        i += 1
        before = time()
    print(f'average time (ignoring first iter): {sum(times[1:]) / (len(times) - 1)}')


def main():
    dataset = FakeData(size=32 * 11, transform=ToTensor(), num_classes=1000)
    print('Single process times:')
    measure_dataloader_times(dataset)
    print(f'Multiprocessing times:')
    ctx = mp.get_context('spawn')
    process = ctx.Process(target=measure_dataloader_times, args=(dataset,))
    process.start()
    process.join()


if __name__ == '__main__':
    main()
```

Output:
>Single process times:
iter 0: time taken = 1.5959210395812988
iter 1: time taken = 0.009970903396606445
iter 2: time taken = 0.00590062141418457
iter 3: time taken = 0.006066799163818359
iter 4: time taken = 0.07842516899108887
iter 5: time taken = 0.008443117141723633
iter 6: time taken = 0.06877660751342773
iter 7: time taken = 0.00980687141418457
iter 8: time taken = 0.06483817100524902
iter 9: time taken = 0.00932002067565918
iter 10: time taken = 0.06453561782836914
**average time (ignoring first iter): 0.03260838985443115**
Multiprocessing times:
iter 0: time taken = 34.27748417854309
iter 1: time taken = 5.0067901611328125e-05
iter 2: time taken = 0.9114985466003418
iter 3: time taken = 5.53131103515625e-05
iter 4: time taken = 2.170708417892456
iter 5: time taken = 5.2928924560546875e-05
iter 6: time taken = 2.159883975982666
iter 7: time taken = 2.2172927856445312e-05
iter 8: time taken = 2.065272331237793
iter 9: time taken = 1.7642974853515625e-05
iter 10: time taken = 1.979146957397461
**average time (ignoring first iter): 0.9286708354949951**

The first iteration is much slower than the rest of the iterations in both cases, I'm assuming some initialization is performed in that iteration. Regardless, even when ignoring that first iteration, you can clearly see that dataloading in the spawned process is almost x30 slower than in the main process. You can also see a weird pattern in the spawned process, where it cycles between quick and slow iterations.

## Expected behavior

Speed should be similar to the main process, 

## Environment

Collecting environment information...
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect (CUDA Version 10.2.89, checked manually using the /usr/local/cuda/version.txt file)
GPU models and configuration:
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti
GPU 4: GeForce RTX 2080 Ti
GPU 5: GeForce RTX 2080 Ti
GPU 6: GeForce RTX 2080 Ti
GPU 7: GeForce RTX 2080 Ti

Nvidia driver version: 440.33.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.7.1
[pip3] torchaudio==0.7.0a0+a853dff
[pip3] torchvision==0.8.2
[conda] blas 1.0 mkl
[conda] cudatoolkit 10.2.89 hfd86e86_1
[conda] mkl 2020.2 256
[conda] mkl-service 2.3.0 py38he904b0f_0
[conda] mkl_fft 1.2.0 py38h23d657b_0
[conda] mkl_random 1.1.1 py38h0573a6f_0
[conda] numpy 1.19.2 py38h54aff64_0
[conda] numpy-base 1.19.2 py38hfa32c7d_0
[conda] pytorch 1.7.1 py3.8_cuda10.2.89_cudnn7.6.5_0 pytorch
[conda] torchaudio 0.7.2 py38 pytorch
[conda] torchvision 0.8.2 py38_cu102 pytorch

## Additional context

I can't use `fork` since my main process initializes `CUDA` prior to starting the child process.

cc @SsnL @VitalyFedyunin @ejguan @ngimel