Repository: pytorch
Issue: PyTorch multiprocessing.spawn seems slow with list of tensors
Labels: module: performance,module: multiprocessing,triaged

## 🐛 Bug

When I use torch.multiprocessing.spawn in distributed GPU training (on a single machine), I observe much slower training times than starting the processes independently.

This is even more true when my Dataset contains a list of tensors. It seems that the shared memory creates a file pointer for each entry in the list and all these filepointers make the training very slow.

In particular, I need to increase the ulimit: `ulimit -n 9999`.

When training exactly the same data and model, but using a tensor of tensors vs a list of tensors, the list of tensors takes around 4 times as long (!) to train.

## To Reproduce

I created a repository with a minimal example (~100 lines of code including everything needed): https://github.com/mpaepper/pytorch_multiprocessing_list_of_tensors

When using the `--use_spawn` flag, it's using multiprocessing.spawn, else you start the processes manually with `--rank`:

```
python custom.py --use_spawn                         # Training time: 17 seconds
python custom.py --use_spawn --use_lists             # Training time: 72 seconds (!)

# Instead of using spawn, start each process independently:
python custom.py --rank 0               # Training time: 14 seconds
python custom.py --rank 1

python custom.py --rank 0 --use_lists   # Training time: 14 seconds
python custom.py --rank 1 --use_lists
```

Basically this is the main change in the Dataset:
```
        if use_lists:
            self.list = []
            for i in range(self.length):
                self.list.append(torch.rand((3, 224, 224)))
        else:
            self.list = torch.rand((self.length, 3, 224, 224))
```

## Expected behavior

I would expect to have `python custom.py --use_spawn` and `python custom.py --use_spawn --use_lists` run in the same amount of time, i.e. just having a list of tensors shouldn't completely slow down my training.

## Environment

Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: Could not collect

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti

Nvidia driver version: 440.33.01
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] numpydoc==0.9.1
[pip] pytorch-lightning==0.7.5
[pip] torch==1.4.0
[pip] torch-lr-finder==0.1.2
[pip] torchbearer==0.5.3
[pip] torchfile==0.1.0
[pip] torchvision==0.4.2
[conda] _pytorch_select           0.1                       cpu_0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.2.89              hfd86e86_0
[conda] mkl                       2019.4                      243
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.0.14           py37ha843d7b_0
[conda] mkl_random                1.1.0            py37hd6b4f25_0
[conda] numpy                     1.18.1                   pypi_0    pypi
[conda] numpy-base                1.17.2           py37hde5b4d6_0
[conda] numpydoc                  0.9.1                      py_0
[conda] pytorch                   1.3.1           cpu_py37h62f834f_0
[conda] pytorch-lightning         0.7.5                    pypi_0    pypi
[conda] torch                     1.4.0                    pypi_0    pypi
[conda] torch-lr-finder           0.1.2                    pypi_0    pypi
[conda] torchbearer               0.5.3                    pypi_0    pypi
[conda] torchfile                 0.1.0                    pypi_0    pypi
[conda] torchvision               0.4.2           cpu_py37h9ec355b_0


cc @VitalyFedyunin @ngimel