Repository: pytorch
Issue: Parallel data loader performance degradation for IterableDataset with num_workers > 1 (but not for Dataset).
Labels: module: dataloader,triaged

## 🐛 Bug

Data loader performance degrades with > 1 data loader worker when using `IterableDataset` but not when using (map-style) `Dataset`.

## To Reproduce

Steps to reproduce the behavior:

My experiment (code below) consisted of:

1. Make a (Iterable)Dataset class. This synthesizes dummy data and possibly adds a time delay to simulate batch loading work.
2. Make a Dataloader that consumes the above dataset instance. The number of workers varied from 0 (loading in the main process) to 4.
3. Iterate through N=10 batches of data yielded by the dataloader, with possibly a time delay to simulate batch processing work (e.g. training).

Here are my measurements for `IterableDataset`:

```
(num_workers=0, batch_loading_time=1,  batch_process_time=1) execution finished. Time elapsed = 20.021s.
(num_workers=1, batch_loading_time=1,  batch_process_time=1) execution finished. Time elapsed = 11.433s.
(num_workers=2, batch_loading_time=1,  batch_process_time=1) execution finished. Time elapsed = 21.469s.
(num_workers=3, batch_loading_time=1,  batch_process_time=1) execution finished. Time elapsed = 31.517s.
(num_workers=4, batch_loading_time=1,  batch_process_time=1) execution finished. Time elapsed = 41.545s.

(num_workers=0, batch_loading_time=1,  batch_process_time=0.1) execution finished. Time elapsed = 11.014s.
(num_workers=1, batch_loading_time=1,  batch_process_time=0.1) execution finished. Time elapsed = 10.518s.
(num_workers=2, batch_loading_time=1,  batch_process_time=0.1) execution finished. Time elapsed = 10.643s.
(num_workers=3, batch_loading_time=1,  batch_process_time=0.1) execution finished. Time elapsed = 10.745s.
(num_workers=4, batch_loading_time=1,  batch_process_time=0.1) execution finished. Time elapsed = 10.862s.

(num_workers=0, batch_loading_time=0.1, batch_process_time=1) execution finished. Time elapsed = 11.013s.
(num_workers=1, batch_loading_time=0.1, batch_process_time=1) execution finished. Time elapsed = 10.515s.
(num_workers=2, batch_loading_time=0.1, batch_process_time=1) execution finished. Time elapsed = 20.541s.
(num_workers=3, batch_loading_time=0.1, batch_process_time=1) execution finished. Time elapsed = 30.617s.
(num_workers=4, batch_loading_time=0.1, batch_process_time=1) execution finished. Time elapsed = 40.610s.

(num_workers=0, batch_loading_time=0.1, batch_process_time=0.1) execution finished. Time elapsed = 2.004s.
(num_workers=1, batch_loading_time=0.1, batch_process_time=0.1) execution finished. Time elapsed = 1.543s.
(num_workers=2, batch_loading_time=0.1, batch_process_time=0.1) execution finished. Time elapsed = 2.553s.
(num_workers=3, batch_loading_time=0.1, batch_process_time=0.1) execution finished. Time elapsed = 3.593s.
(num_workers=4, batch_loading_time=0.1, batch_process_time=0.1) execution finished. Time elapsed = 4.649s.
```
This is the code I used to run this experiment:

```
import os
import time
import itertools
import numpy as np
import torch


class TimedBlock:
    """
    Context manager to measure wall time.
    """

    def __init__(self, name):
        self.name = name

    def __enter__(self):
        self.t_start = time.perf_counter()

    def __exit__(self, *args):
        print("(%d) %s execution finished. Time elapsed = %.3fs." % (os.getpid(), self.name,
                                                                     time.perf_counter() - self.t_start))


class IterableDataset(torch.utils.data.IterableDataset):
    """
    Synthetic iterable dataset that simulates data loading work.
    """

    def __init__(self, n_items, shape, batch_loading_time):
        """
        Args:
            n_items: number of items to return.
            shape: shape of batch tensor to return.
            batch_loading_time: each batch will take at least this many seconds to return.
        """
        self.n_items = n_items
        self.shape = shape
        self.batch_loading_time = batch_loading_time

    def __iter__(self):
        self.i = 0
        while self.i < self.n_items:
            t_start = time.perf_counter()
            X = np.random.randn(*self.shape)
            t_remaining =  self.batch_loading_time - (time.perf_counter() - t_start)
            if t_remaining > 0:
                time.sleep(t_remaining)
            yield X
            self.i += 1


class Dataset(torch.utils.data.Dataset):
    """
    Synthetic map-style dataset that simulates data loading work.
    """

    def __init__(self, n_items, shape, batch_loading_time):
        """
        Args:
            n_items: number of items to return.
            shape: shape of batch tensor to return.
            batch_loading_time: each batch will take at least this many seconds to return.
        """
        self.n_items = n_items
        self.shape = shape
        self.batch_loading_time = batch_loading_time

    def __len__(self):
        return self.n_items

    def __getitem__(self, idx):
        t_start = time.perf_counter()
        X = np.random.randn(*self.shape)
        t_remaining =  self.batch_loading_time - (time.perf_counter() - t_start)
        if t_remaining > 0:
            time.sleep(t_remaining)
        return X


def test_simpleloader(n_iters=int(1e2), shape=(1,), num_workers=0, batch_loading_time=0, batch_process_time=0,
                      exp_name="simpleloader", dataset_cls=IterableDataset):
    """
    Load and process n_iters batches of data using a dataloader and one of the above dataset classes. We simulate
    training by waiting batch_process_time seconds per batch.
    """

    dataset = dataset_cls(n_iters, shape, batch_loading_time)
    data_loader = torch.utils.data.DataLoader(dataset,
                                             batch_size=None,
                                             num_workers=num_workers,
                                             collate_fn=None,
                                             pin_memory=False,
                                             worker_init_fn=None,
                                             multiprocessing_context=None)
    
    with TimedBlock(exp_name):
        for i, x in enumerate(data_loader):
            time.sleep(batch_process_time)
            pass


def simpleloader_grid(num_workers_grid=[0, 1, 2, 3, 4],
                      batch_loading_time_grid=[10**i for i in range(0, -2, -1)],
                      batch_process_time_grid=[10**i for i in range(0, -2, -1)],
                      dataset_cls=IterableDataset):

    n_iters = int(1e1)
    shape = (1,)

    params = itertools.product(num_workers_grid, batch_loading_time_grid, batch_process_time_grid)

    for p in params:
        test_simpleloader(n_iters, shape, p[0], p[1], p[2], str(p), dataset_cls=dataset_cls)


if __name__ == "__main__":
    torch.multiprocessing.set_start_method("spawn")
    t_start = time.perf_counter()
    simpleloader_grid(dataset_cls=IterableDataset)
    print("Main execution time = %.3fs" % (time.perf_counter() - t_start))
```



## Expected behavior

Here are my measurements for (map-style)`Dataset`, which I would also expect from `IterableDataset`:

```
(num_workers=0, batch_loading_time=1,   batch_process_time=1) execution finished. Time elapsed = 20.022s.
(num_workers=1, batch_loading_time=1,   batch_process_time=1) execution finished. Time elapsed = 11.432s.
(num_workers=2, batch_loading_time=1,   batch_process_time=1) execution finished. Time elapsed = 11.455s.
(num_workers=3, batch_loading_time=1,   batch_process_time=1) execution finished. Time elapsed = 11.493s.
(num_workers=4, batch_loading_time=1,   batch_process_time=1) execution finished. Time elapsed = 11.479s.

(num_workers=0, batch_loading_time=1,   batch_process_time=0.1) execution finished. Time elapsed = 11.014s.
(num_workers=1, batch_loading_time=1,   batch_process_time=0.1) execution finished. Time elapsed = 10.529s.
(num_workers=2, batch_loading_time=1,   batch_process_time=0.1) execution finished. Time elapsed = 5.640s.
(num_workers=3, batch_loading_time=1,   batch_process_time=0.1) execution finished. Time elapsed = 4.537s.
(num_workers=4, batch_loading_time=1,   batch_process_time=0.1) execution finished. Time elapsed = 3.652s.

(num_workers=0, batch_loading_time=0.1, batch_process_time=1) execution finished. Time elapsed = 11.012s.
(num_workers=1, batch_loading_time=0.1, batch_process_time=1) execution finished. Time elapsed = 10.502s.
(num_workers=2, batch_loading_time=0.1, batch_process_time=1) execution finished. Time elapsed = 10.538s.
(num_workers=3, batch_loading_time=0.1, batch_process_time=1) execution finished. Time elapsed = 10.594s.
(num_workers=4, batch_loading_time=0.1, batch_process_time=1) execution finished. Time elapsed = 10.604s.

(num_workers=0, batch_loading_time=0.1, batch_process_time=0.1) execution finished. Time elapsed = 2.005s.
(num_workers=1, batch_loading_time=0.1, batch_process_time=0.1) execution finished. Time elapsed = 1.518s.
(num_workers=2, batch_loading_time=0.1, batch_process_time=0.1) execution finished. Time elapsed = 1.540s.
(num_workers=3, batch_loading_time=0.1, batch_process_time=0.1) execution finished. Time elapsed = 1.601s.
(num_workers=4, batch_loading_time=0.1, batch_process_time=0.1) execution finished. Time elapsed = 1.572s
```

## Environment

```Collecting environment information...
PyTorch version: 1.3.0
Is debug build: No
CUDA used to build PyTorch: 10.1.243

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: Could not collect

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration: 
GPU 0: GeForce GTX 1060 6GB
GPU 1: GeForce GTX 1060 6GB

Nvidia driver version: 430.50
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.4

Versions of relevant libraries:
[pip] numpy==1.17.2
[pip] torch==1.3.0
[pip] torch-cluster==1.4.3
[pip] torch-geometric==1.3.0
[pip] torch-scatter==1.4.0
[pip] torch-sparse==0.4.3
[pip] torch-spline-conv==1.1.0
[pip] torchcontrib==0.0.2
[pip] torchvision==0.2.2
[conda] _tflow_select             2.3.0                       mkl  
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.3                      199  
[conda] mkl_fft                   1.0.10           py37ha843d7b_0  
[conda] mkl_random                1.0.2            py37hd81dba3_0  
[conda] tensorflow                1.13.1          mkl_py37h54b294f_0  
[conda] tensorflow-base           1.13.1          mkl_py37h7ce6ba3_0  
[conda] torch                     1.3.0                    pypi_0    pypi
[conda] torch-cluster             1.4.3                     dev_0    <develop>
[conda] torch-geometric           1.3.0                    pypi_0    pypi
[conda] torch-scatter             1.4.0                    pypi_0    pypi
[conda] torch-sparse              0.4.3                    pypi_0    pypi
[conda] torch-spline-conv         1.1.0                    pypi_0    pypi
[conda] torchcontrib              0.0.2                    pypi_0    pypi
[conda] torchvision               0.2.2                      py_3    pytorch
```

## Additional context

<!-- Add any other context about the problem here. -->


cc @SsnL