Repository: pytorch
Issue: The speed of scatter is influenced by the data size while using nn.DataParallel
Labels: module: performance,module: dataloader,triaged,module: data parallel

## 🐛 Bug

I am using `nn.DataParallel` for a Neural Machine Translation task, when the number of parallel sentences is more than 10 millions, the training speed becomes much slower. After some analysis, I find that the cost of time of  **scatter** operation increased strangely  for some batches.

<!-- A clear and concise description of what the bug is. -->

## Experiment Result

## 100 thousand sentence pairs

Experiment on this small dataset shows that all the batches work well as expected.

```
src size: torch.Size([256, 32, 1])    // model input
tgt size: torch.Size([256, 17, 1])    // model input
src device: 0    // device number of src tensor
tgt device: 0    // device number of tgt tensor
time consumption of scatter: 0.013    // time for nn.DataParallel to scatter data to multi-gpu
time consumption of model processing: 0.141    // time for module to process data
time consumption of gather: 0.001    // time for nn.DataParallel to gather data
time consumption of batch: 0.396    // total time for this batch, which considered some additional operation besides operations mentioned above.
```

## 10 millions sentence pairs

While most of the batches work well as previous experiment, some batches will spend more time on scatter operation on this dataset. As a consequence, the overall training speed becomes slower. I pick a strange batch as below:

```
src size: torch.Size([112, 73, 1])
tgt size: torch.Size([112, 53, 1])
src device: 0
tgt device: 0
time consumption of scatter: 0.714    // which growth from 0.01 to around 0.7 on this dataset
time consumption of model processing: 0.131
time consumption of gather: 0.001
time consumption of batch: 71.116
```

## 70 millions sentence pairs

Compared with the second experiment, some batches spend even much more time on data scatter, and the overall training speed is much slower.
```
src size: torch.Size([88, 93, 1])
tgt size: torch.Size([88, 66, 1])
src device: 0
tgt device: 0
time consumption of scatter: 6.630
time consumption of model processing : 0.133
time consumption of gather: 0.001
time consumption of batch : 7.038
```

## Note:

1. The behavior of strange batches are similar on a dataset.
2. It seems that there is no correlation between strange time consumption and the batch size.
3. The consumption for scatter is calculated by subtract the time before the model wrapped by `DataParallel` is called from the time that `forward` method of real model is called.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
I expect that the time consumption shouldn't be influenced by the dataset size.


## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

PyTorch version: 0.4.1
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: Could not collect

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration:
GPU 0: Tesla P40
GPU 1: Tesla P40

Nvidia driver version: 384.59
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.1.4

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect

## Additional context

<!-- Add any other context about the problem here. -->


cc @SsnL @VitalyFedyunin @ngimel