Repository: pytorch
Issue: One GPU is more memory efficient than Multiple GPUs
Labels: module: multi-gpu,triaged,module: data parallel

## Issue description

Multiple GPUs runs out of memory with ``DataParallel`` while one GPU handles the load. 

```
Hidden Size: 7150
Input Size: 1024
Sequence Length: 1024
Batch Size: 64
====================================================================================================
One GPU...
====================================================================================================
Multiple GPUs...
abc.py:16: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().
  output, hidden_state = self.rnn(input_[i].unsqueeze(0))
Traceback (most recent call last):
  File "abc.py", line 50, in <module>
    module=model, inputs=input_, dim=1, output_device=device.index)[-1]
  File "/home/michaelp/.local/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 168, in data_parallel
    outputs = parallel_apply(replicas, inputs, module_kwargs, used_device_ids)
  File "/home/michaelp/.local/lib/python3.5/site-packages/torch/nn/parallel/parallel_apply.py", line 77, in parallel_apply
    raise output
  File "/home/michaelp/.local/lib/python3.5/site-packages/torch/nn/parallel/parallel_apply.py", line 53, in _worker
    output = module(*input, **kwargs)
  File "/home/michaelp/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "abc.py", line 16, in forward
    output, hidden_state = self.rnn(input_[i].unsqueeze(0))
  File "/home/michaelp/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/michaelp/.local/lib/python3.5/site-packages/torch/nn/modules/rnn.py", line 192, in forward
    output, hidden = func(input, self.all_weights, hx, batch_sizes)
  File "/home/michaelp/.local/lib/python3.5/site-packages/torch/nn/_functions/rnn.py", line 324, in forward
    return func(input, *fargs, **fkwargs)
  File "/home/michaelp/.local/lib/python3.5/site-packages/torch/nn/_functions/rnn.py", line 288, in forward
    dropout_ts)
RuntimeError: CUDA error: out of memory
```

## Code example

This is a minimal code example of my actual model. I am unable to use multiple GPUs due to this issue.

```python
import torch

from torch import nn


class Model(nn.Module):

    def __init__(self, *args):
        super().__init__()
        self.rnn = nn.LSTM(*args)

    def forward(self, input_):
        hidden_state = None
        outputs = []
        for i in range(input_.shape[0]):
            output, hidden_state = self.rnn(input_[i].unsqueeze(0))
            outputs.append(output)
        return outputs


hidden_size = 7150
input_size = 1024
sequence_length = 1024
batch_size = 64
model = Model(input_size, hidden_size)
input_ = torch.FloatTensor(sequence_length, batch_size, input_size).uniform_(0, 1)

device = torch.device('cuda')
model = model.to(device)
input_ = input_.to(device)

print('Hidden Size:', hidden_size)
print('Input Size:', input_size)
print('Sequence Length:', sequence_length)
print('Batch Size:', batch_size)

print('=' * 100)

# One GPU
print('One GPU...')
output = model(input_)[-1]
model.zero_grad()
output.sum().backward()

print('=' * 100)

# Multiple GPUs
print('Multiple GPUs...')
output = torch.nn.parallel.data_parallel(
    module=model, inputs=input_, dim=1, output_device=device.index)[-1]
model.zero_grad()
output.sum().backward()
```

## System Info

```
Collecting environment information...
PyTorch version: 0.4.1
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.4 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.5
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: Tesla P100-PCIE-16GB
GPU 1: Tesla P100-PCIE-16GB
GPU 2: Tesla P100-PCIE-16GB
GPU 3: Tesla P100-PCIE-16GB

Nvidia driver version: 390.30
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.1.3

Versions of relevant libraries:
[pip3] numpy (1.14.3)
[pip3] pytorch-nlp (0.3.5)
[pip3] torch (0.4.0)
[conda] Could not collect
```