Repository: pytorch
Issue: DataParallel gives different gradients when using LSTMs
Labels: triaged,module: determinism,module: data parallel

## 🐛 Bug

When using DataParallel on a model with LSTMs the losses obtained compared to the same model run on a single GPU are different. 

## To Reproduce

Here is a sample code block that seeds everything to ensure the weights are the same.
Model: 
``` python
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.input_transform = nn.Linear(100, 512)
        self.lstm = nn.LSTM(512,512,batch_first=True)
        self.output_transform = nn.Linear(512, 10)
    def forward(self, x):
        self.lstm.flatten_parameters()
        input_t = self.input_transform(x)
        output, hidden = self.lstm(input_t)
        return self.output_transform(hidden[0][0])
```

Running on a single GPU
```python
torch.manual_seed(1234)
np.random.seed(1234)
device = torch.device("cuda:0")
model = Model().to(device)
optimizer = torch.optim.Adam(model.parameters())

for i in range(5):
    input = torch.rand(128,10,100).to(device)
    output = torch.from_numpy(np.random.randint(10, size=(128,))).to(device)
    pred = model(input)
    loss = nn.functional.cross_entropy(pred, output)
    print (loss)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
```
These are the losses:
```
tensor(2.3043, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(2.3364, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(2.3563, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(2.2963, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(2.3315, device='cuda:0', grad_fn=<NllLossBackward>)
```

Running on 4 P100s
```python
torch.manual_seed(1234)
np.random.seed(1234)
device = torch.device("cuda:0")
model = Model().to(device)
model = DataParallel(model)
optimizer = torch.optim.Adam(model.parameters())

for i in range(5):
    input = torch.rand(128,10,100).to(device)
    output = torch.from_numpy(np.random.randint(10, size=(128,))).to(device)
    pred = model(input)
    loss = nn.functional.cross_entropy(pred, output)
    print (loss)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

```
These are the results:
```
tensor(2.3043, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(2.3068, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(2.3137, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(2.2964, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(2.3243, device='cuda:0', grad_fn=<NllLossBackward>)
```

This behaviour is on the stable version of PyTorch 1.4. I face convergence issues on my actual code that uses multiple LSTMs when using dataparallel.
Things work fine when I downgrade to PyTorch 1.2. 


## Environment
```
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.0

OS: Ubuntu 18.04.2 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: Could not collect

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: Tesla P100-PCIE-12GB
GPU 1: Tesla P100-PCIE-12GB
GPU 2: Tesla P100-PCIE-12GB
GPU 3: Tesla P100-PCIE-12GB

Nvidia driver version: 418.40.04
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] torch==1.4.0
[pip] torchvision==0.5.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.0                      166
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.0.15           py37ha843d7b_0
[conda] mkl_random                1.1.0            py37hd6b4f25_0
[conda] pytorch                   1.4.0           py3.7_cuda10.0.130_cudnn7.6.3_0    pytorch
[conda] torchvision               0.5.0                py37_cu100    pytorch
```