Repository: pytorch
Issue: Exceedingly different F.conv2d outputs on cuda/cpu
Labels: high priority,module: cudnn,module: cuda,module: convolution,triaged,module: correctness (silent)

## 🐛 Bug

I'm running a simple conv2d with padding and dilation, and i notice big difference in results between the cpu and cuda runs (10% of samples are different - i.e. not close enough)

## To Reproduce

```
import torch
from torch import nn
from torch.nn import functional as F
import numpy as np

inps = 2
outs = 512
w, h = 16, 8
ks = 5
b = 4

data_np = np.random.random([b, inps, h, w])
par = nn.Parameter(torch.empty(size=[outs, inps, ks, ks]))
nn.init.kaiming_normal_(par, a=2)

res = {}

for device in ["cpu", "cuda"]:
    tensor = torch.from_numpy(data_np).to(device).to(torch.float32)
    kernel = par.to(device)

    output = F.conv2d(tensor, weight=kernel, padding=(4, 4), dilation=[2, 2])

    res[device] = output

np.testing.assert_allclose(*[r.detach().cpu().numpy() for r in res.values()],
                           rtol=2e-3,
                           atol=1e-3)
```

## Expected behavior

I would expect the outputs to broadly match within the given tolerance (`rtol=1e-1, atol=1e-1`) yet 10% of the output is *consistently* different across runs,

```
AssertionError:
Not equal to tolerance rtol=0.1, atol=0.1

Mismatched elements: 26351 / 262144 (10.1%)
Max absolute difference: 0.67974156
Max relative difference: 11097.508
```


## Environment

PyTorch version: 1.9.0.dev20210302
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: 10.0.0-4ubuntu1
CMake version: version 3.15.4

Python version: 3.7 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: TITAN RTX
GPU 1: TITAN RTX
GPU 2: TITAN RTX

Nvidia driver version: 450.102.04
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.9.0.dev20210302
[pip3] torchaudio==0.8.0a0+5566138
[pip3] torchvision==0.9.0.dev20210302
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.2.89              hfd86e86_1
[conda] mkl                       2020.2                      256
[conda] mkl-service               2.3.0            py37he8ac12f_0
[conda] mkl_fft                   1.3.0            py37h54f3939_0
[conda] mkl_random                1.1.1            py37h0573a6f_0
[conda] numpy                     1.19.2           py37h54aff64_0
[conda] numpy-base                1.19.2           py37hfa32c7d_0
[conda] pytorch                   1.9.0.dev20210302 py3.7_cuda10.2_cudnn7.6.5_0    pytorch-nightly
[conda] torchaudio                0.8.0.dev20210302            py37    pytorch-nightly
[conda] torchvision               0.9.0.dev20210302      py37_cu102    pytorch-nightly

## 

What am I doing wrong?


cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @anjali411 @ngimel @csarofeen @ptrblck @xwang233 @aocsa