Repository: pytorch
Issue: Backward pass for a nn.Conv2d with half-precision on Quadro RTX 8000 leads to CUDNN_STATUS_INTERNAL_ERROR
Labels: module: cudnn,triaged

## 🐛 Bug

I was trying to launch training with half precision on Quadro RTX 8000, but it failed with `CUDNN_STATUS_INTERNAL_ERROR` error.

## To reproduce

```python
import torch
torch.backends.cuda.matmul.allow_tf32 = False
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = False
data = torch.randn([16, 256, 128, 128], dtype=torch.half, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(256, 256, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().half()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()
```

This code sample fails with the exception:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/skoroki/fc-gan/env/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/skoroki/fc-gan/env/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = False
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = False
data = torch.randn([16, 256, 128, 128], dtype=torch.half, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(256, 256, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().half()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams
    data_type = CUDNN_DATA_HALF
    padding = [1, 1, 0]
    stride = [1, 1, 0]
    dilation = [1, 1, 0]
    groups = 1
    deterministic = false
    allow_tf32 = false
input: TensorDescriptor 0x560c0008a040
    type = CUDNN_DATA_HALF
    nbDims = 4
    dimA = 16, 256, 128, 128,
    strideA = 4194304, 16384, 128, 1,
output: TensorDescriptor 0x153108016570
    type = CUDNN_DATA_HALF
    nbDims = 4
    dimA = 16, 256, 128, 128,
    strideA = 4194304, 16384, 128, 1,
weight: FilterDescriptor 0x153108048490
    type = CUDNN_DATA_HALF
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 256, 256, 3, 3,
Pointer addresses:
    input: 0x1539ae000000
    output: 0x15310e000000
    weight: 0x1539ac000000
```

## Expected behavior

There shouldn't be any errors or those errors should have a clear description.

## Environment

The provided script for env collection gives the following output:

```
Collecting environment information...
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.18.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 9.1.85
GPU models and configuration:
GPU 0: Quadro RTX 8000
GPU 1: Quadro RTX 8000
GPU 2: Quadro RTX 8000
GPU 3: Quadro RTX 8000

Nvidia driver version: 450.119.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.1
[pip3] torch==1.7.1
[pip3] torchvision==0.8.2
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.2.89              hfd86e86_1
[conda] mkl                       2021.2.0           h06a4308_296
[conda] mkl-service               2.3.0            py38h27cfd23_1
[conda] mkl_fft                   1.3.0            py38h42c9631_2
[conda] mkl_random                1.2.1            py38ha9443f7_2
[conda] numpy                     1.20.1           py38h93e21f0_0
[conda] numpy-base                1.20.1           py38h7d8b39e_0
[conda] pytorch                   1.7.1           py3.8_cuda10.2.89_cudnn7.6.5_0    pytorch
[conda] torchvision               0.8.2                py38_cu102    pytorch
```

## Additional context

<!-- Add any other context about the problem here. -->


cc @csarofeen @ptrblck @xwang233