Repository: pytorch
Issue: RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
Labels: module: cudnn,module: convolution,triaged

## 🐛 Bug

A simple code snippet throws the aforementioned error.

## To Reproduce

Steps to reproduce the behavior:

Run the following code (autogenerated from a previous error message in a big codebase)
```python
import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([1, 8, 8, 480, 640], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv3d(8, 1, kernel_size=[3, 3, 3], padding=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()
```

## Expected behavior

Normally, this should just run through. Instead, the following output is generated
```
Traceback (most recent call last):
  File "test_fail.py", line 10, in <module>
    out.backward(torch.randn_like(out))
  File "/usr/wiss/koestlel/miniconda3/envs/t1.7/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/wiss/koestlel/miniconda3/envs/t1.7/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([1, 8, 8, 480, 640], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv3d(8, 1, kernel_size=[3, 3, 3], padding=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams 
    data_type = CUDNN_DATA_FLOAT
    padding = [1, 1, 1]
    stride = [1, 1, 1]
    dilation = [1, 1, 1]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0x5579b26c3d70
    type = CUDNN_DATA_FLOAT
    nbDims = 5
    dimA = 1, 8, 8, 480, 640, 
    strideA = 19660800, 2457600, 307200, 640, 1, 
output: TensorDescriptor 0x7fc91800f5b0
    type = CUDNN_DATA_FLOAT
    nbDims = 5
    dimA = 1, 1, 8, 480, 640, 
    strideA = 2457600, 2457600, 307200, 640, 1, 
weight: FilterDescriptor 0x7fc91803ae90
    type = CUDNN_DATA_FLOAT
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 5
    dimA = 1, 8, 3, 3, 3, 
Pointer addresses: 
    input: 0x7fc92e000000
    output: 0x7fc934f60000
    weight: 0x7fc932c00800
```

## Environment
The output of script:
```
Collecting environment information...
PyTorch version: 1.7.0
Is debug build: True
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.19.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: Quadro P400
GPU 1: Quadro RTX 5000

Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.4
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] pytorch-lightning==0.7.6
[pip3] torch==1.7.0
[pip3] torchaudio==0.7.0a0+ac17b64
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.8.1
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.0.221             h6bb024c_0  
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.2.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.2           py38h54aff64_0  
[conda] numpy-base                1.19.2           py38hfa32c7d_0  
[conda] pytorch                   1.7.0           py3.8_cuda11.0.221_cudnn8.0.3_0    pytorch
[conda] pytorch-lightning         0.7.6                    pypi_0    pypi
[conda] torchaudio                0.7.0                      py38    pytorch
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.8.1                py38_cu110    pytorch
```


Thank you very much for your help!
Lukas


cc @csarofeen @ptrblck @xwang233