Repository: pytorch
Issue: Conv3D error : CUDNN_STATUS_INTERNAL_ERROR
Labels: module: cudnn,module: convolution,triaged

## 🐛 Bug

A cudnn error is raised using the code below. I suppose that the issue is related to #4632 #13636.

## To Reproduce

```
import torch
import torch.nn.functional as F

torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True

class Simpler3DNet(torch.nn.Module):
   
    def __init__(self, large_bn=False):
        super(Simpler3DNet, self).__init__()
        self.conv1 = torch.nn.Conv3d(1, 32, kernel_size=5, padding=2)
        self.conv2 = torch.nn.Conv3d(32, 64, kernel_size=5, padding=2)
        self.conv3 = torch.nn.Conv3d(64, 128, kernel_size=3, padding=1)
        self.fc1 = torch.nn.Linear(128, 64)
        self.fc2 = torch.nn.Linear(64, 1)
        self.bn1 = torch.nn.BatchNorm3d(32)
        self.bn2 = torch.nn.BatchNorm3d(64)
        self.bn3 = torch.nn.BatchNorm3d(128)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = self.bn1(x)
        x = F.max_pool3d(x, 5)
        x = F.relu(self.conv2(x))
        x = self.bn2(x)
        x = F.max_pool3d(x, 5)
        x = F.relu(self.conv3(x))
        x = self.bn3(x)
        x = F.max_pool3d(x, 4)

        x = x.view(-1, 128)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.relu(x).flatten()


if __name__ == '__main__':
    data = torch.randn([70, 1, 100, 100, 100], dtype=torch.half, device='cuda', requires_grad=True)
    net = Simpler3DNet()
    net = net.cuda().half()
    out = net(data)
    out.backward(torch.randn_like(out))
    torch.cuda.synchronize()
```

The error is 
```
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([70, 32, 20, 20, 20], dtype=torch.half, device='cuda', requires_grad=True)
net = torch.nn.Conv3d(32, 64, kernel_size=[5, 5, 5], padding=[2, 2, 2], stride=[1, 1, 1], dilation=[1, 1, 1], groups=1)
net = net.cuda().half()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams 
    data_type = CUDNN_DATA_HALF
    padding = [2, 2, 2]
    stride = [1, 1, 1]
    dilation = [1, 1, 1]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0x555b8925c250
    type = CUDNN_DATA_HALF
    nbDims = 5
    dimA = 70, 32, 20, 20, 20, 
    strideA = 256000, 8000, 400, 20, 1, 
output: TensorDescriptor 0x555b8925c300
    type = CUDNN_DATA_HALF
    nbDims = 5
    dimA = 70, 64, 20, 20, 20, 
    strideA = 512000, 8000, 400, 20, 1, 
weight: FilterDescriptor 0x555b8925c2c0
    type = CUDNN_DATA_HALF
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 5
    dimA = 64, 32, 5, 5, 5, 
Pointer addresses: 
    input: 0x7f4690000000
    output: 0x7f4680000000
    weight: 0x7f49c2800000
```
Note that the snippet code is ok...

If I disable `cudnn` adding the following line
```
torch.backends.cudnn.enabled = False
```

I have another error
```
Traceback (most recent call last):
  File "/home/irsrvhome1/R11/desrozis/issue.py", line 63, in <module>
    out = net(data)
  File "/disk1/work/desrozis/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/irsrvhome1/R11/desrozis/issue.py", line 45, in forward
    x = F.relu(self.conv2(x))
  File "/disk1/work/desrozis/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/disk1/work/desrozis/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 572, in forward
    return F.conv3d(input, self.weight, self.bias, self.stride,
RuntimeError: CUDA error: an illegal memory access was encountered
```

## Expected behavior

The code should run without error

## Environment

PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: CentOS Linux release 7.9.2009 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.9 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: Tesla V100-PCIE-16GB
GPU 1: Tesla V100-PCIE-16GB

Nvidia driver version: 440.33.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.1
[pip3] torch==1.7.1
[conda] numpy                     1.20.1                   pypi_0    pypi
[conda] torch                     1.7.1                    pypi_0    pypi

Help is welcome.


cc @csarofeen @ptrblck @xwang233