Repository: pytorch
Issue: RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
Labels: module: cudnn,triaged

## 🐛 RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR

I have run train.py in SSD project, I got the error message as title description.

## To Reproduce
My sample code as the link:  https://github.com/xijier/SSD_resnet_pytorch/blob/main/train.py

The error message as below:

## Error message:

Traceback (most recent call last):
  File "train.py", line 291, in <module>
    train()
  File "train.py", line 206, in train
    out = net(images)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\parallel\data_parallel.py", line 166, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "E:\kg\SSD_resnet_pytorch-master\SSD_resnet_pytorch-master\ssd.py", line 96, in forward
    x = v(x)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\conv.py", line 443, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\conv.py", line 440, in _conv_forward
    self.padding, self.dilation, self.groups)
**RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR**
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([8, 3, 300, 300], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(3, 64, kernel_size=[7, 7], padding=[3, 3], stride=[2, 2], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams
    data_type = CUDNN_DATA_FLOAT
    padding = [3, 3, 0]
    stride = [2, 2, 0]
    dilation = [1, 1, 0]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 00000248BB74D840
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 8, 3, 300, 300,
    strideA = 270000, 90000, 300, 1,
output: TensorDescriptor 00000248BB74D8B0
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 8, 64, 150, 150,
    strideA = 1440000, 22500, 150, 1,
weight: FilterDescriptor 00000248B5DBF3A0
    type = CUDNN_DATA_FLOAT
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 64, 3, 7, 7,
Pointer addresses:
    input: 000000072EE00000
    output: 0000000730200000
    weight: 0000000704A00000

## My Action:

I have already run the code snippet as below which provided by error message. It's running successfully. No error.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([8, 3, 300, 300], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(3, 64, kernel_size=[7, 7], padding=[3, 3], stride=[2, 2], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

## Environment Collection

Collecting environment information...
PyTorch version: 1.9.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 家庭中文版
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.6 (64-bit runtime)
Python platform: Windows-10-10.0.19041-SP0
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration: GPU 0: GeForce RTX 2060
Nvidia driver version: 456.81
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip] numpy==1.14.3
[pip] numpydoc==0.8.0
[pip] torch==1.9.0+cu111
[pip] torchaudio==0.9.0
[pip] torchfile==0.1.0
[pip] torchvision==0.10.0+cu111
[conda] blas                      1.0                         mkl
[conda] mkl                       2018.0.2                      1
[conda] mkl-service               1.1.2            py36h57e144c_4
[conda] mkl_fft                   1.0.1            py36h452e1ab_0
[conda] mkl_random                1.0.1            py36h9258bd6_0
[conda] numpy                     1.14.3           py36h9fa60d3_1
[conda] numpy-base                1.14.3           py36h555522e_1
[conda] numpydoc                  0.8.0                    py36_0
[conda] torch                     1.9.0+cu111               <pip>
[conda] torchaudio                0.9.0                     <pip>
[conda] torchfile                 0.1.0                     <pip>
[conda] torchvision               0.10.0+cu111              <pip>

## Additional context
My sample code as below link:  https://github.com/xijier/SSD_resnet_pytorch/blob/main/train.py


Looking for your reply.
Thanks a lot.


cc @csarofeen @ptrblck @xwang233