Repository: pytorch
Issue: Process hangs after calling conv2d() in pytorch 1.11.0 with CUDA 11.3
Labels: module: cudnn,module: convolution,triaged,module: deadlock

### 🐛 Describe the bug

When running the following code, the python process hangs and becomes unresponsive to Ctrl-C, forcing me to manually kill the process in the task manager.

Monitoring the resources shows the memory for data is allocated, but no GPU activity is monitored.

```python
import torch
#from hanging_threads import start_monitoring
#start_monitoring()
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([4, 1, 1, 262656], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(1, 16, kernel_size=[1, 513], padding=[0, 0], stride=[1, 16], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()
```

Using the hanging_threads module to get information on what code causes the problem yields the following: 

```
----------    Thread 22344 "MainThread" hangs     ----------
        File "D:\AI wobble\RAVE\example.py", line 11, in <module>
                out = net(data)
        File "D:\Anaconda3\envs\rave\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
                return forward_call(*input, **kwargs)
        File "D:\Anaconda3\envs\rave\lib\site-packages\torch\nn\modules\conv.py", line 447, in forward
                return self._conv_forward(input, self.weight, self.bias)
        File "D:\Anaconda3\envs\rave\lib\site-packages\torch\nn\modules\conv.py", line 443, in _conv_forward
                return F.conv2d(input, weight, bias, self.stride,
```

Running the same code on a python 3.7 environment with pytorch 1.8.1 cuda10.2 cudnn7.0 (see below) runs fine; execution finishes after a few seconds.

Hardware details: NVIDIA RTX2060

### Versions

This is the environment where the problem occurs:
```
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Home
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.22.4
Libc version: N/A

Python version: 3.9.12 (main, Apr  4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19043-SP0
Is CUDA available: True
CUDA runtime version: 11.6.124
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2060
Nvidia driver version: 512.59
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.20.3
[pip3] pytorch-lightning==1.6.1
[pip3] torch==1.11.0
[pip3] torchmetrics==0.8.2
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h59b6b97_2
[conda] mkl                       2022.0.0           haa95532_115
[conda] numpy                     1.20.3                   pypi_0    pypi
[conda] pytorch                   1.11.0          py3.9_cuda11.3_cudnn8_0    pytorch
[conda] pytorch-lightning         1.6.1                    pypi_0    pypi
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchmetrics              0.8.2                    pypi_0    pypi
```

This is an environment where the problem does not occur:
```
PyTorch version: 1.8.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Home
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.22.4
Libc version: N/A

Python version: 3.7.10 (default, Feb 26 2021, 13:06:18) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19041-SP0
Is CUDA available: True
CUDA runtime version: 11.6.124
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2060
Nvidia driver version: 512.59
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.20.3
[pip3] pytorch-lightning==1.6.1
[pip3] torch==1.8.1
[pip3] torchaudio==0.8.1
[pip3] torchmetrics==0.8.2
[pip3] torchvision==0.9.1
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.2.89              h74a9793_1
[conda] mkl                       2021.2.0           haa95532_296
[conda] mkl-service               2.3.0            py37h2bbff1b_1
[conda] mkl_fft                   1.3.0            py37h277e83a_2
[conda] mkl_random                1.2.1            py37hf11a4ad_2
[conda] numpy                     1.20.3                   pypi_0    pypi
[conda] numpy-base                1.20.2           py37hc2deb75_0
[conda] pytorch                   1.8.1           py3.7_cuda10.2_cudnn7_0    pytorch
[conda] pytorch-lightning         1.6.1                    pypi_0    pypi
[conda] torchaudio                0.8.1                      py37    pytorch
[conda] torchmetrics              0.8.2                    pypi_0    pypi
[conda] torchvision               0.9.1                py37_cu102    pytorch
```

cc @csarofeen @ptrblck @xwang233 @ngimel