Repository: pytorch
Issue: Conv2D with large different number of input and output channels gives a CUDNN_STATUS_INTERNAL_ERROR
Labels: module: cudnn,triaged

### 🐛 Describe the bug

I'm trying to perform a convolution just to increase the dimensionality of the input tensor from `512*32` (i.e. 16384) dimensions to `512*512` (i.e. 262144) dimesions, and I got the following message with a code snippet which indeed reproduces the error:

```
Exception has occurred: RuntimeError
cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([4, 16384, 1, 1], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(16384, 262144, kernel_size=[1, 1], padding=[0, 0], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams 
    data_type = CUDNN_DATA_FLOAT
    padding = [0, 0, 0]
    stride = [1, 1, 0]
    dilation = [1, 1, 0]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0x55eae4955610
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 4, 16384, 1, 1, 
    strideA = 16384, 1, 1, 1, 
output: TensorDescriptor 0x55eae494f370
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 4, 262144, 1, 1, 
    strideA = 262144, 1, 1, 1, 
weight: FilterDescriptor 0x55eae4955520
    type = CUDNN_DATA_FLOAT
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 262144, 16384, 1, 1, 
Pointer addresses: 
    input: 0x7f5103540000
    output: 0x7f55ae000000
    weight: 0x7f4cf0000000
  File "/scratch/arturao/GANSketching22/train_hypernet.py", line 159, in training_loop
    pred_weights = trainer.gan_model.hyper_net.weight_predictor(feature)
  File "/scratch/arturao/GANSketching22/train_hypernet.py", line 240, in <module>
    training_loop()
```

### Versions

Collecting environment information...
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:57:06)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.13.0-41-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.6.112
GPU models and configuration: 
GPU 0: NVIDIA RTX A5000
GPU 1: NVIDIA RTX A5000

Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.3.2
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.11.0
[pip3] torch-tb-profiler==0.4.0
[pip3] torchaudio==0.11.0
[pip3] torchvision==0.12.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py38h95df7f1_0    conda-forge
[conda] mkl_fft                   1.3.1            py38h8666266_1    conda-forge
[conda] mkl_random                1.2.2            py38h1abd341_0    conda-forge
[conda] numpy                     1.19.5                   pypi_0    pypi
[conda] numpy-base                1.21.2           py38h79a1101_0  
[conda] pytorch                   1.11.0          py3.8_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch                     1.6.0                    pypi_0    pypi
[conda] torch-tb-profiler         0.4.0                    pypi_0    pypi
[conda] torchaudio                0.11.0               py38_cu113    pytorch
[conda] torchvision               0.7.0                    pypi_0    pypi

cc @csarofeen @ptrblck @xwang233