Repository: pytorch
Issue: cudnn_convolution_add_relu fails under basic conditions
Labels: module: dependency bug,module: cudnn,triaged

## 🐛 Bug

cudnn_convolution_add_relu seems to raise ```CUDNN_STATUS_NOT_SUPPORTED``` under basic inputs. 

## To Reproduce

```
import torch
device = torch.device('cuda')
dtype = torch.float
inp = torch.rand(1, 1, 8, 8, dtype=dtype, device=device)
w = torch.randn(1, 1, 1, 1, dtype=dtype, device=device)
conv2d_out = torch.conv2d(inp, w, None, (1, 1), (0, 0), (1, 1), 1)
print(conv2d_out.size())
print(inp.size())
cudnn_out = torch.cudnn_convolution_add_relu(inp, w, inp, 1.0, None, (1, 1), (0, 0), (1, 1), 1)
torch.cuda.synchronize()
```

Results in output

```
torch.Size([1, 1, 8, 8])
torch.Size([1, 1, 8, 8])
Traceback (most recent call last):
  File "/data/home/cpuhrsch/tmp/tmp15.py", line 9, in <module>
    cudnn_out = torch.cudnn_convolution_add_relu(inp, w, inp, 1.0, None, (1, 1), (0, 0), (1, 1), 1)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([1, 1, 8, 8], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(1, 1, kernel_size=[1, 1], padding=[0, 0], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams
    data_type = CUDNN_DATA_FLOAT
    padding = [0, 0, 0]
    stride = [1, 1, 0]
    dilation = [1, 1, 0]
    groups = 1
    deterministic = false
    allow_tf32 = true
input: TensorDescriptor 0x55a9a350c9f0
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 1, 1, 8, 8,
    strideA = 64, 64, 8, 1,
output: TensorDescriptor 0x55a9a350caa0
    type = CUDNN_DATA_FLOAT
    nbDims = 4
    dimA = 1, 1, 8, 8,
    strideA = 64, 64, 8, 1,
weight: FilterDescriptor 0x55a9a350ca60
    type = CUDNN_DATA_FLOAT
    tensor_format = CUDNN_TENSOR_NCHW
    nbDims = 4
    dimA = 1, 1, 1, 1,
Pointer addresses:
    input: 0x7f14a7e00000
    output: 0x7f14a7e00600
    weight: 0x7f14a7e00200
cudnnConvolutionBiasActivationForward: 1
```

## Expected behavior

Should result in same output as conv2d + add + relu

## Environment

```
$ python collect_env.py
Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.18.4
Libc version: glibc-2.17

Python version: 3.7.10 (default, Jun  4 2021, 14:48:32)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-1041-aws-x86_64-with-debian-buster-sid
Is CUDA available: N/A
CUDA runtime version: 11.0.221
GPU models and configuration:
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB
GPU 3: Tesla V100-SXM2-16GB

Nvidia driver version: 450.80.02
cuDNN version: Probably one of the following:
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7.6.5
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn.so.7.6.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.3
[pip3] torch==1.10.0.dev20210729
[pip3] torch2trt==0.2.0
[pip3] torchaudio==0.10.0a0+2475e25
[pip3] torchvision==0.11.0.dev20210729
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] mkl                       2021.3.0           h06a4308_520
[conda] mkl-service               2.4.0            py37h7f8727e_0
[conda] mkl_fft                   1.3.0            py37h42c9631_2
[conda] mkl_random                1.2.2            py37h51133e4_0
[conda] numpy                     1.20.3           py37hf144106_0
[conda] numpy-base                1.20.3           py37h74d4b33_0
[conda] pytorch                   1.10.0.dev20210729 py3.7_cuda11.1_cudnn8.0.5_0    pytorch-nightly
[conda] torch2trt                 0.2.0                     dev_0    <develop>
[conda] torchaudio                0.10.0.dev20210729            py37    pytorch-nightly
[conda] torchvision               0.11.0.dev20210729      py37_cu111    pytorch-nightly
```

## Additional context

cuDNN version v8005

```
$ CUDNN_LOGDEST_DBG=stdout CUDNN_LOGINFO_DBG=1 python script.py | head -n 10

I! CuDNN (v8005) function cudnnCreate() called:
i!     handle: location=host; addr=0x55db75a8af30;
i! Time: 2021-08-06T21:32:13.527246 (0d+0h+0m+4s since start)
i! Process=27337; Thread=27337; GPU=NULL; Handle=NULL; StreamId=NULL.


I! CuDNN (v8005) function cudnnSetStream() called:
i!     handle: type=cudnnHandle_t; streamId=(nil) (defaultStream);
i!     streamId: type=cudaStream_t; streamId=(nil) (defaultStream);
[...]
```


cc @csarofeen @ptrblck @xwang233