Repository: pytorch
Issue: CuDNN 8 with benchmark=True takes minutes to execute for certain configurations
Labels: module: performance,module: cudnn,triaged

## 🐛 Bug

CuDNN v8 can take >100x longer than v7 to execute the first call to a ConvTranspose module when torch.backends.cudnn.benchmark=True

## To Reproduce

```
import torch
import time
import sys

torch.backends.cudnn.benchmark = True

tpconv = torch.nn.ConvTranspose1d(1000, 1, kernel_size=10000, stride=1000)
tpconv.cuda()

x = torch.rand(16,1000,100).cuda()

start = time.time()
y = tpconv(x)
end = time.time()

print(f'call took {end-start:.5f} seconds')

```
Using CuDNN 8, I get: `call took 173.81815 seconds`
On the same machine using CuDNN 7, I get: `call took 0.97816 seconds`

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

CuDNN 8 benchmark time to be within an order of magnitude of CuDNN 7

## Environment

CuDNN 8 set-up uses nvcr.io/nvidia/pytorch:20.12-py3 docker image:
```
PyTorch version: 1.8.0a0+1606899
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.18.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2080 Ti
Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] pytorch-transformers==1.1.0
[pip3] torch==1.8.0a0+1606899
[pip3] torchtext==0.9.0a0
[pip3] torchvision==0.9.0a0
[conda] magma-cuda110             2.5.2                         5    local
[conda] mkl                       2019.4                      243
[conda] mkl-include               2019.4                      243
[conda] nomkl                     3.0                           0
[conda] numpy                     1.19.2           py38h6163131_0
[conda] numpy-base                1.19.2           py38h75fe3a5_0
[conda] pytorch-transformers      1.1.0                    pypi_0    pypi
[conda] torch                     1.8.0a0+1606899          pypi_0    pypi
[conda] torchtext                 0.9.0a0                  pypi_0    pypi
[conda] torchvision               0.9.0a0                  pypi_0    pypi
```
CuDNN 7 set-up on same machine, using conda installation of pytorch:
```
PyTorch version: 1.7.0
Is debug build: True
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 16.04.6 LTS (x86_64)
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
Clang version: 3.8.0-2ubuntu4 (tags/RELEASE_380/final)
CMake version: version 3.14.0

Python version: 3.7 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce RTX 2080 Ti

Nvidia driver version: 460.32.03
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] pytorch-ranger==0.1.1
[pip3] torch==1.7.0
[pip3] torchaudio==0.7.0a0+ac17b64
[pip3] torchvision==0.8.1
[conda] blas                      1.0                         mkl    anaconda
[conda] cudatoolkit               10.2.89              hfd86e86_1
[conda] mkl                       2020.1                      217    anaconda
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.1.0            py37h23d657b_0
[conda] mkl_random                1.1.1            py37h0573a6f_0    anaconda
[conda] numpy                     1.19.1           py37hbc911f0_0
[conda] numpy-base                1.19.1           py37hfa32c7d_0
[conda] pytorch                   1.7.0           py3.7_cuda10.2.89_cudnn7.6.5_0    pytorch
[conda] pytorch-ranger            0.1.1                    pypi_0    pypi
[conda] torchaudio                0.7.0                      py37    pytorch
[conda] torchvision               0.8.1                py37_cu102    pytorch
```
## Additional context

* if torch.backends.cudnn.benchmark=False, then the first call with CuDNN 8 is roughly 5x slower than CuDNN 7
* Although I use the docker image for CuDNN 8, I have observed the same behavior using the conda installation
* The benchmark time scales with the dimensionality and stride of the TPConv tensor and the input tensor; time scales roughly linearly with batch size; reducing the stride to 1 is enough to reduce the time by ~10x


cc @csarofeen @ptrblck @xwang233 @VitalyFedyunin @ngimel