Repository: pytorch
Issue: cudnn.determinstic=True causes dilated convolution to be >10x slower
Labels: module: cudnn,triaged

## 🐛 Bug

If you set `torch.backends.cudnn.deterministic=True`, and then create and use a Conv2d layer, it's MUCH slower than if you use `torch.backends.cudnn.deterministic=False`. My tests found it to be between 10-80x slower.

## To Reproduce

I wrote a timing script to see how much slower:

```python
# dilated_test.py
import time
import torch
import torch.nn as nn

import sys
B = int(sys.argv[1]) or 32
C = int(sys.argv[2]) or 64
det = bool(int(sys.argv[3])) or False
dilation = int(sys.argv[4]) or 0

torch.backends.cudnn.deterministic=det

inp = torch.rand((B, C, 32, 32), device='cuda:0')
if dilation == 0:
  conv = nn.Conv2d(C, 64, 3, padding=1, stride=1).cuda()
else:
  conv = nn.Conv2d(C, 64, 3, padding=dilation, dilation=dilation, stride=1).cuda()


# Warmup
for x in range(10):
  outp = conv(inp)

# Test
a = time.perf_counter()
for x in range(1000):
  outp = conv(inp)
b = time.perf_counter()
print(f'{b-a:8.4f}')
```

Which I coordinated with a bash script:

```bash
# !/bin/bash

AAAA=`python dilated_test.py 32 64 0 0`
AAAB=`python dilated_test.py 32 64 0 2`
AABA=`python dilated_test.py 32 64 1 0`
AABB=`python dilated_test.py 32 64 1 2`

ABAA=`python dilated_test.py 32 512 0 0`
ABAB=`python dilated_test.py 32 512 0 2`
ABBA=`python dilated_test.py 32 512 1 0`
ABBB=`python dilated_test.py 32 512 1 2`

BAAA=`python dilated_test.py 4 64 0 0`
BAAB=`python dilated_test.py 4 64 0 2`
BABA=`python dilated_test.py 4 64 1 0`
BABB=`python dilated_test.py 4 64 1 2`

CAAA=`python dilated_test.py 128 64 0 0`
CAAB=`python dilated_test.py 128 64 0 2`
CABA=`python dilated_test.py 128 64 1 0`
CABB=`python dilated_test.py 128 64 1 2`

echo '                 deterministic        deterministic'
echo '                     false                 true    '
echo '                not dil     dil       not dil     dil '
echo "B= 32, C= 64:  ${AAAA} | ${AAAB} | ${AABA} | ${AABB} "
echo "B= 32, C=512:  ${ABAA} | ${ABAB} | ${ABBA} | ${ABBB} "
echo "B=  4, C= 64:  ${BAAA} | ${BAAB} | ${BABA} | ${BABB} "
echo "B=128, C= 64:  ${CAAA} | ${CAAB} | ${CABA} | ${CABB} "
```

Which gives:

```
                 deterministic        deterministic
                     false                 true    
                not dil     dil       not dil     dil 
B= 32, C= 64:    0.2016 |   0.4260 |   0.3227 |  24.2022 
B= 32, C=512:    0.9506 |   3.1413 |   2.1431 |  31.4578 
B=  4, C= 64:    0.0472 |   0.0606 |   0.0633 |   3.2975 
B=128, C= 64:    0.7815 |   1.6887 |   1.2011 |  97.4557 
```


## Expected behavior

Although I expect it to be a little slower, I would expect that the difference in speed would be at most 2x.

## Environment

```
PyTorch version: 1.3.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: GeForce GTX 1080
Nvidia driver version: 410.104
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.17.2
[pip] torch==1.3.0
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.4                      243  
[conda] mkl-service               2.3.0            py36he904b0f_0  
[conda] mkl_fft                   1.0.14           py36ha843d7b_0  
[conda] mkl_random                1.1.0            py36hd6b4f25_0  
[conda] pytorch                   1.3.0           py3.6_cuda10.0.130_cudnn7.6.3_0    pytorch

```

## Additional context

Built with Dockerfile:

```Dockerfile
FROM nvidia/cuda:10.0-devel-ubuntu18.04

RUN apt-get update \
 && apt-get install -y curl cmake \
 && rm -rf /var/lib/apt/lists/*

# Install Miniconda and Python 3.6.5
ENV CONDA_AUTO_UPDATE_CONDA=false
ENV PATH=/root/miniconda/bin:$PATH
RUN curl -so ~/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh \
 && chmod +x ~/miniconda.sh \
 && ~/miniconda.sh -b -p ~/miniconda \
 && rm ~/miniconda.sh \
 && conda install -y python==3.6.5 \
 && conda clean -ya

RUN conda install pytorch cudatoolkit=10.0 -c pytorch
```

But I observe similar results if I replace the last line with:

```Dockerfile
RUN conda install pytorch==1.2.0
```

I do NOT observe these slow-downs with:

```Dockerfile
RUN conda install pytorch==1.1.0
```