Repository: pytorch
Issue: Performance bug with convolutions with weights and inputs of similar spatial size
Labels: module: performance,module: cudnn,module: convolution,triaged

This is a performance bug in `conv2d`, when spatial dimensions of the weight and input are the same (or similar).  Such convolutions get 5 to 10 times slower when transitioning from image sizes of 30 to 32.  There may be an issue with the selection of the CUDNN convolution algorithm.

One possibly route to a solution is to allow manual selection of the convolution algorithm.

## To Reproduce

Full minimal working example:
```
import torch as t
t.backends.cudnn.benchmark=True
import torch.nn.functional as F
from timeit import default_timer as timer

device = "cuda"
Cin = 128
Cout = 256
B = 1000

for W in range(20, 40):
    input = t.randn(B, Cin, W, W, device=device)
    weight = t.randn(Cout, Cin, W, W, device=device)
    output = F.conv2d(input, weight, padding=1)
    t.cuda.synchronize(device=device)

    start_time = timer()
    output = F.conv2d(input, weight, padding=1)
    t.cuda.synchronize(device=device)
    print(f"W: {W}, W**2: {W**2:4d}, time: {1000*(timer()-start_time)}")
```

## Expected behavior

The time required should scale linearly with `W**2`, instead, there is around a factor of 5 jump between W=30 and W=32,
```
W: 20, W**2:  400, time: 14.050104655325413
W: 21, W**2:  441, time: 13.91313411295414
W: 22, W**2:  484, time: 13.899956829845905
W: 23, W**2:  529, time: 13.991509564220905
W: 24, W**2:  576, time: 14.103865250945091
W: 25, W**2:  625, time: 14.099820517003536
W: 26, W**2:  676, time: 14.254673384130001
W: 27, W**2:  729, time: 14.209367334842682
W: 28, W**2:  784, time: 14.332166872918606
W: 29, W**2:  841, time: 14.398249797523022
W: 30, W**2:  900, time: 14.406020753085613
W: 31, W**2:  961, time: 53.21667902171612
W: 32, W**2: 1024, time: 86.8472745642066
W: 33, W**2: 1089, time: 84.1836640611291
W: 34, W**2: 1156, time: 113.77124954015017
W: 35, W**2: 1225, time: 84.53960344195366
W: 36, W**2: 1296, time: 125.90546812862158
W: 37, W**2: 1369, time: 84.77605786174536
W: 38, W**2: 1444, time: 141.37933682650328
W: 39, W**2: 1521, time: 149.50636308640242
```
The performance with `benchmark=True` commented out is even more exciting,
```
W: 20, W**2:  400, time: 17.404177226126194
W: 21, W**2:  441, time: 17.446410842239857
W: 22, W**2:  484, time: 17.630738206207752
W: 23, W**2:  529, time: 17.792406491935253
W: 24, W**2:  576, time: 16.371975652873516
W: 25, W**2:  625, time: 13.557913713157177
W: 26, W**2:  676, time: 13.596143573522568
W: 27, W**2:  729, time: 13.673634268343449
W: 28, W**2:  784, time: 13.687632977962494
W: 29, W**2:  841, time: 13.748884201049805
W: 30, W**2:  900, time: 13.820515014231205
W: 31, W**2:  961, time: 53.05525008589029
W: 32, W**2: 1024, time: 118.49690321832895
W: 33, W**2: 1089, time: 84.05695110559464
W: 34, W**2: 1156, time: 84.2417012900114
W: 35, W**2: 1225, time: 84.5922939479351
W: 36, W**2: 1296, time: 84.85969807952642
W: 37, W**2: 1369, time: 85.06182674318552
W: 38, W**2: 1444, time: 84.93016473948956
W: 39, W**2: 1521, time: 85.2658236399293
```

This is pretty painful, because it means my algorithm is very fast on MNIST, but disproportionately slower on any other dataset, such as CIFAR10.

## Environment

```
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.10.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration:
GPU 0: NVS 510
GPU 1: TITAN V
GPU 2: TITAN V

Nvidia driver version: 418.74
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip] numpy==1.17.2
[pip] numpydoc==0.9.1
[pip] torch==1.4.0
[pip] torchvision==0.5.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.4                      243
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.0.14           py37ha843d7b_0
[conda] mkl_random                1.1.0            py37hd6b4f25_0
[conda] pytorch                   1.3.0           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] torchvision               0.4.1                py37_cu101    pytorch
```


cc @csarofeen @ptrblck @VitalyFedyunin @ngimel