Repository: pytorch
Issue: Backward `Functional.conv3d` is slow when cuDNN is enabled
Labels: module: cudnn,module: cuda,triaged

## 🐛 Bug

Profiling shows that my backward pass through `conv3d` is between 7 and 15 times slower than forward. Setting `torch.backends.cudnn.enabled = False` brings it down to parity with forward.

## To Reproduce

```Python3
torch.backends.cudnn.enabled = True

input = torch.rand(1, 2, 33, 33, 33, device="cuda")
kernel = torch.rand(729, 2, 2, 2, 2, device="cuda", requires_grad=True)
target = torch.rand(1, 729, 32, 32, 32, device="cuda")

# Warmup
for i in range(100):
  output = F.conv3d(input, kernel)
  loss = F.mse_loss(output, target)
  loss.backward()
  kernel.grad.data.zero_()

with torch.autograd.profiler.profile(use_cuda=True) as prof:
  output = F.conv3d(input, kernel)
print(prof.key_averages().table(sort_by="cuda_time_total"))

with torch.autograd.profiler.profile(use_cuda=True) as prof:
  loss = F.mse_loss(output, target)
print(prof.key_averages().table(sort_by="cuda_time_total"))

with torch.autograd.profiler.profile(use_cuda=True) as prof:
  loss.backward()
print(prof.key_averages().table(sort_by="cuda_time_total"))
```

```
---------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Name                   Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     CUDA total %     CUDA total       CUDA time avg    Number of Calls  
---------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
conv3d                 6.22%            13.086us         100.00%          210.505us        210.505us        25.36%           1.296ms          1.296ms          1                
convolution            3.75%            7.893us          93.78%           197.419us        197.419us        25.15%           1.285ms          1.285ms          1                
_convolution           13.04%           27.443us         90.03%           189.526us        189.526us        25.00%           1.277ms          1.277ms          1                
cudnn_convolution      74.50%           156.819us        74.50%           156.819us        156.819us        24.40%           1.246ms          1.246ms          1                
contiguous             2.50%            5.264us          2.50%            5.264us          5.264us          0.10%            4.928us          4.928us          1                
---------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Self CPU time total: 210.505us
CUDA time total: 5.108ms

---------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Name                   Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     CUDA total %     CUDA total       CUDA time avg    Number of Calls  
---------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
mse_loss               93.99%           100.818us        93.99%           100.818us        100.818us        99.18%           783.104us        783.104us        1                
broadcast_tensors      6.01%            6.445us          6.01%            6.445us          6.445us          0.82%            6.464us          6.464us          1                
---------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Self CPU time total: 107.263us
CUDA time total: 789.568us

-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Name                                 Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     CUDA total %     CUDA total       CUDA time avg    Number of Calls  
-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
CudnnConvolutionBackward             5.09%            11.710us         39.36%           90.593us         90.593us         47.20%           12.720ms         12.720ms         1                
cudnn_convolution_backward           34.27%           78.883us         34.27%           78.883us         78.883us         47.19%           12.716ms         12.716ms         1                
MseLossBackward                      9.65%            22.209us         42.82%           98.559us         98.559us         2.80%            754.976us        754.976us        1                
mse_loss_backward                    33.17%           76.350us         33.17%           76.350us         76.350us         2.74%            738.176us        738.176us        1                
torch::autograd::AccumulateGrad      4.04%            9.295us          13.45%           30.961us         30.961us         0.03%            8.801us          8.801us          1                
torch::autograd::GraphRoot           4.36%            10.045us         4.36%            10.045us         10.045us         0.02%            6.336us          6.336us          1                
add_                                 9.41%            21.666us         9.41%            21.666us         21.666us         0.02%            4.609us          4.609us          1                
-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Self CPU time total: 230.158us
CUDA time total: 26.949ms
```

Setting `torch.backends.cudnn.enabled = False` gives this instead:

```
------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Name                      Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     CUDA total %     CUDA total       CUDA time avg    Number of Calls  
------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
conv3d                    6.22%            8.673us          100.00%          139.483us        139.483us        20.64%           661.504us        661.504us        1                
convolution               4.38%            6.103us          93.78%           130.810us        130.810us        20.37%           653.024us        653.024us        1                
_convolution              13.59%           18.962us         89.41%           124.707us        124.707us        20.17%           646.496us        646.496us        1                
_convolution_nogroup      4.91%            6.847us          71.96%           100.368us        100.368us        19.43%           622.624us        622.624us        1                
slow_conv_dilated3d       67.05%           93.521us         67.05%           93.521us         93.521us         19.22%           616.192us        616.192us        1                
contiguous                3.85%            5.377us          3.85%            5.377us          5.377us          0.17%            5.344us          5.344us          1                
------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Self CPU time total: 139.483us
CUDA time total: 3.205ms

---------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Name                   Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     CUDA total %     CUDA total       CUDA time avg    Number of Calls  
---------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
mse_loss               93.46%           75.576us         93.46%           75.576us         75.576us         99.40%           774.272us        774.272us        1                
broadcast_tensors      6.54%            5.289us          6.54%            5.289us          5.289us          0.60%            4.704us          4.704us          1                
---------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Self CPU time total: 80.865us
CUDA time total: 778.976us

-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Name                                 Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     CUDA total %     CUDA total       CUDA time avg    Number of Calls  
-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
SlowConvDilated3DBackward            6.15%            9.507us          35.73%           55.222us         55.222us         28.88%           1.020ms          1.020ms          1                
slow_conv_dilated3d_backward         29.58%           45.715us         29.58%           45.715us         45.715us         28.76%           1.016ms          1.016ms          1                
MseLossBackward                      10.33%           15.963us         41.21%           63.705us         63.705us         21.12%           745.760us        745.760us        1                
mse_loss_backward                    30.89%           47.742us         30.89%           47.742us         47.742us         20.74%           732.544us        732.544us        1                
torch::autograd::AccumulateGrad      4.89%            7.565us          14.64%           22.636us         22.636us         0.25%            8.800us          8.800us          1                
add_                                 9.75%            15.071us         9.75%            15.071us         15.071us         0.13%            4.640us          4.640us          1                
torch::autograd::GraphRoot           8.41%            13.006us         8.41%            13.006us         13.006us         0.12%            4.192us          4.192us          1                
-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  
Self CPU time total: 154.569us
CUDA time total: 3.531ms
```

## Expected behavior

Leaving `torch.backends.cudnn.enabled = True` as per the default should not cause a large slow down in the backward pass.

## Environment

The environment is Google Colab with a fresh GPU instance on which `!pip install -q torch==1.4.0 torchvision==0.5.0` has been run as Colab is still on `1.3.1` by default.

```
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.12.0

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: Tesla P100-PCIE-16GB
Nvidia driver version: 418.67
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip3] numpy==1.17.5
[pip3] torch==1.4.0
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.3.1
[pip3] torchvision==0.5.0
[conda] Could not collect
```

cc @ngimel @csarofeen @ptrblck