Repository: pytorch
Issue: Double backward 3 times slower for conv2d with padding = 1
Labels: module: dependency bug,module: performance,module: double backwards,module: nn,module: convolution,triaged,has workaround,quansight-nack

## 🐛 Bug

Hi, I'm trying to compute the jacobian vector product using a trick from https://j-towns.github.io/2017/06/12/A-new-trick.html. This involves computing the gradient twice. It seems like using conv2d with padding =1 slows down the computation by almost 3 times, compared to padding =0:

padding= 0:  2.962866 s
padding=1:  9.922703s

In case only one backward pass is computed both give the same times:

padding =0: 1.056120s
padding = 1: 1.180639s


## To Reproduce
```python
import  torch as tr
from torch.autograd import Variable
import torch.nn as nn
from torch import autograd

import time
from copy import deepcopy
from datetime import datetime
startTime = datetime.now()

import torch.nn as nn
import torch.nn.functional as F
from models.resnet import *

tr.backends.cudnn.benchmark=True

class Net(nn.Module):
    def __init__(self,padding= 0):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, 3,padding=padding)
        self.conv2 = nn.Conv2d(64, 64, 3,padding=padding)

    def forward(self, x):
        out = F.relu(self.conv1(x))
        out = F.relu(self.conv2(out))
        return out.sum()


def jvp(outputs,variable,direction):
    dummy_u =  Variable(tr.ones_like(outputs), requires_grad=True)
    aux_grad_loss = tr.autograd.grad(outputs = outputs,inputs = variable, grad_outputs=dummy_u,  create_graph=True,only_inputs=True)
    out = tr.autograd.grad(outputs= aux_grad_loss,inputs = dummy_u,grad_outputs=direction,create_graph=True,only_inputs=True)[0]
    return out

device = 'cuda'
dist = tr.distributions.Normal(tr.tensor(4.0,dtype= tr.float32, device = device), tr.tensor(0.5,dtype= tr.float32, device = device))


b_size = 128
num_iter = 100
X = dist.sample([b_size,3,32,32])

print("no padding")
net = Net(padding=0).to(device)

params = list(net.parameters())
directions = deepcopy(params)

torch.cuda.synchronize()
startTime = datetime.now()
for k in range(num_iter):

    Y = tr.sum(net(X))
    dd = jvp(Y,params,directions)

torch.cuda.synchronize()
print(datetime.now() - startTime)

print("with padding")

net = Net(padding=1).to(device)
params = list(net.parameters())
directions = deepcopy(params)

torch.cuda.synchronize()
startTime = datetime.now()
for k in range(num_iter):
    Y = tr.sum(net(X))
    dd = jvp(Y,params,directions)
torch.cuda.synchronize()
print(datetime.now() - startTime)
```
## Expected behavior

The computation time should be roughly similar with or without padding.

## Environment


PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 16.04.5 LTS
GCC version: (crosstool-NG 1.23.0.449-a04d0) 7.3.0
CMake version: version 3.7.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: Quadro P5000
GPU 1: GeForce RTX 2080

Nvidia driver version: 410.73
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.3.1

Versions of relevant libraries:
[pip] numpy==1.16.4
[pip] torch==1.1.0
[pip] torchvision==0.3.0
[conda] mkl                       2019.4                      243
[conda] torch                     1.1.0                     <pip>
[conda] torchvision               0.3.0                     <pip>

Edit [t-vi]: add backticks around source

cc @ezyang @gchanan @zou3519 @jerryzh168 @VitalyFedyunin @ngimel @mruberry