Repository: pytorch
Issue: conv_transpose3d returns different result when the input and kernel are mkldnn tensors
Labels: high priority,module: dependency bug,module: convolution,triaged,module: mkldnn

## 🐛 Bug

We implemented a upsampler using conv_transpose3d, it works correctly with CPU and GPU backend, but when I tested it with mkldnn, the output is incorrect. In the example below, upsampling a 1x4x4 tensor returns a 1x8x8 tensor, but when the input and kernel are converted using to_mkldnn, the output is a 1x2x2 tensor

## To Reproduce

Here is the sample code:
```
import torch
from torch.nn import functional as F
                                                                                                                                             

input_data = torch.arange(1, 17, dtype=torch.float32).view(1, 1, 1, 4, 4)
factor=(1,4,4)
padding=(0,2,2)
kernel = torch.ones(1, 1, *factor)
print("input tensor:")
print(input_data)
print("kernel:")
print(kernel)
output=F.conv_transpose3d(input_data, kernel, stride=factor, padding=padding, groups=1)
print("output [dense input and weight]:")
print(output)
output2=F.conv_transpose3d(input_data.to_mkldnn(), kernel.to_mkldnn(), stride=factor, padding=padding, groups=1)
print("output [mkldnn input and weight]:")
print(output2)
```
The output:

```
input tensor:
tensor([[[[[ 1.,  2.,  3.,  4.],
           [ 5.,  6.,  7.,  8.],
           [ 9., 10., 11., 12.],
           [13., 14., 15., 16.]]]]])
kernel:
tensor([[[[[1., 1., 1., 1.],
           [1., 1., 1., 1.],
           [1., 1., 1., 1.],
           [1., 1., 1., 1.]]]]])
output [dense input and weight]:
tensor([[[[[ 1.,  1.,  2.,  2.,  2.,  2.,  3.,  3.,  3.,  3.,  4.,  4.],
           [ 1.,  1.,  2.,  2.,  2.,  2.,  3.,  3.,  3.,  3.,  4.,  4.],
           [ 5.,  5.,  6.,  6.,  6.,  6.,  7.,  7.,  7.,  7.,  8.,  8.],
           [ 5.,  5.,  6.,  6.,  6.,  6.,  7.,  7.,  7.,  7.,  8.,  8.],
           [ 5.,  5.,  6.,  6.,  6.,  6.,  7.,  7.,  7.,  7.,  8.,  8.],
           [ 5.,  5.,  6.,  6.,  6.,  6.,  7.,  7.,  7.,  7.,  8.,  8.],
           [ 9.,  9., 10., 10., 10., 10., 11., 11., 11., 11., 12., 12.],
           [ 9.,  9., 10., 10., 10., 10., 11., 11., 11., 11., 12., 12.],
           [ 9.,  9., 10., 10., 10., 10., 11., 11., 11., 11., 12., 12.],
           [ 9.,  9., 10., 10., 10., 10., 11., 11., 11., 11., 12., 12.],
           [13., 13., 14., 14., 14., 14., 15., 15., 15., 15., 16., 16.],
           [13., 13., 14., 14., 14., 14., 15., 15., 15., 15., 16., 16.]]]]])
output [mkldnn input and weight]:
tensor([[[[[14., 22.],
           [46., 54.]]]]], layout=torch._mkldnn)
```

## Expected behavior

The output stays the same when I use mkldnn tensors.

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

> Collecting environment information...
> PyTorch version: 1.7.0
> Is debug build: True
> CUDA used to build PyTorch: 11.0
> ROCM used to build PyTorch: N/A
> 
> OS: Ubuntu 20.04.1 LTS (x86_64)
> GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
> Clang version: Could not collect
> CMake version: version 3.16.3
> 
> Python version: 3.8 (64-bit runtime)
> Is CUDA available: True
> CUDA runtime version: Could not collect
> GPU models and configuration: GPU 0: GeForce RTX 2070 with Max-Q Design
> Nvidia driver version: 450.80.02
> cuDNN version: Could not collect
> HIP runtime version: N/A
> MIOpen runtime version: N/A
> 
> Versions of relevant libraries:
> [pip3] numpy==1.19.2
> [pip3] torch==1.7.0
> [pip3] torchaudio==0.7.0a0+ac17b64
> [pip3] torchvision==0.8.1
> [conda] blas                      1.0                         mkl  
> [conda] cudatoolkit               11.0.221             h6bb024c_0  
> [conda] mkl                       2020.2                      256  
> [conda] mkl-service               2.3.0            py38he904b0f_0  
> [conda] mkl_fft                   1.2.0            py38h23d657b_0  
> [conda] mkl_random                1.1.1            py38h0573a6f_0  
> [conda] numpy                     1.19.2           py38h54aff64_0  
> [conda] numpy-base                1.19.2           py38hfa32c7d_0  
> [conda] pytorch                   1.7.0           py3.8_cuda11.0.221_cudnn8.0.3_0    pytorch
> [conda] torchaudio                0.7.0                      py38    pytorch
> [conda] torchvision               0.8.1                py38_cu110    pytorch



cc @ezyang @gchanan @zou3519 @bdhirsh @gujinghui @PenghuiCheng @XiaobingSuper @jianyuh @VitalyFedyunin