Repository: pytorch
Issue: MKLDNN_conv2d 2X slower than the native TH implementation
Labels: module: performance,module: cpu,triaged,module: mkldnn

## 🐛 Bug
It seems the mkldnn_convolution runs 2X slower than native thnn_conv2d_forward(use MKL for gemm).

<!-- A clear and concise description of what the bug is. -->

## To Reproduce

```

import torch
import torch.nn.functional as F

import numpy as np

from timeit import Timer


num = 50

N = 1
C = 512
H = 4
W = 4

M = 512
kernel_h = 3
kernel_w = 3
stride_h = 1
stride_w = 1
padding_h = 1
padding_w = 1

X_np = np.random.randn(N, C, H, W).astype(np.float32)
W_np = np.random.randn(M, C, kernel_h, kernel_w).astype(np.float32)
X = torch.from_numpy(X_np)

conv2d_pt = torch.nn.Conv2d(
    C, M, (kernel_h, kernel_w), stride=(stride_h, stride_w),
    padding=(padding_h, padding_w), groups=1, bias=False)

class ConvNet(torch.nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.conv2d = conv2d_pt

    def forward(self, x):
        return self.conv2d(x)

model = ConvNet()

def pt_forward():
    with torch.autograd.profiler.profile(record_shapes=True) as prof:
        model(X)
    print(prof.key_averages().table(sort_by="self_cpu_time_total"))

# torch._C._set_mkldnn_enabled(False)

t = Timer("pt_forward()", "from __main__ import pt_forward, X")
print("pt time = {}".format(t.timeit(num) / num * 1000.0))
```

Steps to reproduce the behavior:

1. You need to build Pytorch with USE_BLAS=ON, USE_MKL=ON, USE_MKLDNN=ON
2. Then you can  torch._C._set_mkldnn_enabled(False) to switch between MKLDNN convolution and native convolution.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior
```
----------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------
Name                    Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     Number of Calls
----------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------
mkldnn_convolution      99.26%           3.037ms          99.26%           3.037ms          3.037ms          1
_convolution            0.46%            14.153us         99.77%           3.053ms          3.053ms          1
conv2d                  0.13%            3.880us          100.00%          3.060ms          3.060ms          1
convolution             0.10%            3.083us          99.87%           3.056ms          3.056ms          1
contiguous              0.05%            1.403us          0.05%            1.403us          0.701us          2
----------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------
Self CPU time total: 3.060ms
```
```
------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------
Name                      Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     Number of Calls
------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------
thnn_conv2d_forward       98.52%           1.619ms          98.52%           1.619ms          1.619ms          1
_convolution              0.57%            9.306us          99.63%           1.638ms          1.638ms          1
_convolution_nogroup      0.27%            4.363us          99.00%           1.627ms          1.627ms          1
conv2d                    0.22%            3.551us          100.00%          1.644ms          1.644ms          1
thnn_conv2d               0.19%            3.154us          98.71%           1.622ms          1.622ms          1
convolution               0.16%            2.559us          99.78%           1.640ms          1.640ms          1
contiguous                0.07%            1.087us          0.07%            1.087us          1.087us          1
_nnpack_available         0.02%            0.366us          0.02%            0.366us          0.366us          1
------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------
Self CPU time total: 1.644ms

```
<!-- A clear and concise description of what you expected to happen. -->

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

 - PyTorch Version (e.g., 1.0):
 - OS (e.g., Linux):
 - How you installed PyTorch (`conda`, `pip`, source):
 - Build command you used (if compiling from source):
 - Python version:
 - CUDA/cuDNN version:
 - GPU models and configuration:
 - Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


cc @VitalyFedyunin @gujinghui @PenghuiCheng @XiaobingSuper @jianyuh @ngimel