Repository: pytorch
Issue: Quantized conv2d with dilation and groups much slower than float32
Labels: module: performance,oncall: quantization,low priority,module: convolution,triaged

## 🐛 Bug

Convolutions with dilation and groups are 1~2 orders of magnitude slower than their float32 counterparts.

With the following, script, the float32 conv2d runs in ~25ms while the quantized conv2d runs in ~725ms.

## Code To Reproduce

```python
from copy import deepcopy
from typing import Tuple

import numpy as np
import torch
from torch import nn
from torch.quantization import QuantWrapper


def measure_inference_latency(model: nn.Module,
                              device: torch.device,
                              input_size: Tuple[int, ...] = (1, 1, 512, 512),
                              num_samples: int = 100) -> float:
    model.eval()
    x: torch.Tensor = torch.rand(size=input_size, device=device)

    start, end = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
    timings = []
    for _ in range(num_samples):
        start.record()
        model(x)
        end.record()
        torch.cuda.synchronize()
        timings.append(start.elapsed_time(end))

    return float(np.median(timings))


if __name__ == '__main__':
    torch.set_num_threads(4)
    device = torch.device('cpu')
    dilation = 2
    kernel_size = 3
    in_channels = 64
    out_channels = 64
    groups = 64

    padding = (kernel_size - 1) // 2 * dilation
    model = nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, padding=padding, dilation=dilation,
                      bias=False, groups=groups)

    q_model = QuantWrapper(module=deepcopy(model))
    q_model.qconfig = torch.quantization.get_default_qconfig('fbgemm')
    torch.quantization.prepare(q_model, inplace=True)
    torch.quantization.convert(q_model, inplace=True)

    latency_model = measure_inference_latency(model, device=device, input_size=(1, in_channels, 512, 512),
                                              num_samples=10)
    print('Model latency: {:.1f} '.format(latency_model))

    latency_quantized = measure_inference_latency(q_model, device=device, input_size=(1, in_channels, 512, 512),
                                                  num_samples=10)
    print('Quant latency: {:.1f} '.format(latency_quantized))
```

## Expected behavior

Quantized convolutions should 2~4 times faster.

## Environment

```
PyTorch version: 1.8.1
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8 (64-bit runtime)
Python platform: Linux-5.8.0-55-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.2.67
GPU models and configuration: GPU 0: GeForce RTX 3090
Nvidia driver version: 460.80
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.1
[pip3] torch==1.8.1
[pip3] torch-dct==0.1.5
[pip3] torchaudio==0.8.0a0+e4e171a
[pip3] torchfile==0.1.0
[pip3] torchvision==0.9.1
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.1.1               h6406543_8    conda-forge
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.3.0            py38h54f3939_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.20.1                   pypi_0    pypi
[conda] numpy-base                1.19.2           py38hfa32c7d_0  
[conda] pytorch                   1.8.1           py3.8_cuda11.1_cudnn8.0.5_0    pytorch
[conda] torch-dct                 0.1.5                    pypi_0    pypi
[conda] torchaudio                0.8.1                      py38    pytorch
[conda] torchfile                 0.1.0                    pypi_0    pypi
[conda] torchvision               0.9.1                py38_cu111    pytorch
```
## Additional context

CPU Model: Ryzen 7 3700X


cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel @VitalyFedyunin @ngimel @heitorschueroff