Repository: pytorch
Issue: Dynamic quantified Conv2d have accuracy issue
Labels: oncall: quantization,low priority,triaged

### 🐛 Describe the bug

Dynamic quantified Conv2d have accuracy issue. After quantify Conv2d of resnet18, the accuracy on imagenet will drop to 0. 

```python
import sys
from PIL import Image
import torch
import torchvision.models as models
import torchvision.transforms as transforms


model = models.__dict__['resnet18'](pretrained=True)
model.eval()
preprocess_transform = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])

if sys.argv[1] == 'old_int8':
    model = torch.quantization.quantize_dynamic(
            model,  # the original model
            {torch.nn.Linear},  # a set of layers to dynamically quantize
            dtype=torch.qint8)

if sys.argv[1] == 'new_int8':
    model = torch.quantization.quantize_dynamic(
            model,  # the original model
            {torch.nn.Linear, torch.nn.Conv2d},  # a set of layers to dynamically quantize
            dtype=torch.qint8)

#print(model)

img_path = 'ILSVRC2012_val_00000997.JPEG'

image_PIL = Image.open(img_path)
image_tensor = preprocess_transform(image_PIL)
image_tensor.unsqueeze_(0)
image_tensor = image_tensor
out = model(image_tensor)
_, indices = torch.sort(out, descending=True)
percentage = torch.nn.functional.softmax(out, dim=1)[0] * 100

top5_result = [(idx, percentage[idx].item()) for idx in indices[0][:5]]

print('Top5 result: ')
print('\tclass\tpercentage')
for res in top5_result:
    print('\t', int(res[0]), '\t', round(res[1], 4))
```

Result to reproduce:
image: 
![ILSVRC2012_val_00000997](https://user-images.githubusercontent.com/83260933/147624136-3331d821-1b48-48df-affd-5d8e1256a184.JPEG)


```
python test_strategy.py fp32
Top5 result: 
        class   percentage
         264     87.434
         195     7.6018
         232     2.5542
         180     0.4697
         245     0.4228

python test_strategy.py old_int8                                                                                    
Top5 result: 
        class   percentage
         264     87.5294
         195     7.7584
         232     2.4107
         180     0.4372
         245     0.3927

python test_strategy.py new_int8                                                                                    
Top5 result: 
        class   percentage
         558     11.8976
         776     11.8902
         513     10.1926
         822     6.5963
         684     5.8715
```
The result of command `python test_strategy.py new_int8` should be close to fp32 or old int8, isn't it?

### Versions

Collecting environment information...
PyTorch version: 1.11.0a0+git1d09458
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 7.2.1 20170829 (Red Hat 7.2.1-1)
Clang version: Could not collect
CMake version: version 3.19.6
Libc version: glibc-2.17

Python version: 3.8.11 (default, Sep 14 2021, 14:36:04)  [GCC 9.3.0] (64-bit runtime)
Python platform: Linux-3.10.0-862.el7.x86_64-x86_64-with-glibc2.15
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.2
[pip3] torch==1.11.0a0+git1d09458
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.12.0a0+65676b4
[pip3] warpctc-pytorch==0.1
[conda] mkl                       2022.0.1              intel_117    intel
[conda] mkl-include               2022.0.1              intel_117    intel
[conda] mkl-service               2.4.0           py38h4119f30_11    intel
[conda] mkl_fft                   1.3.1            py38h8344fd8_7    intel
[conda] mkl_random                1.2.2            py38h4ac99d2_7    intel
[conda] mkl_umath                 0.1.1           py38h03fa629_17    intel
[conda] mypy-extensions           0.4.3                    pypi_0    pypi
[conda] numpy                     1.21.2           py38hec4e512_7    intel
[conda] numpy-base                1.21.2           py38h40791c5_7    intel
[conda] torch                     1.10.0a0+git36449ea          pypi_0    pypi
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.12.0a0+65676b4           dev_0    <develop>
[conda] warpctc-pytorch           0.1                      pypi_0    pypi


cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel @ezyang @gchanan @zou3519 @bdhirsh