Repository: pytorch
Issue: How to exclude some modules from quantization?
Labels: oncall: quantization

### 🐛 Describe the bug

Hi there, I am newcomer to model quantization.  I have some problems and hope to get some advice and help from community. Thanks in advance!

Here is a demo model:

```python
class DemoModel(nn.Module):
    def __init__(self):
        super(DemoModel, self).__init__()
        self.conv = nn.Conv2d(3, 3, kernel_size=(3, 3))
        self.bn = nn.BatchNorm2d(3)
        self.fc = nn.Linear(3 * 26 * 26, 10)
        # comment following code if we use fx mode
        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.conv(x)
        x = self.bn(x)
        x = torch.reshape(x, (1, -1))
        output = self.fc(torch.relu(x))
        output = self.dequant(output)
        return output
```
I want to quantize it and export it as onnx format I got error messge:

```
Exporting the operator 'quantized::batch_norm2d' to ONNX opset version 17 is not supported
```
So there are compatibility issues between onnx and quantized op in pytroch. Is there any way to exclude some modules from quanzation and let rest modules be quantized ? `nn.batchNorm2d` here is a example. I found one solution is to fuse `nn.Conv2d` and `nn.BatchNorm2D`, like:

```
torch.quantization.fuse_modules(model, ['conv', 'bn'], inplace=True)
```
Or try to use FX mode intsead because it fuses conv and bn automatically. Howerver, I encountered another problem, FX mode would also fuse Linear and relu and then similiar error comes again :(. 
```
Exporting the operator 'quantized::linear_relu' to ONNX opset version 17 is not supported.
```

The demo model **not shown here**, just add linear and relu, quantize it in FX model then export to onnx format should reproduce it. 

In all, my question are:
1. **Is there any way to exclude some modules from quanzation and let rest modules be quantized?** 
 
` torch.quantization.prepare` provides a `allow_list` , if I filter out `nn.BatchNorm2d` , I got error message:

```
AttributeError: 'BatchNorm2d' object has no attribute 'activation_post_process'
```
2. **How to set some modules in FX mode not be fused?** 
3. **why `object_type` in qconfig_dict does not work?**
According to a SO [answer](https://stackoverflow.com/questions/72730969/pytorch-eager-quantization-skipping-modules#comment128471477_72733206),  using dict like :
```
    qconfig_dict = {"": torch.quantization.get_default_qconfig(backend),
                    "object_type": [
                        (torch.nn.Linear, None),
                        (torch.nn.ReLU, None)
                    ]}
```
could skip quantize `torch.nn.Linear` and `torch.nn.ReLU` but seems does not work , I still got  `Exporting the operator 'quantized::linear_relu' to ONNX`.

4. **How to quantize a sophiscated model (e.g. using other model as backbone)?**
In this scenario, if I choose eager mode , do I need to insert `quantStub` and `deQuantStub` to every backbone modules? If it's true, so FX mode is a better choose to quantize complex model, right ?
### Versions

version: 2.1.1+cu118


cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel