Repository: pytorch
Issue: [PT2E Quantization] _fuse_conv_bn_ | RuntimeError: !argument.default_value() INTERNAL ASSERT FAILED
Labels: oncall: quantization

### 🐛 Describe the bug

PT2E Quantization results incorrect.
After debugging, seems like `torch.ao.quantization.pt2e.utils._fuse_conv_bn_` has a bug.

To reproduce:
```python
import torch
import torch.ao.quantization.pt2e.utils
import torch.export._trace
from torch.ao.quantization.quantize_pt2e import convert_pt2e, prepare_pt2e
from torch.ao.quantization.quantizer.xnnpack_quantizer import (
    XNNPACKQuantizer,
    get_symmetric_quantization_config,
)

input = torch.rand(125, 3, 224, 224)
with torch.no_grad():
    model = torch.export.load('van-tiny_3rdparty_in1k.pt2').module()
    output_gt = model(input)  # works

    # After debugging, seems that the following line is the cause of the bug
    torch.ao.quantization.pt2e.utils._fuse_conv_bn_(model)
    model(input)  # RuntimeError: !argument.default_value() INTERNAL ASSERT FAILED


    # To compare quantization results
    # model = torch.export._trace._export(model, (input,), pre_dispatch=True).module()
    # quantizer = XNNPACKQuantizer().set_global(
    #     get_symmetric_quantization_config()
    # )
    # model = prepare_pt2e(model, quantizer)
    # model(input)
    # model = convert_pt2e(model)
    # output_quant = model(input)
    # print(f'{output_gt.flatten()[:10]=}')
    # print(f'{output_quant.flatten()[:10]=}')
```

[van-tiny_3rdparty_in1k.zip](https://github.com/pytorch/pytorch/files/15302077/van-tiny_3rdparty_in1k.zip)



Traceback
```
Traceback (most recent call last):
  File "reproduce_fuse_conv_bn_bug.py", line 18, in <module>
    model(input)  # RuntimeError: !argument.default_value() INTERNAL ASSERT FAILED
  File "/home/chensf/git/pytorch-quantization/env-torch2.3/lib/python3.8/site-packages/torch/fx/graph_module.py", line 737, in call_wrapped
    return self._wrapped_call(self, *args, **kwargs)
  File "/home/chensf/git/pytorch-quantization/env-torch2.3/lib/python3.8/site-packages/torch/fx/graph_module.py", line 317, in __call__
    raise e
  File "/home/chensf/git/pytorch-quantization/env-torch2.3/lib/python3.8/site-packages/torch/fx/graph_module.py", line 304, in __call__
    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
  File "/home/chensf/git/pytorch-quantization/env-torch2.3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/chensf/git/pytorch-quantization/env-torch2.3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1582, in _call_impl
    result = forward_call(*args, **kwargs)
  File "<eval_with_key>.5", line 347, in forward
  File "/home/chensf/git/pytorch-quantization/env-torch2.3/lib/python3.8/site-packages/torch/_ops.py", line 594, in __call__
    return self_._op(*args, **kwargs)
RuntimeError: !argument.default_value() INTERNAL ASSERT FAILED at "../aten/src/ATen/core/function_schema_inl.h":367, please report a bug to PyTorch.
```

Here is the diff of model tabulars before and after `_fuse_conv_bn_`
```diff
$ diff van_no_fuse.txt van_fuse_first_conv_bn.txt
239,240d238
< get_attr       model_backbone_patch_embed1_norm_weight                             model.backbone.patch_embed1.norm.weight                             ()                                                                                                                                                                                                    {}
< get_attr       model_backbone_patch_embed1_norm_bias                               model.backbone.patch_embed1.norm.bias                               ()                                                                                                                                                                                                    {}
315,316d312
< get_attr       model_backbone_patch_embed1_norm_running_mean                       model.backbone.patch_embed1.norm.running_mean                       ()                                                                                                                                                                                                    {}
< get_attr       model_backbone_patch_embed1_norm_running_var                        model.backbone.patch_embed1.norm.running_var                        ()                                                                                                                                                                                                    {}
376,379c372,374
< call_function  conv2d                                                              aten.conv2d.default                                                 (arg402_1, model_backbone_patch_embed1_projection_weight)                                                                                                                                             {'bias': model_backbone_patch_embed1_projection_bias, 'stride': [4, 4], 'padding': [3, 3]}
< call_function  _native_batch_norm_legit_no_training_default                        aten._native_batch_norm_legit_no_training.default                   (conv2d, model_backbone_patch_embed1_norm_weight, model_backbone_patch_embed1_norm_bias, model_backbone_patch_embed1_norm_running_mean, model_backbone_patch_embed1_norm_running_var, 0.1, 1e-05)     {}
< call_function  getitem                                                             <built-in function getitem>                                         (_native_batch_norm_legit_no_training_default, 0)                                                                                                                                                     {}
< call_function  _native_batch_norm_legit_no_training_default_1                      aten._native_batch_norm_legit_no_training.default                   (getitem, model_backbone_blocks1_0_norm1_weight, model_backbone_blocks1_0_norm1_bias, model_backbone_blocks1_0_norm1_running_mean, model_backbone_blocks1_0_norm1_running_var, 0.1, 1e-05)            {}
---
> get_attr       model_backbone_patch_embed1_projection_weight_bias                  model.backbone.patch_embed1.projection.weight_bias                  ()                                                                                                                                                                                                    {}
> call_function  conv2d                                                              aten.conv2d.default                                                 (arg402_1, model_backbone_patch_embed1_projection_weight, model_backbone_patch_embed1_projection_weight_bias)                                                                                         {'bias': model_backbone_patch_embed1_projection_bias, 'stride': [4, 4], 'padding': [3, 3]}
> call_function  _native_batch_norm_legit_no_training_default_1                      aten._native_batch_norm_legit_no_training.default                   (conv2d, model_backbone_blocks1_0_norm1_weight, model_backbone_blocks1_0_norm1_bias, model_backbone_blocks1_0_norm1_running_mean, model_backbone_blocks1_0_norm1_running_var, 0.1, 1e-05)             {}
394c389
< call_function  add_1                                                               aten.add.Tensor                                                     (getitem, mul_1)                                                                                                                                                                                      {}
---
> call_function  add_1                                                               aten.add.Tensor                                                     (conv2d, mul_1)                                                                                                                                                                                       {}
```

### Versions

```
Collecting environment information...
PyTorch version: 2.3.0
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0
Clang version: Could not collect
CMake version: version 3.10.2
Libc version: glibc-2.27

Python version: 3.8.19 (default, Mar 20 2024, 19:58:24) 
```

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel