Repository: pytorch
Issue: (prototype) Graph Mode Dynamic Quantization on BERT failure on quantize_dynamic_jit(...) call
Labels: triaged,oncall: mobile

## 🐛 Bug

Attempted to reproduce results from
https://pytorch.org/tutorials/intermediate/dynamic_quantization_bert_tutorial.html#install-pytorch-and-huggingface-transformers

## To Reproduce

Steps to reproduce the behavior:

1. Follow the tutorial under the link above:

`tokenizer = BertTokenizer.from_pretrained(configs.output_dir, do_lower_case=configs.do_lower_case)`
`model = BertForSequenceClassification.from_pretrained(configs.output_dir, torchscript=True)
model.to(configs.device)`

`input_ids = ids_tensor([8, 128], 2)`
`token_type_ids = ids_tensor([8, 128], 2)`
`attention_mask = ids_tensor([8, 128], vocab_size=2)`
`dummy_input = (input_ids, attention_mask, token_type_ids)`
`traced_model = torch.jit.trace(model, dummy_input)`

`qconfig_dict = {'': per_channel_dynamic_qconfig}`

`quantized_model = quantize_dynamic_jit(traced_model, qconfig_dict)`

## Expected behavior

The instruction `quantized_model = quantize_dynamic_jit(traced_model, qconfig_dict)` executed

## Actual result

RuntimeError: 
Module 'MinMaxDynamicQuantObserver' has no attribute '_calculate_qmin_qmax' :
  File "/usr/local/lib/python3.6/dist-packages/torch/quantization/observer.py", line 254
            )

        quant_min, quant_max = self._calculate_qmin_qmax()
                               ~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
        min_val_neg = torch.min(min_val, torch.zeros_like(min_val))
        max_val_pos = torch.max(max_val, torch.zeros_like(max_val))

## Environment

PyTorch version: 1.7.0.dev20200921+cpu
Is debug build: True
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.12.0

Python version: 3.6 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: 10.1.243
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] torch==1.7.0.dev20200921+cpu
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.3.1
[conda] Could not collect

## Additional context

Full Stack trace:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-11-d266318b4136> in <module>()
----> 1 quantized_model = quantize_dynamic_jit(traced_model, qconfig_dict)

10 frames
/usr/local/lib/python3.6/dist-packages/torch/quantization/quantize_jit.py in quantize_dynamic_jit(model, qconfig_dict, inplace, debug)

/usr/local/lib/python3.6/dist-packages/torch/quantization/quantize_jit.py in _quantize_jit(model, qconfig_dict, run_fn, run_args, inplace, debug, quant_type)
    101         model = prepare_jit(model, qconfig_dict, inplace)
    102         run_fn(model, *run_args)
--> 103         model = convert_jit(model, True, debug)
    104 
    105     return model

/usr/local/lib/python3.6/dist-packages/torch/quantization/quantize_jit.py in prepare_dynamic_jit(model, qconfig_dict, inplace)
     73     model_c = model._c
     74     model_c = torch._C._jit_pass_insert_quant_dequant(model_c, 'forward', inplace, debug, quant_type)
---> 75     if not debug:
     76         # Moving model parameters to CPU since quantized operators
     77         # are only supported on CPU right now

/usr/local/lib/python3.6/dist-packages/torch/quantization/quantize_jit.py in _prepare_jit(model, qconfig_dict, inplace, quant_type)
     56                                                   inplace,
     57                                                   quant_type)
---> 58     if inplace:
     59         model._reconstruct(model_c)
     60     else:

/usr/local/lib/python3.6/dist-packages/torch/quantization/quantize_jit.py in script_qconfig_dict(qconfig_dict)
     34 
     35     Args:
---> 36         model: TorchScript model from scripting or tracing
     37     """
     38     model_c = model._c

/usr/local/lib/python3.6/dist-packages/torch/quantization/quantize_jit.py in <dictcomp>(.0)
     34 
     35     Args:
---> 36         model: TorchScript model from scripting or tracing
     37     """
     38     model_c = model._c

/usr/local/lib/python3.6/dist-packages/torch/quantization/quantize_jit.py in script_qconfig(qconfig)
     26     Apply `script_qconfig` for all entries in `qconfig_dict` that is
     27     not None.
---> 28     """
     29     return {k: script_qconfig(v) if v else None for k, v in qconfig_dict.items()}
     30 

/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py in script(obj, optimize, _frames_up, _rcb)

/usr/local/lib/python3.6/dist-packages/torch/jit/_recursive.py in create_script_module(nn_module, stubs_fn, share_types)
    316         # Get a concrete type directly, without trying to re-use an existing JIT
    317         # type from the type store.
--> 318         concrete_type_builder = infer_concrete_type_builder(nn_module, share_types)
    319         concrete_type_builder.set_poisoned()
    320         concrete_type = concrete_type_builder.build()

/usr/local/lib/python3.6/dist-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
    374                 # use the default recursive rule to compile the module
    375                 scripted = create_script_module_impl(orig_value, sub_concrete_type, infer_methods_to_compile)
--> 376 
    377             cpp_module.setattr(name, scripted)
    378             script_module._modules[name] = scripted

/usr/local/lib/python3.6/dist-packages/torch/jit/_recursive.py in create_methods_from_stubs(concrete_type, stubs)
    290 
    291     concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
--> 292 
    293 
    294 def get_module_concrete_type(nn_module, share_types=True):

RuntimeError: 
Module 'MinMaxDynamicQuantObserver' has no attribute '_calculate_qmin_qmax' :
  File "/usr/local/lib/python3.6/dist-packages/torch/quantization/observer.py", line 254
            )

        quant_min, quant_max = self._calculate_qmin_qmax()
                               ~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
        min_val_neg = torch.min(min_val, torch.zeros_like(min_val))
        max_val_pos = torch.max(max_val, torch.zeros_like(max_val))


cc @gmagogsfm @jerryzh168 @jianyuh @dzhulgakov @raghuramank100 @jamesr66a @vkuzo