Repository: mlcommons/inference
Issue: mobilenet int8  model not meeting 99% accuracy target. 
Root cause: Validation Gap


I've measured the [Top1/Top5 accuracy with TF-C++ via CK](https://github.com/dividiti/inference/blob/update-ck/v0.5/classification_and_detection/optional_harness_ck/classification/tf-cpp/README.md#reference-accuracy) with the official TF image classification models using 3 preprocessing methods:
- Pillow (aka PIL);
- OpenCV (the new official);
- TensorFlow (ResNet-specific).

| Model                   | Metric | Pillow  | OpenCV  | TensorFlow |
|-|-|-|-|-|
| ResNet                  |  Top1  | 0.76170 | 0.76458 | 0.76522 |
|                         |  Top5  | 0.92866 | 0.93014 | 0.93066 |
| MobileNet non-quantized |  Top1  | 0.71226 | 0.71516 | N/A     |
|                         |  Top5  | 0.89834 | 0.90004 | N/A     |
| MobileNet quantized     |  Top1  | 0.70348 | 0.70654 | N/A     |
|                         |  Top5  | 0.89376 | 0.89514 | N/A     |


Let's concentrate on MobileNet, and also add Top1 measured through the official application:

| Model                   | Metric | Pillow  | OpenCV  | Official |
|-|-|-|-|-|
| MobileNet non-quantized |  Top1  | 0.71226 | 0.71516 | 0.71676 |
| MobileNet quantized     |  Top1  | 0.70348 | 0.70654 | 0.70676 |

Expressed in percent from the official MobileNet non-quantized score (100.00%):

| Model                   | Metric | Pillow  | OpenCV  | Official |
|-|-|-|-|-|
| MobileNet non-quantized |  Top1  | 99.37% | 99.97% | 100.00% |
| MobileNet quantized     |  Top1  | 98.14% | 98.57% | 98.74% |

Expressed in percent from the MobileNet non-quantized score with Pillow (100.00%):

| Model                   | Metric | Pillow  | OpenCV  | Official |
|-|-|-|-|-|
| MobileNet non-quantized |  Top1  | 100.00% | 100.41% | 100.64% |
| MobileNet quantized     |  Top1  | 98.77% | 99.20% | 99.23% |

Expressed in percent from the MobileNet non-quantized score with OpenCV (100.00%):

| Model                   | Metric | Pillow  | OpenCV  | Official |
|-|-|-|-|-|
| MobileNet non-quantized |  Top1  | 99.59% | 100.00% | 100.23% |
| MobileNet quantized     |  Top1  | 98.37% | 98.79% | 98.83% |

I think it would be fair not to outlaw using Pillow, as OpenCV and TensorFlow are not easy to install on non-x86 platforms. In this case, no matter which non-quantized baseline we choose, we would need to increase the allowed accuracy drop to 2% from the baseline.

Another option would seem to be to obtain a more accurate quantized model, but it's probably out of the question for v0.5.

@psyhtest is 'MobileNet quantized' MobileNet_v1_1.0_224_quant from https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md ? Were you able to measure with the INT8 model provided in the readme?

@christ1ne Yes, that's the one. I believe it should also be added to the table. 

I am not sure yet what's required to make https://zenodo.org/record/3353417/files/Quantized%20MobileNet.zip work, but our focus is on TF/TFLite anyway. 

From #309, it appears that an older version of the MobileNet TF model (2018_02_22) might have been used instead of the most recent one (2018_08_02). Perhaps this explains the accuracy difference between the official code and CK.

I've got the same accuracy with the older model (2018_02_22) as with the latest one (2018_08_02). I now recall that the difference between the two was due to the TFLite model, which can be quickly validated with CK:
```
anton@diviniti:~/projects/mlperf/dividiti_inference$ md5sum `ck locate env --tags=model,tf,mlperf,mobilenet,2018_08_02`/mobilenet_v1_1.0_224_frozen.pb
7f564c782f0141e4e7bdc5161a6cc2f4  /home/anton/CK_TOOLS/model-tf-mlperf-mobilenet-downloaded/mobilenet_v1_1.0_224_frozen.pb
anton@diviniti:~/projects/mlperf/dividiti_inference$ md5sum `ck locate env --tags=model,tf,mlperf,mobilenet,2018_02_22`/mobilenet_v1_1.0_224_frozen.pb
7f564c782f0141e4e7bdc5161a6cc2f4  /home/anton/CK_TOOLS/model-tf-mlperf-mobilenet-downloaded-2018_02_22/mobilenet_v1_1.0_224_frozen.pb

anton@diviniti:~/projects/mlperf/dividiti_inference$ md5sum `ck locate env --tags=model,tf,mlperf,mobilenet,2018_02_22`/mobilenet_v1_1.0_224.tflite
8ac1abc6bb388135b8ae9343460482fe  /home/anton/CK_TOOLS/model-tf-mlperf-mobilenet-downloaded-2018_02_22/mobilenet_v1_1.0_224.tflite
anton@diviniti:~/projects/mlperf/dividiti_inference$ md5sum `ck locate env --tags=model,tf,mlperf,mobilenet,2018_08_02`/mobilenet_v1_1.0_224.tflite
b59fffd212856b84de2acc55be9f0693  /home/anton/CK_TOOLS/model-tf-mlperf-mobilenet-downloaded/mobilenet_v1_1.0_224.tflite
```

Do we have a script that could reproduce the 99% of baseline accuracy from quantized model?  Since Nvidia provide the quantized mobilenet model, does Nvidia also have script for reproduce the accuracy number?

Sorry, we don't. 

My understanding is that NVIDIA provide a [quantized ONNX model](https://zenodo.org/record/3353417). If you use the [official quantized TF model](https://zenodo.org/record/2269307), you can measure its accuracy [via CK](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection/optional_harness_ck/classification) or [via the official reference code](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection).

Meanwhile, the accuracy of the quantized MobileNet TFLite model with OpenCV preprocessing is 0.70776, or a bit higher than that of the quantized MobileNet TF model. But that's still 98.97% from 0.71516 - the accuracy of the non-quantized MobileNet TF model.

According to the [latest data](https://github.com/ctuning/ck-env/tree/master/package/dataset-imagenet-preprocessed-using-opencv#accuracy-on-the-imagenet-2012-validation-set) (with OpenCV preprocessing via CK to match that of the official app), the non-quantized **TF** MobileNet model achieves 0.71676, while the quantized **TF** MobileNet model achieves 0.70700, or 98.6%.

It's peculiar that the quantized **TFLite** model achieves 0.70762 (< 0.70776 which was with mismatching preprocessing).

closed as we changed the accuracy to 98%