Repository: mlcommons/inference
Issue: Why input images are quantized for mobilenetSSD

For ssd mobilenet we observed input images are quantized during the input pre-processing step, 
https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/dataset.py#L219

We would like to know, why ssd Mobilenet_SSD’s input images are quantized while no other model has input images quantized?  

thanks 
Will


Hi Will,

While the images preprocessed for SSD-MobileNet are stored as 8-bit integers, I wouldn't say they are quantized. After all, the original JPEG images are also stored as 8-bit integers :).

I think it's useful to reflect why images preprocessed for ResNet and MobileNet are stored as 32-bit floats. I believe it's because ResNet and MobileNet require different preprocessing after resizing: ResNet requires [means to be subtracted](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/dataset.py#L178), while MobileNet requires [normalization to the (-1, 1) range](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/dataset.py#L195). In addition, the official ResNet preprocessing uses [area interpolation](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/dataset.py#L172), instead of the default [bilinear interpolation](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/dataset.py#L154). That's why preprocessed images for both MobileNet and ResNet require 58G on disk. Preprocessing for SSD-MobileNet needs neither means to be subtracted nor normalization. Therefore, images preprocessed for SSD-MobileNet can be stored as 8-bit integers.

As an aside, with the (unofficial) [Collective Knowledge workflows](https://github.com/dividiti/inference/edit/updateaccuracy/v0.5/classificatiaon_and_detection/optional_harness_ck/classification/README.md), we support an alternative preprocesing method, dubbed OpenCV (universal), which uses bilinear interpolation and stores pixels as 8-bit integers. At load time, however, minor additional processing may be required depending on the model (e.g. see sample [code for normalization](https://github.com/ctuning/ck-tensorflow/blob/master/program/image-classification-tflite/benchmark.h#L463) and [code for subtracting means](https://github.com/ctuning/ck-tensorflow/blob/master/program/image-classification-tflite/benchmark.h#L469)).

The table below shows the accuracy on the ImageNet 2012 validation set
(50,000 images) measured [via TensorFlow (C++)](tf-cpp/README.md).

| | Pillow  | OpenCV universal | OpenCV for ResNet | OpenCV for MobileNet | TensorFlow |
|-|-|-|-|-|-|
| Matches official?    | No                | No                       | Yes                          | Yes                       | No                 |
| Additional CK package tags   | `using-pillow`    | `using-opencv,universal` | `using-opencv,for-resnet` | `using-opencv,for-mobilenet` | `using-tensorflow` |
| Supported models     | ResNet, MobileNet | ResNet, MobileNet        | ResNet only               | MobileNet only               | ResNet only        |
| Supported platforms  | x86, arm          | x86                      | x86                          | x86                       | x86 (prebuilt TF)  |
| Data format          | rgb8 (int8)       | rgb8 (int8)              | rgbf32 (float32)             | rgbf32 (float32)          | rgbf32 (float32)   |
| Data size            | 7.1G              | 7.1G                     | 29G                          | 29G                       | 29G                |
| ResNet Top1  | 0.76170 | 0.76422          | 0.76456           | N/A                  | 0.76522    |
| ResNet Top5  | 0.92866 | 0.93074          | 0.93016           | N/A                  | 0.93066    |
| MobileNet non-quantized Top1  | 0.71226 | 0.71676          | N/A               | 0.71676              | N/A        |
| MobileNet non-quantized Top5  | 0.89834 | 0.90118          | N/A               | 0.90118              | N/A        |
| MobileNet quantized Top1  | 0.70348 | 0.70700          | N/A               | 0.70694              | N/A        |
| MobileNet quantized Top5  | 0.89376 | 0.89594          | N/A               | 0.89594              | N/A        |

Considering Top1, the universal OpenCV method is slightly less accurate for ResNet, but slightly more accurate for MobileNet quantized than the official code. The TensorFlow method is most accurate for ResNet, but is not suitable for MobileNet. The Pillow method is least accurate, but can be used on Arm platforms. The difference between the universal OpenCV and the TensorFlow methods on ResNet is exactly 0.1% or 50 images.

For more information, please see [here](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection/optional_harness_ck/classification#preprocess-datasets).

Anton,

Thanks for tracking these differences. Since we do have an "official"
version -- openCV based -- can we update and indicate that in the table
above on the README? Just so that people know which one MLPerf is using?

On Tue, Aug 20, 2019 at 8:08 AM Anton Lokhmotov <notifications@github.com>
wrote:

> Hi Will,
>
> While the images preprocessed for SSD-MobileNet are stored as 8-bit
> integers, I wouldn't say they are quantized. After all, the original JPEG
> images are also stored as 8-bit integers :).
>
> I think it's useful to reflect why images preprocessed for ResNet and
> MobileNet are stored as 32-bit floats. I believe it's because ResNet and
> MobileNet require different preprocessing after resizing: ResNet requires means
> to be subtracted
> <https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/dataset.py#L178>,
> while MobileNet requires normalization to the (-1, 1) range
> <https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/dataset.py#L195>.
> In addition, the official ResNet preprocessing uses area interpolation
> <https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/dataset.py#L172>,
> instead of the default bilinear interpolation
> <https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/dataset.py#L154>.
> That's why preprocessed images for both MobileNet and ResNet require 58G on
> disk. Preprocessing for SSD-MobileNet needs neither means to be subtracted
> nor normalization. Therefore, images preprocessed for SSD-MobileNet can be
> stored as 8-bit integers.
>
> As an aside, with unofficial Collective Knowledge workflows for MLPerf
> <https://github.com/dividiti/inference/edit/update-accuracy/v0.5/classification_and_detection/optional_harness_ck/classification/README.md>,
> we support an alternative preprocesing method, dubbed OpenCV (universal),
> uses bilinear interpolation and stores pixels as 8-bit integers. At load
> time, however, minor additional processing may be required depending on the
> model (e.g. see sample code for normalization
> <https://github.com/ctuning/ck-tensorflow/blob/master/program/image-classification-tflite/benchmark.h#L463>
> and code for subtracting means
> <https://github.com/ctuning/ck-tensorflow/blob/master/program/image-classification-tflite/benchmark.h#L469>
> ).
>
> The table below shows the accuracy on the ImageNet 2012 validation set
> (50,000 images) measured via TensorFlow (C++) <http://tf-cpp/README.md>.
> Pillow OpenCV universal OpenCV for ResNet OpenCV for MobileNet TensorFlow
> Matches official? No No Yes Yes No
> Additional CK package tags using-pillow using-opencv,universal
> using-opencv,for-resnet using-opencv,for-mobilenet using-tensorflow
> Supported models ResNet, MobileNet ResNet, MobileNet ResNet only MobileNet
> only ResNet only
> Supported platforms x86, arm x86 x86 x86 x86 (prebuilt TF)
> Data format rgb8 (int8) rgb8 (int8) rgbf32 (float32) rgbf32 (float32) rgbf32
> (float32)
> Data size 7.1G 7.1G 29G 29G 29G
> ResNet Top1 0.76170 0.76422 0.76456 N/A 0.76522
> ResNet Top5 0.92866 0.93074 0.93016 N/A 0.93066
> MobileNet non-quantized Top1 0.71226 0.71676 N/A 0.71676 N/A
> MobileNet non-quantized Top5 0.89834 0.90118 N/A 0.90118 N/A
> MobileNet quantized Top1 0.70348 0.70700 N/A 0.70694 N/A
> MobileNet quantized Top5 0.89376 0.89594 N/A 0.89594 N/A
>
> Considering Top1, the universal OpenCV method is slightly less accurate
> for ResNet, but slightly more accurate for MobileNet quantized than the
> official code. The TensorFlow method is most accurate for ResNet, but is
> not suitable for MobileNet. The Pillow method is least accurate, but can be
> used on Arm platforms. The difference between the universal OpenCV and the
> TensorFlow methods on ResNet is exactly 0.1% or 50 images.
>
> For more information, please see here
> <https://github.com/dividiti/inference/tree/update-accuracy/v0.5/classification_and_detection/optional_harness_ck/classification#preprocess-datasets>
> .
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/350?email_source=notifications&email_token=ABT6DFFUJ3ZIC6FI554J3QLQFQCINA5CNFSM4IMNUFP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4WT7HA#issuecomment-523059100>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABT6DFA2UZXKQUEAIDBUXIDQFQCINANCNFSM4IMNUFPQ>
> .
>


-- 
Vijay Janapa Reddi, Ph. D. | Associate Professor
33 Oxford St. | Maxwell Dworkin 147 | Cambridge, MA 02138
John A. Paulson School of Engineering and Applied Sciences | Harvard
University
Email: vj@eecs.harvard.edu | Web:
http://scholar.harvard.edu/vijay-janapa-reddi


None of the versions above are "official" but OpenCV for ResNet and OpenCV for MobileNet fully match the official code as the first row of the table suggests. 

I believe any preprocessing method here can be used for submission, as the accuracy drops within the agreed tolerances (99% for ResNet and 98% for MobileNet). 

Ah, you are right. Though our references are in openCV we can indeed use
any version. That slipped my mind. Sorry about that.

On Tue, Aug 20, 2019 at 5:50 PM Anton Lokhmotov <notifications@github.com>
wrote:

> I believe any version of preprocessing can be used for submission, as the
> accuracy drops within the agreed tolerances (99% for ResNet and 98% for
> MobileNet).
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/350?email_source=notifications&email_token=ABT6DFH7I7PPHWFQRZWFACLQFSGLPA5CNFSM4IMNUFP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4YCUVA#issuecomment-523250260>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABT6DFHZNYKX7TFLOE4U5TDQFSGLPANCNFSM4IMNUFPQ>
> .
>


-- 
Vijay Janapa Reddi, Ph. D. | Associate Professor
33 Oxford St. | Maxwell Dworkin 147 | Cambridge, MA 02138
John A. Paulson School of Engineering and Applied Sciences | Harvard
University
Email: vj@eecs.harvard.edu | Web:
http://scholar.harvard.edu/vijay-janapa-reddi


I've checked that SSD-ResNet also needs transformations (rather different [for ONNX](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/dataset.py#L236) and [for TF](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/dataset.py#L250), so a single preprocessed image takes 17MB, or 82 GB for the whole COCO 2017 validation set.