Repository: mlcommons/inference
Issue: MobileNet fails via TFLite
Root cause: Validation Gap

If I install the quantized MobileNet model via CK and set up symbolic links (file names are hardcoded in the official app):
```
$ ck install package --tags=mlperf,image-classification,model,tf,mobilenet,quantized
$ ln -s `ck locate env --tags=mobilenet,quantized`/mobilenet_v1_1.0_224{_quant,}_frozen.pb
$ ln -s `ck locate env --tags=mobilenet,quantized`/mobilenet_v1_1.0_224{_quant,}.tflite
```
and then other environment variables as follows:
```
$ export MODEL_DIR=`ck locate env --tags=model,tf,mobilenet,quantized`
$ export DATA_DIR=`ck locate env --tags=dataset,imagenet,val`
$ export EXTRA_OPS="--accuracy --count 50000 --scenario Offline"
```
I can run the TF model:
```
$ ./run_and_time.sh tf mobilenet cpu
...
TestScenario.Offline qps=128.83, mean=7.5497, time=12.419, acc=70.676, queries=1600, tiles=50.0:7.8294,80.0:11.1442,90.0:11.7616,95.0:12.1174,99.0:12.9126,99.9:13.1641
```
but not the TFLite model:
```
$ ./run_and_time.sh tflite mobilenet cpu
...
INFO:main:Namespace(accuracy=True, backend='tflite', cache=0, count=50000, data_format=None, dataset='imagenet_mobilenet', dataset_list=None, dataset_path='/datasets/dataset-imagenet-ilsvrc2012-val', inpu
ts=['input:0'], max_batchsize=32, max_latency=[0.01], model='/home/anton/CK_TOOLS/model-tf-mlperf-mobilenet-quantized-downloaded/mobilenet_v1_1.0_224.tflite', output='/output/results.json', outputs=['Mobi
lenetV1/Predictions/Reshape_1:0'], profile='mobilenet-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.Offline], threads=8, time=None)
WARNING: Logging before flag parsing goes to stderr.
I0814 10:34:54.607473 140213712041728 imagenet.py:82] loaded 50000 images, cache=0, took=2.1sec
INFO: Initialized TensorFlow Lite runtime.
Traceback (most recent call last):
  File "python/main.py", line 553, in <module>
    main()
  File "python/main.py", line 457, in main
    _ = backend.predict({backend.inputs[0]: img})
  File "/mlperf/python/backend_tflite.py", line 46, in predict
    self.sess.set_tensor(v, feed[k])
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/lite/python/interpreter.py", line 197, in set_tensor
    self._interpreter.SetTensor(tensor_index, value)
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/lite/python/interpreter_wrapper/tensorflow_wrap_interpreter_wrapper.py", line 136, in SetTensor
    return _tensorflow_wrap_interpreter_wrapper.InterpreterWrapper_SetTensor(self, i, value)
ValueError: Cannot set tensor: Got tensor of type FLOAT32 but expected type UINT8 for input 88, name: input
```

The Docker image is a fresh one built from:
```
$ git log -1
commit 68e7e9aa1152dee2504f266c373c1fb9c331b64c (HEAD -> master, upstream/master, origin/master, origin/HEAD)
Author: Aaron Zhong <aaron.zhong@alibaba-inc.com>
Date:   Wed Aug 14 10:03:23 2019 +0800

    LoadGen: fix some issues of c api interface
    1. need to include QuerySamplesComplete in c api for SUT to report result
    2. need to include the whole test_settings.h for detailed struct and enum definition of TestSettings
```

By the way, @guschmue, I have a [bunch of instructions](https://github.com/dividiti/inference/commit/9e6a213e4cf1372773f927066ad3dea05d09ea25) to run other models.

Actually, running the non-quantized MobileNet:
```
$ ck install package --tags=mlperf,image-classification,model,tf,mobilenet,non-quantized
$ export MODEL_DIR=`ck locate env --tags=model,tf,mobilenet,non-quantized`
$ export DATA_DIR=`ck locate env --tags=dataset,imagenet,val`
$ export EXTRA_OPS="--accuracy --count 50000 --scenario Offline"
```
also fails via TFLite:
```
$ ./run_and_time.sh tflite mobilenet cpu
...
STARTING RUN AT 2019-08-14 10:47:45 AM
INFO:main:Namespace(accuracy=True, backend='tflite', cache=0, count=50000, data_format=None, dataset='imagenet_mobilenet', dataset_list=None, dataset_path='/datasets/dataset-imagenet-ilsvrc2012-val', inputs=['input:0'], max_batchsize=32, max_latency=[0.01], model='/home/anton/CK_TOOLS/model-tf-mlperf-mobilenet-downloaded-from-zenodo/mobilenet_v1_1.0_224.tflite', output='/output/results.json', outputs=['MobilenetV1/Predictions/Reshape_1:0'], profile='mobilenet-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.Offline], threads=8, time=None)
WARNING: Logging before flag parsing goes to stderr.
I0814 10:47:49.392686 139794081322752 imagenet.py:82] loaded 50000 images, cache=0, took=2.1sec
INFO: Initialized TensorFlow Lite runtime.
I0814 10:47:49.533662 139794081322752 main.py:527] starting TestScenario.Offline
E0814 10:47:50.657735 139793266636544 main.py:302] thread: failed on contentid=['/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00033260.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00009373.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00040888.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00016958.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00016445.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00001267.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00008941.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00020008.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00001314.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00033661.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00031714.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00018221.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00018598.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00044171.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00044701.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00028877.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00004247.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00035079.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00012588.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00023268.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00049148.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00045608.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00005835.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00026851.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00032662.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00023285.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00046628.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00005151.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00006500.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00012406.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00044058.JPEG', '/datasets/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00015496.JPEG'], Cannot set tensor: Dimension mismatch. Got 32 but expected 1 for dimension 0 of input 88.
```
although works fine via TF:
```
$ ./run_and_time.sh tf mobilenet cpu
...
TestScenario.Offline qps=352.92, mean=3.2609, time=4.534, acc=71.676, queries=1600, tiles=50.0:2.9725,80.0:4.0271,90.0:4.0907,95.0:4.3719,99.0:4.4811,99.9:4.5173
```

'Cannot set tensor: Dimension mismatch. Got 32 but expected 1 for dimension 0 of input 88'
sounds like either the model or tflite doesn't support batchsize > 1. For offline we'd use 32 by default. See if ```--max-batchsize 1``` will make it happy. If it does we can add a profile for tflite here:
https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/main.py#L98

For the root cause: I don't know toco that well but is it possible that it uses a fix shape for the input ?

This message for TFLite MobileNet quantized is a bit clearer:
```
ValueError: Cannot set tensor: Got tensor of type FLOAT32 but expected type UINT8 for input 88, name: input
```

Hm, tflite was somewhat a afterthought in the reference app and I did not implement the changes needed for quantitized models. If you already have a fix send the PR :).
Else I'll take a look how to implement it.

To be honest, this is not terribly important or urgent, as we can run all the TFLite models [via CK](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection/optional_harness_ck/classification/tflite).

But I would like us (the WG) to get a better understanding of what is working today, across all the models and backends. For example, I've tried running MobileNet with TFLite, I've failed. Someone else may try the same, and discover it's a known issue. It's all part of building collective knowledge (lowercase intended :))

The answer is to modify line 193 in dataset.py. This converts the data type to float32 and it should be uint8. The second line below should be changed for TFLite, so we need code to check for backend and then set dtype accordingly.

```    
img = center_crop(img, output_height, output_width)
img = np.asarray(img, dtype='float32')
img /= 255.0
```

Actually - we shouldn't tie this to the backend, but rather tie it to the data type for the input of the network. 

Let me modify what I wrote... Not only should the dtype by uint8, but also the preprocessing should correspondingly be updated for INT8.

Anton,

I agree we should get a good grip on what is working today. Can we briefly
discuss this on the call on Wednesday?

On Fri, Aug 16, 2019 at 6:55 AM Anton Lokhmotov <notifications@github.com>
wrote:

> To be honest, this is not terribly important or urgent, as we can run all
> the TFLite models via CK
> <https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection/optional_harness_ck/classification/tflite>
> .
>
> But I would like us (the WG) to get a better understanding of what is
> working today, across all the models and backends. For example, I've tried
> running MobileNet with TFLite, I've failed. Someone else may try the same,
> and discover it's a known issue. It's all part of building collective
> knowledge (lowercase intended :))
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/334?email_source=notifications&email_token=ABT6DFDMKKFND5SLIMTTQMDQE2WWVA5CNFSM4ILTQDE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4OVV5I#issuecomment-522017525>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABT6DFHLPYB4L644DEHHQCLQE2WWVANCNFSM4ILTQDEQ>
> .
>


-- 
Vijay Janapa Reddi, Ph. D. | Associate Professor
33 Oxford St. | Maxwell Dworkin 147 | Cambridge, MA 02138
John A. Paulson School of Engineering and Applied Sciences | Harvard
University
Email: vj@eecs.harvard.edu | Web:
http://scholar.harvard.edu/vijay-janapa-reddi


@jveejay Unfortunately, I'm away on Wednesday, so won't be joining the call.

Ok, we will defer this to next week then.

On Tue, Aug 20, 2019 at 7:36 AM Anton Lokhmotov <notifications@github.com>
wrote:

> @jveejay <https://github.com/jveejay> Unfortunately, I'm away on
> Wednesday, so won't be joining the call.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/334?email_source=notifications&email_token=ABT6DFANJPZZT4L4U6MGOETQFP6PTA5CNFSM4ILTQDE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4WQMEQ#issuecomment-523044370>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABT6DFBKVLVSBTCOBXR5RMTQFP6PTANCNFSM4ILTQDEQ>
> .
>


-- 
Vijay Janapa Reddi, Ph. D. | Associate Professor
33 Oxford St. | Maxwell Dworkin 147 | Cambridge, MA 02138
John A. Paulson School of Engineering and Applied Sciences | Harvard
University
Email: vj@eecs.harvard.edu | Web:
http://scholar.harvard.edu/vijay-janapa-reddi