Repository: mlcommons/inference
Issue: mobilenet FP32 accuracy for TF
Root cause: Configuration Error

Readme says 71.68, I got 71.66.

$ ./run_local.sh tf mobilenet gpu --accuracy

TestScenario.SingleStream qps=22011.51, mean=0.0023, time=2.27, acc=**71.66**, queries=50000, tiles=50.0:0.0019,80.0:0.0023,90.0:0.0031,95.0:0.0059,99.0:0.0060,99.9:0.0062

$ ./run_local.sh tf mobilenet cpu --accuracy
TestScenario.SingleStream qps=6187.80, mean=0.0078, time=8.08, acc=**71.66**, queries=50000, tiles=50.0:0.0078,80.0:0.0082,90.0:0.0084,95.0:0.0086,99.0:0.0090,99.9:0.0096

same issue, I got 71.23

./run_local.sh tf mobilenet gpu --accuracy
TestScenario.SingleStream qps=19803.51, mean=0.0023, time=2.52, acc=71.23, queries=50000, tiles=50.0:0.0023,80.0:0.0024,90.0:0.0024,95.0:0.0024,99.0:0.0026,99.9:0.0031

./run_local.sh tf mobilenet cpu --accuracy
TestScenario.SingleStream qps=4451.07, mean=0.0109, time=11.23, acc=71.23, queries=50000, tiles=50.0:0.0108,80.0:0.0112,90.0:0.0114,95.0:0.0116,99.0:0.0123,99.9:0.0233

@christ1ne @zhuochenKIDD As I assume you are using the latest code with OpenCV preprocessing, could you please provide more details about your OpenCV version and system?

opencv-python: 4.1.0.25
os: Ubuntu 16.04.6

The numbers in the readme I got with opencv-python-headless 4.1.0.25.
I got the same number with tf/gpu, tf/cpu and onnxruntime/cpu.
Only thing special in my case is that all runtimes use avx2 which might result in a slight difference.
I could try to run this on a different machine.

personally I don't think it is a big issue as long as we have the min. accuracy threasholds clearly specified.

Here is the system info @psyhtest 

I0724 08:56:14.753673 140647163713344 main.py:527] starting TestScenario.SingleStream
TestScenario.SingleStream qps=16028.21, mean=0.0029, time=3.12, acc=71.66, queries=50000, tiles=50.0:0.0020,80.0:0.0057,90.0:0.0058,95.0:0.0058,99.0:0.0059,99.9:0.0061
~~~~
$ python
Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print(cv2.__version__)
4.1.0
>>> 
~~~~
$ lsb_release -a 
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.2 LTS
Release:	18.04
Codename:	bionic

Using a clean docker image (https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/Dockerfile.gpu) I got:
```
TestScenario.SingleStream qps=15014.59, mean=0.0033, time=3.33, acc=71.68, queries=50000, tiles=50.0:0.0034,80.0:0.0038,90.0:0.0040,95.0:0.0041,99.0:0.0042,99.9:0.0063
```


@zhuochenKIDD Your number 71.23% is the same we had before the switch to OpenCV: https://github.com/mlperf/inference/pull/243#issuecomment-508300383 Are you sure you are using the latest version of the code?

@christ1ne uses Ubuntu 18.04, while @guschmue uses (a Docker image based on) Ubuntu 16.04.

@zhuochenKIDD, maybe nuke the preprocessed folder ... the preprocessed images are cached there and if you ever ran this with the older version ...

@guschmue you are right. I removed the preprocessed folder and I got 71.68 exactly. Can we add to the code to remove the folder whenever you need to have an accuracy run? 

pretty slow if you have multiple runs.
And now that the pre-processing is hopefully done the cache saves a lot of time.
I could maybe add a ```--cache flag``` that official runs would set to 0.


@guschmue, @psyhtest  I got 71.68 when preprocessed folder removed, thanks for your help.

Great, so this issue can be resolved then!

closed as not a bug.