Repository: mlcommons/inference
Issue: SSD-MobileNet processes 4885 images
Root cause: Validation Gap

When analyzing the LoadGen output from running SSD-MobileNet on 5,000 images in the Offline mode, we've noticed that only 4,885 image ids appear in the results:
```
$ ipython

In [1]: import json

In [2]: with open('TestScenario.Offline.json') as f: j = json.load(f)

In [3]: len(j)
Out[3]: 21981

In [4]: len(set([ v['image_id'] for v in j ]))
Out[4]: 4885
```

The same for SingleStream.

/cc @me2x

I can confirm this happens also in the Server and MultiStream scenarios.

A couple questions:
1) Is the performance sample count 5000?
2) Does the situation improve as you increase the min_duration or min_queries in TestSettings?

The samples are chosen randomly (distributed uniformly and with replacement), so the expected number of queries we'd need before all samples are visited will be very high (I think on the order of 5000^2).

Maybe we could change the rules for 0.6 so that we don't sample uniformly and used some kind of stratisfied sampling, but I don't think we'd change it for 0.5 at this point.

@briandersn Are you saying this behaviour is expected? The accuracy mode works as (we) expected for image classification, i.e. prints the Top1 accuracy over the ImageNet validation set.

```
$ ck install package --tags=dataset,coco,2017,val
$ export DATA_DIR=`ck locate env --tags=dataset,coco,2017,val`

$ ck install package --tags=mlperf,object-detection,model,tf,ssd-mobilenet,non-quantized
$ export MODEL_DIR=`ck locate env --tags=mlperf,object-detection,model,tf,ssd-mobilenet,non-quantized`

$ export EXTRA_OPS="--accuracy --count 5000 --scenario Offline"
$ ./run_and_time.sh tf ssd-mobilenet cpu
...
STARTING RUN AT 2019-08-15 10:03:39 AM
INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=5000, data_format=None, dataset='coco-300', dataset_list=None, dataset_path='/home/anton/CK_TOOLS/dataset-coco-2017-val', inputs=['image_tensor:0'], max_batchsize=32, max_latency=[0.01], model='/home/anton/CK_TOOLS/model-tf-mlperf-ssd-mobilenet/ssd_mobilenet_v1_coco_2018_01_28.pb', output='/output/results.json', outputs=['num_detections:0', 'detection_boxes:0', 'detection_scores:0', 'detection_classes:0'], profile='ssd-mobilenet-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.Offline], threads=8, time=None)
WARNING: Logging before flag parsing goes to stderr.
I0815 10:03:42.282268 140175865046784 coco.py:96] loaded 5000 images, cache=0, took=0.9sec
W0815 10:03:43.189779 140175865046784 deprecation_wrapper.py:119] From /mlperf/python/backend_tf.py:41: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-08-15 10:03:43.191289: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-08-15 10:03:43.218427: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2808000000 Hz
2019-08-15 10:03:43.218904: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55f1655abf50 executing computations on platform Host. Devices:
2019-08-15 10:03:43.218926: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-08-15 10:03:45.778459: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.
 If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0815 10:03:46.133154 140175865046784 main.py:527] starting TestScenario.Offline
loading annotations into memory...
Done (t=0.69s)
creating index...
index created!
Loading and preparing results...
Converting ndarray to lists...
(21981, 7)
0/21981
DONE (t=0.10s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=18.24s).
Accumulating evaluation results...
DONE (t=2.90s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.235
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.355
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.258
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.018
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.171
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.534
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.212
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.267
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.267
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.022
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.196
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.610
TestScenario.Offline qps=3.26, mean=16.6867, time=49.094, acc=93.312%, mAP=0.235%, queries=160, tiles=50.0:16.9979,80.0:25.4770,90.0:26.4886,95.0:27.9844,99.0:32.4735,99.9:32.4885
ENDING RUN AT 2019-08-15 10:06:28 AM
```
Here's the generated file (ignore the 1h timestamp difference)
```
$ ls -la output/ssd-mobilenet-tf-cpu/
total 7496
drwxr-xr-x 2 anton anton    4096 Aug 15 11:06 .
drwxr-xr-x 3 anton anton    4096 Aug 15 11:02 ..
-rw-r--r-- 1 anton anton    3716 Aug 15 11:06 output.txt
-rw-r--r-- 1 root  root     1317 Aug 15 11:06 results.json
-rw-r--r-- 1 root  root  7659312 Aug 15 11:06 TestScenario.Offline.json
```
And here's the same result:
```
$ ipython
Python 3.6.9 (default, Jul  3 2019, 07:38:46) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import json

In [2]: with open('output/ssd-mobilenet-tf-cpu/TestScenario.Offline.json') as f: j = json.load(f)

In [3]: len(set([ v['image_id'] for v in j ]))
Out[3]: 4885
```

The Docker image was built from:
```
$ git log -1
commit 68e7e9aa1152dee2504f266c373c1fb9c331b64c (HEAD -> master, upstream/master, origin/master, origin/HEAD)
Author: Aaron Zhong <aaron.zhong@alibaba-inc.com>
Date:   Wed Aug 14 10:03:23 2019 +0800

    LoadGen: fix some issues of c api interface
    1. need to include QuerySamplesComplete in c api for SUT to report result
    2. need to include the whole test_settings.h for detailed struct and enum defini
```

And yes, 5,000 images get loaded, but only 4,885 processed. This affects the mAP score: 0.235 gets calculated but it should be around 0.231. (Ignore % after 0.235 - this is fixed in https://github.com/mlperf/inference/pull/342.)

On 50 images, I get 49 images processed (and [51 loaded](https://github.com/mlperf/inference/issues/324)):
```
ipython
Python 3.6.9 (default, Jul  3 2019, 07:38:46) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import json

In [2]: with open('output/ssd-mobilenet-tf-cpu/TestScenario.Offline.json') as f: j = json.load(f)

In [3]: len(set([ v['image_id'] for v in j ]))
Out[3]: 49
```

> A couple questions:
> 
>     1. Is the performance sample count 5000?
> 
>     2. Does the situation improve as you increase the min_duration or min_queries in TestSettings?

I tried to change the following parameters:
--time (from 60 to 120)
--max_latency (from 0.1 to 1)
--queries-offline (from default=24576, to 49152)
and nothing changed.

Did you mean one of these or some other parameter?

btw I also noticed that even if I increase the count (let's say at 10k), with accuracy it stops at 5k.

It is expected that performance mode will not necessarily visit every loaded sample.

The mAP score in performance mode will not be used to evaluate whether a SUT meets the accuracy targets. Only the mAP score in the accuracy mode will. Also note that accuracy mode may visit the same sample twice in some cases - the accuracy scripts fo each model should ignore duplicates.

I thought that running as follows:
```
$ export EXTRA_OPS="--accuracy --count 5000 --scenario Offline"
$ ./run_and_time.sh tf ssd-mobilenet cpu
```
should launch the accuracy mode and traverse all the samples? 

Can you attach the detail loadgen log? It logs every sample loaded and what mode it thinks it's in.

- mlperf_log_*
[mlperf_log_detail.txt](https://github.com/mlperf/inference/files/3509143/mlperf_log_detail.txt)
[mlperf_log_summary.txt](https://github.com/mlperf/inference/files/3509145/mlperf_log_summary.txt)
[mlperf_log_trace.json](https://github.com/mlperf/inference/files/3509147/mlperf_log_trace.json.txt)
[mlperf_log_accuracy.json](https://github.com/mlperf/inference/files/3509148/mlperf_log_accuracy.json.txt)
- output/ssd-mobilenet-tf-cpu/*
[TestScenario.Offline.json](https://github.com/mlperf/inference/files/3509151/TestScenario.Offline.json.txt)
[results.json](https://github.com/mlperf/inference/files/3509152/results.json.txt)
[output.txt](https://github.com/mlperf/inference/files/3509153/output.txt)

For the ONNX SSD-ResNet model, I see only 44 images being processed:
```
$ export MODEL_DIR=`ck locate env --tags=model,onnx,mlperf,ssd-resnet`
$ export DATA_DIR=`ck locate env --tags=dataset,coco,2017,val`
$ export EXTRA_OPS="--accuracy --count 50 --scenario SingleStream"
$ ./run_and_time.sh onnxruntime ssd-resnet34 cpu
...
TestScenario.SingleStream qps=0.35, mean=2.8108, time=144.507, acc=69.531%, mAP=5.249%, queries=51, tiles=50.0:3.0323,80.0:3.1321,90.0:3.1538,95.0:3.1933,99.0:3.3024,99.9:3.3315

$ ipython
Python 3.6.9 (default, Jul  3 2019, 07:38:46) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import json

In [2]: with open('output/ssd-resnet34-onnxruntime-cpu/TestScenario.SingleStream.json') as f: j = json.load(f)

In [3]: len(set([ v['image_id'] for v in j ]))
Out[3]: 44
```
How weird!

I've a recent Docker image built from:
```
$ git log -1
commit 2a4cbf75d886ed0880ee317ef90c86ea568c5497 (HEAD -> master, upstream/master, origin/master, origin/HEAD)
Merge: ffe78dc cfb3706
Author: Brian C. Anderson <brianderson@google.com>
Date:   Fri Aug 23 00:27:52 2019 -0700

    Merge pull request #349 from kstreee-furiosa/binary-search
    
    Implements binary search for MultiStream & MultiStreamFree & Server scenarios.
```
and gives the same result for SSD-MobileNet as before (i.e. processes 49 images).

I strongly suggest we label this issue as "urgent". Otherwise, we can't reliably calculate accuracy for object detection.

I don't seen anything wrong with the loadgen - though we should figure out how the loadgen is given more samples than exist. Details below.

> * mlperf_log_*
>   [mlperf_log_detail.txt](https://github.com/mlperf/inference/files/3509143/mlperf_log_detail.txt)
>   [mlperf_log_summary.txt](https://github.com/mlperf/inference/files/3509145/mlperf_log_summary.txt)
>   [mlperf_log_trace.json](https://github.com/mlperf/inference/files/3509147/mlperf_log_trace.json.txt)
>   [mlperf_log_accuracy.json](https://github.com/mlperf/inference/files/3509148/mlperf_log_accuracy.json.txt)
> * output/ssd-mobilenet-tf-cpu/*
>   [TestScenario.Offline.json](https://github.com/mlperf/inference/files/3509151/TestScenario.Offline.json.txt)
>   [results.json](https://github.com/mlperf/inference/files/3509152/results.json.txt)
>   [output.txt](https://github.com/mlperf/inference/files/3509153/output.txt)

I took a look at these files.
mlperf_log_detail.txt indicates the QSL told the loadgen there were 51 total samples.
mlperf_log_accuracy.json has exactly 51 unique responses recorded for samples [0,50].

Were there cases where mlperf_log_accuracy.json didn't have all the responses recorded? You should be able to run this file through the model's accuracy script to get a valid accuracy value.

In one of @psyhtest 's posts, he indicates TestScenario.Offline.json doesn't have all the samples. It's not in a format that I recognize. Is this a file recorded by the reference implementation in the performance run?

@briandersn Do you mean this [TestScenario.Offline.json](https://github.com/mlperf/inference/files/3509151/TestScenario.Offline.json.txt)? Yes, it was recorded by the reference implementation with params:
```
$ export EXTRA_OPS="--accuracy --count 5000 --scenario Offline"
$ ./run_and_time.sh tf ssd-mobilenet cpu
```

mlperf_log_accuracy.json is the official file to use for accuracy.
The one attached in a comment to this issue has all the samples included. Do you have a version that does not?

TestScenario.Offline.json for accuracy is not supported.

So TestScenario.Offline.json was some debug thing that I used together with ```tools/coco-analyze.py``` when we had issues with accuracy. It wasn't really meant to be used for something else but it does have the info people would want.
The official command for accuracy is:
```
python tools/accuracy-coco.py  --coco-dir /data/NN/coco --mlperf-accuracy-file output/ssd-mobilenet-tf-gpu/mlperf_log_accuracy.json
```
which reports for ```--count 500```
```
found and ignored 0 dupes
501 seen, 501 results
```
Basically, looks good (ok, I should fix the +1 error).

```tools/accuracy-coco.py``` does have a ```--output-file``` option that generates the same format as ```TestScenario.Offline.json```. The funny thing is that file has the same issue as Anton reports.

But I now understand why: that format is as a flat array of all results. I has no empty {} in case there are no results for a query and sure enough in mlperf_accuracy_log.json I find entries like:
```
{ "seq_id" : 25, "qsl_idx" : 317, "data" : "" }
```
aka the image[317] does not have results. The full list (added a print into accuracy-coco):
```
29 has no results
317 has no results
284 has no results
273 has no results
393 has no results
138 has no results
373 has no results
235 has no results
449 has no results
278 has no results
25 has no results
403 has no results
```
I'll take a look at those images to make sure but I think there is no issue here.

Btw if I say 'that format' I mean what is used in TestScenario.Offline.json. It is the format used by pycoco.

@psyhtest  , does that makes sense ?

@guschmue I would not be that concerned about having no detections for some images (some images may have no detectable objects, while some models may fail to detect any objects) if it were not for the higher than expected accuracy figures. For example, for the non-quantized SSD-MobileNet model we have previously measured 23.11% via CK, while we are measuring 23.46% via the official app.

I feel uneasy about discarding images having no detections when calculating the accuracy. I believe @me2x has a an idea how to fix that.

So, the issue is that when creating the detection list to use with cocoEval library, only the images that have a detection are inserted in the list of images to evaluate. and the reason is that the empty images ids are not inserted into the ones to evaluate.

This can be easily find if you add the following two lines at the end of accuracy-coco.py
```
        print("processed {} images".format(len(results) ))
        print("found {} different images".format(len(image_ids) ))
```

the problem is that when creating the cocoEval object, the ids of the images to evaluate are the one in the image_ids list:

```
cocoDt = cocoGt.loadRes(detections)
    cocoEval = COCOeval(cocoGt, cocoDt, iouType='bbox')
    cocoEval.params.imgIds = list(image_ids)
    cocoEval.evaluate()
    cocoEval.accumulate()
    cocoEval.summarize()
```

As for the solution, my idea is to somehow add a "dummy detection" whenever no detection are done by the model, in order to save the image id (that is lost in the entries like ```{ "seq_id" : 25, "qsl_idx" : 317, "data" : "" }``` and be able to insert it in the image_ids list. However, I don't have anything working at the moment (and I don't know if the solution could be acceptable).

Is there a way to find the image id (as per id in the coco dataset) starting from the qsl_idx? I think that would be a cleaner solution.

If COCOeval is going to look at empty entries ... it its easy enough to add those. Let me check pycoco if it would handle those.

Ok, I have a fix for tools/accuracy-coco.py to deal with no results. Still need to deal with the mAP for the reference app. Because I can not reflect the empty images in the json generated by the reference app I was thinking to remove that so people are using the loadgen log.
The diff in mAP is:
```
with the fix: 
mAP=23.173%
found 5000 results
found 5000 images
found 115 images with no results
ignored 0 dupes
```
```
without the fix:
mAP=23.461%
found 5000 results
found 4885 images
found 115 images with no results
ignored 0 dupes
```

Ok, sent a PR, mAP is now slightly lower at 23.173% for all images.

Does anybody know how does this change affect SSD-R34? 

not yet.

for tf/resnet34:
```
with fix mAP=0.021%
without fix mAP=0.021%

found 4952 results
found 4952 images
found 0 images with no results
ignored 0 dupes
```
I guess ```--use-inv-map``` already takes care of this.

@guschmue  is there a bug? I see the following 2 code snippts. Is the tf/resnet accuracy almost 0?

--> this looks ok.
with the fix: 
mAP=23.173%
found 5000 results
found 5000 images
found 115 images with no results
ignored 0 dupes


--> this does not look OK
with fix mAP=0.021%
without fix mAP=0.021%

found 4952 results
found 4952 images
found 0 images with no results
ignored 0 dupes


There is something else going on here. I see the wrong mAP on tensorflow/resnet34.
It was marked to use the mapping, not sure it needs it but when I run it without the mAP is 3.x, still bad. And the reference benchmark itself has the same (aka the issue is not in the accuracy-coco script).
This model was updated at one point and I'm not sure if I ever tested it after that. Trying to find some old logs.

For tensorflow/ssd-resnet34 I can get to the desired accuracy with by setting use_label_map to False.
Basically this line seems wrong:
https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/main.py#L60
But I'm still looking at it.


@guschmue  all fixed. Please re-open if needed.