Repository: mlcommons/inference
Issue: SSD-MobileNet loads (count+1) images
Root cause: Algorithmic Error

I've noticed with SSD-MobileNet that when any number of images is via `--count <count>` , then `count+1` images get loaded e.g.:
```
$ ck install package --tags=mlperf,object-detection,model,tf,ssd-mobilenet,non-quantized
$ ln -s `ck locate env --tags=model,tf,ssd-mobilenet,non-quantized`/{frozen_inference_graph.pb,ssd_mobilenet_v1_coco_2018_01_28.pb}
$ export MODEL_DIR=`ck locate env --tags=model,tf,ssd-mobilenet,non-quantized`
$ export DATA_DIR=`ck locate env --tags=dataset,coco,2017,val`
$ export EXTRA_OPS="--accuracy --count 5000 --scenario Offline"
$ ./run_and_time.sh tf ssd-mobilenet cpu
...
STARTING RUN AT 2019-08-09 01:26:54 PM                                                                                                                                                                      
INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=1, data_format=None, dataset='coco-300', dataset_list=None, dataset_path='/home/anton/CK_TOOLS/dataset-coco-2017-val', inputs=['imag
e_tensor:0'], max_batchsize=32, max_latency=[0.01], model='/home/anton/CK_TOOLS/model-tf-mlperf-ssd-mobilenet/ssd_mobilenet_v1_coco_2018_01_28.pb', output='/output/results.json', outputs=['num_detections:
0', 'detection_boxes:0', 'detection_scores:0', 'detection_classes:0'], profile='ssd-mobilenet-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.SingleStream], threads=8, time=None)    
WARNING: Logging before flag parsing goes to stderr.                 
I0809 13:26:56.778456 139672093333248 coco.py:96] loaded 2 images, cache=0, took=0.6sec
...
```
Only when `count` is exactly 5000, then `loaded 5000 images` gets printed.

Why is that? May this affect the accuracy calculation?

/cc @guschmue @briandersn 

For example, when `count` is 1:
```
$ export EXTRA_OPS="--accuracy --count 1 --scenario SingleStream"
$ ./run_and_time.sh tf ssd-mobilenet cpu
...
INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=1, data_format=None, dataset='coco-300', dataset_list=None, dataset_path='/home/anton/CK_TOOLS/dataset-coco-2017-val', inputs=['ima$
e_tensor:0'], max_batchsize=32, max_latency=[0.01], model='/home/anton/CK_TOOLS/model-tf-mlperf-ssd-mobilenet/ssd_mobilenet_v1_coco_2018_01_28.pb', output='/output/results.json', outputs=['num_detections$
0', 'detection_boxes:0', 'detection_scores:0', 'detection_classes:0'], profile='ssd-mobilenet-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.SingleSt$
eam], threads=8, time=None)    
WARNING: Logging before flag parsing goes to stderr.
I0809 13:41:54.014330 140040341505792 coco.py:96] loaded 2 images, cache=0, took=0.6sec
...
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.133
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.211
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.147
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.004
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.200
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.700
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.133
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.133
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.133
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.004
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.200
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.700
TestScenario.SingleStream qps=2.20, mean=0.0300, time=0.91, acc=100.00, mAP=0.13, queries=2, tiles=50.0:0.0300,80.0:0.0305,90.0:0.0307,95.0:0.0307,99.0:0.0308,99.9:0.0308
ENDING RUN AT 2019-08-09 01:41:58 PM
```
two images get processed with `image_id`s 37777 and 397133 (but because of #323 `image_loc` is `/home/anton/CK_TOOLS/dataset-coco-2017-val/val2017/000000037777.jpg` for all the detected bounding boxes).

It's similar with the `Offline` scenario:
```
$ export EXTRA_OPS="--accuracy --count 1 --scenario Offline"
$ ./run_and_time.sh tf ssd-mobilenet cpu
...
INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=1, data_format=None, dataset='coco-300', dataset_list=None, dataset_path='/home/anton/CK_TOOLS/dataset-coco-2017-val', inputs=['image_tensor:0'], max_batchsize=32, max_latency=[0.01], model='/home/anton/CK_TOOLS/model-tf-mlperf-ssd-mobilenet/ssd_mobilenet_v1_coco_2018_01_28.pb', output='/output/results.json', outputs=['num_detections:0', 'detection_boxes:0', 'detection_scores:0', 'detection_classes:0'], profile='ssd-mobilenet-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.Offline], threads=8, time=None)
WARNING: Logging before flag parsing goes to stderr.
I0809 13:55:53.923795 139724925429504 coco.py:96] loaded 2 images, cache=0, took=0.6sec
...
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.133
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.211
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.147
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.004
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.200
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.700
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.133
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.133
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.133
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.004
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.200
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.700
TestScenario.Offline qps=1.13, mean=0.0541, time=0.88, acc=100.00, mAP=0.13, queries=1, tiles=50.0:0.0541,80.0:0.0541,90.0:0.0541,95.0:0.0541,99.0:0.0541,99.9:0.0541
ENDING RUN AT 2019-08-09 01:55:58 PM
```
but the number of queries changes from 2 to 1 (although 2 images still get loaded).

The `AP` decreases a bit and `AR` increases a bit for the `MultiStream` scenario:
```
$ export EXTRA_OPS= "--accuracy --count 1 --scenario MultiStream"
$ ./run_and_time.sh tf ssd-mobilenet cpu
...
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.132 < 0.133
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.211
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.147
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.004
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.196 < 0.200
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.700
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.133
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.137 > 0.133
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.137 > 0.133
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.004
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.200
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.700
TestScenario.MultiStream qps=0.92, mean=0.1807, time=1.09, acc=100.00, mAP=0.13, queries=1, tiles=50.0:0.1807,80.0:0.1807,90.0:0.1807,95.0:0.1807,99.0:0.1807,99.9:0.1807
```
(1 query like for `Offline`.)

Another observation for `MultiStream` is that the results seem to be repeated twice:
```
$ grep -c 397133 TestScenario.*.json -R
TestScenario.MultiStream.json:10
TestScenario.Offline.json:5
TestScenario.Server.json:5

$ grep -c 37777 TestScenario.*.json -R
TestScenario.MultiStream.json:42
TestScenario.Offline.json:21
TestScenario.Server.json:21
```

Apologies if what I'm asking is completely obvious!

But here's another question. The files seemed to be picked up in random order from disk:
```
$ ls -la preprocessed/coco-300/NHWC/val2017/* | grep -n 37777
346:-rw-r--r-- 1 root root 270128 Aug  9 13:14 preprocessed/coco-300/NHWC/val2017/000000037777.jpg.npy

$ ls -la preprocessed/coco-300/NHWC/val2017/* | grep -n 397133
3430:-rw-r--r-- 1 root root 270128 Aug  9 13:14 preprocessed/coco-300/NHWC/val2017/000000397133.jpg.npy
```
We can't guarantee that the same two files will be picked on a different machine.

But if we fix the order (e.g. by sorting the list of file names in `DATA_DIR`), we should make partial accuracy results (e.g. on 50 images) meaningful.

Hi Anton, can you attach the loadgen's detail log?

Certainly. From which command(s)? 

Here's a log on 50 images:
```
$ ck install package --tags=dataset,coco,2017,val
$ export DATA_DIR=`ck locate env --tags=dataset,coco,2017,val`

$ ck install package --tags=mlperf,object-detection,model,tf,ssd-mobilenet,non-quantized
$ export MODEL_DIR=`ck locate env --tags=mlperf,object-detection,model,tf,ssd-mobilenet,non-quantized`

$ export EXTRA_OPS="--accuracy --count 50 --scenario Offline"
$ ./run_and_time.sh tf ssd-mobilenet cpu
...
STARTING RUN AT 2019-08-15 10:32:02 AM                                         
INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=50, data_format=None, dataset='coco-300', dataset_list=None, dataset_path='/home/anton/CK_TOOLS/dataset-coco-2017-val', inputs=['ima
ge_tensor:0'], max_batchsize=32, max_latency=[0.01], model='/home/anton/CK_TOOLS/model-tf-mlperf-ssd-mobilenet/ssd_mobilenet_v1_coco_2018_01_28.pb', output='/output/results.json', outputs=['num_detections
:0', 'detection_boxes:0', 'detection_scores:0', 'detection_classes:0'], profile='ssd-mobilenet-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.Offline]
, threads=8, time=None)                      
WARNING: Logging before flag parsing goes to stderr.         
I0815 10:32:04.532219 140132828227328 coco.py:96] loaded 51 images, cache=0, took=0.6sec
W0815 10:32:05.410545 140132828227328 deprecation_wrapper.py:119] From /mlperf/python/backend_tf.py:41: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
                   
2019-08-15 10:32:05.411426: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-08-15 10:32:05.438432: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2808000000 Hz
2019-08-15 10:32:05.439010: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5607427a6db0 executing computations on platform Host. Devices:
2019-08-15 10:32:05.439030: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-08-15 10:32:07.890320: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.
 If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not vi
a TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
I0815 10:32:08.229867 140132828227328 main.py:527] starting TestScenario.Offline                                                                                                                            
loading annotations into memory...
Done (t=0.61s)
creating index...
index created!
Loading and preparing results...
Converting ndarray to lists...
(270, 7)
0/270
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.18s).
Accumulating evaluation results...
DONE (t=0.26s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.258
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.409
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.273
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.023
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.228
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.613
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.226
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.271
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.273
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.028
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.231
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.648
TestScenario.Offline qps=0.84, mean=1.1605, time=2.394, acc=92.222%, mAP=0.258%, queries=2, tiles=50.0:1.1605,80.0:1.1930,90.0:1.2038,95.0:1.2092,99.0:1.2135,99.9:1.2145
ENDING RUN AT 2019-08-15 10:32:10 AM
```
As can be seen from this log, 51 images get loaded (but only [49 processed](https://github.com/mlperf/inference/issues/329#issuecomment-521597153)).

Can you attach the summary and detail logs when you get 51 images loaded (or any other case where the number of samples loaded is more than you'd expect)?

Please see [here](https://github.com/mlperf/inference/issues/329#issuecomment-521964533) files from running:
```
$ ck install package --tags=dataset,coco,2017,val
$ export DATA_DIR=`ck locate env --tags=dataset,coco,2017,val`

$ ck install package --tags=mlperf,object-detection,model,tf,ssd-mobilenet,non-quantized
$ export MODEL_DIR=`ck locate env --tags=mlperf,object-detection,model,tf,ssd-mobilenet,non-quantized`

$ export EXTRA_OPS="--accuracy --count 50 --scenario Offline"
$ ./run_and_time.sh tf ssd-mobilenet cpu

$ grep loaded output/ssd-mobilenet-tf-cpu/output.txt
I0816 10:22:05.610579 140553496745728 coco.py:96] loaded 51 images, cache=0, took=0.6sec

$ grep count output/ssd-mobilenet-tf-cpu/output.txt
output.txt:INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=50, data_format=None, dataset='coco-300', dataset_list=None, dataset_path='/home/anton/CK_TOOLS/dataset-coco-2017-val', inputs=['image_tensor:0'], max_batchsize=32, max_latency=[0.01], model='/home/anton/CK_TOOLS/model-tf-mlperf-ssd-mobilenet/ssd_mobilenet_v1_coco_2018_01_28.pb', output='/output/results.json', outputs=['num_detections:0', 'detection_boxes:0', 'detection_scores:0', 'detection_classes:0'], profile='ssd-mobilenet-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.Offline], threads=8, time=None)

$ grep LoadSamples mlperf_log_*
mlperf_log_trace.json.txt:{ "name": "LoadSamples", "ph": "X", "pid": 10, "tid": 140553496745728, "ts": 32451, "dur": 43494761, "args": { "count" : 51 }},
```

From the detail log in #324, the loadgen QSL is reporting a size of 51. Somehow, the number passed into ConstructQSL is or becomes 51. It's not clear from a quick manual look at the code where that might happen.

"""
"pid": 10, "tid": 140553496745728, "ts": 17455ns : Date + time of test: 2019-08-16T10:22:09Z
"pid": 10, "tid": 140553496745728, "ts": 17455ns : System Under Test (SUT) name: PySUT
"pid": 10, "tid": 140553496745728, "ts": 17455ns : Query Sample Library (QSL) name: PyQSL
"pid": 10, "tid": 140553496745728, "ts": 17455ns : QSL total size: 51
"pid": 10, "tid": 140553496745728, "ts": 17455ns : QSL performance size: 51
"""


@briandersn OK, but can you at least reproduce (via Docker)? 

this is most likely my bad, must be checking ```--count``` wrong. Going to fix it.