Repository: mlcommons/inference
Issue: SSD-MobileNet outputs the same image_loc
Root cause: Algorithmic Error

I've run full evaluation on the COCO 2017 validation set using SSD-MobileNet as follows:
```
$ ck install package --tags=mlperf,object-detection,model,tf,ssd-mobilenet,non-quantized
$ ln -s `ck locate env --tags=model,tf,ssd-mobilenet,non-quantized`/{frozen_inference_graph.pb,ssd_mobilenet_v1_coco_2018_01_28.pb}
$ export MODEL_DIR=`ck locate env --tags=model,tf,ssd-mobilenet,non-quantized`
$ export DATA_DIR=`ck locate env --tags=dataset,coco,2017,val`
$ export EXTRA_OPS="--accuracy --count 5000 --scenario Offline"
$ ./run_and_time.sh tf ssd-mobilenet cpu
```
When I examine `output/ssd-mobilenet-tf-cpu/TestScenario.Offline.json`, I see that all the 21981 bounding boxes detected for the 5000 images have the same `image_loc`:
```
[
    {
        "bbox": [
            147.42318153381348,
            142.23948097229004,
            315.95194816589355,
            275.6081118583679
        ],
        "category_id": 61,
        "image_id": 255824,
        "image_loc": "/home/anton/CK_TOOLS/dataset-coco-2017-val/val2017/000000003661.jpg",
        "score": 0.9780274629592896
    },
    {
        "bbox": [
            292.1623992919922,
            0.14655780792236328,
            282.27500915527344,
            197.58756136894226
        ],
        "category_id": 47,
        "image_id": 255824,
        "image_loc": "/home/anton/CK_TOOLS/dataset-coco-2017-val/val2017/000000003661.jpg",
        "score": 0.7211153507232666
    },
    {
        "bbox": [
            348.78807067871094,
            59.856241285800934,
            265.01808166503906,
            281.6300727725029
        ],
        "category_id": 54,
        "image_id": 417608,
        "image_loc": "/home/anton/CK_TOOLS/dataset-coco-2017-val/val2017/000000003661.jpg",
        "score": 0.9599727392196655
    },
...
```
In a similarly obtained `SingleStream` moded, `image_loc` is something else: `/home/anton/CK_TOOLS/dataset-coco-2017-val/val2017/000000186980.jpg`.

@jveejay  @psyhtest What is requested here from Nvidia?

for some reason I feel guilty of something :)
I have a hunch that I grab the wrong url when dealing with batchsize != 1. Will take a look.

@DilipSequeira Nothing I can think of :)

Beats me, you must've popped into my head and I thought oh fun I should have @DilipSequeira look at this. LOL. Just kidding, not sure why I did that Dilip. Sorry about that. 

Hello, it's a silly bug due to using the index of the previous loop in line 200 of coco.py:
            "image_loc": ds.get_item_loc(image_ids[idx]), 

I fixed it by changing the get_item_loc function as follows:
       def get_item_loc(self, nr):
            src = os.path.join(self.data_path, nr)
            return src

and changing the wrong line into:
       "image_loc": ds.get_item_loc("{:012d}.jpg".format(int(detection[0]))),

In this way i can use the detection[0] value that contains the image id, and i don't need to insert another loop variable.

Can you please issue a PR for this?

On Thu, Aug 15, 2019 at 10:24 AM Emanuele Vitali <notifications@github.com>
wrote:

> Hello, it's a silly bug due to using the index of the previous loop in
> line 200 of coco.py:
> "image_loc": ds.get_item_loc(image_ids[idx]),
>
> I fixed it by changing the get_item_loc function as follows:
> def get_item_loc(self, nr):
> src = os.path.join(self.data_path, nr)
> return src
>
> and changing the wrong line into:
> "image_loc": ds.get_item_loc("{:012d}.jpg".format(int(detection[0]))),
>
> In this way i can use the detection[0] value that contains the image id,
> and i don't need to insert another loop variable.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/323?email_source=notifications&email_token=ABT6DFAMEJAQXRJZBFLOAJLQEVRJ5A5CNFSM4IKT6X52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4L6L5Y#issuecomment-521659895>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABT6DFAQNMWKCV3R2NLXTK3QEVRJ5ANCNFSM4IKT6X5Q>
> .
>


-- 
Vijay Janapa Reddi, Ph. D. | Associate Professor
33 Oxford St. | Maxwell Dworkin 147 | Cambridge, MA 02138
John A. Paulson School of Engineering and Applied Sciences | Harvard
University
Email: vj@eecs.harvard.edu | Web:
http://scholar.harvard.edu/vijay-janapa-reddi


Not sure, get_item_loc looks like this:

def get_item_loc(self, nr):
        src = os.path.join(self.data_path, self.image_list[nr])
        return src

It takes the image index from the coco dataset, not a string.

I belief nr is wrong in some cases.

done. https://github.com/mlperf/inference/pull/346



> It takes the image index from the coco dataset, not a string.

It is explained in the pull request, i had also to modify the get_item_loc function.