Repository: mlcommons/inference
Issue: Accurancy(fp32) of SSD-ResNet can not reach to mAP=0.2
Root cause: Validation Gap

# Question
I have test **SSD-ResNet34** with Offline mode,   while got the **mAP=19.618%** which can not reach to 99%*FP32(mAP=0.2).
When testing, print some ERROR logs, COCO-2017 dataset seams 48  missmatched images

```
....
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
```


## ssd-ResNet34 result (datacenter + Offline, 1 x nvidia GPU)
**ssd-ResNet34** with **PyTorch** backend, get the result:
```
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
ERROR:coco:image_idx missmatch, lg=3631 / result=396
loading annotations into memory...
Done (t=3.23s)
creating index...
index created!
Loading and preparing results...
Converting ndarray to lists...
(800080, 7)
0/800080
DONE (t=32.43s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=479.35s).
Accumulating evaluation results...
DONE (t=3884.66s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.196
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.374
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.180
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.113
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.252
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.233
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.202
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.327
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.341
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.169
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.398
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.420
TestScenario.Offline qps=0.03, mean=6.9694, time=4518.978, acc=35.167%, mAP=19.618%, queries=155, tiles=50.0:6.9320,80.0:7.3238,90.0:7.5072,95.0:7.8464,99.0:8.6547,99.9:9.0154
```

## Test code
repo: https://github.com/mlcommons/inference/tree/r0.7
Git commit id: 5ba3acfd3932c18d91235eddbc0910b7818f4dd5
Dataset: coco2017  (original, not resized to 1200x1200)

## S/W
* torch==1.7.0
* torchvision==0.8.1
* pycocotools==2.0.2
* cuda==10.2