Repository: mlcommons/inference
Issue: v0.5/v0.7 SSD-ResNet34 PyTorch Accuracy
Root cause: Algorithmic Error

I'm trying to validate the reference `fp32` accuracy of the SSD-ResNet34 model using the PyTorch backend.
The quoted accuracy is `mAP 0.20`.
However with PyTorch (v1.5.1) I'm seeing:
```
python3 python/main.py --backend=pytorch-native --dataset-path=/datasets/mlperf-v0.5/dataset-coco-2017-val-1200/ --profile=ssd-resnet34-pytorch --max-batchsize=1 --model=/datasets/mlperf-v0.5/models/resnet34-ssd1200.pytorch --model-name=ssd-resnet34 --scenario=SingleStream --threads=2 --cache=1 --output=/workspace/ --accuracy --count=5000
```
Result in `0.196 mAP`:
``` 
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.196
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.373
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.180
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.038
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.172
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.246
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.202
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.327
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.341
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.059
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.263
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.428
TestScenario.SingleStream qps=28.26, mean=1.5273, time=175.237, acc=35.223%, mAP=19.628%, queries=4952, tiles=50.0:1.5328,80.0:1.6218,90.0:1.6688,95.0:1.7007,99.0:1.7841,99.9:2.0040
```

This is using the upsampled validation COCO dataset per https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection#prepare-the-coco-dataset except we are using `jpeg` instead of `png`.

I noticed its reporting `queries=4952` while I requested `count=5000` which appears consistent with https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/coco.py#L83. I'm trying to determine if my dataset is corrupted.

If anyone can report their table of accuracies from the PyTorch backend that would be greatly appreciated.

@itayhubara  can you do a quick check on this issue?
otherwise we set the new accuracy threshold to 99% of 0.196 for SSD-Resnet34 for v0.7.

If I update the model's post-processing, to use the post-processing (`decode_batch_with_nms_trace`) from the branch used to create the MLPerf ONNX version for this model (https://github.com/BowenBao/inference/tree/master/cloud/single_stage_detector/pytorch) 
I get `0.200`:
```
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.200
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.381
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.184
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.040
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.181
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.249
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.202
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.329
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.350
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.064
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.281
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.433
TestScenario.SingleStream qps=28.32, mean=1.4867, time=174.852, acc=37.697%, mAP=20.000%, queries=4952, tiles=50.0:1.4867,80.0:1.5403,90.0:1.5718,95.0:1.6028,99.0:1.6810,99.9:2.0902
```

My changes can be found in:
https://github.com/parvizp/inference/blob/ssd_r34_update_post_processing/v0.5/classification_and_detection/python/models/ssd_r34.py

I think the onnx exporter has improved a lot since - possible that this converts out of the box these days.
We also can convert from the tensorflow model - I can send some instructions if desired.

Thanks for the offer @guschmue -- in this case we would actually prefer to submit using PyTorch native without exporting to ONNX.