Repository: mlcommons/inference
Issue: ssd-resnet34
Root cause: External Dependency Breakage

Does anyone remember where the mAP came from for ssd-resnet34? The table says the reference accuracy is mAP 0.20 which is lower than the than mAP 0.23 for ssd-mobilenet. I have a vague recollection that there is a known issue with NMS that artificially lowers the accuracy. I'm asking because we swapped out our NMS implementation recently, and without retraining, calibration, or quantization the accuracy results look like:

```
Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.226
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.411
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.221
Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.037
Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.219
Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.279
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.217
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.356
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.390
Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.038
Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.349
Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.480
mAP=22.584%
found 5000 results
found 5000 images
found 0 images with no results
ignored 0 dupes
```

I thought perhaps I swapped the ssd-resnet34 and ssd-mobilenet, but the graph I am working with takes a [1200, 1200, 3] as input and is based on https://zenodo.org/record/3345892/files/tf_ssd_resnet34_22.1.zip. For the successors of ssd-resnet34 and ssd-mobilenet, maybe MLPerf ought to own the NMS implementation to avoid this sort of variability.

@tjablin I've recently run the corresponding ONNX model, and [found](https://github.com/krai/ck-mlperf/blob/master/program/object-detection-onnx-py/README.md#5000-images) the accuracy to be just under 20%:
<pre>
DONE (t=14.66s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.200
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.381
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.183
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.119
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.257
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.233
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.200
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.321
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.344
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.174
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.406
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.416

Summary:
-------------------------------
All images loaded in 176.739452s
Average image load time: 0.035348s
All images detected in 5474.896789s
Average detection time: 1.094935s
Total NMS time: 0.000000s
Average NMS time: 0.000000s
mAP: 0.19952640873605498
Recall: 0.343745110610767
--------------------------------
</pre>

Perhaps the ONNX and TF models are simply not equivalent? From experience, I doubt NMS alone can improve the accuracy by 2.5% (perhaps 0.5% sounds plausible, but I don't have any hard data).

If I read the results correctly, the TF model performs better on larger objects, while the ONNX model on smaller objects.

So the official onnx model for ssd-resnet34 came from pytorch and had a bunch of pain to make that work in onnx (long time ago and I forgot the details).

I have another way to convert it directly from the tensorflow model via tf2onnx.
The nms in onnx should be identical to tensorflow, we have bunch of unit test that say so and never had complains on real models.