mAP Drop
#1
by
mhyatt000
- opened
I tried to reproduce the results mentioned on this model card. Seems like my results do not match the claimed mAP in the model card. I cannot figure out how to get the correct numbers, can you help me find my mistake?
- Claimed mAP: 42.0
- Recieved mAP: 36.6
Here are the details for my validation:
- I instantiate pre-trained model with
transformers.pipeline()and use COCO API to calculate AP from detection bboxes. - Evaluation was performed on macOS CPU.
- Dataset was downloaded from cocodataset.org
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.366
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.567
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.379
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.144
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.397
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.585
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.303
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.441
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.452
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.193
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.491
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.687