Repository: mlcommons/inference
Issue: Openimages accuracy script not working for retinanet pytorch model
Root cause: Environment Incompatibility

I'm running the pytorch retinanet model in accuracy mode. The result looks fine but when I call the accuracy script, reported accuracy is bad. Am I missing something? 
```
cd '/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/b29aef91b8bd445b/inference/vision/classification_and_detection' && OUTPUT_DIR='/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/b29aef91b8bd445b/inference/vision/classification_and_detection/test_results/pytorch-cpu/retinanet/offline/accuracy' ./run_local.sh pytorch retinanet cpu --scenario Offline  --mlperf_conf '/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/b29aef91b8bd445b/inference/mlperf.conf' --threads 8 --count 10 --user_conf '/home/arjun/CM/repos/octoml@ck/cm-mlops/script/app-mlperf-inference/tmp/d4b8414a458e4558849e5d7faf773680.conf' --accuracy --cache_dir /media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/a806bacbd0084d6e
python3 python/main.py --profile retinanet-pytorch --mlperf_conf ../../mlperf.conf --model "/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/d1a2d5e9e91c492c/resnext50_32x4d_fpn.pth" --dataset-path /media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/c146b41e505e4831/install --output "/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/b29aef91b8bd445b/inference/vision/classification_and_detection/test_results/pytorch-cpu/retinanet/offline/accuracy" --scenario Offline --mlperf_conf /media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/b29aef91b8bd445b/inference/mlperf.conf --threads 8 --count 10 --user_conf /home/arjun/CM/repos/octoml@ck/cm-mlops/script/app-mlperf-inference/tmp/d4b8414a458e4558849e5d7faf773680.conf --accuracy --cache_dir /media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/a806bacbd0084d6e
INFO:main:Namespace(dataset='openimages-800-retinanet', dataset_path='/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/c146b41e505e4831/install', dataset_list=None, data_format=None, profile='retinanet-pytorch', scenario='Offline', max_batchsize=32, model='/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/d1a2d5e9e91c492c/resnext50_32x4d_fpn.pth', output='/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/b29aef91b8bd445b/inference/vision/classification_and_detection/test_results/pytorch-cpu/retinanet/offline/accuracy', inputs=['image'], outputs=['boxes', 'labels', 'scores'], backend='pytorch-native', model_name='retinanet', threads=8, qps=None, cache=0, cache_dir='/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/a806bacbd0084d6e', accuracy=True, find_peak_performance=False, debug=False, mlperf_conf='/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/b29aef91b8bd445b/inference/mlperf.conf', user_conf='/home/arjun/CM/repos/octoml@ck/cm-mlops/script/app-mlperf-inference/tmp/d4b8414a458e4558849e5d7faf773680.conf', audit_conf='audit.config', time=None, count=10, performance_sample_count=None, max_latency=None, samples_per_query=8)
INFO:coco:loaded 10 images, cache=0, took=0.0sec
/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/b29aef91b8bd445b/inference/vision/classification_and_detection/python/openimages.py:109: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  self.label_list = np.array(self.label_list)
/home/arjun/.local/lib/python3.10/site-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)
  warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"
/home/arjun/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1110: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2228.)
  return forward_call(*input, **kwargs)
code/__torch__/model/retinanet.py:159: UserWarning: RetinaNet always returns a (Losses, Detections) tuple in scripting
INFO:main:starting TestScenario.Offline
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
Loading and preparing results...
Converting ndarray to lists...
(981, 7)
0/981
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.11s).
Accumulating evaluation results...
DONE (t=0.11s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.54804
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.78624
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.71423
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.00000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.30370
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.63093
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.43288
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.64808
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.66236
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.00000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.34111
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.73062
TestScenario.Offline qps=0.02, mean=56.6429, time=57.029, acc=47.910%, mAP=54.804%, queries=1, tiles=50.0:56.6429,80.0:56.6429,90.0:56.6429,95.0:56.6429,99.0:56.6429,99.9:56.6429



No warnings encountered during test.

No errors encountered during test.
```

Result from calling the accuracy script.
```
python3 '/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/b29aef91b8bd445b/inference/vision/classification_and_detection/tools/accuracy-openimages.py' --mlperf-accuracy-file '/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/b29aef91b8bd445b/inference/vision/classification_and_detection/test_results/pytorch-cpu/retinanet/offline/accuracy/mlperf_log_accuracy.json' --openimages-dir '/home/arjun/CM/repos/local/cache/347a3211af1a4ae0/install' > '/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/b29aef91b8bd445b/inference/vision/classification_and_detection/test_results/pytorch-cpu/retinanet/offline/accuracy/accuracy.txt'
Loading annotations into memory...
Done (t=0.96s)
Creating index...
index created!
Loading and preparing results...
DONE (t=0.02s)
Creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.06s).
Accumulating evaluation results...
DONE (t=0.08s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.00156
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.00591
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.00000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.00000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.00000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.00298
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.01000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.01000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.01000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.00000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.00000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.01000
mAP=0.156%
```

@pgmpablo157321 Can you please see if I'm missing something here?

@arjunsuresh I am not sure what is happening there. I tried to run the benchmark several times but couldn't replicate the error. Here is one run (500 image):
```
code/__torch__/model/retinanet.py:159: UserWarning: RetinaNet always returns a (Losses, Detections) tuple in scripting
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py:1130: UserWarning: operator() profile_node %2995 : int[] = prim::profile_ivalue[profile_failed="varying profile values"](%2993)
 does not have profile information (Triggered internally at  ../torch/csrc/jit/codegen/cuda/graph_fuser.cpp:104.)
  return forward_call(*input, **kwargs)
INFO:main:starting TestScenario.SingleStream
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
Loading and preparing results...
Converting ndarray to lists...
(56252, 7)
0/56252
DONE (t=0.29s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=6.01s).
Accumulating evaluation results...
DONE (t=1.64s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.500
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.642
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.547
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.054
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.186
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.543
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.499
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.637
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.663
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.099
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.308
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.706
TestScenario.SingleStream qps=29.35, mean=0.1304, time=16.863, acc=40.452%, mAP=49.955%, queries=495, tiles=50.0:0.1301,80.0:0.1356,90.0:0.1391,95.0:0.1415,99.0:0.1458,99.9:0.2078
```
Result from calling the accuracy script
```
python tools/accuracy-openimages.py \
 --mlperf-accuracy-file output/pytorch-gpu/retinanet/mlperf_log_accuracy.json \
 --openimages-dir open-images-v6-mlperf/

loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.39s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=6.05s).
Accumulating evaluation results...
DONE (t=1.70s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.500
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.642
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.547
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.054
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.186
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.543
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.499
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.637
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.663
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.099
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.308
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.706
mAP=49.955%
```

I have two guesses about what could be happening:
1. You seem to be using this folder `/home/arjun/CM/repos/local/cache/347a3211af1a4ae0/install` as the location of the open-images dataset. Is this the folder where the script downloads the dataset?
2. What version of pycocotools are you using? If you are using the [NVIDIA fork](https://github.com/NVIDIA/cocoapi.git), I recall that some previous versions had issues computing the mAP (in fact you commented about that in this [issue](https://github.com/mlcommons/inference/issues/1240)). I tested it with the current [head](https://github.com/NVIDIA/cocoapi/tree/fa44301f7a8b3f95a9f2751d19bfd735b0f6c65d) and it works fine.

A couple of questions:
1. What subset of open-images are you using? Is it the one you get when running `./openimages_mlperf.sh -m 50`?
2. In the reference implementation when you give the `--accuracy` flag, the benchmark runs in SingleStream mode. But the scenario that you ran is Offline. Is there a reason for this?

@arjunsuresh I did another experiment where I was able to get a mismatch between the metrics. When I run the benchmark into offline mode (`!./run_local.sh pytorch retinanet gpu --scenario Offline --accuracy`), I get a memory allocation error:

```
RuntimeError: CUDA out of memory. Tried to allocate 3.78 GiB (GPU 0; 14.76 GiB total capacity; 7.46 GiB already allocated; 573.75 MiB free; 13.12 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```
And I get an mAP mismatch between the one computed by the benchmark and by the script:
```
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
Loading and preparing results...
Converting ndarray to lists...
(1739, 7)
0/1739
DONE (t=0.01s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.15s).
Accumulating evaluation results...
DONE (t=0.16s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.490
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.647
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.534
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.120
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.459
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.564
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.428
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.559
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.566
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.120
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.497
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.643
TestScenario.Offline qps=0.07, mean=14.3813, time=14.966, acc=40.311%, mAP=48.969%, queries=1, tiles=50.0:14.3813,80.0:14.3813,90.0:14.3813,95.0:14.3813,99.0:14.3813,99.9:14.3813
```
Running the accuracy script:
```
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.01s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.22s).
Accumulating evaluation results...
DONE (t=0.21s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.194
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.247
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.216
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.011
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.106
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.191
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.167
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.209
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.213
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.011
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.109
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.212
mAP=19.408%
```
Do you think something like this could be the issue?

Thank you @pgmpablo157321 for checking. I verified my `pycocotools` installation and it seems fine. Infact I'm getting the expected accuracy by using `onnxruntime` backend. But @gfursin got the expected accuracy using `pytorch` backend. So, it should be something to do with my environment - let me check that and I'll update you.

Our [run on github actions](https://github.com/mlcommons/ck/actions/runs/3481112301/jobs/5821777831) went fine. So, I'll check again my environment. Thanks @pgmpablo157321 for looking into this.