Repository: mlcommons/inference
Issue: RuntimeError: list index out of range on RetinaNet
Root cause: Validation Gap

(mlc) ➜  mlcommons mlc run script --tags=run-mlperf,inference,_r5.0-dev \
   --model=retinanet \
   --implementation=reference \
   --framework=pytorch \
   --category=edge \
   --scenario=Offline \
   --execution_mode=valid \
   --device=cpu \
   --quiet                   
```                                
[2025-02-11 16:55:23,348 main.py:159 WARNING] - Warning: /home/junikki/MLC/repos/GATEOverflow@mlperf-automations/meta.yaml not found. Skipping...
....
...
......
.......
Using MLCommons Inference source from '/home/junikki/MLC/repos/local/cache/get-git-repo_509d019e/inference'

[2025-02-11 16:56:38,692 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-dataset-openimages_77ccb076
[2025-02-11 16:56:38,693 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-dataset-openimages/run.sh from tmp-run.sh
./openimages_mlperf.sh -d /home/junikki/MLC/repos/local/cache/get-dataset-openimages_77ccb076/install -m 50
Downloading images: 100%|███████████████████████| 50/50 [00:13<00:00,  3.80it/s]
Converting annotations to COCO format...
/home/junikki/MLC/repos/local/cache/get-git-repo_509d019e/inference/vision/classification_and_detection/tools/openimages.py:159: FutureWarning: factorize with argument that is not not a Series, Index, ExtensionArray, or np.ndarray is deprecated and will raise in a future version.
  annotations["image_id"] = pd.factorize(annotations["ImageID"].tolist())[0]
[2025-02-11 16:57:07,463 module.py:5481 INFO] -              ! call "postprocess" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-dataset-openimages/customize.py
[2025-02-11 16:57:07,517 module.py:560 INFO] -       * mlcr mlperf,mlcommons,inference,source,src
[2025-02-11 16:57:07,525 module.py:1274 INFO] -            ! load /home/junikki/MLC/repos/local/cache/get-mlperf-inference-src_7e7276f9/mlc-cached-state.json
[2025-02-11 16:57:07,593 module.py:560 INFO] -       * mlcr get,generic-python-lib,_pycocotools
[2025-02-11 16:57:07,609 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:07,610 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:07,611 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:07,611 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:07,612 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:07,612 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:07,794 module.py:5481 INFO] -              ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 2.0.8
[2025-02-11 16:57:07,819 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:07,821 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:07,822 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:07,822 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:07,825 module.py:1274 INFO] -            ! load /home/junikki/MLC/repos/local/cache/get-generic-python-lib_1077e419/mlc-cached-state.json
[2025-02-11 16:57:07,891 module.py:560 INFO] -       * mlcr get,generic-python-lib,_opencv-python
[2025-02-11 16:57:07,905 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:07,906 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:07,907 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:07,907 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:07,908 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:07,908 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:08,031 module.py:5481 INFO] -              ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 4.11.0.86
[2025-02-11 16:57:08,050 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:08,051 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:08,052 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:08,052 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:08,055 module.py:1274 INFO] -            ! load /home/junikki/MLC/repos/local/cache/get-generic-python-lib_1791e3ac/mlc-cached-state.json
[2025-02-11 16:57:08,113 module.py:560 INFO] -       * mlcr get,generic-python-lib,_pillow
[2025-02-11 16:57:08,128 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:08,129 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:08,129 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:08,130 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:08,130 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:08,130 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:08,240 module.py:5481 INFO] -              ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 11.1.0
[2025-02-11 16:57:08,259 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:08,261 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:08,261 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:08,261 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:08,264 module.py:1274 INFO] -            ! load /home/junikki/MLC/repos/local/cache/get-generic-python-lib_7fe994bd/mlc-cached-state.json
[2025-02-11 16:57:08,333 module.py:560 INFO] -       * mlcr get,generic-python-lib,_package.ujson
[2025-02-11 16:57:08,347 module.py:560 INFO] -         * mlcr detect,os
[2025-02-11 16:57:08,353 module.py:5334 INFO] -                ! cd /home/junikki/MLC/repos/local/cache/get-generic-python-lib_dcb2ca9c
[2025-02-11 16:57:08,354 module.py:5335 INFO] -                ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-11 16:57:08,403 module.py:5481 INFO] -                ! call "postprocess" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-02-11 16:57:08,421 module.py:560 INFO] -         * mlcr detect,cpu
[2025-02-11 16:57:08,430 module.py:560 INFO] -           * mlcr detect,os
[2025-02-11 16:57:08,436 module.py:5334 INFO] -                  ! cd /home/junikki/MLC/repos/local/cache/get-generic-python-lib_dcb2ca9c
[2025-02-11 16:57:08,436 module.py:5335 INFO] -                  ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-02-11 16:57:08,466 module.py:5481 INFO] -                  ! call "postprocess" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-02-11 16:57:08,483 module.py:5334 INFO] -                ! cd /home/junikki/MLC/repos/local/cache/get-generic-python-lib_dcb2ca9c
[2025-02-11 16:57:08,483 module.py:5335 INFO] -                ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/run.sh from tmp-run.sh
[2025-02-11 16:57:08,542 module.py:5481 INFO] -                ! call "postprocess" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/detect-cpu/customize.py
[2025-02-11 16:57:08,559 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:08,560 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:08,561 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:08,561 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:08,635 module.py:560 INFO] -         * mlcr get,generic-python-lib,_pip
[2025-02-11 16:57:08,647 module.py:560 INFO] -           * mlcr get,python3
[2025-02-11 16:57:08,648 module.py:1274 INFO] -                ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:08,649 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:08,649 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:08,650 module.py:5334 INFO] -                ! cd /home/junikki/MLC/repos/local/cache/get-generic-python-lib_dcb2ca9c
[2025-02-11 16:57:08,650 module.py:5335 INFO] -                ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:08,771 module.py:5481 INFO] -                ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
              Detected version: 24.0
[2025-02-11 16:57:08,789 module.py:560 INFO] -           * mlcr get,python3
[2025-02-11 16:57:08,790 module.py:1274 INFO] -                ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:08,791 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:08,791 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:08,792 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-generic-python-lib_9e5b1656/mlc-cached-state.json
[2025-02-11 16:57:08,802 module.py:5334 INFO] -                  ! cd /home/junikki/MLC/repos/local/cache/get-generic-python-lib_dcb2ca9c
[2025-02-11 16:57:08,802 module.py:5335 INFO] -                  ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/run.sh from tmp-run.sh
[2025-02-11 16:57:08,923 module.py:5481 INFO] -                  ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py

            Extra PIP CMD:   --break-system-packages 

[2025-02-11 16:57:08,931 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-generic-python-lib_dcb2ca9c
[2025-02-11 16:57:08,931 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/install.sh from tmp-run.sh

/media/junikki/research/mlcommons/mlc/bin/python3 -m pip install "ujson" --break-system-packages
Collecting ujson
  Using cached ujson-5.10.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.3 kB)
Using cached ujson-5.10.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (53 kB)
Installing collected packages: ujson
Successfully installed ujson-5.10.0
[2025-02-11 16:57:11,081 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-generic-python-lib_dcb2ca9c
[2025-02-11 16:57:11,082 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/run.sh from tmp-run.sh
[2025-02-11 16:57:11,185 module.py:5481 INFO] -              ! call "postprocess" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 5.10.0
[2025-02-11 16:57:11,278 module.py:560 INFO] -       * mlcr get,generic-python-lib,_numpy
[2025-02-11 16:57:11,291 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:11,292 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:11,293 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:11,293 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:11,294 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:11,294 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:11,400 module.py:5481 INFO] -              ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 1.26.4
[2025-02-11 16:57:11,415 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:11,416 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:11,416 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:11,417 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:11,417 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:11,417 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:11,523 module.py:5481 INFO] -              ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 1.26.4
[2025-02-11 16:57:11,550 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:11,552 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:11,552 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:11,552 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:11,555 module.py:1274 INFO] -            ! load /home/junikki/MLC/repos/local/cache/get-generic-python-lib_b0eede39/mlc-cached-state.json
[2025-02-11 16:57:11,644 module.py:560 INFO] -       * mlcr get,generic-python-lib,_numpy
[2025-02-11 16:57:11,660 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:11,661 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:11,661 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:11,662 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:11,662 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:11,662 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:11,772 module.py:5481 INFO] -              ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 1.26.4
[2025-02-11 16:57:11,790 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:11,791 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:11,792 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:11,792 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:11,793 module.py:5334 INFO] -              ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:11,793 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/validate_cache.sh from tmp-run.sh
[2025-02-11 16:57:11,891 module.py:5481 INFO] -              ! call "detect_version" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
            Detected version: 1.26.4
[2025-02-11 16:57:11,919 module.py:560 INFO] -         * mlcr get,python3
[2025-02-11 16:57:11,921 module.py:1274 INFO] -              ! load /home/junikki/MLC/repos/local/cache/get-python3_1165d524/mlc-cached-state.json
[2025-02-11 16:57:11,921 module.py:2220 INFO] - Path to Python: /media/junikki/research/mlcommons/mlc/bin/python3
[2025-02-11 16:57:11,921 module.py:2220 INFO] - Python version: 3.12.3
[2025-02-11 16:57:11,924 module.py:1274 INFO] -            ! load /home/junikki/MLC/repos/local/cache/get-generic-python-lib_b0eede39/mlc-cached-state.json
Using MLCommons Inference source from '/home/junikki/MLC/repos/local/cache/get-git-repo_509d019e/inference'
[2025-02-11 16:57:11,933 module.py:5334 INFO] -            ! cd /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9
[2025-02-11 16:57:11,933 module.py:5335 INFO] -            ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-preprocessed-dataset-openimages/run.sh from tmp-run.sh
INFO:coco:loaded 50 images, cache=True, already_preprocessed=False, took=4.2sec
[2025-02-11 16:57:16,447 module.py:5481 INFO] -            ! call "postprocess" from /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/get-preprocessed-dataset-openimages/customize.py
[2025-02-11 16:57:16,480 module.py:560 INFO] -     * mlcr generate,user-conf,mlperf,inference
[2025-02-11 16:57:16,492 module.py:560 INFO] -       * mlcr detect,os
[2025-02-11 16:57:16,501 module.py:5334 INFO] -              ! cd /media/junikki/research/mlcommons
[2025-02-11 16:57:16,501 module.py:5335 INFO] -              ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
....
....
***************************************************************************
CM script::benchmark-program/run.sh

Run Directory: /home/junikki/MLC/repos/local/cache/get-git-repo_509d019e/inference/vision/classification_and_detection

CMD: ./run_local.sh pytorch retinanet cpu --scenario Offline    --max-batchsize 1 --threads 8 --user_conf '/home/junikki/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/d60c5984361c4900b3dcedf09242df27.conf' --use_preprocessed_dataset --cache_dir /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9 --dataset-list /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9/annotations/openimages-mlperf.json 2>&1 | tee '/home/junikki/MLC/repos/local/cache/get-mlperf-inference-results-dir_7dbef0e6/valid_results/junikki-reference-cpu-pytorch-v2.6.0-default_config/retinanet/offline/performance/run_1/console.out'; echo \${PIPESTATUS[0]} > exitstatus

[2025-02-11 16:57:19,099 module.py:5334 INFO] -          ! cd /media/junikki/research/mlcommons
[2025-02-11 16:57:19,099 module.py:5335 INFO] -          ! call /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/benchmark-program/run-ubuntu.sh from tmp-run.sh

./run_local.sh pytorch retinanet cpu --scenario Offline    --max-batchsize 1 --threads 8 --user_conf '/home/junikki/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/d60c5984361c4900b3dcedf09242df27.conf' --use_preprocessed_dataset --cache_dir /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9 --dataset-list /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9/annotations/openimages-mlperf.json 2>&1 | tee '/home/junikki/MLC/repos/local/cache/get-mlperf-inference-results-dir_7dbef0e6/valid_results/junikki-reference-cpu-pytorch-v2.6.0-default_config/retinanet/offline/performance/run_1/console.out'; echo ${PIPESTATUS[0]} > exitstatus
python3 python/main.py --profile retinanet-pytorch --model "/home/junikki/MLC/repos/local/cache/download-file_3313d856/resnext50_32x4d_fpn.pth" --dataset-path /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9 --output "/home/junikki/MLC/repos/local/cache/get-mlperf-inference-results-dir_7dbef0e6/valid_results/junikki-reference-cpu-pytorch-v2.6.0-default_config/retinanet/offline/performance/run_1" --scenario Offline --max-batchsize 1 --threads 8 --user_conf /home/junikki/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/d60c5984361c4900b3dcedf09242df27.conf --use_preprocessed_dataset --cache_dir /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9 --dataset-list /home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9/annotations/openimages-mlperf.json
INFO:main:Namespace(dataset='openimages-800-retinanet', dataset_path='/home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9', dataset_list='/home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9/annotations/openimages-mlperf.json', data_format=None, profile='retinanet-pytorch', scenario='Offline', max_batchsize=1, model='/home/junikki/MLC/repos/local/cache/download-file_3313d856/resnext50_32x4d_fpn.pth', output='/home/junikki/MLC/repos/local/cache/get-mlperf-inference-results-dir_7dbef0e6/valid_results/junikki-reference-cpu-pytorch-v2.6.0-default_config/retinanet/offline/performance/run_1', inputs=['image'], outputs=['boxes', 'labels', 'scores'], backend='pytorch-native', device=None, model_name='retinanet', threads=8, qps=None, cache=0, cache_dir='/home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9', preprocessed_dir=None, use_preprocessed_dataset=True, accuracy=False, find_peak_performance=False, debug=False, user_conf='/home/junikki/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/d60c5984361c4900b3dcedf09242df27.conf', audit_conf='audit.config', time=None, count=None, performance_sample_count=None, max_latency=None, samples_per_query=8)
INFO:coco:loaded 50 images, cache=0, already_preprocessed=True, took=0.0sec
/media/junikki/research/mlcommons/mlc/lib/python3.12/site-packages/torch/serialization.py:1434: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)
  warnings.warn(
/media/junikki/research/mlcommons/mlc/lib/python3.12/site-packages/torch/nn/modules/module.py:1750: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3637.)
  return forward_call(*args, **kwargs)
code/__torch__/model/retinanet.py:159: UserWarning: RetinaNet always returns a (Losses, Detections) tuple in scripting
INFO:main:starting TestScenario.Offline
ERROR:main:thread: failed on contentid=['/home/junikki/MLC/repos/local/cache/get-preprocessed-dataset-openimages_4eb26ce9/01e714e8fbbc0d0c.jpg'], The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/model/anchor_utils.py", line 76, in forward
      size = grid_sizes[_26]
      stride = strides[_26]
      base_anchors = cell_anchors[_26]
                     ~~~~~~~~~~~~~~~~~ <--- HERE
      grid_height, grid_width, = size
      stride_height, stride_width, = stride

Traceback of TorchScript, original code (most recent call last):
  File "/Users/pablogonzalez/Desktop/MLCommons/training/single_stage_detector/ssd/model/anchor_utils.py", line 91, in forward
                             "feature maps passed and the number of sizes / aspect ratios specified.")
    
        for size, stride, base_anchors in zip(
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            grid_sizes, strides, cell_anchors
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        ):
        ~~
            grid_height, grid_width = size
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            stride_height, stride_width = stride
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            device = base_anchors.device
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
            # For output anchor, compute [x_center, y_center, x_center, y_center]
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            shifts_x = torch.arange(
            ~~~~~~~~~~~~~~~~~~~~~~~~
                0, grid_width, dtype=torch.float32, device=device
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            ) * stride_width
            ~~~~~~~~~~~~~~~~
            shifts_y = torch.arange(
            ~~~~~~~~~~~~~~~~~~~~~~~~
                0, grid_height, dtype=torch.float32, device=device
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            ) * stride_height
            ~~~~~~~~~~~~~~~~~
            shift_y, shift_x = torch.meshgrid(shifts_y, shifts_x)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            shift_x = shift_x.reshape(-1)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            shift_y = shift_y.reshape(-1)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            shifts = torch.stack((shift_x, shift_y, shift_x, shift_y), dim=1)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
            # For every (base anchor, output anchor) pair,
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            # offset each zero-centered base anchor by the center of the output anchor.
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            anchors.append(
            ~~~~~~~~~~~~~~~
                (shifts.view(-1, 1, 4) + base_anchors.view(1, -1, 4)).reshape(-1, 4)
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
            )
    
RuntimeError: list index out of range
```

This is because of not using the "_full" variation and so only 50 images are loaded. Can you please try with this variation?

@anandhu-eng - can we make "_full" the default for retinanet as otherwise it always fails for >50 inputs.