Repository: mlcommons/inference
Issue: [Bug] RetinaNet+ONNXRuntime Reference (r5.0-dev): CUDA Error (misaligned/Xid 31) in MultiStream/Offline & ValueError (NaN iscrowd) in Accuracy
Root cause: Resource Mishandling

**Environment:**

*   **OS:** `Ubuntu 20.04.6 LTS`
*   **GPU Model:** `NVIDIA A800 80GB PCIe`
*   **NVIDIA Driver Version:** `530.30.02`
*   **CUDA Toolkit Version:** `12.1 (V12.1.66)`(Installed via system package or runfile)
*   **cuDNN Version:**`8.9.7` (Manually installed in user space)
*   **ONNX Runtime Version:** `1.18.0` (GPU version, installed via pip in conda env ml_perf)
*   **Python Version:** `3.10.16` (via Miniconda environment `ml_perf`)
*   **pycocotools Version:** `2.0.8` (installed via pip)
*   **MLPerf Inference Repo Version:** Branch/Tag `r5.0-dev` (Cloned/managed via MLCommons CM)
*   **MLCommons CM Version:** `mlc-scripts 1.0.2` / `mlcflow 1.0.10`

**Command Used to Reproduce:**

The primary command intended to run all scenarios in valid mode:

```bash
mlcr run-mlperf,inference,_full,_r5.0-dev,_all-scenarios \
   --model=retinanet \
   --implementation=reference \
   --framework=onnxruntime \
   --category=edge \
   --execution_mode=valid \
   --device=cuda \
   --quiet
```

Commands used for debugging specific scenarios (after clearing results cache):

```bash
# For SingleStream (Performance runs, Accuracy fails)
mlcr run-mlperf,inference,_full,_r5.0-dev \
   --model=retinanet \
   --implementation=reference \
   --framework=onnxruntime \
   --category=edge \
   --scenario=SingleStream \
   --execution_mode=valid \
   --device=cuda \
   --quiet

# For Offline (Fails during Performance run)
mlcr run-mlperf,inference,_full,_r5.0-dev \
   --model=retinanet \
   --implementation=reference \
   --framework=onnxruntime \
   --category=edge \
   --scenario=Offline \
   --execution_mode=valid \
   --device=cuda \
   --quiet

# For MultiStream (Fails during Performance run)
mlcr run-mlperf,inference,_full,_r5.0-dev \
   --model=retinanet \
   --implementation=reference \
   --framework=onnxruntime \
   --category=edge \
   --scenario=MultiStream \
   --execution_mode=valid \
   --device=cuda \
   --quiet
```

**Contrast: Successful ResNet50 Commands (Same Environment):**

The following commands run successfully to completion (Performance and Accuracy) in the **exact same environment**:

```bash
# ResNet50 Edge Valid
mlcr run-mlperf,inference,_full,_r5.0-dev,_all-scenarios \
   --model=resnet50 \
   --implementation=reference \
   --framework=onnxruntime \
   --category=edge  \
   --execution_mode=valid \
   --device=cuda \
   --quiet

# ResNet50 Datacenter Valid (SERVER_TARGET_QPS needs to be set)
# mlcr run-mlperf,inference,_full,_r5.0-dev,_all-scenarios \
#    --model=resnet50 \
#    --implementation=reference \
#    --framework=onnxruntime \
#    --category=datacenter \
#    --server_target_qps=<SERVER_TARGET_QPS> \
#    --execution_mode=valid \
#    --device=cuda \
#    --quiet
```

**Observed Behavior:**

When running RetinaNet with the reference implementation and ONNX Runtime backend in `valid` mode, multiple distinct failures occur depending on the scenario:

1.  **SingleStream Scenario:**
    *   The **Performance** part runs successfully. Example summary output:
        ```log
        ================================================
        MLPerf Results Summary
        ================================================
        SUT name : PySUT
        Scenario : SingleStream
        Mode     : PerformanceOnly
        90.0th percentile latency (ns) : 21396647
        Result is : VALID
        ... (rest of summary) ...
        ================================================
        ```
    *   However, the subsequent **Accuracy** evaluation phase fails consistently with a `ValueError`. This happens even after clearing the results cache (`rm -rf .../retinanet`) and re-downloading/re-processing the dataset. The error occurs within `pycocotools` when processing the ground truth annotations.
    *   **Accuracy Traceback:**
        ```python
        loading annotations into memory...
        Done (t=0.46s)
        creating index...
        index created!
        Loading and preparing results...
        Converting ndarray to lists...
        (1773424, 7) # Note: Prediction count might vary slightly between runs
        0/1773424
        1000000/1773424
        DONE (t=10.84s)
        creating index...
        index created!
        Running per image evaluation...
        Evaluate annotation type *bbox*
        Traceback (most recent call last):
          File "/home/wanghaonan/MLC/repos/local/cache/get-git-repo_ecbc20bf/inference/vision/classification_and_detection/python/main.py", line 781, in <module>
            main()
          File "/home/wanghaonan/MLC/repos/local/cache/get-git-repo_ecbc20bf/inference/vision/classification_and_detection/python/main.py", line 757, in main
            post_proc.finalize(result_dict, ds, output_dir=args.output)
          File "/home/wanghaonan/MLC/repos/local/cache/get-git-repo_ecbc20bf/inference/vision/classification_and_detection/python/openimages.py", line 295, in finalize
            cocoEval.evaluate()
          File "/home/wanghaonan/miniconda3/envs/ml_perf/lib/python3.10/site-packages/pycocotools/cocoeval.py", line 148, in evaluate
            self.ious = {(imgId, catId): computeIoU(imgId, catId) \
          File "/home/wanghaonan/miniconda3/envs/ml_perf/lib/python3.10/site-packages/pycocotools/cocoeval.py", line 148, in <dictcomp>
            self.ious = {(imgId, catId): computeIoU(imgId, catId) \
          File "/home/wanghaonan/miniconda3/envs/ml_perf/lib/python3.10/site-packages/pycocotools/cocoeval.py", line 188, in computeIoU
            iscrowd = [int(o['iscrowd']) for o in gt]
          File "/home/wanghaonan/miniconda3/envs/ml_perf/lib/python3.10/site-packages/pycocotools/cocoeval.py", line 188, in <listcomp>
            iscrowd = [int(o['iscrowd']) for o in gt]
        ValueError: cannot convert float NaN to integer
        ```
    *   This suggests the COCO annotation file generated by `tools/openimages.py` contains `NaN` values for the `iscrowd` field.

2.  **Offline Scenario:**
    *   The run fails **during the Performance test** itself, before even reaching accuracy evaluation.
    *   It throws a CUDA error related to memory access.
    *   **CUDA Error Log Snippet:**
        ```log
        ERROR:main:thread: failed on contentid=['/home/wanghaonan/MLC/repos/local/cache/get-preprocessed-dataset-openimages_2952321f/7aa0b17d541db028.jpg'], [ONNXRuntimeError] : 1 : FAIL : CUDA failure 716: misaligned address ; GPU=0 ; hostname=graph-2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=63 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));
        2025-04-23 10:41:28.753762954 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 716: misaligned address ; GPU=0 ; hostname=graph-2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=63 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));
        2025-04-23 10:41:28.753789958 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 716: misaligned address ; GPU=0 ; hostname=graph-2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=446 ; expr=cudaStreamSynchronize(static_cast<cudaStream_t>(stream_));
        ```
    *   This CUDA error often coincides with **Xid 31 errors** reported in `dmesg` for the corresponding GPU and python3 process, indicating an MMU fault likely related to accessing an invalid address (e.g., near NULL).
    *   **Relevant `dmesg` Output:**
        ```log
        [10348879.702796] NVRM: Xid (PCI:0000:34:00): 31, pid=2992993, name=python3, Ch 00000008, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_6 faulted @ 0x0_00000000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
        ```
        *(Note: PID will change between runs, but the pattern of Xid 31 / MMU Fault / python3 / 0x0 address was observed)*

3.  **MultiStream Scenario:**
    *   Similar to the Offline scenario, the run fails **during the Performance test**.
    *   It exhibits the **same** `CUDA failure 716: misaligned address` error and corresponding `Xid 31` errors in `dmesg`.
    *   **CUDA Error Log Snippet:**
        ```log
        ERROR:main:thread: failed on contentid=['/home/wanghaonan/MLC/repos/local/cache/get-preprocessed-dataset-openimages_2952321f/776b458d0d0e412a.jpg'], [ONNXRuntimeError] : 1 : FAIL : CUDA failure 716: misaligned address ; GPU=0 ; hostname=graph-2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=63 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));
        2025-04-23 11:13:47.424419492 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 716: misaligned address ; GPU=0 ; hostname=graph-2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=63 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));
        ```

**Expected Behavior:**

The `mlcr` command for RetinaNet using the reference implementation and ONNX Runtime in `valid` mode should successfully complete both Performance and Accuracy runs for all specified scenarios (SingleStream, MultiStream, Offline), similar to how ResNet50 runs successfully in the same environment. At a minimum, the Performance runs for MultiStream and Offline should not trigger CUDA errors or Xid events.

**Attempts to Resolve:**

*   Cleared the relevant CM results cache (`rm -rf /home/wanghaonan/MLC/repos/local/cache/get-mlperf-inference-results-dir_65ab07b1/valid_results/graph_2-reference-gpu-onnxruntime-v1.18.0-cu121/retinanet`) multiple times.
*   Allowed CM to re-download and re-process the OpenImages dataset.
*   The `ValueError` in accuracy persists after data reprocessing.
*   The CUDA errors in MultiStream/Offline persist after clearing results.

**Conclusion:**

Given that the environment successfully runs ResNet50 benchmarks, and the RetinaNet benchmark fails in multiple, specific ways (data generation issue for accuracy, CUDA memory issue for MultiStream/Offline) even after cache clearing, this strongly suggests potential bugs within the MLPerf reference scripts' handling of RetinaNet data preparation/batching for ONNX Runtime, or an issue in how ONNX Runtime's CUDA EP handles this specific model/data combination.

Could you please investigate these issues? Let me know if full logs for specific runs are needed.

@haonan0x3a1 Can you please do `mlc pull repo` and retry the commands? It seems retinanet implementation for cuda has an issue with `--threads>1`.