Repository: mlcommons/inference
Issue: ONNX Runtime ResNet50 Inference on NVIDIA A30 Uses CPU Instead of GPU
Root cause: Configuration Error

Hi, everyone!

I was testing ResNet50 MLPerf benchmarking on Nvidia A30. I was following this docs: https://docs.mlcommons.org/inference/benchmarks/image_classification/resnet50/#__tabbed_32_1

When running the MLPerf Inference v5.1 benchmark, the inference appears to be running on the CPU instead of the GPU. Throughput is around ~160 samples/sec, significantly below expected GPU performance (~1000+).

Here's the related outputs and logs:

<img width="1909" height="1033" alt="Image" src="https://github.com/user-attachments/assets/2b28158f-cc0b-4b2a-bb88-a63c995b1ba2" />
- No GPU utilization seen in nvidia-smi
- htop confirms heavy CPU usage

This is final console output:
Graph visualization saved as /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_d5e35de0/valid_results/bf9334fda04a-reference-gpu-onnxruntime-v1.22.0-cu124/resnet50/offline/TEST04/mlc-deps.png
Mermaid syntax saved to /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_d5e35de0/valid_results/bf9334fda04a-reference-gpu-onnxruntime-v1.22.0-cu124/resnet50/offline/TEST04/mlc-deps.mmd
Graph data saved as graph.graphml
[2025-07-10 09:32:29,312 module.py:5274 INFO] -        ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/draw-graph-from-json-data/customize.py
[2025-07-10 09:32:29,326 customize.py:339 INFO] - bf9334fda04a-reference-gpu-onnxruntime-v1.22.0-cu124
[2025-07-10 09:32:29,326 customize.py:342 INFO] - +----------+----------+----------+------------+-----------------+---------------------------------+--------+--------+
|  Model   | Scenario | Accuracy | Throughput | Latency (in ms) | Power Efficiency (in samples/J) | TEST01 | TEST04 |
+----------+----------+----------+------------+-----------------+---------------------------------+--------+--------+
| resnet50 | Offline  |  76.456  |  162.262   |        -        |                                 | passed | passed |
+----------+----------+----------+------------+-----------------+---------------------------------+--------+--------+

The MLPerf inference results are stored at /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_d5e35de0/valid_results

[2025-07-10 09:32:29,327 module.py:5274 INFO] -        ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py
[2025-07-10 09:32:29,330 customize.py:401 INFO] -
Path to the MLPerf inference benchmark reference sources: /home/mlcuser/MLC/repos/local/cache/get-git-repo_inference-src_48b4ca01/inference
Path to the MLPerf inference reference configuration file: /home/mlcuser/MLC/repos/local/cache/get-git-repo_inference-src_48b4ca01/inference/mlperf.conf
[2025-07-10 09:32:29,330 customize.py:406 INFO] -
root@bf9334fda04a:~#
As we can see from the console output both TEST01 and TEST04 are successfully passed.

Environment:

GPU: NVIDIA A30 (24GB)
Driver: 535.230.02
CUDA version: 12.2
ONNX Runtime version: 1.22.0
MLPerf Inference version: v5.1
MLCFlow / Docker: Yes (inside container)
Model: ResNet50 (ONNX)
Scenario: Offline
Batch Size: 32 (default)

Could you help identify the following:

1. Why is onnxruntime-gpu not utilizing the GPU during inference?
2. How can we ensure that the CUDAExecutionProvider is activated and used inside the Docker container?
3. Is the reference implementation of ONNX Runtime in MLPerf Inference designed to run only on the CPU, regardless of GPU availability?
4. Approximately, what throughput can be expected when running a ResNet ONNX model on an NVIDIA A30 GPU with a batch size of 32? Also, what is the default batch size?

I’d really appreciate any help or insights on this issue. Please let me know if more logs or environment details are needed.

Thanks in advance!

Hey @xdilshod , the issue was with the incorrect installation order between `onnxruntime` and `onnxruntime-gpu`. Could you please do the following step and retry:

If you are not in the docker container, please add the following tags along with the docker run command:

```
--docker --docker_cache=no --docker_rebuild
```

if you are already in the docker container, please pull the latest `mlperf-automations` repo `dev` branch changes, uninstall both the libraries and then execute the run command without docker flags:

```
mlc pull repo
```

```
pip uninstall onnxruntime onnxruntime-gpu
```

Hi, @anandhu-eng, thanks for the response!

This one fixed the issue.
`pip uninstall onnxruntime onnxruntime-gpu
`

One follow-up question:
Where can we check the default batch size being used during the run? I couldn’t locate it in the logs or configuration files. Any pointer would be appreciated!

Hi @xdilshod , If I'm correct it's using a batch size of 128 by default. (referred [here](https://github.com/mlcommons/inference/blob/1bc3e998cb29a2ccb7635a5c74c875bf0c3b6432/vision/classification_and_detection/python/main.py#L505))

Since requested, I have raised a PR in automation repo that adds batch size arg along with the run command which is yet to be merged. Please note that it's not tested from our side.


## Root Cause Analysis for ONNX GPU Issue

I've analyzed this issue and found the root cause. The problem is **ONNX Runtime library installation order**.

### The Issue
When `onnxruntime` (CPU-only) is installed before `onnxruntime-gpu`, the CUDA provider gets listed in `get_all_providers()` but remains non-functional in `get_available_providers()`.

**Code Location**: The logic in `vision/classification_and_detection/python/backend_onnxruntime.py` (lines 39-43) only checks if CUDA provider exists, not if it works:

```python
if len(rt.get_all_providers()) > 1:  # Checks existence, not functionality
    providers=["CUDAExecutionProvider"]  # Assumes it works
```

### Quick Fix
```bash
pip uninstall onnxruntime onnxruntime-gpu -y
pip install onnxruntime-gpu  # This includes CPU fallback
```

### Verification
```python
import onnxruntime as rt
print("Available providers:", rt.get_available_providers())
# Should show: ['CUDAExecutionProvider', 'CPUExecutionProvider']
```

### Expected A30 Performance
- **GPU**: ~1000-2000 samples/sec for ResNet50
- **CPU**: ~100-200 samples/sec  
- Your ~160 samples/sec confirms CPU execution

### Why This Happens
1. Docker environments often have complex Python package installation sequences
2. ONNX Runtime silently falls back to CPU when CUDA provider fails to initialize
3. The current code doesn't validate provider functionality

### Long-term Solution
Consider adding provider validation to the backend code and improving installation documentation to prevent this common issue.

**Platform Background**: 20+ years platform technology strategy | Docker/GPU configuration expertise