Repository: mlcommons/inference
Issue: [BUG] ResNet50 TensorRT build hangs at 92% in NMSOptPlugin during MLCFlow Inference (CUDA 12.2 + A30)
Root cause: Environment Incompatibility

While attempting to benchmark ResNet50 under the Datacenter / Offline scenario using MLCFlow with NVIDIA implementation (TensorRT), the build hangs indefinitely during the compilation of the NMSOptPlugin.
This happens at [ 92%] with no CPU or GPU activity. It seems to hang at or right after building softmaxScore.cpp.o. No error is printed, only warnings.

OS: Ubuntu 22.04

I was following this docs:
https://docs.mlcommons.org/inference/benchmarks/image_classification/resnet50/#__tabbed_1_2

What I've done so far:

python3 -m venv mlc
source mlc/bin/activate
pip install mlcflow && mlc pull repo --url=mlcommons@cm4mlops --branch=mlperf-inference
mlc pull repo mlcommons@mlperf-automations --branch=dev
mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev
--model=resnet50
--implementation=nvidia
--framework=tensorrt
--category=datacenter
--scenario=Offline
--execution_mode=test
--device=cuda
--gpu_name=a30
--docker
--docker_cache=no
--quiet
--test_query_count=5000
--rerun
This is the freezing phase, I've waited for over an hour:
/home/mlcuser/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_851d6966/repo/closed/NVIDIA/code/plugin/NMSOptPlugin/src/nmsPluginOpt.h(136): warning https://github.com/mlcommons/inference/pull/997: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const __nv_bool , const __nv_bool , nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::plugin::DetectionOutputOptDynamic::configurePlugin" -- virtual function override intended?
void configurePlugin(const DynamicPluginTensorDesc in, int nbInputs, const DynamicPluginTensorDesc out,
^

/home/mlcuser/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_851d6966/repo/closed/NVIDIA/code/plugin/NMSOptPlugin/src/nmsPluginOpt.h:101:46: warning: ‘IPluginV2Ext’ is deprecated [-Wdeprecated-declarations]
101 | IPluginV2Ext* clone() const noexcept override;
| ^~~~~
/usr/include/x86_64-linux-gnu/NvInferRuntimePlugin.h:447:22: note: declared here
447 | class TRT_DEPRECATED IPluginV2Ext : public IPluginV2
| ^~~~~~~~~~~~
/home/mlcuser/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_851d6966/repo/closed/NVIDIA/code/plugin/NMSOptPlugin/src/nmsPluginOpt.h:158:95: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations]
158 | IPluginV2* createPlugin(const char* name, const PluginFieldCollection* fc) noexcept override;
| ^~~
/usr/include/x86_64-linux-gnu/NvInferRuntimePlugin.h:117:22: note: declared here
117 | class TRT_DEPRECATED IPluginV2
| ^~~~~~~~~
/home/mlcuser/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_851d6966/repo/closed/NVIDIA/code/plugin/NMSOptPlugin/src/nmsPluginOpt.h:160:112: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations]
160 | IPluginV2* deserializePlugin(const char* name, const void* serialData, size_t serialLength) noexcept override;
| ^~~
/usr/include/x86_64-linux-gnu/NvInferRuntimePlugin.h:117:22: note: declared here
117 | class TRT_DEPRECATED IPluginV2
| ^~~~~~~~~

Forcefully stopped:
Traceback (most recent call last):
File "/usr/local/bin/mlcr", line 8, in
sys.exit(mlcr())
File "/usr/local/lib/python3.8/dist-packages/mlc/main.py", line 86, in mlcr
main()
File "/usr/local/lib/python3.8/dist-packages/mlc/main.py", line 273, in main
res = method(run_args)
File "/usr/local/lib/python3.8/dist-packages/mlc/script_action.py", line 309, in run
return self.call_script_module_function("run", run_args)
File "/usr/local/lib/python3.8/dist-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 227, in run
r = self._run(i)
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1858, in _run
r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3322, in _call_run_deps
r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3495, in _run_deps
r = self.action_object.access(ii)
File "/usr/local/lib/python3.8/dist-packages/mlc/action.py", line 56, in access
result = method(options)
File "/usr/local/lib/python3.8/dist-packages/mlc/script_action.py", line 309, in run
return self.call_script_module_function("run", run_args)
File "/usr/local/lib/python3.8/dist-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 227, in run
r = self._run(i)
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1643, in _run
r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3322, in _call_run_deps
r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3495, in _run_deps
r = self.action_object.access(ii)
File "/usr/local/lib/python3.8/dist-packages/mlc/action.py", line 56, in access
result = method(options)
File "/usr/local/lib/python3.8/dist-packages/mlc/script_action.py", line 309, in run
return self.call_script_module_function("run", run_args)
File "/usr/local/lib/python3.8/dist-packages/mlc/script_action.py", line 244, in call_script_module_function
raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = build-mlperf-inference-server-nvidia, return code = 2)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.

Expected Behavior:
MLCFlow should finish building the plugin and proceed to inference.

I’d really appreciate any help or insights on this issue. Please let me know if more logs or environment details are needed.

Thanks in advance!

Hi @xdilshod , could you try with `_r5.0` instead of `_r5.0-dev` in the run command? This should run the recent Nvidia code from submission round 5.0.

I will also update the docs website regarding this.

Edit: Do you have information about the hardware usage through `htop` or `Nvidia-smi` at the time it got stuck? 

Hi, @anandhu-eng, thank you for the quick response!

Regarding hardware usage: yes, I had checked htop and nvidia-smi when the issue occurred — both showed no activity during the hang, which confirmed it was stuck.

After the freezing phase during the NMSOptPlugin build, I forcefully stopped the process and decided to clean the environment and start from scratch. Here's what I did:

python3 -m venv mlc
source mlc/bin/activate

pip install mlcflow && mlc pull repo --url=mlcommons@cm4mlops --branch=mlperf-inference
mlc pull repo mlcommons@mlperf-automations --branch=dev

Then, I ran the following command (slightly different from the documentation):
mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
  --model=resnet50 \
  --implementation=nvidia \
  --framework=tensorrt \
  --category=datacenter \
  --scenario=Offline \
  --execution_mode=valid \
  --division=closed \
  --compliance=yes \
  --device=cuda \
  --gpu_name=a30 \
  --docker \
  --docker_cache=no \
  --quiet \
  --rerun

It successfully built the Docker container and gave me access to an interactive shell inside the container. Then, from inside the container, I ran this command:
mlcr run-mlperf,inference,_full,_r5.0-dev \
  --model=resnet50 \
  --implementation=nvidia \
  --framework=tensorrt \
  --category=datacenter \
  --scenario=Offline \
  --execution_mode=valid \
  --division=closed \
  --compliance=yes \
  --device=cuda \
  --gpu_name=a30 \
  --quiet \
  --rerun

The tests ran successfully. The final results showed that both TEST01 and TEST04 passed:
+----------+----------+----------+------------+-----------------+---------------------------------+--------+--------+
|  Model   | Scenario | Accuracy | Throughput | Latency (in ms) | Power Efficiency (in samples/J) | TEST01 | TEST04 |
+----------+----------+----------+------------+-----------------+---------------------------------+--------+--------+
| resnet50 | Offline  |  76.078  |  17419.1   |        -        |                                 | passed | passed |
+----------+----------+----------+------------+-----------------+---------------------------------+--------+--------+

Everything seems to have worked as expected?

Here's the log files:
[mlperf_log_summary-TEST01.txt](https://github.com/user-attachments/files/21030890/mlperf_log_summary-TEST01.txt)

One question I have: since we're using the --docker flag and running everything inside a container, why do we need to set up the Python virtual environment and install mlcflow outside of Docker initially? Is that step only required for host-side orchestration?



> Everything seems to have worked as expected?

From the output, yes it's looking good:)

> One question I have: since we're using the --docker flag and running everything inside a container, why do we need to set up the Python virtual environment and install mlcflow outside of Docker initially? Is that step only required for host-side orchestration?

Yes, that’s correct. When we run the **MLCFlow automation command** to execute the entire workflow inside a Docker container, it still requires some orchestration steps to be performed on the host beforehand.

These steps include setting up the results directory, downloading the model and dataset(if specified) which then later is mounted in docker container being launched, and launching the container with the correct parameters. To enable this, we need both the **MLCFlow Python library** and the **mlperf-automations repository** available in the host environment.

These components are essential not just for inference, but for running any MLC script. So, in short — whether you're running inside Docker or directly on the host, you’ll need both the library and the repository present in the environment initiating the automation.