Repository: mlcommons/inference
Issue: The test result for the inference on 3d-unet-99 model seems abnormal.
Root cause: Environment Incompatibility

Hi @arjunsuresh,

it's successful to build the docker container by running the be below command but still report the errors pasted **at the end.**

mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
   --model=3d-unet-99 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=edge \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --docker --quiet \
   --test_query_count=50

In the new-created docker container, run the below command successfully and get the final result.

_mlcr run-mlperf,inference,_full,_r5.0-dev \
   --model=3d-unet-99 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=edge \
   --scenario=Offline \
   --execution_mode=valid \
   --device=cuda \
   --quiet_ 

<pre>+------------+----------+----------+------------+-----------------+
|   Model    | Scenario | Accuracy | Throughput | Latency (in ms) |
+------------+----------+----------+------------+-----------------+
| 3d-unet-99 | Offline  | 0.86234  |   2.049    |        -        |
+------------+----------+----------+------------+-----------------+

The MLPerf inference results are stored at /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_c369e3b3/valid_results

[2025-03-27 20:50:41,091 module.py:5256 <font color="#26A269">INFO</font>] -        ! call &quot;postprocess&quot; from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py
mlcuser@3e124278351a:~$</pre>

I have two questions:
1. I just want to figure out what the error message occurred during the docker building process is and why this error will not impact the following inference testing.
2. why the final result is so bad? Accuracy < 1% and Throughput = 2, that seems abnormal and how to improve it?


<pre> "the below log is for the docker building process"
......
......
......
0, GPU 2247 (MiB)
[I] Creating batcher thread: 0 EnableBatcherThreadPerDevice: true
Finished setting up SUT.
Starting warmup. Running for a minimum of 5 seconds.
Finished warmup. Ran for 5.83001s.
Starting running actual test.
================================================
MLPerf Results Summary
================================================
SUT name : Server_3DUNet
Scenario : Offline
Mode     : PerformanceOnly
Samples per second: 2.0352
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes

================================================
Additional Stats
================================================
Min latency (ns)                : 473988432
Max latency (ns)                : 24567628500
Mean latency (ns)               : 12511875273
50.00 percentile latency (ns)   : 12985068001
90.00 percentile latency (ns)   : 22595759637
95.00 percentile latency (ns)   : 23949121811
97.00 percentile latency (ns)   : 24327248185
99.00 percentile latency (ns)   : 24567628500
99.90 percentile latency (ns)   : 24567628500

================================================
Test Parameters Used
================================================
samples_per_query : 50
target_qps : 0.01
target_latency (ns): 0
max_async_queries : 1
min_duration (ms): 0
max_duration (ms): 0
min_query_count : 1
max_query_count : 50
qsl_rng_seed : 6023615788873153749
sample_index_rng_seed : 15036839855038426416
schedule_rng_seed : 9933818062894767841
accuracy_log_rng_seed : 0
accuracy_log_probability : 0
accuracy_log_sampling_target : 0
print_timestamps : 0
performance_issue_unique : 0
performance_issue_same : 0
performance_issue_same_index : 0
performance_sample_count : 43

No warnings encountered during test.

No errors encountered during test.
Finished running actual test.
Device Device:0 processed:
  9 batches of size 2
  2 batches of size 3
  8 batches of size 4
  4 batches of size 5
  400 batches of size 8
  Memcpy Calls: 0
  PerSampleCudaMemcpy Calls: 50
  BatchedCudaMemcpy Calls: 0
&amp;&amp;&amp;&amp; PASSED MLPerf_Inference_3DUNet_Harness # ./build/bin/harness_3dunet
[2025-03-27 20:00:08,401 run_harness.py:166 INFO] Result: result_samples_per_second: 2.0352, Result is VALID
 
======================== Result summaries: ========================

[2025-03-27 20:00:08,831 module.py:5256 <font color="#26A269">INFO</font>] -          ! call &quot;postprocess&quot; from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/benchmark-program/customize.py
[2025-03-27 20:00:08,832 module.py:5319 DEBUG] -     - Running postprocess ...
[2025-03-27 20:00:08,835 module.py:2200 <font color="#26A269">INFO</font>] -     - running time of script &quot;program,benchmark,benchmark
[2025-03-27 20:27:08,646 index.py:177 DEBUG] - Shared index for script saved to /home/mlcuser/MLC/repos/index_script.json.
[2025-03-27 20:27:08,647 index.py:177 DEBUG] - Shared index for cache saved to /home/mlcuser/MLC/repos/index_cache.json.
[2025-03-27 20:27:08,647 index.py:177 DEBUG] - Shared index for experiment saved to /home/mlcuser/MLC/repos/index_experiment.json.
[2025-03-27 20:27:08,651 index.py:177 DEBUG] - Shared index for script saved to /home/mlcuser/MLC/repos/index_script.json.
[2025-03-27 20:27:08,652 index.py:177 DEBUG] - Shared index for cache saved to /home/mlcuser/MLC/repos/index_cache.json.
[2025-03-27 20:27:08,652 index.py:177 DEBUG] - Shared index for experiment saved to /home/mlcuser/MLC/repos/index_experiment.json.
[2025-03-27 20:27:08,652 module.py:1402 DEBUG] -     - Changing to /home/mlcuser/MLC/repos/local/cache/get-cache-dir_e3afe6f0
[2025-03-27 20:27:08,653 module.py:1762 DEBUG] -     - Running preprocess ...
[2025-03-27 20:27:08,659 module.py:5256 <font color="#26A269">INFO</font>] -          ! call &quot;postprocess&quot; from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-cache-dir/customize.py
[2025-03-27 20:27:08,660 module.py:5319 DEBUG] -     - Running postprocess ...
[2025-03-27 20:27:08,673 module.py:2014 DEBUG] -     - Removing tmp tag in the script cached output e3afe6f077774eed ...
[2025-03-27 20:27:08,678 index.py:177 DEBUG] - Shared index for script saved to /home/mlcuser/MLC/repos/index_script.json.
[2025-03-27 20:27:08,679 index.py:177 DEBUG] - Shared index for cache saved to /home/mlcuser/MLC/repos/index_cache.json.
[2025-03-27 20:27:08,679 index.py:177 DEBUG] - Shared index for experiment saved to /home/mlcuser/MLC/repos/index_experiment.json.
[2025-03-27 20:27:08,679 module.py:2158 <font color="#26A269">INFO</font>] -     - cache UID: e3afe6f077774eed
[2025-03-27 20:27:08,679 module.py:2200 <font color="#26A269">INFO</font>] -     - running time of script &quot;get,cache,dir,directory&quot;: 0.05 sec.
[2025-03-27 20:27:08,680 module.py:1640 DEBUG] -   - Processing env after dependencies ...
[2025-03-27 20:27:08,680 module.py:1762 DEBUG] -   - Running preprocess ...
Generating SUT description file for 3e124278351a-tensorrt
HW description file for 3e124278351a not found. Copying from default!!!
[2025-03-27 20:27:08,687 module.py:5256 <font color="#26A269">INFO</font>] -        ! call &quot;postprocess&quot; from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-sut-description/customize.py
[2025-03-27 20:27:08,687 module.py:5319 DEBUG] -   - Running postprocess ...
[2025-03-27 20:27:08,692 module.py:2200 <font color="#26A269">INFO</font>] -   - running time of script &quot;get,mlperf,sut,description,system-under-test,system-description&quot;: 1619.82 sec.
[2025-03-27 20:27:08,693 module.py:5319 DEBUG] -   - Running postprocess ...

......
......
......



[2025-03-27 20:27:18,705 module.py:1640 DEBUG] -     - Processing env after dependencies ...
[2025-03-27 20:27:18,705 module.py:1762 DEBUG] -     - Running preprocess ...
[2025-03-27 20:27:18,712 module.py:5102 DEBUG] -         - Running native script &quot;/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/run.sh&quot; from temporal script &quot;tmp-run.sh&quot; in &quot;/home/mlcuser/MLC/repos/local/cache/get-generic-python-lib_c7805511&quot; ...
[2025-03-27 20:27:18,713 module.py:5109 <font color="#26A269">INFO</font>] -              ! cd /home/mlcuser/MLC/repos/local/cache/get-generic-python-lib_c7805511
[2025-03-27 20:27:18,713 module.py:5110 <font color="#26A269">INFO</font>] -              ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/run.sh from tmp-run.sh
[2025-03-27 20:27:18,810 module.py:5256 <font color="#26A269">INFO</font>] -              ! call &quot;detect_version&quot; from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py

        Extra PIP CMD: 

[2025-03-27 20:27:18,820 module.py:5102 DEBUG] -     - Running native script &quot;/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/install.sh&quot; from temporal script &quot;tmp-run.sh&quot; in &quot;/home/mlcuser/MLC/repos/local/cache/get-generic-python-lib_c7805511&quot; ...
[2025-03-27 20:27:18,820 module.py:5109 <font color="#26A269">INFO</font>] -          ! cd /home/mlcuser/MLC/repos/local/cache/get-generic-python-lib_c7805511
[2025-03-27 20:27:18,820 module.py:5110 <font color="#26A269">INFO</font>] -          ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/install.sh from tmp-run.sh

/usr/bin/python3 -m pip install &quot;matplotlib&quot;
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host &apos;pypi.ngc.nvidia.com&apos;. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
Collecting matplotlib
  Downloading matplotlib-3.7.5-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.2 MB)
     |████████████████████████████████| 9.2 MB 634 kB/s 
Requirement already satisfied: packaging&gt;=20.0 in /home/mlcuser/.local/lib/python3.8/site-packages (from matplotlib) (24.2)
/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host &apos;pypi.ngc.nvidia.com&apos;. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
Collecting kiwisolver&gt;=1.0.1
  Downloading kiwisolver-1.4.7-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
     |████████████████████████████████| 1.2 MB 596 kB/s 
Requirement already satisfied: importlib-resources&gt;=3.2.0; python_version &lt; &quot;3.10&quot; in /home/mlcuser/.local/lib/python3.8/site-packages (from matplotlib) (6.4.5)
Requirement already satisfied: numpy&lt;2,&gt;=1.20 in /home/mlcuser/.local/lib/python3.8/site-packages (from matplotlib) (1.23.5)
Requirement already satisfied: python-dateutil&gt;=2.7 in /home/mlcuser/.local/lib/python3.8/site-packages (from matplotlib) (2.9.0.post0)
Requirement already satisfied: pillow&gt;=6.2.0 in /home/mlcuser/.local/lib/python3.8/site-packages (from matplotlib) (10.4.0)
/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host &apos;pypi.ngc.nvidia.com&apos;. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
Collecting fonttools&gt;=4.22.0
  Downloading fonttools-4.56.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
     |████████████████████████████████| 4.7 MB 583 kB/s 
/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host &apos;pypi.ngc.nvidia.com&apos;. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
Collecting cycler&gt;=0.10
  Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB)
/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host &apos;pypi.ngc.nvidia.com&apos;. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
Collecting contourpy&gt;=1.0.1
  Downloading contourpy-1.1.1-cp38-cp38-manylinux_2_17_x[2025-03-27 20:27:08,189 module.py:1250 DEBUG] -           - Checking prehook dependencies on other MLC scripts:
[2025-03-27 20:27:08,189 module.py:1263 DEBUG] -             - Loading state from cached entry ...
[2025-03-27 20:27:08,189 module.py:1276 <font color="#26A269">INFO</font>] -            ! load /home/mlcuser/MLC/repos/local/cache/get-python3_2ae8ade1/mlc-cached-state.json
[2025-03-27 20:27:08,189 module.py:1311 DEBUG] -           - Checking posthook dependencies on other MLC scripts:
[2025-03-27 20:27:08,189 module.py:1324 DEBUG] -           - Checking post dependencies on other MLC scripts:
......
......
......
automations/script/draw-graph-from-json-data/run.sh from tmp-run.sh
Running: 
/usr/bin/python3 &apos;/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/draw-graph-from-json-data/process-mlc-deps.py&apos; &apos;/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_c369e3b3/test_results/3e124278351a-nvidia_original-gpu-tensorrt-vdefault-default_config/3d-unet-99/offline/performance/run_1/mlc-version-info.json&apos;  --output_image &apos;/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_c369e3b3/test_results/3e124278351a-nvidia_original-gpu-tensorrt-vdefault-default_config/3d-unet-99/offline/performance/run_1/mlc-deps.png&apos;  --output_mermaid &apos;/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_c369e3b3/test_results/3e124278351a-nvidia_original-gpu-tensorrt-vdefault-default_config/3d-unet-99/offline/performance/run_1/mlc-deps.mmd&apos; 

Graph visualization saved as /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_c369e3b3/test_results/3e124278351a-nvidia_original-gpu-tensorrt-vdefault-default_config/3d-unet-99/offline/performance/run_1/mlc-deps.png
Mermaid syntax saved to /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_c369e3b3/test_results/3e124278351a-nvidia_original-gpu-tensorrt-vdefault-default_config/3d-unet-99/offline/performance/run_1/mlc-deps.mmd
Graph data saved as graph.graphml
[2025-03-27 20:28:06,065 module.py:5256 <font color="#26A269">INFO</font>] -        ! call &quot;postprocess&quot; from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/draw-graph-from-json-data/customize.py
[2025-03-27 20:28:06,065 module.py:5319 DEBUG] -   - Running postprocess ...
[2025-03-27 20:28:06,071 module.py:2200 <font color="#26A269">INFO</font>] -   - running time of script &quot;draw,graph,from-json,from-json-data&quot;: 56.27 sec.
[2025-03-27 20:28:06,072 module.py:3638 <font color="#26A269">INFO</font>] - mlcr detect,os
[2025-03-27 20:28:06,073 module.py:3638 <font color="#26A269">INFO</font>] - mlcr get,sys-utils-cm
[2025-03-27 20:28:06,073 module.py:3638 <font color="#26A269">INFO</font>] - mlcr get,python
[2025-03-27 20:28:06,073 module.py:3638 <font color="#26A269">INFO</font>] - mlcr get,mlcommons,inference,src
[2025-03-27 20:28:06,073 module.py:3638 <font color="#26A269">INFO</font>] - mlcr get,mlperf,inference,utils
[2025-03-27 20:28:06,073 module.py:3638 <font color="#26A269">INFO</font>] - mlcr get,cuda-devices,_with-pycuda
[2025-03-27 20:28:06,073 module.py:3638 <font color="#26A269">INFO</font>] - mlcr reproduce,mlperf,nvidia,inference,_run_harness,_cuda,_tensorrt,_offline,_3d-unet-99,_gpu_memory.48
[2025-03-27 20:28:06,073 module.py:3638 <font color="#26A269">INFO</font>] - mlcr get,mlperf,sut,description
[2025-03-27 20:28:06,073 module.py:3638 <font color="#26A269">INFO</font>] - mlcr draw,graph,from-json
[2025-03-27 20:28:06,076 module.py:2200 <font color="#26A269">INFO</font>] -   - running time of script &quot;app,vision,language,mlcommons,mlperf,inference,generic&quot;: 10887.38 sec.
mlcuser@3e124278351a:~$ mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
&gt;    --model=3d-unet-99 \
&gt;    --implementation=nvidia \
&gt;    --framework=tensorrt \
&gt;    --category=edge \
&gt;    --scenario=Offline \
&gt;    --execution_mode=test \
&gt;    --device=cuda  \
&gt;    --docker --quiet \
&gt;    --test_query_count=50mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
&gt;    --model=3d-unet-99 \
&gt;    --implementation=nvidia \
&gt;    --framework=tensorrt \
&gt;    --category=edge \
&gt;    --scenario=Offline \
&gt;    --execution_mode=test \
&gt;    --device=cuda  \
&gt;    --docker --quiet \
&gt;    --test_query_count=50mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
&gt;    --model=3d-unet-99 \
&gt;    --implementation=nvidia \
&gt;    --framework=tensorrt \
&gt;    --category=edge \
&gt;    --scenario=Offline \
&gt;    --execution_mode=test \
&gt;    --device=cuda  \
&gt;    --docker --quiet \
&gt;    --test_query_count=50mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
&gt;    --model=3d-unet-99 \
&gt;    --implementation=nvidia \
&gt;    --framework=tensorrt \
&gt;    --category=edge \
&gt;    --scenario=Offline \
&gt;    --execution_mode=test \
&gt;    --device=cuda  \
&gt;    --docker --quiet \
&gt;    --test_query_count=50mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
&gt;    --model=3d-unet-99 \
&gt;    --implementation=nvidia \
&gt;    --framework=tensorrt \
&gt;    --category=edge \
&gt;    --scenario=Offline \
&gt;    --execution_mode=test \
&gt;    --device=cuda  \
&gt;    --docker --quiet \
&gt;    --test_query_count=50mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
&gt;    --model=3d-unet-99 \
&gt;    --implementation=nvidia \
&gt;    --framework=tensorrt \
&gt;    --category=edge \
&gt;    --scenario=Offline \
&gt;    --execution_mode=test \
&gt;    --device=cuda  \
&gt;    --docker --quiet \
&gt;    --test_query_count=50mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
&gt;    --model=3d-unet-99 \
&gt;    --implementation=nvidia \
&gt;    --framework=tensorrt \
&gt;    --category=edge \
&gt;    --scenario=Offline \
&gt;    --execution_mode=test \
&gt;    --device=cuda  \
&gt;    --docker --quiet \
&gt;    --test_query_count=50mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
&gt;    --model=3d-unet-99 \
&gt;    --implementation=nvidia \
&gt;    --framework=tensorrt \
&gt;    --category=edge \
&gt;    --scenario=Offline \
&gt;    --execution_mode=test \
&gt;    --device=cuda  \
&gt;    --docker --quiet \
&gt;    --test_query_count=5013522989755
[2025-03-27 20:28:07,460 module.py:561 <font color="#26A269">INFO</font>] - * mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev
[2025-03-27 20:28:07,475 module.py:561 <font color="#26A269">INFO</font>] -   * mlcr get,mlcommons,inference,src
[2025-03-27 20:28:07,475 module.py:1276 <font color="#26A269">INFO</font>] -        ! load /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-src_9b6bb300/mlc-cached-state.json
[2025-03-27 20:28:07,484 module.py:561 <font color="#26A269">INFO</font>] -   * mlcr get,mlperf,inference,results,dir,_version.r5.0-dev
[2025-03-27 20:28:07,485 module.py:1276 <font color="#26A269">INFO</font>] -        ! load /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_c369e3b3/mlc-cached-state.json
[2025-03-27 20:28:07,491 module.py:561 <font color="#26A269">INFO</font>] -   * mlcr install,pip-package,for-mlc-python,_package.tabulate
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: tabulate in /home/mlcuser/.local/lib/python3.8/site-packages (0.9.0)
[2025-03-27 20:28:09,145 module.py:5256 <font color="#26A269">INFO</font>] -          ! call &quot;postprocess&quot; from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/install-pip-package-for-mlc-python/customize.py
[2025-03-27 20:28:09,165 module.py:561 <font color="#26A269">INFO</font>] -   * mlcr get,mlperf,inference,utils
[2025-03-27 20:28:09,179 module.py:561 <font color="#26A269">INFO</font>] -     * mlcr get,mlperf,inference,src
[2025-03-27 20:28:09,180 module.py:1276 <font color="#26A269">INFO</font>] -          ! load /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-src_9b6bb300/mlc-cached-state.json
[2025-03-27 20:28:09,182 module.py:5256 <font color="#26A269">INFO</font>] -          ! call &quot;postprocess&quot; from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-utils/customize.py
Using MLCommons Inference source from /home/mlcuser/MLC/repos/local/cache/get-git-repo_0a774c1a/inference

Running loadgen scenario: Offline and mode: performance
[2025-03-27 20:28:09,282 module.py:561 <font color="#26A269">INFO</font>] - * mlcr detect,os
[2025-03-27 20:28:09,284 module.py:5109 <font color="#26A269">INFO</font>] -        ! cd /home/mlcuser
[2025-03-27 20:28:09,284 module.py:5110 <font color="#26A269">INFO</font>] -        ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-03-27 20:28:09,320 module.py:5256 <font color="#26A269">INFO</font>] -        ! call &quot;postprocess&quot; from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-03-27 20:28:09,340 module.py:561 <font color="#26A269">INFO</font>] - * mlcr build,dockerfile
[2025-03-27 20:28:09,350 module.py:561 <font color="#26A269">INFO</font>] -   * mlcr get,docker
[2025-03-27 20:28:09,366 module.py:561 <font color="#26A269">INFO</font>] -     * mlcr detect,os
[2025-03-27 20:28:09,369 module.py:5109 <font color="#26A269">INFO</font>] -            ! cd /home/mlcuser/MLC/repos/local/cache/get-docker_4530afcd
[2025-03-27 20:28:09,369 module.py:5110 <font color="#26A269">INFO</font>] -            ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-03-27 20:28:09,406 module.py:5256 <font color="#26A269">INFO</font>] -            ! call &quot;postprocess&quot; from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-03-27 20:28:09,439 module.py:5109 <font color="#26A269">INFO</font>] -          ! cd /home/mlcuser/MLC/repos/local/cache/get-docker_4530afcd
[2025-03-27 20:28:09,439 module.py:5110 <font color="#26A269">INFO</font>] -          ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-docker/install-ubuntu.sh from tmp-run.sh
Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease                                      
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB]                   
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB]                                
Get:5 http://security.ubuntu.com/ubuntu focal-security/ms already cached ...ultiverse amd64 Packages [33.2 kB]                       
Get:4 https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64  InRelease [1581 B]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB]                 
Get:7 https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64  Packages [2068 kB]
Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [4291 kB]     
Get:9 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [36.8 kB]         
Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1597 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [4688 kB]
Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1304 kB]    
Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [4491 kB]  
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [4777 kB]           
Fetched 23.7 MB in 5s (4425 kB/s)                                                             
Reading package lists... Done
sudo apt-get install -y ca-certificates curl gnupg
Reading package lists... Done
Building dependency tree       
Reading state information... Done
curl is already the newest version (7.68.0-1ubuntu2.25).
gnupg is already the newest version (2.2.19-3ubuntu2.2).
gnupg set to manually installed.
The following packages will be upgraded:
  ca-certificates
1 upgraded, 0 newly installed, 0 to remove and 76 not upgraded.
Need to get 159 kB of archives.
After this operation, 9216 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 ca-certificates all 20240203~20.04.1 [159 kB]
Fetched 159 kB in 2s (90.4 kB/s)          
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, &lt;&gt; line 1.)
debconf: falling back to frontend: Readline
Preconfiguring packages ...
(Reading database ... 47541 files and directories currently installed.)
Preparing to unpack .../ca-certificates_20240203~20.04.1_all.deb ...
Unpacking ca-certificates (20240203~20.04.1) over (20230311ubuntu0.20.04.1) ...
Setting up ca-certificates (20240203~20.04.1) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
14 added, 5 removed; done.
Processing triggers for ca-certificates (20240203~20.04.1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
Ign:1 https://download.docker.com/linux/ubuntu focal InRelease
Err:2 https://download.docker.com/linux/ubuntu focal Release                                                     
  Could not handshake: Error in the pull function. [IP: 18.172.31.124 443]
Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease                                                           
Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease                                                 
Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease                      
Hit:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:7 https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64  InRelease
Reading package lists... Done
E: The repository &apos;https://download.docker.com/linux/ubuntu focal Release&apos; does not have a Release file.
N: Updating from such a repository can&apos;t be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package docker-ce is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package &apos;docker-ce&apos; has no installation candidate
E: Unable to locate package docker-ce-cli
E: Unable to locate package containerd.io
E: Couldn&apos;t find any package by glob &apos;containerd.io&apos;
E: Couldn&apos;t find any package by regex &apos;containerd.io&apos;
E: Unable to locate package docker-buildx-plugin
E: Unable to locate package docker-compose-plugin
Traceback (most recent call last):
  File &quot;/home/mlcuser/.local/bin/mlcr&quot;, line 8, in &lt;module&gt;
    sys.exit(mlcr())
  File &quot;/home/mlcuser/.local/lib/python3.8/site-packages/mlc/main.py&quot;, line 86, in mlcr
    main()
  File &quot;/home/mlcuser/.local/lib/python3.8/site-packages/mlc/main.py&quot;, line 173, in main
    res = method(run_args)
  File &quot;/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py&quot;, line 141, in run
    return self.call_script_module_function(&quot;run&quot;, run_args)
  File &quot;/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py&quot;, line 121, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File &quot;/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py&quot;, line 225, in run
    r = self._run(i)
  File &quot;/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py&quot;, line 1776, in _run
    r = customize_code.preprocess(ii)
  File &quot;/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py&quot;, line 284, in preprocess
    r = mlc.access(ii)
  File &quot;/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py&quot;, line 56, in access
    result = method(options)
  File &quot;/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py&quot;, line 138, in docker
    return self.call_script_module_function(&quot;docker&quot;, run_args)
  File &quot;/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py&quot;, line 123, in call_script_module_function
    result = automation_instance.docker(run_args)  # Pass args to the run method
  File &quot;/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py&quot;, line 4460, in docker
    return docker_run(self, i)
  File &quot;/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/docker.py&quot;, line 226, in docker_run
    r = dockerfile(self_module, i)
  File &quot;/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/docker.py&quot;, line 187, in dockerfile
    dockerfile_result = self_module.action_object.access(mlc_docker_input)
  File &quot;/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py&quot;, line 56, in access
    result = method(options)
  File &quot;/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py&quot;, line 141, in run
    return self.call_script_module_function(&quot;run&quot;, run_args)
  File &quot;/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py&quot;, line 121, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File &quot;/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py&quot;, line 225, in run
    r = self._run(i)
  File &quot;/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py&quot;, line 1634, in _run
    r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File &quot;/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py&quot;, line 3301, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File &quot;/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py&quot;, line 3471, in _run_deps
    r = self.action_object.access(ii)
  File &quot;/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py&quot;, line 56, in access
    result = method(options)So I just want to figure out what the error message occurred during the docker building process is and why this error will not impact the following valid inference testing.
  File &quot;/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py&quot;, line 141, in run
    return self.call_script_module_function(&quot;run&quot;, run_args)
  File &quot;/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py&quot;, line 131, in call_script_module_function
    raise ScriptExecutionError(f&quot;Script {function_name} execution failed. Error : {error}&quot;)
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = get-docker, return code = 256)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along witSo I just want to figure out what the error message occurred during the docker building process is and why this error will not impact the following valid inference testing.h the full MLC command being run and the relevant
or full console log.

mlcuser@3e124278351a:~$ 

</pre>





I'm having the same issue, have you found the cause of this issue