Repository: mlcommons/inference
Issue: Fail to build the docker for mlc command on Ubuntu 22.04
Root cause: Environment Incompatibility

I failed to run the mlc command on Ubuntu 22.04:

mlcr run-mlperf,inference,_find-performance,_full,_r4.1-dev \
   --model=resnet50 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=edge \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --docker --quiet \
   --test_query_count=5000 \
   --all_models=yes

Failed to resolve 'developer.download.nvidia.com' as below, but in fact I can access the link of developer.download.nvidia.com via firefox manually.

<pre>212.3 The following NEW packages will be installed:
212.3   libcublas-12-3 libcublas-dev-12-3
242.8 0 upgraded, 2 newly installed, 0 to remove and 87 not upgraded.
242.8 Need to get 514 MB of archives.
242.8 After this operation, 1577 MB of additional disk space will be used.
242.8 Ign:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  libcublas-12-3 12.3.4.1-1
262.8 Ign:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  libcublas-dev-12-3 12.3.4.1-1
282.9 Ign:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  libcublas-12-3 12.3.4.1-1
282.9 Ign:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  libcublas-dev-12-3 12.3.4.1-1
284.9 Ign:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  libcublas-12-3 12.3.4.1-1
284.9 Ign:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  libcublas-dev-12-3 12.3.4.1-1
308.9 Err:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  libcublas-12-3 12.3.4.1-1
308.9   Could not connect to developer.download.nvidia.com:443 (23.223.211.90), connection timed out Could not connect to developer.download.nvidia.com:443 (23.223.211.42), connection timed out
328.9 Err:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  libcublas-dev-12-3 12.3.4.1-1
328.9   Temporary failure resolving &apos;developer.download.nvidia.com&apos;
328.9 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/./libcublas-12-3_12.3.4.1-1_amd64.deb  Could not connect to developer.download.nvidia.com:443 (23.223.211.90), connection timed out Could not connect to developer.download.nvidia.com:443 (23.223.211.42), connection timed out
328.9 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/./libcublas-dev-12-3_12.3.4.1-1_amd64.deb  Temporary failure resolving &apos;developer.download.nvidia.com&apos;
328.9 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
------

 <font color="#A2734C">4 warnings found (use docker --debug to expand):</font>
 - FromAsCasing: &apos;as&apos; and &apos;FROM&apos; keywords&apos; casing do not match (line 6)
 - FromAsCasing: &apos;as&apos; and &apos;FROM&apos; keywords&apos; casing do not match (line 14)
 - FromAsCasing: &apos;as&apos; and &apos;FROM&apos; keywords&apos; casing do not match (line 53)
 - FromAsCasing: &apos;as&apos; and &apos;FROM&apos; keywords&apos; casing do not match (line 66)
Dockerfile.multi:32
--------------------
  31 |     COPY docker/common/install_tensorrt.sh install_tensorrt.sh
  32 | &gt;&gt;&gt; RUN bash ./install_tensorrt.sh \
  33 | &gt;&gt;&gt;     --TRT_VER=${TRT_VER} \
  34 | &gt;&gt;&gt;     --CUDA_VER=${CUDA_VER} \
  35 | &gt;&gt;&gt;     --CUDNN_VER=${CUDNN_VER} \
  36 | &gt;&gt;&gt;     --NCCL_VER=${NCCL_VER} \
  37 | &gt;&gt;&gt;     --CUBLAS_VER=${CUBLAS_VER} &amp;&amp; \
  38 | &gt;&gt;&gt;     rm install_tensorrt.sh
  39 |     
--------------------
ERROR: failed to solve: process &quot;/bin/bash -c bash ./install_tensorrt.sh     --TRT_VER=${TRT_VER}     --CUDA_VER=${CUDA_VER}     --CUDNN_VER=${CUDNN_VER}     --NCCL_VER=${NCCL_VER}     --CUBLAS_VER=${CUBLAS_VER} &amp;&amp;     rm install_tensorrt.sh&quot; did not complete successfully: exit code: 100
exit status 1
make: *** [Makefile:55: devel_build] Error 1
make: Leaving directory &apos;/home/bob2/MLC/repos/local/cache/get-git-repo_d790359e/repo/docker&apos;
Traceback (most recent call last):
  File &quot;/home/bob2/mlc/bin/mlcr&quot;, line 8, in &lt;module&gt;
    sys.exit(mlcr())
  File &quot;/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py&quot;, line 1715, in mlcr
    main()
  File &quot;/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py&quot;, line 1797, in main
    res = method(run_args)
  File &quot;/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py&quot;, line 1529, in run
    return self.call_script_module_function(&quot;run&quot;, run_args)
  File &quot;/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py&quot;, line 1509, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File &quot;/home/bob2/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py&quot;, line 225, in run
    r = self._run(i)
  File &quot;/home/bob2/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py&quot;, line 1768, in _run
    r = customize_code.preprocess(ii)
  File &quot;/home/bob2/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py&quot;, line 284, in preprocess
    r = mlc.access(ii)
  File &quot;/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py&quot;, line 92, in access
    result = method(options)
  File &quot;/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py&quot;, line 1526, in docker
    return self.call_script_module_function(&quot;docker&quot;, run_args)
  File &quot;/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py&quot;, line 1511, in call_script_module_function
    result = automation_instance.docker(run_args)  # Pass args to the run method
  File &quot;/home/bob2/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py&quot;, line 4684, in docker
    return docker_run(self, i)
  File &quot;/home/bob2/MLC/repos/mlcommons@mlperf-automations/automation/script/docker.py&quot;, line 308, in docker_run
    r = self_module._run_deps(
  File &quot;/home/bob2/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py&quot;, line 3695, in _run_deps
    r = self.action_object.access(ii)
  File &quot;/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py&quot;, line 92, in access
    result = method(options)
  File &quot;/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py&quot;, line 1529, in run
    return self.call_script_module_function(&quot;run&quot;, run_args)
  File &quot;/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py&quot;, line 1519, in call_script_module_function
    raise ScriptExecutionError(f&quot;Script {function_name} execution failed. Error : {error}&quot;)
mlc.main.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = get-ml-model-gptj, return code = 256)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.
</pre>

Can you please try the same command with `--docker_cache=no`? 

OK, I will try it and share the result later. Thanks.

some error promt as below but the the progress is still going on:

......
......
......
<pre><font color="#A2734C">Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.</font>
<font color="#A2734C">  WARNING: The script ammo-wf-exec is installed in &apos;/home/bob2/.local/bin&apos; which is not on PATH.</font>
<font color="#A2734C">  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.</font>
<font color="#A2734C">  WARNING: The script evaluate-cli is installed in &apos;/home/bob2/.local/bin&apos; which is not on PATH.</font>
<font color="#A2734C">  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.</font>
<font color="#C01C28">ERROR: pip&apos;s dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.</font>
<font color="#C01C28">dask-cuda 23.10.0 requires pynvml&lt;11.5,&gt;=11.0.0, but you have pynvml 12.0.0 which is incompatible.</font>
Successfully installed accelerate-0.25.0 bandit-1.7.7 build-1.2.2.post1 cfgv-3.4.0 colored-2.3.0 coloredlogs-15.0.1 coverage-7.6.12 datasets-3.3.2 diffusers-0.15.0 dill-0.3.8 distlib-0.3.9 evaluate-0.4.3 flatbuffers-25.2.10 graphviz-0.20.3 huggingface-hub-0.29.1 humanfriendly-10.0 identify-2.6.8 janus-2.0.0 lark-1.2.2 multiprocess-0.70.16 mypy-1.15.0 mypy_extensions-1.0.0 nltk-3.9.1 nodeenv-1.9.1 nvidia-ammo-0.7.4 nvidia-ml-py-12.570.86 onnx-graphsurgeon-0.5.5 onnxruntime-1.16.3 optimum-1.24.0 parameterized-0.9.0 pbr-6.1.1 pre-commit-4.1.0 py-1.11.0 pyarrow-19.0.1 pybind11-stubgen-2.5.3 pynvml-12.0.0 pyproject_hooks-1.2.0 pytest-cov-6.0.0 pytest-forked-1.6.0 requests-2.32.3 rouge_score-0.1.2 safetensors-0.5.2 sentencepiece-0.2.0 stevedore-5.4.1 tokenizers-0.15.2 tqdm-4.67.1 transformers-4.36.1 virtualenv-20.29.2 xxhash-3.5.0

<b>[</b><font color="#12488B">notice</font><b>]</b> A new release of pip is available: <font color="#C01C28">23.3.1</font> -&gt; <font color="#26A269">25.0.1</font>
<b>[</b><font color="#12488B">notice</font><b>]</b> To update, run: <font color="#26A269">python3 -m pip install --upgrade pip</font>
-- The CXX compiler identification is GNU 11.4.0
-- Detecting CXX compiler ABI info
</pre>
......
......
......
<pre>[ 97%] <font color="#26A269">Building CUDA object tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/weightOnlyBatchedGemv/weightOnlyBatchedGemvBs3Int4b.cu.o</font>
[ 97%] <font color="#26A269">Building CUDA object tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/weightOnlyBatchedGemv/weightOnlyBatchedGemvBs2Int4b.cu.o</font>
[ 97%] <font color="#26A269">Building CUDA object tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/weightOnlyBatchedGemv/weightOnlyBatchedGemvBs3Int8b.cu.o</font>
[ 97%] <font color="#26A269">Building CUDA object tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/weightOnlyBatchedGemv/weightOnlyBatchedGemvBs4Int8b.cu.o</font>
[ 98%] <font color="#26A269">Building CUDA object tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/weightOnlyBatchedGemv/weightOnlyBatchedGemvBs4Int4b.cu.o</font>
<b>/code/tensorrt_llm/cpp/tensorrt_llm/cutlass_extensions/include/cutlass_extensions/epilogue/threadblock/epilogue_tensor_op_int32.h(97)</b>: <font color="#C01C28"><b>error</b></font>: class template <b>&quot;cutlass::epilogue::threadblock::detail::DefaultIteratorsTensorOp&quot;</b> has already been defined
  struct DefaultIteratorsTensorOp&lt;cutlass::bfloat16_t, int32_t, 8, ThreadblockShape, WarpShape, InstructionShape,
         ^

1 error detected in the compilation of &quot;/code/tensorrt_llm/cpp/tensorrt_llm/kernels/cutlass_kernels/int8_gemm/int8_gemm_fp16.cu&quot;.
gmake[3]: *** [tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/build.make:12932: tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/cutlass_kernels/int8_gemm/int8_gemm_fp16.cu.o] Error 2
gmake[3]: *** Waiting for unfinished jobs....
<b>/code/tensorrt_llm/cpp/tensorrt_llm/cutlass_extensions/include/cutlass_extensions/epilogue/threadblock/epilogue_tensor_op_int32.h(97)</b>: <font color="#C01C28"><b>error</b></font>: class template <b>&quot;cutlass::epilogue::threadblock::detail::DefaultIteratorsTensorOp&quot;</b> has already been defined
  struct DefaultIteratorsTensorOp&lt;cutlass::bfloat16_t, int32_t, 8, ThreadblockShape, WarpShape, InstructionShape,
         ^

1 error detected in the compilation of &quot;/code/tensorrt_llm/cpp/tensorrt_llm/kernels/cutlass_kernels/int8_gemm/int8_gemm_bf16.cu&quot;.
gmake[3]: *** [tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/build.make:12917: tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/cutlass_kernels/int8_gemm/int8_gemm_bf16.cu.o] Error 2
<b>/code/tensorrt_llm/cpp/tensorrt_llm/cutlass_extensions/include/cutlass_extensions/epilogue/threadblock/epilogue_tensor_op_int32.h(97)</b>: <font color="#C01C28"><b>error</b></font>: class template <b>&quot;cutlass::epilogue::threadblock::detail::DefaultIteratorsTensorOp&quot;</b> has already been defined
  struct DefaultIteratorsTensorOp&lt;cutlass::bfloat16_t, int32_t, 8, ThreadblockShape, WarpShape, InstructionShape,
         ^

1 error detected in the compilation of &quot;/code/tensorrt_llm/cpp/tensorrt_llm/kernels/cutlass_kernels/int8_gemm/int8_gemm_int32.cu&quot;.
gmake[3]: *** [tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/build.make:12962: tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/cutlass_kernels/int8_gemm/int8_gemm_int32.cu.o] Error 2
<b>/code/tensorrt_llm/cpp/tensorrt_llm/cutlass_extensions/include/cutlass_extensions/epilogue/threadblock/epilogue_tensor_op_int32.h(97)</b>: <font color="#C01C28"><b>error</b></font>: class template <b>&quot;cutlass::epilogue::threadblock::detail::DefaultIteratorsTensorOp&quot;</b> has already been defined
  struct DefaultIteratorsTensorOp&lt;cutlass::bfloat16_t, int32_t, 8, ThreadblockShape, WarpShape, InstructionShape,
         ^

1 error detected in the compilation of &quot;/code/tensorrt_llm/cpp/tensorrt_llm/kernels/cutlass_kernels/int8_gemm/int8_gemm_fp32.cu&quot;.
gmake[3]: *** [tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/build.make:12947: tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/cutlass_kernels/int8_gemm/int8_gemm_fp32.cu.o] Error 2
[ 98%] Built target layers_src
[ 98%] Built target common_src
[ 98%] Built target runtime_src


</pre>

Quit after the above errors occurred.

mlcr run-mlperf,inference,_find-performance,_full,_r4.1-dev \
   --model=resnet50 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=edge \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --docker --quiet \
   --test_query_count=5000 \
   --all_models=yes \
   --docker_cache=no

oh. Which GPU are you running on?

oh, I made the same mistake again - mix the GPU with different model.

Thank you @arjunsuresh , I will remove one and try it later.

@arjunsuresh Unfortunately the complete same error as above happen and failed to build the docker.

please see the log here:

[mlc-log.txt](https://github.com/user-attachments/files/18975157/mlc-log.txt)