Repository: mlcommons/inference
Issue: GPT-J_NVIDIA_TensorRT_Docker environment Failure
Root cause: Environment Incompatibility

I am running the below command :
```
mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
   --model=gptj-99 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=edge \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --docker --quiet \
   --test_query_count=50 –rerun
```

Attached the Log with the error .
[mlperf_inference_GPT_J.txt](https://github.com/user-attachments/files/21589959/mlperf_inference_GPT_J.txt)

As directed in the MLcommons documentation running in the python environment.
I am trying to test NVIDIA RTX 3070
Host station details:
Ubuntu 20.04

<img width="616" height="298" alt="Image" src="https://github.com/user-attachments/assets/aa3f5023-1c97-4ced-8be4-4afd8baa8b3e" />
Please let me know what is the issue and how to solve it?
Similarly I tried running Resnet as well, I am seeing an issue there as well where the Git pull command is failing.
Is there a FAQ or list of issues repository which I can refer to resolve such issues?




@barsha-barua-sndk can you please do `mlc pull repo` and try the run command again? Please share the error with resnet50 in a new issue as that is supposed to work without any issue. Sometimes retries are needed if package fetch fails inside docker.

**I did mlc pull repo**
Also note i created a virtual env with 3.9 version , still gives the below error
Now it fails with below:
/home/qa/mlc/bin/python3 -m pip install "networkx" --extra-index-url https://download.pytorch.org/whl/cpu
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
Collecting networkx
  Using cached https://download.pytorch.org/whl/networkx-3.2.1-py3-none-any.whl (1.6 MB)
ERROR: Package 'networkx' requires a different Python: 3.8.10 not in '>=3.9'
Traceback (most recent call last):
  File "/home/qa/mlc_1/bin/mlcr", line 8, in <module>
    sys.exit(mlcr())
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/main.py", line 86, in mlcr
    main()
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/main.py", line 273, in main
    res = method(run_args)
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/script_action.py", line 311, in run
    return self.call_script_module_function("run", run_args)
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/script_action.py", line 230, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
    r = self._run(i)
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1780, in _run
    r = customize_code.preprocess(ii)
  File "/root/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py", line 285, in preprocess
    r = mlc.access(ii)
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/script_action.py", line 288, in docker
    return self.call_script_module_function("docker", run_args)
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/script_action.py", line 232, in call_script_module_function
    result = automation_instance.docker(run_args)  # Pass args to the run method
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 4573, in docker
    return docker_run(self, i)
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/docker.py", line 338, in docker_run
    r = self_module._run_deps(
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3552, in _run_deps
    r = self.action_object.access(ii)
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/script_action.py", line 311, in run
    return self.call_script_module_function("run", run_args)
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/script_action.py", line 230, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
    r = self._run(i)
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1638, in _run
    r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3360, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3552, in _run_deps
    r = self.action_object.access(ii)
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/script_action.py", line 311, in run
    return self.call_script_module_function("run", run_args)
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/script_action.py", line 230, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
    r = self._run(i)
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1638, in _run
    r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3360, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3552, in _run_deps
    r = self.action_object.access(ii)
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/script_action.py", line 311, in run
    return self.call_script_module_function("run", run_args)
  File "/home/qa/mlc_1/lib/python3.9/site-packages/mlc/script_action.py", line 246, in call_script_module_function
    raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = get-generic-python-lib, return code = 256)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.





Hi @arjunsuresh  , could you please look into the error log and update what is the issue?
I have tried virtual environment with 3.8,3.9,3.10 .
With 3.8 it throws an error "ERROR: Package 'networkx' requires a different Python: 3.8.10 not in '>=3.9'"
while I try with 3.9 and 3.10 below error is seen
x86_64-linux-gnu-g++ -pthread -Wno-unused-result -Wsign-compare -fwrapv -Wall -O3 -DNDEBUG -shared -Wl,-O1 -Wl,-Bsymbolic-functions /root/MLC/repos/local/cache/install-cuda-prebuilt_b949f23d/install/lib64 build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/converter/arg_to_python_base.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/converter/builtin_converters.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/converter/from_python.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/converter/registry.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/converter/type_id.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/dict.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/errors.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/exec.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/import.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/list.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/long.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/module.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/numeric.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/object/class.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/object/enum.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/object/function.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/object/function_doc_signature.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/object/inheritance.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/object/iterator.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/object/life_support.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/object/pickle_support.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/object/stl_iterator.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/object_operators.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/object_protocol.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/slice.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/str.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/tuple.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/python/src/wrapper.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/smart_ptr/src/sp_collector.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/smart_ptr/src/sp_debug_hooks.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/system/src/error_code.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/thread/src/pthread/once.o build/temp.linux-x86_64-cpython-310/bpl-subset/bpl_subset/libs/thread/src/pthread/thread.o build/temp.linux-x86_64-cpython-310/src/cpp/bitlog.o build/temp.linux-x86_64-cpython-310/src/cpp/cuda.o build/temp.linux-x86_64-cpython-310/src/wrapper/mempool.o build/temp.linux-x86_64-cpython-310/src/wrapper/wrap_cudadrv.o build/temp.linux-x86_64-cpython-310/src/wrapper/wrap_curand.o -L/usr/lib -L/usr/lib64 -L/usr/lib/stubs -L/usr/lib64/stubs -L/usr/lib -L/usr/lib64 -L/usr/lib/stubs -L/usr/lib64/stubs -L/usr/lib/x86_64-linux-gnu -lcuda -lcurand -o build/lib.linux-x86_64-cpython-310/pycuda/_driver.cpython-310-x86_64-linux-gnu.so
      /usr/bin/ld: cannot find /root/MLC/repos/local/cache/install-cuda-prebuilt_b949f23d/install/lib64: file format not recognized
      collect2: error: ld returned 1 exit status
      error: command '/usr/bin/x86_64-linux-gnu-g++' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pycuda
Failed to build pycuda
ERROR: Could not build wheels for pycuda, which is required to install pyproject.toml-based projects

[notice] A new release of pip is available: 23.0.1 -> 25.2
[notice] To update, run: pip install --upgrade pip
Traceback (most recent call last):
  File "/home/qa/mlc/bin/mlcr", line 8, in <module>
    sys.exit(mlcr())
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/main.py", line 86, in mlcr
    main()
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/main.py", line 273, in main
    res = method(run_args)
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 311, in run
    return self.call_script_module_function("run", run_args)
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 230, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
    r = self._run(i)
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1780, in _run
    r = customize_code.preprocess(ii)
  File "/root/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py", line 285, in preprocess
    r = mlc.access(ii)
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 288, in docker
    return self.call_script_module_function("docker", run_args)
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 232, in call_script_module_function
    result = automation_instance.docker(run_args)  # Pass args to the run method
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 4573, in docker
    return docker_run(self, i)
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/docker.py", line 338, in docker_run
    r = self_module._run_deps(
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3552, in _run_deps
    r = self.action_object.access(ii)
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 311, in run
    return self.call_script_module_function("run", run_args)
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 230, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
    r = self._run(i)
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1638, in _run
    r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3360, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3552, in _run_deps
    r = self.action_object.access(ii)
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 311, in run
    return self.call_script_module_function("run", run_args)
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 230, in call_script_module_function
    result = automation_instance.run(run_args)  # Pass args to the run method
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 240, in run
    r = self._run(i)
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1638, in _run
    r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3360, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/root/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3552, in _run_deps
    r = self.action_object.access(ii)
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/action.py", line 56, in access
    result = method(options)
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 311, in run
    return self.call_script_module_function("run", run_args)
  File "/home/qa/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 246, in call_script_module_function
    raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = get-generic-python-lib, return code = 256)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.