Repository: mlcommons/inference
Issue:  CM error: automation script not found!
Root cause: Configuration Error

Hi,
I've followed [this instruction](https://docs.mlcommons.org/inference/benchmarks/image_classification/resnet50/#__tabbed_8_2) to test the inference time of resnet50. However, I've encountered this error:
```
(cm) t@p:~$ cm
cm {action} {automation} {artifact(s)} {--flags} @input.yaml @input.json
(cm) t@p:~$ pip install cm4mlops
Requirement already satisfied: cm4mlops in ./cm/lib/python3.12/site-packages (0.2)
Requirement already satisfied: setuptools>=60 in ./cm/lib/python3.12/site-packages (from cm4mlops) (75.1.0)
Requirement already satisfied: wheel in ./cm/lib/python3.12/site-packages (from cm4mlops) (0.44.0)
Requirement already satisfied: cmind in ./cm/lib/python3.12/site-packages (from cm4mlops) (2.3.7)
Requirement already satisfied: giturlparse in ./cm/lib/python3.12/site-packages (from cm4mlops) (0.12.0)
Requirement already satisfied: requests in ./cm/lib/python3.12/site-packages (from cm4mlops) (2.32.3)
Requirement already satisfied: pyyaml in ./cm/lib/python3.12/site-packages (from cm4mlops) (6.0.2)
Requirement already satisfied: charset-normalizer<4,>=2 in ./cm/lib/python3.12/site-packages (from requests->cm4mlops) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./cm/lib/python3.12/site-packages (from requests->cm4mlops) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./cm/lib/python3.12/site-packages (from requests->cm4mlops) (2.2.3)
Requirement already satisfied: certifi>=2017.4.17 in ./cm/lib/python3.12/site-packages (from requests->cm4mlops) (2024.8.30)
(cm) t@p:~$ cm run script --tags=install,python-venv --name=mlperf

CM error: automation script not found!
```
Can you please give suggestions? Thank you for your help.

Hi, did you do `pip install cmind` as well? Looks like the installed `cm` is from a different python than the one `pip` is using. You can do `cm pull repo mlcommons@cm4mlops --branch=mlperf-inference` to fix the issue. 

Thank you @arjunsuresh , using `cm pull repo mlcommons@cm4mlops --branch=mlperf-inference` helped. However, the next commands yield error `ModuleNotFoundError: No module named 'numpy'`. Here's the log:
```bash
(cm) t@p:~$ cm pull repo mlcommons@cm4mlops --branch=mlperf-inference
=======================================================
Alias:    mlcommons@cm4mlops
URL:      https://github.com/mlcommons/cm4mlops
Branch:   mlperf-inference

Local path: /home/t/CM/repos/mlcommons@cm4mlops

git clone https://github.com/mlcommons/cm4mlops mlcommons@cm4mlops

Cloning into 'mlcommons@cm4mlops'...
remote: Enumerating objects: 73098, done.
remote: Counting objects: 100% (3278/3278), done.
remote: Compressing objects: 100% (1010/1010), done.
remote: Total 73098 (delta 2058), reused 3156 (delta 2010), pack-reused 69820 (from 1)
Receiving objects: 100% (73098/73098), 31.03 MiB | 23.83 MiB/s, done.
Resolving deltas: 100% (51102/51102), done.

git fetch && git checkout mlperf-inference

branch 'mlperf-inference' set up to track 'origin/mlperf-inference'.
Switched to a new branch 'mlperf-inference'

CM alias for this repository: mlcommons@cm4mlops
=======================================================

Reindexing all CM artifacts. Can take some time ...
Took 0.4 sec.
(cm) t@p:~$ cm run script --tags=install,python-venv --name=mlperf
INFO:root:* cm run script "install python-venv"
INFO:root:  * cm run script "get python -virtual"
INFO:root:      * /home/t/cm/bin/python3
INFO:root:             ! cd /home/t/CM/repos/local/cache/d1992f1472414bce
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-python3/run.sh from tmp-run.sh
INFO:root:             ! call "detect_version" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-python3/customize.py
        Detected version: 3.12.3
INFO:root:      # Found artifact in /home/t/cm/bin/python3
INFO:root:         ! cd /home/t/CM/repos/local/cache/d1992f1472414bce
INFO:root:         ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-python3/run.sh from tmp-run.sh
INFO:root:         ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-python3/customize.py
        Detected version: 3.12.3
INFO:root:Path to Python: /home/t/cm/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:       ! cd /home/t/CM/repos/local/cache/9ed2e4b836774e5c
INFO:root:       ! call /home/t/CM/repos/mlcommons@cm4mlops/script/install-python-venv/run.sh from tmp-run.sh
Collecting virtualenv
  Using cached virtualenv-20.26.5-py3-none-any.whl.metadata (4.5 kB)
Collecting distlib<1,>=0.3.7 (from virtualenv)
  Using cached distlib-0.3.8-py2.py3-none-any.whl.metadata (5.1 kB)
Collecting filelock<4,>=3.12.2 (from virtualenv)
  Using cached filelock-3.16.1-py3-none-any.whl.metadata (2.9 kB)
Collecting platformdirs<5,>=3.9.1 (from virtualenv)
  Using cached platformdirs-4.3.6-py3-none-any.whl.metadata (11 kB)
Using cached virtualenv-20.26.5-py3-none-any.whl (6.0 MB)
Using cached distlib-0.3.8-py2.py3-none-any.whl (468 kB)
Using cached filelock-3.16.1-py3-none-any.whl (16 kB)
Using cached platformdirs-4.3.6-py3-none-any.whl (18 kB)
Installing collected packages: distlib, platformdirs, filelock, virtualenv
Successfully installed distlib-0.3.8 filelock-3.16.1 platformdirs-4.3.6 virtualenv-20.26.5
INFO:root:       ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/install-python-venv/customize.py
INFO:root:* cm run script "get python3"
INFO:root:       ! cd /home/t/CM/repos/local/cache/dcf9d50abd684c73
INFO:root:       ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-python3/run.sh from tmp-run.sh
INFO:root:       ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-python3/customize.py
      Detected version: 3.12.3
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
(cm) t@p:~$ export CM_SCRIPT_EXTRA_CMD="--adr.python.name=mlperf"
(cm) t@p:~$ cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1-dev \
   --model=resnet50 \
   --implementation=reference \
   --framework=onnxruntime \
   --category=edge \
   --scenario=Offline \
   --execution_mode=test \
   --device=cuda  \
   --quiet \
   --test_query_count=5000
INFO:root:* cm run script "run-mlperf inference _find-performance _full _r4.1-dev"
INFO:root:  * cm run script "detect os"
INFO:root:         ! cd /home/t
INFO:root:         ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:         ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:  * cm run script "detect cpu"
INFO:root:    * cm run script "detect os"
INFO:root:           ! cd /home/t
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:         ! cd /home/t
INFO:root:         ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:         ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:  * cm run script "get python3"
INFO:root:       ! load /home/t/CM/repos/local/cache/dcf9d50abd684c73/cm-cached-state.json
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:  * cm run script "get mlcommons inference src"
INFO:root:    * cm run script "detect os"
INFO:root:           ! cd /home/t/CM/repos/local/cache/fb13e0e973fd4843
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:    * cm run script "get python3"
INFO:root:         ! load /home/t/CM/repos/local/cache/dcf9d50abd684c73/cm-cached-state.json
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:    * cm run script "get git repo _branch.master _repo.https://github.com/mlcommons/inference"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /home/t/CM/repos/local/cache/3afa84a5de7948c3
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:           ! cd /home/t/CM/repos/local/cache/3afa84a5de7948c3
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-git-repo/run.sh from tmp-run.sh
/home/t/CM/repos/local/cache/3afa84a5de7948c3
******************************************************
Current directory: /home/t/CM/repos/local/cache/3afa84a5de7948c3

Cloning inference from https://github.com/mlcommons/inference

git clone  -b master https://github.com/mlcommons/inference --depth 5 inference

Cloning into 'inference'...
remote: Enumerating objects: 792, done.
remote: Counting objects: 100% (792/792), done.
remote: Compressing objects: 100% (621/621), done.
remote: Total 792 (delta 195), reused 479 (delta 103), pack-reused 0 (from 0)
Receiving objects: 100% (792/792), 32.97 MiB | 20.42 MiB/s, done.
Resolving deltas: 100% (195/195), done.
git rev-parse HEAD >> ../tmp-cm-git-hash.out
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-git-repo/customize.py
INFO:root:CM cache path to the Git repo: /home/t/CM/repos/local/cache/3afa84a5de7948c3/inference
INFO:root:         ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-src/customize.py
INFO:root:  * cm run script "get sut description"
INFO:root:    * cm run script "detect os"
INFO:root:           ! cd /home/t
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:    * cm run script "detect cpu"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /home/t
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:           ! cd /home/t
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:    * cm run script "get python3"
INFO:root:         ! load /home/t/CM/repos/local/cache/dcf9d50abd684c73/cm-cached-state.json
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:    * cm run script "get compiler"
INFO:root:      * cm run script "install llvm"
INFO:root:        * cm run script "detect os"
INFO:root:               ! cd /home/t/CM/repos/local/cache/2605b106924b4ebb
INFO:root:               ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
          # Requested version: 15.0.6
          # Prepared package URL: https://github.com/llvm/llvm-project/releases/download/llvmorg-15.0.6/clang+llvm-15.0.6-x86_64-linux-gnu-ubuntu-18.04.tar.xz

Downloading from https://github.com/llvm/llvm-project/releases/download/llvmorg-15.0.6/clang+llvm-15.0.6-x86_64-linux-gnu-ubuntu-18.04.tar.xz ...
Downloading to /home/t/CM/repos/local/cache/2605b106924b4ebb/clang+llvm-15.0.6-x86_64-linux-gnu-ubuntu-18.04.tar.xz

Downloaded: 100%
INFO:root:             ! cd /home/t/CM/repos/local/cache/2605b106924b4ebb
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/install-llvm-prebuilt/run.sh from tmp-run.sh

Unarchiving clang+llvm-15.0.6-x86_64-linux-gnu-ubuntu-18.04.tar.xz ...
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/install-llvm-prebuilt/customize.py
INFO:root:           ! cd /home/t/CM/repos/local/cache/4e36c64695ac435a
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-llvm/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-llvm/customize.py
        Detected version: 15.0.6
INFO:root:    * cm run script "get compiler-flags"
INFO:root:      * cm run script "detect cpu"
INFO:root:        * cm run script "detect os"
INFO:root:               ! cd /home/t/CM/repos/local/cache/4e36c64695ac435a
INFO:root:               ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:             ! cd /home/t/CM/repos/local/cache/4e36c64695ac435a
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:    * cm run script "get cuda-devices _with-pycuda"
INFO:root:      * cm run script "get cuda _toolkit"
INFO:root:        * cm run script "detect os"
INFO:root:               ! cd /home/t/CM/repos/local/cache/c0916442873f4b81
INFO:root:               ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:          # Requested paths: /home/t/cm/bin:/home/t/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/usr/local/cuda/bin:/usr/cuda/bin:/usr/local/cuda-11/bin:/usr/cuda-11/bin:/usr/local/cuda-12/bin:/usr/cuda-12/bin:/usr/local/packages/cuda
INFO:root:          * /usr/bin/nvcc
INFO:root:                 ! cd /home/t/CM/repos/local/cache/c0916442873f4b81
INFO:root:                 ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-cuda/run.sh from tmp-run.sh
INFO:root:                 ! call "detect_version" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-cuda/customize.py
          Detected version: 12.0
INFO:root:          # Found artifact in /usr/bin/nvcc
INFO:root:             ! cd /home/t/CM/repos/local/cache/c0916442873f4b81
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-cuda/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-cuda/customize.py
          Detected version: 12.0
INFO:root:ENV[CM_CUDA_PATH_LIB_CUDNN_EXISTS]: no
INFO:root:ENV[CM_CUDA_VERSION]: 12.0
INFO:root:ENV[CM_CUDA_VERSION_STRING]: cu120
INFO:root:ENV[CM_NVCC_BIN_WITH_PATH]: /usr/bin/nvcc
INFO:root:ENV[CUDA_HOME]: /usr
INFO:root:      * cm run script "get python3"
INFO:root:           ! load /home/t/CM/repos/local/cache/dcf9d50abd684c73/cm-cached-state.json
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:      * cm run script "get generic-python-lib _package.pycuda"
INFO:root:        * cm run script "detect os"
INFO:root:               ! cd /home/t/CM/repos/local/cache/c16bbe01f6bc4455
INFO:root:               ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:        * cm run script "detect cpu"
INFO:root:          * cm run script "detect os"
INFO:root:                 ! cd /home/t/CM/repos/local/cache/c16bbe01f6bc4455
INFO:root:                 ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:                 ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:               ! cd /home/t/CM/repos/local/cache/c16bbe01f6bc4455
INFO:root:               ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:        * cm run script "get python3"
INFO:root:             ! load /home/t/CM/repos/local/cache/dcf9d50abd684c73/cm-cached-state.json
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:        * cm run script "get generic-python-lib _pip"
INFO:root:          * cm run script "detect os"
INFO:root:                 ! cd /home/t/CM/repos/local/cache/ee3df69597124b7d
INFO:root:                 ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:                 ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:          * cm run script "detect cpu"
INFO:root:            * cm run script "detect os"
INFO:root:                   ! cd /home/t/CM/repos/local/cache/ee3df69597124b7d
INFO:root:                   ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:                   ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:                 ! cd /home/t/CM/repos/local/cache/ee3df69597124b7d
INFO:root:                 ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:                 ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:          * cm run script "get python3"
INFO:root:               ! load /home/t/CM/repos/local/cache/dcf9d50abd684c73/cm-cached-state.json
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:                   ! cd /home/t/CM/repos/local/cache/ee3df69597124b7d
INFO:root:                   ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
INFO:root:                   ! call "detect_version" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py
              Detected version: 24.0
INFO:root:               ! cd /home/t/CM/repos/local/cache/ee3df69597124b7d
INFO:root:               ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py
INFO:root:                 ! cd /home/t/CM/repos/local/cache/c16bbe01f6bc4455
INFO:root:                 ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
INFO:root:                 ! call "detect_version" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py

            Extra PIP CMD:   --break-system-packages 

INFO:root:             ! cd /home/t/CM/repos/local/cache/c16bbe01f6bc4455
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/install.sh from tmp-run.sh

/home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3 -m pip install "pycuda" --break-system-packages
Collecting pycuda
  Using cached pycuda-2024.1.2-cp312-cp312-linux_x86_64.whl
Collecting pytools>=2011.2 (from pycuda)
  Using cached pytools-2024.1.14-py3-none-any.whl.metadata (3.0 kB)
Collecting platformdirs>=2.2.0 (from pycuda)
  Using cached platformdirs-4.3.6-py3-none-any.whl.metadata (11 kB)
Collecting mako (from pycuda)
  Using cached Mako-1.3.5-py3-none-any.whl.metadata (2.9 kB)
Collecting typing-extensions>=4 (from pytools>=2011.2->pycuda)
  Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
Collecting MarkupSafe>=0.9.2 (from mako->pycuda)
  Using cached MarkupSafe-2.1.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Using cached platformdirs-4.3.6-py3-none-any.whl (18 kB)
Using cached pytools-2024.1.14-py3-none-any.whl (89 kB)
Using cached Mako-1.3.5-py3-none-any.whl (78 kB)
Using cached MarkupSafe-2.1.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28 kB)
Using cached typing_extensions-4.12.2-py3-none-any.whl (37 kB)
Installing collected packages: typing-extensions, platformdirs, MarkupSafe, pytools, mako, pycuda
Successfully installed MarkupSafe-2.1.5 mako-1.3.5 platformdirs-4.3.6 pycuda-2024.1.2 pytools-2024.1.14 typing-extensions-4.12.2
INFO:root:             ! cd /home/t/CM/repos/local/cache/c16bbe01f6bc4455
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py
            Detected version: 2024.1.2
INFO:root:           ! cd /home/t
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/detect.sh from tmp-run.sh
Traceback (most recent call last):
  File "/home/t/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/detect.py", line 1, in <module>
    import pycuda.driver as cuda
  File "/home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/lib/python3.12/site-packages/pycuda/driver.py", line 7, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'

CM error: Portable CM script failed (name = get-cuda-devices, return code = 256)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Note that it is often a portability issue of a third-party tool or a native script
wrapped and unified by this CM script (automation recipe). Please re-run
this script with --repro flag and report this issue with the original
command line, cm-repro directory and full log here:

https://github.com/mlcommons/cm4mlops/issues

The CM concept is to collaboratively fix such issues inside portable CM scripts
to make existing tools and native scripts more portable, interoperable
and deterministic. Thank you!
```

Any suggestion is appreciated. Thank you for your help!

Hi @pvti Can you please do `cm pull repo` and retry the command? We have just fixed the `numpy` dependency for `pycuda`.

But while running on `cuda` device, it is better to use the `docker` option as otherwise there are manual steps needed to fix the cuda dependencies like cuDNN and TensorRT. 

Thank you @arjunsuresh . Here's the next error:
```
(cm) t@p:~$ cm pull repo
=======================================================
Alias:    mlcommons@cm4mlops

Local path: /home/t/CM/repos/mlcommons@cm4mlops

git pull

remote: Enumerating objects: 103, done.
remote: Counting objects: 100% (103/103), done.
remote: Compressing objects: 100% (53/53), done.
remote: Total 103 (delta 64), reused 84 (delta 50), pack-reused 0 (from 0)
Receiving objects: 100% (103/103), 58.88 KiB | 2.35 MiB/s, done.
Resolving deltas: 100% (64/64), completed with 10 local objects.
From https://github.com/mlcommons/cm4mlops
   75911cdca..1c2d924a7  mlperf-inference -> origin/mlperf-inference
 + 6b9861eb1...6e428b572 gh-pages         -> origin/gh-pages  (forced update)
Updating 75911cdca..1c2d924a7
Fast-forward
 .github/workflows/code-review.yml                         |  4 ++--
 script/app-mlperf-inference-mlcommons-python/_cm.yaml     |  9 +++++++++
 script/app-mlperf-inference-mlcommons-python/customize.py |  6 +++---
 script/app-mlperf-inference/_cm.yaml                      | 17 +++++++++++++++++
 script/get-cuda-devices/_cm.yaml                          |  3 +++
 script/get-preprocessed-dataset-kits19/customize.py       |  1 +
 6 files changed, 35 insertions(+), 5 deletions(-)

CM alias for this repository: mlcommons@cm4mlops
=======================================================

Reindexing all CM artifacts. Can take some time ...
Took 0.4 sec.
(cm) t@p:~$ cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1-dev    --model=resnet50    --implementation=reference    --framework=onnxruntime    --category=edge    --scenario=Offline    --execution_mode=test    --device=cuda     --quiet    --test_query_count=5000
INFO:root:* cm run script "run-mlperf inference _find-performance _full _r4.1-dev"
INFO:root:  * cm run script "detect os"
INFO:root:         ! cd /home/t
INFO:root:         ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:         ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:  * cm run script "detect cpu"
INFO:root:    * cm run script "detect os"
INFO:root:           ! cd /home/t
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:         ! cd /home/t
INFO:root:         ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:         ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:  * cm run script "get python3"
INFO:root:       ! load /home/t/CM/repos/local/cache/dcf9d50abd684c73/cm-cached-state.json
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:  * cm run script "get mlcommons inference src"
INFO:root:       ! load /home/t/CM/repos/local/cache/fb13e0e973fd4843/cm-cached-state.json
INFO:root:  * cm run script "get sut description"
INFO:root:    * cm run script "detect os"
INFO:root:           ! cd /home/t
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:    * cm run script "detect cpu"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /home/t
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:           ! cd /home/t
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:    * cm run script "get python3"
INFO:root:         ! load /home/t/CM/repos/local/cache/dcf9d50abd684c73/cm-cached-state.json
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:    * cm run script "get compiler"
INFO:root:         ! load /home/t/CM/repos/local/cache/4e36c64695ac435a/cm-cached-state.json
INFO:root:    * cm run script "get cuda-devices _with-pycuda"
INFO:root:      * cm run script "get cuda _toolkit"
INFO:root:           ! load /home/t/CM/repos/local/cache/c0916442873f4b81/cm-cached-state.json
INFO:root:ENV[CM_CUDA_PATH_LIB_CUDNN_EXISTS]: no
INFO:root:ENV[CM_CUDA_VERSION]: 12.0
INFO:root:ENV[CM_CUDA_VERSION_STRING]: cu120
INFO:root:ENV[CM_NVCC_BIN_WITH_PATH]: /usr/bin/nvcc
INFO:root:ENV[CUDA_HOME]: /usr
INFO:root:      * cm run script "get python3"
INFO:root:           ! load /home/t/CM/repos/local/cache/dcf9d50abd684c73/cm-cached-state.json
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:      * cm run script "get generic-python-lib _package.pycuda"
INFO:root:           ! load /home/t/CM/repos/local/cache/c16bbe01f6bc4455/cm-cached-state.json
INFO:root:      * cm run script "get generic-python-lib _package.numpy"
INFO:root:        * cm run script "detect os"
INFO:root:               ! cd /home/t/CM/repos/local/cache/68aeff96e63e49ab
INFO:root:               ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:        * cm run script "detect cpu"
INFO:root:          * cm run script "detect os"
INFO:root:                 ! cd /home/t/CM/repos/local/cache/68aeff96e63e49ab
INFO:root:                 ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:                 ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:               ! cd /home/t/CM/repos/local/cache/68aeff96e63e49ab
INFO:root:               ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:        * cm run script "get python3"
INFO:root:             ! load /home/t/CM/repos/local/cache/dcf9d50abd684c73/cm-cached-state.json
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:        * cm run script "get generic-python-lib _pip"
INFO:root:             ! load /home/t/CM/repos/local/cache/ee3df69597124b7d/cm-cached-state.json
INFO:root:                 ! cd /home/t/CM/repos/local/cache/68aeff96e63e49ab
INFO:root:                 ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
INFO:root:                 ! call "detect_version" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py

            Extra PIP CMD:   --break-system-packages 

INFO:root:             ! cd /home/t/CM/repos/local/cache/68aeff96e63e49ab
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/install.sh from tmp-run.sh

/home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3 -m pip install "numpy" --break-system-packages
Collecting numpy
  Using cached numpy-2.1.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
Using cached numpy-2.1.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.0 MB)
Installing collected packages: numpy
Successfully installed numpy-2.1.1
INFO:root:             ! cd /home/t/CM/repos/local/cache/68aeff96e63e49ab
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py
            Detected version: 2.1.1
INFO:root:           ! cd /home/t
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/detect.sh from tmp-run.sh
GPU 0:
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/customize.py
INFO:root:    * cm run script "get generic-python-lib _package.dmiparser"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /home/t/CM/repos/local/cache/4ba47b859e1b4e65
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:      * cm run script "detect cpu"
INFO:root:        * cm run script "detect os"
INFO:root:               ! cd /home/t/CM/repos/local/cache/4ba47b859e1b4e65
INFO:root:               ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:             ! cd /home/t/CM/repos/local/cache/4ba47b859e1b4e65
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:      * cm run script "get python3"
INFO:root:           ! load /home/t/CM/repos/local/cache/dcf9d50abd684c73/cm-cached-state.json
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:      * cm run script "get generic-python-lib _pip"
INFO:root:           ! load /home/t/CM/repos/local/cache/ee3df69597124b7d/cm-cached-state.json
INFO:root:               ! cd /home/t/CM/repos/local/cache/4ba47b859e1b4e65
INFO:root:               ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
INFO:root:               ! call "detect_version" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py

          Extra PIP CMD:   --break-system-packages 

INFO:root:           ! cd /home/t/CM/repos/local/cache/4ba47b859e1b4e65
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/install.sh from tmp-run.sh

/home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3 -m pip install "dmiparser" --break-system-packages
Collecting dmiparser
  Downloading dmiparser-5.1-py3-none-any.whl.metadata (5.7 kB)
Downloading dmiparser-5.1-py3-none-any.whl (8.3 kB)
Installing collected packages: dmiparser
Successfully installed dmiparser-5.1
INFO:root:           ! cd /home/t/CM/repos/local/cache/4ba47b859e1b4e65
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py
          Detected version: 5.1
INFO:root:    * cm run script "get cache dir _name.mlperf-inference-sut-descriptions"
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-cache-dir/customize.py
Generating SUT description file for p-onnxruntime
HW description file for p not found. Copying from default!!!
INFO:root:         ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-sut-description/customize.py
INFO:root:  * cm run script "get mlperf inference results dir _version.r4_1-dev"
INFO:root:         ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-results-dir/customize.py
INFO:root:  * cm run script "install pip-package for-cmind-python _package.tabulate"
Collecting tabulate
  Using cached tabulate-0.9.0-py3-none-any.whl.metadata (34 kB)
Using cached tabulate-0.9.0-py3-none-any.whl (35 kB)
Installing collected packages: tabulate
Successfully installed tabulate-0.9.0
INFO:root:         ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/install-pip-package-for-cmind-python/customize.py
INFO:root:  * cm run script "get mlperf inference utils"
INFO:root:    * cm run script "get mlperf inference src"
INFO:root:         ! load /home/t/CM/repos/local/cache/fb13e0e973fd4843/cm-cached-state.json
INFO:root:         ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-utils/customize.py
Using MLCommons Inference source from /home/t/CM/repos/local/cache/3afa84a5de7948c3/inference

Running loadgen scenario: Offline and mode: performance
INFO:root:* cm run script "app mlperf inference generic _reference _resnet50 _onnxruntime _cuda _test _r4.1-dev_default _offline"
INFO:root:  * cm run script "detect os"
INFO:root:         ! cd /home/t
INFO:root:         ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:         ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:  * cm run script "get sys-utils-cm"
WARNING:root:=================================================
WARNING:root:WARNINGS:
WARNING:root:  This CM script will install extra OS system utils required for CM automation workflows!
WARNING:root:=================================================
INFO:root:    * cm run script "detect os"
INFO:root:           ! cd /home/t/CM/repos/local/cache/010dba3dfd6248f8
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:  * cm run script "get python"
INFO:root:       ! load /home/t/CM/repos/local/cache/dcf9d50abd684c73/cm-cached-state.json
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:  * cm run script "get mlcommons inference src"
INFO:root:       ! load /home/t/CM/repos/local/cache/fb13e0e973fd4843/cm-cached-state.json
INFO:root:  * cm run script "get mlperf inference utils"
INFO:root:    * cm run script "get mlperf inference src"
INFO:root:         ! load /home/t/CM/repos/local/cache/fb13e0e973fd4843/cm-cached-state.json
INFO:root:         ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-utils/customize.py
INFO:root:  * cm run script "get dataset-aux imagenet-aux"
INFO:root:    * cm run script "download-and-extract _extract _wget _url.https://www.dropbox.com/s/92n2fyej3lzy3s3/caffe_ilsvrc12.tar.gz"
INFO:root:      * cm run script "download file _wget _url.https://www.dropbox.com/s/92n2fyej3lzy3s3/caffe_ilsvrc12.tar.gz"
INFO:root:        * cm run script "detect os"
INFO:root:               ! cd /home/t/CM/repos/local/cache/f38f76082ef64699
INFO:root:               ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py

Downloading from https://www.dropbox.com/s/92n2fyej3lzy3s3/caffe_ilsvrc12.tar.gz
wget -nc  --tries=3 -O 'caffe_ilsvrc12.tar.gz'  https://www.dropbox.com/s/92n2fyej3lzy3s3/caffe_ilsvrc12.tar.gz || ((rm -f caffe_ilsvrc12.tar.gz || true) && wget -nc  --tries=3 -O 'caffe_ilsvrc12.tar.gz'  http://dl.caffe.berkeleyvision.org/caffe_ilsvrc12.tar.gz)
INFO:root:             ! cd /home/t/CM/repos/local/cache/f38f76082ef64699
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/download-file/run.sh from tmp-run.sh

Downloading: wget -nc  --tries=3 -O 'caffe_ilsvrc12.tar.gz'  https://www.dropbox.com/s/92n2fyej3lzy3s3/caffe_ilsvrc12.tar.gz || ((rm -f caffe_ilsvrc12.tar.gz || true) && wget -nc  --tries=3 -O 'caffe_ilsvrc12.tar.gz'  http://dl.caffe.berkeleyvision.org/caffe_ilsvrc12.tar.gz)
--2024-09-27 13:10:26--  https://www.dropbox.com/s/92n2fyej3lzy3s3/caffe_ilsvrc12.tar.gz
Resolving www.dropbox.com (www.dropbox.com)... 162.125.69.18, 2620:100:6025:18::a27d:4512
Connecting to www.dropbox.com (www.dropbox.com)|162.125.69.18|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://www.dropbox.com/scl/fi/pdhe77hzkuix2tf4rwc9v/caffe_ilsvrc12.tar.gz?rlkey=6p4gu3s6ajewrlwcrx7mp4re8 [following]
--2024-09-27 13:10:26--  https://www.dropbox.com/scl/fi/pdhe77hzkuix2tf4rwc9v/caffe_ilsvrc12.tar.gz?rlkey=6p4gu3s6ajewrlwcrx7mp4re8
Reusing existing connection to www.dropbox.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://ucf0b100353a085515e7088c910a.dl.dropboxusercontent.com/cd/0/inline/CbXk_bn0GkoWmsLcaYozxLuSurD7u3KbezQsyRHe76VsuAhgcd1hqHEUEnMMutHTwdXrCmB5iiZY9mm-1m3boz63y-5PjqDRzrFRigToXZk8P0CHIowq_j-7SmEVPa9M4bA/file# [following]
--2024-09-27 13:10:27--  https://ucf0b100353a085515e7088c910a.dl.dropboxusercontent.com/cd/0/inline/CbXk_bn0GkoWmsLcaYozxLuSurD7u3KbezQsyRHe76VsuAhgcd1hqHEUEnMMutHTwdXrCmB5iiZY9mm-1m3boz63y-5PjqDRzrFRigToXZk8P0CHIowq_j-7SmEVPa9M4bA/file
Resolving ucf0b100353a085515e7088c910a.dl.dropboxusercontent.com (ucf0b100353a085515e7088c910a.dl.dropboxusercontent.com)... 162.125.69.15, 2620:100:6025:15::a27d:450f
Connecting to ucf0b100353a085515e7088c910a.dl.dropboxusercontent.com (ucf0b100353a085515e7088c910a.dl.dropboxusercontent.com)|162.125.69.15|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /cd/0/inline2/CbUAq4LYLXxojiFP2NxmZQDOdPAtzu-RSV9lRr3BQukP9Qq7lU3HejwCPavx4VdPnwmeMIxoN8scd1xQfDj8_UHU3v5_rWJnZA5lvvHt1nEFvecE-dwI0cU48hDIiQEbmSkFF9Ml_uqyI59CvJaWo3z63CztE8m5IKlZnXYzICEsHdGigQftrywJnwHCzc_URdbIJZcodbJj17732eUj5XYyTBLUzNtAD5EfhR67Y9ef-OgBidcPSARoUs1Z4eWBQyCDQLLuj0g-_3KmyM2WxPbAzbp8w96bqD-XVtkVShbbCSDOMbg0RNmnkiGnzdoyYUy-F9uVdJ7jDzct0A3WRG3TrFXpiprmmHDW1oxACYj2jg/file [following]
--2024-09-27 13:10:28--  https://ucf0b100353a085515e7088c910a.dl.dropboxusercontent.com/cd/0/inline2/CbUAq4LYLXxojiFP2NxmZQDOdPAtzu-RSV9lRr3BQukP9Qq7lU3HejwCPavx4VdPnwmeMIxoN8scd1xQfDj8_UHU3v5_rWJnZA5lvvHt1nEFvecE-dwI0cU48hDIiQEbmSkFF9Ml_uqyI59CvJaWo3z63CztE8m5IKlZnXYzICEsHdGigQftrywJnwHCzc_URdbIJZcodbJj17732eUj5XYyTBLUzNtAD5EfhR67Y9ef-OgBidcPSARoUs1Z4eWBQyCDQLLuj0g-_3KmyM2WxPbAzbp8w96bqD-XVtkVShbbCSDOMbg0RNmnkiGnzdoyYUy-F9uVdJ7jDzct0A3WRG3TrFXpiprmmHDW1oxACYj2jg/file
Reusing existing connection to ucf0b100353a085515e7088c910a.dl.dropboxusercontent.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 17852691 (17M) [application/octet-stream]
Saving to: ‘caffe_ilsvrc12.tar.gz’

caffe_ilsvrc12.tar.gz 100%[======================>]  17.03M  24.0MB/s    in 0.7s    

2024-09-27 13:10:29 (24.0 MB/s) - ‘caffe_ilsvrc12.tar.gz’ saved [17852691/17852691]


Verifying checksum after download: echo ee346d67141e476df9c1a3f813552503 'caffe_ilsvrc12.tar.gz' | md5sum  -c - || echo f963098ea0e785a968ca1eb634003a90 'caffe_ilsvrc12.tar.gz' | md5sum  -c -
caffe_ilsvrc12.tar.gz: OK
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/download-file/customize.py
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/download-and-extract/customize.py
INFO:root:    * cm run script "extract file _path./home/t/CM/repos/local/cache/f38f76082ef64699/caffe_ilsvrc12.tar.gz"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /home/t/CM/repos/local/cache/d170116a3af34611
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py

Current directory: /home/t/CM/repos/local/cache/d170116a3af34611
Command line: "tar    --skip-old-files -xvzf  /home/t/CM/repos/local/cache/f38f76082ef64699/caffe_ilsvrc12.tar.gz"

INFO:root:           ! cd /home/t/CM/repos/local/cache/d170116a3af34611
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/extract-file/run.sh from tmp-run.sh

tar    --skip-old-files -xvzf  /home/t/CM/repos/local/cache/f38f76082ef64699/caffe_ilsvrc12.tar.gz
det_synset_words.txt
imagenet.bet.pickle
imagenet_mean.binaryproto
synsets.txt
synset_words.txt
test.txt
train.txt
val.txt


INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/extract-file/customize.py
INFO:root:  * cm run script "get cuda-devices _with-pycuda"
INFO:root:    * cm run script "get cuda _toolkit"
INFO:root:         ! load /home/t/CM/repos/local/cache/c0916442873f4b81/cm-cached-state.json
INFO:root:ENV[CM_CUDA_PATH_LIB_CUDNN_EXISTS]: no
INFO:root:ENV[CM_CUDA_VERSION]: 12.0
INFO:root:ENV[CM_CUDA_VERSION_STRING]: cu120
INFO:root:ENV[CM_NVCC_BIN_WITH_PATH]: /usr/bin/nvcc
INFO:root:ENV[CUDA_HOME]: /usr
INFO:root:    * cm run script "get python3"
INFO:root:         ! load /home/t/CM/repos/local/cache/dcf9d50abd684c73/cm-cached-state.json
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:    * cm run script "get generic-python-lib _package.pycuda"
INFO:root:         ! load /home/t/CM/repos/local/cache/c16bbe01f6bc4455/cm-cached-state.json
INFO:root:    * cm run script "get generic-python-lib _package.numpy"
INFO:root:         ! load /home/t/CM/repos/local/cache/68aeff96e63e49ab/cm-cached-state.json
INFO:root:         ! cd /home/t/CM/repos/local/cache/010dba3dfd6248f8
INFO:root:         ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/detect.sh from tmp-run.sh
GPU 0:
INFO:root:         ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/customize.py
/home/t/CM/repos/mlcommons@cm4mlops/script/app-mlperf-inference/customize.py:143: SyntaxWarning: invalid escape sequence '\('
  pattern["SingleStream"] = "Mean latency \(ns\)\s*:(.*)"
/home/t/CM/repos/mlcommons@cm4mlops/script/app-mlperf-inference/customize.py:144: SyntaxWarning: invalid escape sequence '\('
  pattern["MultiStream"] = "Mean latency \(ns\)\s*:(.*)"
/home/t/CM/repos/mlcommons@cm4mlops/script/app-mlperf-inference/customize.py:155: SyntaxWarning: invalid escape sequence '\('
  if "\(ns\)" in pattern[scenario]:
INFO:root:  * cm run script "app mlperf reference inference _onnxruntime _offline _resnet50 _cuda _fp32"
INFO:root:    * cm run script "detect os"
INFO:root:           ! cd /home/t/CM/repos/local/cache/010dba3dfd6248f8
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:    * cm run script "detect cpu"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /home/t/CM/repos/local/cache/010dba3dfd6248f8
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:           ! cd /home/t/CM/repos/local/cache/010dba3dfd6248f8
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:    * cm run script "get sys-utils-cm"
WARNING:root:=================================================
WARNING:root:WARNINGS:
WARNING:root:  This CM script will install extra OS system utils required for CM automation workflows!
WARNING:root:=================================================
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /home/t/CM/repos/local/cache/010dba3dfd6248f8
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:    * cm run script "get python"
INFO:root:         ! load /home/t/CM/repos/local/cache/dcf9d50abd684c73/cm-cached-state.json
INFO:root:Path to Python: /home/t/CM/repos/local/cache/9ed2e4b836774e5c/mlperf/bin/python3
INFO:root:Python version: 3.12.3
INFO:root:    * cm run script "get cuda _cudnn"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /home/t/CM/repos/local/cache/d8d182a76d5a49f0
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:        # Requested paths: /home/t/cm/bin:/home/t/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/usr/local/cuda/bin:/usr/cuda/bin:/usr/local/cuda-11/bin:/usr/cuda-11/bin:/usr/local/cuda-12/bin:/usr/cuda-12/bin:/usr/local/packages/cuda
INFO:root:        * /usr/bin/nvcc
INFO:root:               ! cd /home/t/CM/repos/local/cache/d8d182a76d5a49f0
INFO:root:               ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-cuda/run.sh from tmp-run.sh
INFO:root:               ! call "detect_version" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-cuda/customize.py
        Detected version: 12.0
INFO:root:        # Found artifact in /usr/bin/nvcc
INFO:root:           ! cd /home/t/CM/repos/local/cache/d8d182a76d5a49f0
INFO:root:           ! call /home/t/CM/repos/mlcommons@cm4mlops/script/get-cuda/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/get-cuda/customize.py
        Detected version: 12.0
INFO:root:    * cm run script "get nvidia cudnn"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /home/t/CM/repos/local/cache/1f2fb4172a364ec6
INFO:root:             ! call /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /home/t/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:        # Requested paths: /home/t/cm/bin:/home/t/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/usr/local/cuda/bin:/usr/cuda/bin:/usr/local/cuda-11/bin:/usr/cuda-11/bin:/usr/local/cuda-12/bin:/usr/cuda-12/bin:/usr/local/packages/cuda:/usr/local/cuda/lib64:/usr/cuda/lib64:/usr/local/cuda/lib:/usr/cuda/lib:/usr/local/cuda-11/lib64:/usr/cuda-11/lib:/usr/local/cuda-12/lib:/usr/cuda-12/lib:/usr/local/packages/cuda/lib:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/lib64:/usr/lib64:/usr/local/lib:/lib:/usr/lib

CM error: Please envoke cm run script "get cudnn" --tar_file={full path to the cuDNN tar file}!
(cm) t@p:~$ nvidia-smi
Fri Sep 27 13:11:07 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.107.02             Driver Version: 550.107.02     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060        Off |   00000000:01:00.0  On |                  N/A |
| 60%   71C    P2            135W /  170W |    5837MiB /  12288MiB |    100%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      2329      G   /usr/lib/xorg/Xorg                           1310MiB |
|    0   N/A  N/A      2548      G   /usr/bin/gnome-shell                          127MiB |
|    0   N/A  N/A      3113      G   /usr/libexec/xdg-desktop-portal-gnome         138MiB |
|    0   N/A  N/A      8422      G   ...erProcess --variations-seed-version        265MiB |
|    0   N/A  N/A     18091      C   python                                       1942MiB |
|    0   N/A  N/A     18454      C   python                                       1584MiB |
|    0   N/A  N/A     70177      G   ...erProcess --variations-seed-version        175MiB |
|    0   N/A  N/A     90289      G   ...seed-version=20240926-050110.326000        223MiB |
|    0   N/A  N/A    597270      G   /usr/bin/gnome-calculator                       7MiB |
+-----------------------------------------------------------------------------------------+
(cm) t@p:~$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Jan__6_16:45:21_PST_2023
Cuda compilation tools, release 12.0, V12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0
```

As you mentioned there are manual steps.
Thank you so much!

yes, that's what I had mentioned earlier - you need to use the docker option - or else will need to manually download cuDNN and TensorRT from Nvidia website and pass their paths - even then, if there are package conflicts GPU run may not happen. In short, [docker](https://docs.mlcommons.org/inference/benchmarks/image_classification/resnet50/#__tabbed_14_1) is mandatory for Nvidia GPU runs unless you are already on a CUDA setup.