Repository: mlcommons/inference
Issue: Image building fail during executing make prebuild command.
Root cause: External Dependency Breakage

I tried to execute make prebuild command on /closed/NVIDIA directory.
How to solve this problem?

775.6 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
780.0 Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com/
780.1 Collecting apex@ git+https://github.com/nvidia/apex@0da3ffb92ee6fbe5336602f0e3989db1cd16f880 (from -r requirements.x86_64.2.txt (lin e 1))
780.1 Cloning https://github.com/nvidia/apex (to revision 0da3ffb92ee6fbe5336602f0e3989db1cd16f880) to ./pip-install-177uc4fo/apex_06721 f2a06e34e55be55df8d71638642
780.1 Running command git clone --filter=blob:none --quiet https://github.com/nvidia/apex /tmp/pip-install-177uc4fo/apex_06721f2a06e34e5 5be55df8d71638642
781.6 Running command git rev-parse -q --verify 'sha^0da3ffb92ee6fbe5336602f0e3989db1cd16f880'
781.6 Running command git fetch -q https://github.com/nvidia/apex 0da3ffb92ee6fbe5336602f0e3989db1cd16f880
782.1 Running command git checkout -q 0da3ffb92ee6fbe5336602f0e3989db1cd16f880
783.2 Resolved https://github.com/nvidia/apex to commit 0da3ffb92ee6fbe5336602f0e3989db1cd16f880
783.2 Running command git submodule update --init --recursive -q
793.7 Preparing metadata (setup.py): started
795.3 Preparing metadata (setup.py): finished with status 'error'
795.3 error: subprocess-exited-with-error
795.3
795.3 × python setup.py egg_info did not run successfully.
795.3 │ exit code: 1
795.3 ╰─> [8 lines of output]
795.3 Traceback (most recent call last):
795.3 File "", line 2, in
795.3 File "", line 34, in
795.3 File "/tmp/pip-install-177uc4fo/apex_06721f2a06e34e55be55df8d71638642/setup.py", line 10, in
795.3 from torch.utils.cpp_extension import BuildExtension, CppExtension, CUDAExtension, CUDA_HOME, load
795.3 File "/usr/local/lib/python3.8/dist-packages/torch/utils/cpp_extension.py", line 25, in
795.3 from pkg_resources import packaging # type: ignore[attr-defined]
795.3 ImportError: cannot import name 'packaging' from 'pkg_resources' (/usr/local/lib/python3.8/dist-packages/pkg_resources/_init _.py)
795.3 [end of output]
795.3
795.3 note: This error originates from a subprocess, and is likely not a problem with pip.
795.3 error: metadata-generation-failed
795.3
795.3 × Encountered error while generating package metadata.
795.3 ╰─> See above for output.
795.3
795.3 note: This is an issue with the package mentioned above, not pip.
795.3 hint: See above for details.
Dockerfile.x86_64:129
128 | # Break requirements into two lists because some of them require that other packages be fully installed first.
129 | >>> RUN python3 -m pip install -r requirements.x86_64.1.txt
130 | >>> && python3 -m pip install -r requirements.x86_64.2.txt
131 |
ERROR: failed to solve: process "/bin/bash -c python3 -m pip install -r requirements.x86_64.1.txt && python3 -m pip install -r requiremen ts.x86_64.2.txt" did not complete successfully: exit code: 1
make[1]: *** [Makefile.docker:259: build_docker] Error 1
make[1]: Leaving directory '/mnt/sdb/mlperf/code/inference_results_v3.1/closed/NVIDIA'
make: *** [/mnt/sdb/mlperf/code/inference_results_v3.1/closed/NVIDIA/Makefile.docker:247: prebuild] Error 2

All the models except Stable diffusion and LLama2 from Nvidia 4.0 submissions can be reproduced by using CM. You can see the run commands here:

https://docs.mlcommons.org/inference/benchmarks/image_classification/resnet50/#__tabbed_29_1

Thank you for quick responce. 
I took steps that you recommanded but i got some error message like below.


      * cm run script "get generic-python-lib _pip"
           ! load /home/atc01/CM/repos/local/cache/660268239d814597/cm-cached-state.json
               ! cd /home/atc01/CM/repos/local/cache/cdfb20bf94a94159
               ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
               ! call "detect_version" from /home/atc01/CM/repos/gateoverflow@cm4mlops/script/get-generic-python-lib/customize.py

          Extra PIP CMD:

           ! cd /home/atc01/CM/repos/local/cache/cdfb20bf94a94159
           ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/get-generic-python-lib/install.sh from tmp-run.sh

/mnt/sdb/cm/bin/python3 -m pip install "dmiparser"
Collecting dmiparser
  Using cached dmiparser-5.1-py3-none-any.whl (8.3 kB)
Installing collected packages: dmiparser
Successfully installed dmiparser-5.1
WARNING: Target directory /mnt/sdb/my_packages/dmiparser-5.1.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /mnt/sdb/my_packages/dmiparser already exists. Specify --upgrade to force replacement.
WARNING: Target directory /mnt/sdb/my_packages/bin already exists. Specify --upgrade to force replacement.
           ! cd /home/atc01/CM/repos/local/cache/cdfb20bf94a94159
           ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
           ! call "postprocess" from /home/atc01/CM/repos/gateoverflow@cm4mlops/script/get-generic-python-lib/customize.py

CM error: version was not detected (
)!


Thank you for trying. Which command gave this error? 

Dataset and model download was fine but below command caused above error.

cm run script --tags=run-mlperf,inference,_find-performance,_full \
   --model=resnet50 \
   --implementation=nvidia \
   --framework=tensorrt \
   --category=datacenter \
   --scenario=Offline \
   --execution-mode=test \
   --device=cuda  \
   --docker --quiet \
   --test_query_count=100

Thank you. Can you please try the below command and then retry the same CM command?
```
/mnt/sdb/cm/bin/python3 -m pip install --upgrade "dmiparser"
```

Thank you for quick reply. Unfortunately it take a time to apply your solution because i'm off-duty now. When i am available to access my AI environment i will post the result to you.
On the other hand the current version of dmipaser  isn't compatible?

 /mnt/sdb/cm/bin/python3 -m pip install "dmiparser"
Collecting dmiparser
Using cached dmiparser-5.1-py3-none-any.whl (8.3 kB)
Installing collected packages: dmiparser
Successfully installed dmiparser-5.1

No worries. Actually this is a new error we have seen. For some reason dmiparser is shown as already installed but its version info is missing. The command I gave is a way to override this issue.

Alternatively, you can use the following command to launch the docker (it replaces resnet50 with bert-99) but once inside the container you can use the resnet50 commands. This will avoid downloading of the resnet50 dataset in the host and will do it inside the container. 

```
cm run script --tags=run-mlperf,inference,_find-performance,_full
--model=bert-99
--implementation=nvidia
--framework=tensorrt
--category=datacenter
--scenario=Offline
--execution-mode=test
--device=cuda
--docker --quiet
--test_query_count=100
```

I tried to solve this situation with your solution but the results was same. Could you check more based on below results?

atc01@atc01:/mnt/sdb/cm/bin$ python3 -m pip install --upgrade "dmiparser"
Collecting dmiparser
  Using cached dmiparser-5.1-py3-none-any.whl.metadata (5.7 kB)
Using cached dmiparser-5.1-py3-none-any.whl (8.3 kB)
Installing collected packages: dmiparser
Successfully installed dmiparser-5.1


atc01@atc01:~/CM/repos/gateoverflow@cm4mlops/automation/script$ cm run script --tags=run-mlperf,inference,_find-performance,_full
--model=bert-99
--implementation=nvidia
--framework=tensorrt
--category=datacenter
--scenario=Offline
--execution-mode=test
--device=cuda
--docker --quiet
--test_query_count=100

* cm run script "run-mlperf inference _find-performance _full"

  * cm run script "detect os"
         ! cd /home/atc01/CM/repos/gateoverflow@cm4mlops/automation/script
         ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-os/run.sh from tmp-run.sh
         ! call "postprocess" from /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-os/customize.py

  * cm run script "detect cpu"

    * cm run script "detect os"
           ! cd /home/atc01/CM/repos/gateoverflow@cm4mlops/automation/script
           ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-os/run.sh from tmp-run.sh
           ! call "postprocess" from /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-os/customize.py
         ! cd /home/atc01/CM/repos/gateoverflow@cm4mlops/automation/script
         ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
         ! call "postprocess" from /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-cpu/customize.py

  * cm run script "get python3"
       ! load /home/atc01/CM/repos/local/cache/f99b48709eba4a29/cm-cached-state.json

Path to Python: /usr/bin/python3
Python version: 3.10.12


  * cm run script "get mlcommons inference src"
       ! load /home/atc01/CM/repos/local/cache/7b9d625568c946da/cm-cached-state.json

  * cm run script "get sut description"

    * cm run script "detect os"
           ! cd /home/atc01/CM/repos/gateoverflow@cm4mlops/automation/script
           ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-os/run.sh from tmp-run.sh
           ! call "postprocess" from /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-os/customize.py

    * cm run script "detect cpu"

      * cm run script "detect os"
             ! cd /home/atc01/CM/repos/gateoverflow@cm4mlops/automation/script
             ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-os/run.sh from tmp-run.sh
             ! call "postprocess" from /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-os/customize.py
           ! cd /home/atc01/CM/repos/gateoverflow@cm4mlops/automation/script
           ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
           ! call "postprocess" from /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-cpu/customize.py

    * cm run script "get python3"
         ! load /home/atc01/CM/repos/local/cache/f99b48709eba4a29/cm-cached-state.json

Path to Python: /usr/bin/python3
Python version: 3.10.12


    * cm run script "get compiler"
         ! load /home/atc01/CM/repos/local/cache/8d32fa6309e242d5/cm-cached-state.json

    * cm run script "get generic-python-lib _package.dmiparser"

      * cm run script "detect os"
             ! cd /home/atc01/CM/repos/local/cache/a5704355393f46b3
             ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-os/run.sh from tmp-run.sh
             ! call "postprocess" from /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-os/customize.py

      * cm run script "detect cpu"

        * cm run script "detect os"
               ! cd /home/atc01/CM/repos/local/cache/a5704355393f46b3
               ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-os/run.sh from tmp-run.sh
               ! call "postprocess" from /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-os/customize.py
             ! cd /home/atc01/CM/repos/local/cache/a5704355393f46b3
             ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
             ! call "postprocess" from /home/atc01/CM/repos/gateoverflow@cm4mlops/script/detect-cpu/customize.py

      * cm run script "get python3"
           ! load /home/atc01/CM/repos/local/cache/f99b48709eba4a29/cm-cached-state.json

Path to Python: /usr/bin/python3
Python version: 3.10.12


      * cm run script "get generic-python-lib _pip"
           ! load /home/atc01/CM/repos/local/cache/e41d2ff40ef740e2/cm-cached-state.json
               ! cd /home/atc01/CM/repos/local/cache/a5704355393f46b3
               ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
/home/atc01/CM/repos/gateoverflow@cm4mlops/script/get-generic-python-lib/detect-version.py:23: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
               ! call "detect_version" from /home/atc01/CM/repos/gateoverflow@cm4mlops/script/get-generic-python-lib/customize.py

          Extra PIP CMD:   --break-system-packages

           ! cd /home/atc01/CM/repos/local/cache/a5704355393f46b3
           ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/get-generic-python-lib/install.sh from tmp-run.sh

/usr/bin/python3 -m pip install "dmiparser" --break-system-packages
Collecting dmiparser
  Using cached dmiparser-5.1-py3-none-any.whl.metadata (5.7 kB)
Using cached dmiparser-5.1-py3-none-any.whl (8.3 kB)
Installing collected packages: dmiparser
Successfully installed dmiparser-5.1
WARNING: Target directory /mnt/sdb/my_packages/dmiparser-5.1.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /mnt/sdb/my_packages/dmiparser already exists. Specify --upgrade to force replacement.
WARNING: Target directory /mnt/sdb/my_packages/bin already exists. Specify --upgrade to force replacement.
           ! cd /home/atc01/CM/repos/local/cache/a5704355393f46b3
           ! call /home/atc01/CM/repos/gateoverflow@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
/home/atc01/CM/repos/gateoverflow@cm4mlops/script/get-generic-python-lib/detect-version.py:23: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
           ! call "postprocess" from /home/atc01/CM/repos/gateoverflow@cm4mlops/script/get-generic-python-lib/customize.py

CM error: version was not detected (
)!
--model=bert-99: command not found
--implementation=nvidia: command not found
--framework=tensorrt: command not found
--category=datacenter: command not found
--scenario=Offline: command not found
--execution-mode=test: command not found
--device=cuda: command not found
--docker: command not found
--test_query_count=100: command not found


I just reolved above issue about version checking. The issue was caused by pip download directory changed. I modified the directory with modifying ~/.bashrc file to add $PIP_TARGET=/path/a direcotry. I guess CM tool expected to check the version based on pip default direcory rather than changed new directory. After reverting the value to default the version checking was worked well and passed to other steps. But i have some problem to finish spwanning up container, that was SSD space not enough to accomodate whole data and docker images. For that reason i got error messages during creating a container due to no space remain to complete creating the container. Is there any solution to change the directory that support enough space (/mnt/sdb) than SSD storage that mounted root directory?

Glad that you sorted the pip issue. We'll check for better ways to handle this.

For space issue, are you trying to run all the Nvidia implementations or a specific one? If we ignore dlrm_v2, the space requirement for all others will be around 700 GB and dlrm_v2 in itself needs 7TB for datapreprocessing (once done about 500GB) is enough. Now, the container build space is not controlled by CM - you can use the docker settings to manage this. The path where data and models go is controlled by CM - you can use `$CM_REPOS` `ENV` variable to move the CM repos to a new disk. 

I just focused on resnet50 and offline scenario. dlrm_v2 isn't considering point for mlperf test now. Based on steps to building container image and downloading dataset, the data space to cache them is on $HOME/default path. My root file system is only 200GB and /mnt/sdb file system is 2TB. For that reason i'd like to change the cache or storing dataset path to /mnt/sdb/path. Could you provide exact example to change the path to me?

You can just use `export CM_REPOS=/mnt/sdb/CM` and do the remaining commands as usual. But, if docker build is failing, this may not work as docker will be taking space from the default partition. `docker system prune` can help in clearing the unused docker space. My estimate is at least 100GB free space on the default root for the container build or else you can change the default partition for docker -- I haven't done this recently but Google should help you here. 

Thanks to your solution, i could change the cach directory to other directory. The test seemed well and procceeded well. But in almost finish phase of test, i faced with some error like below.

Config file missing for given hw_name: '365e4de79521', implementation: 'nvidia_original', device: 'gpu,  backend: 'tensorrt', copying from default
        - running time of script "get,mlperf,inference,sut,configs,sut-configs": 0.10 sec.
      - Processing env after dependencies ...
      - Running preprocess ...
Using MLCommons Inference source from '/home/cmuser/CM/repos/local/cache/0341978d6e564966/inference'
Original configuration value 1.0 target_qps
Adjusted configuration value 1.01 target_qps
Output Dir: '/home/cmuser/CM/repos/local/cache/79b5cf0d98af44e3/test_results/365e4de79521-nvidia_original-gpu-tensorrt-vdefault-default_config/resnet50/offline/performance/run_1'
resnet50.Offline.target_qps = 1.0
resnet50.Offline.max_query_count = 1000
resnet50.Offline.min_query_count = 1000
resnet50.Offline.min_duration = 0
resnet50.Offline.sample_concatenate_permutation = 0

Traceback (most recent call last):
  File "/home/cmuser/.local/bin/cm", line 8, in <module>
    sys.exit(run())
  File "/home/cmuser/.local/lib/python3.8/site-packages/cmind/cli.py", line 37, in run
    r = cm.access(argv, out='con')
  File "/home/cmuser/.local/lib/python3.8/site-packages/cmind/core.py", line 602, in access
    r = action_addr(i)
  File "/home/cmuser/CM/repos/gateoverflow@cm4mlops/automation/script/module.py", line 211, in run
    r = self._run(i)
  File "/home/cmuser/CM/repos/gateoverflow@cm4mlops/automation/script/module.py", line 1553, in _run
    r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta,  env, state, const, const_state, add_deps_recursive,
  File "/home/cmuser/CM/repos/gateoverflow@cm4mlops/automation/script/module.py", line 2909, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/home/cmuser/CM/repos/gateoverflow@cm4mlops/automation/script/module.py", line 3080, in _run_deps
    r = self.cmind.access(ii)
  File "/home/cmuser/.local/lib/python3.8/site-packages/cmind/core.py", line 602, in access
    r = action_addr(i)
  File "/home/cmuser/CM/repos/gateoverflow@cm4mlops/automation/script/module.py", line 211, in run
    r = self._run(i)
  File "/home/cmuser/CM/repos/gateoverflow@cm4mlops/automation/script/module.py", line 1380, in _run
    r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "/home/cmuser/CM/repos/gateoverflow@cm4mlops/automation/script/module.py", line 2909, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "/home/cmuser/CM/repos/gateoverflow@cm4mlops/automation/script/module.py", line 3080, in _run_deps
    r = self.cmind.access(ii)
  File "/home/cmuser/.local/lib/python3.8/site-packages/cmind/core.py", line 602, in access
    r = action_addr(i)
  File "/home/cmuser/CM/repos/gateoverflow@cm4mlops/automation/script/module.py", line 211, in run
    r = self._run(i)
  File "/home/cmuser/CM/repos/gateoverflow@cm4mlops/automation/script/module.py", line 1490, in _run
    r = customize_code.preprocess(ii)
  File "/home/cmuser/CM/repos/gateoverflow@cm4mlops/script/generate-mlperf-inference-user-conf/customize.py", line 348, in preprocess
    os.makedirs(OUTPUT_DIR, exist_ok=True)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  [Previous line repeated 2 more times]
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/cmuser/CM/repos/local/cache/79b5cf0d98af44e3/test_results'
       ! call "postprocess" from /root/CM/repos/gateoverflow@cm4mlops/script/run-mlperf-inference-app/customize.py

Oh, is the CM_REPOS now pointing to an NFS mount? Can you please try ```chmod -R 777 $CM_REPOS/CM/repos/local/cache/``` on the host machine? This shouldn't need a rebuild of the container.

After changed write permission for the cache directory the test was good. I really appreciate your thoroughly support.

cat mlperf_log_summary.txt
================================================
MLPerf Results Summary
================================================
SUT name : LWIS_Server
Scenario : Offline
Mode     : PerformanceOnly
Samples per second: 25219.9
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes


That's glad to hear :) Please keep this issue open until we resolve the cache permission issue for mounted dirs (we'll handle this in CM).

Sure. I am looking forward to  hearing the enhancement of permission issue. 

Thank you. We have added the fix to use the host userid inside the docker container. It should be visible if you do `cm pull repo`.

I tried to test again after cm pull repo. But i got some issue like below.

 => ERROR [ 3/12] RUN apt-get install -y python3 python3-pip git sudo wget python3-venv                                                                                                   0.2s
------
 > [ 3/12] RUN apt-get install -y python3 python3-pip git sudo wget python3-venv:
0.187 Reading package lists...
0.196 Building dependency tree...
0.197 Reading state information...
0.197 Package git is not available, but is referred to by another package.
0.197 This may mean that the package is missing, has been obsoleted, or
0.197 is only available from another source
0.197
0.197 Package python3-venv is not available, but is referred to by another package.
0.197 This may mean that the package is missing, has been obsoleted, or
0.197 is only available from another source
0.197
0.197 E: Package 'git' has no installation candidate
0.197 E: Unable to locate package sudo
0.197 E: Package 'python3-venv' has no installation candidate
------
mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public.Dockerfile:20
--------------------
  18 |     # Install system dependencies
  19 |     RUN apt-get update -y
  20 | >>> RUN apt-get install -y python3 python3-pip git sudo wget python3-venv
  21 |
  22 |     # Setup docker environment
--------------------
ERROR: failed to solve: process "/bin/bash -c apt-get install -y python3 python3-pip git sudo wget python3-venv" did not complete successfully: exit code: 100

CM error: Portable CM script failed (name = build-docker-image, return code = 256)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Note that it is often a portability issue of a third-party tool or a native script
wrapped and unified by this CM script (automation recipe). Please re-run
this script with --repro flag and report this issue with the original
command line, cm-repro directory and full log here:

https://github.com/mlcommons/cm4mlops/issues

The CM concept is to collaboratively fix such issues inside portable CM scripts
to make existing tools and native scripts more portable, interoperable
and deterministic. Thank you!


That looks like a network connection issue - can you please give a retry?

Thank you for update. And test was good.
I already created inference related image like below. How can i run the image when i just execute mlperf inference sceinario. I already tried to execute docker run command then the image kept exited status.
Also how to clearn up existing all data and information for taking new test again.

 docker images
REPOSITORY                                                 TAG                   IMAGE ID       CREATED       SIZE
cknowledge/cm-script-app-mlperf-inference                  ubuntu-20.04-latest   8ee88cf5c604   6 days ago    44GB
ghcr.io/k8snetworkplumbingwg/sriov-network-device-plugin   latest                36cc0b497250   2 weeks ago   57.3MB
pytorch_nccl_work                                          latest                6f33c66a8f17   3 weeks ago   20.4GB
pytorch_nccl_master                                        latest                9ca71f68bcd1   3 weeks ago   20.4GB
rdma/sriov-plugin                                          latest                17072d351ad6   5 years ago   64.5MB


We suggest to follow the CM commands to run the docker as it takes care of all the needed volume mounts, device mappings etc. The only `docker` command needed will be `docker start <ID>` incase the docker container crashes. 

 To do a rerun, you can just use `--rerun` flag. And to clean all the cache you can do `cm rm cache -f`. 

Thank you for your support. I got good experience to handle this case with you.
Thank you again.