Repository: mlcommons/inference
Issue: Fail to build and launch the docker in closed/NVIDIA folder
Root cause: Environment Incompatibility

After installing the docer engine and the NVIDIA Container with rootless mode configured,  and then clone mlcommons/inference_results_v4.0.git and configure the MLPERF_SCRATCH_PATH, and start to build and launch a docker container but failed with the below log errors: the 1st error is related to conda configuration, and I don't know how to solve it and the more details in the attached log.
[Commands log.txt](https://github.com/user-attachments/files/15528210/Commands.log.txt)


****************************************************************************************************************
tomcat@tomcat-Lenovo-Product:~/mlperf4.0/inference_results_v4.0/closed/NVIDIA$ make prebuild
make[1]: Entering directory '/home/tomcat/mlperf4.0/inference_results_v4.0/closed/NVIDIA'
Building Docker image
USE_LEGACY: 0
USE_NGC: 1
BASE_IMAGE: nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public
DOCKER_FILENAME: docker/Dockerfile.x86_64
docker pull nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public
mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public: Pulling from nvidia/mlperf/mlperf-inference
Digest: sha256:816bd0cc61b061c9b9e2ebecabd67f710b05740882f04a6a47ac5a5875635759
Status: Image is up to date for nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public
nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public
DOCKER_BUILDKIT=1 docker build -t mlperf-inference:tomcat-x86_64-latest \
	--build-arg BASE_IMAGE=nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public \
	--build-arg CUDA_VER=12.2 \
	--build-arg DRIVER_VER_MAJOR=515 \
	--build-arg USE_NIGHTLY=0 \'m not sure whether there is something wrong with the rootless mode configuration?
	--build-arg USE_NGC=1 \
	--build-arg EXTERNAL_USER=1 \
	--build-arg MITTEN_VER=0.1.3b0 \
	--network host \
	-f docker/Dockerfile.x86_64 docker
[+] Building 1778.9s (27/44)                                                                                                                                                                     docker:rootless
 => [internal] load build definition from Dockerfile.x86_64                                                                                                                                                 0.0s
 => => transferring dockerfile: 12.22kB                                                                                                                                                                     0.0s
 => [internal] load metadata for nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public                                                                             0.0s
 => [internal] load .dockerignore                                                                                                                                                                           0.0s
 => => transferring context: 2B                                                                                                                                                                             0.0s
 => [ 1/40] FROM nvcr.io/nvidia/mlperf/mlperf-inference:mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public                                                                                             0.0s
 => [internal] load build context                                                                                                                                                                           0.0s
 => => transferring context: 92B                                                                                                                                                                            0.0s
 => CACHED [ 2/40] RUN rm -f /etc/apt/sources.list.d/cuda.list  && apt update  && apt install -y --no-install-recommends build-essential autoconf libtool git git-lfs         ccache curl wget pkg-config   0.0s
 => CACHED [ 3/40] RUN apt install -y --no-install-recommends pkg-config zip g++ zlib1g-dev unzip                                                                                                           0.0s
 => CACHED [ 4/40] RUN apt install -y --no-install-recommends libarchive-dev                                                                                                                                0.0s
 => CACHED [ 5/40] RUN apt install -y --no-install-recommends rsync                                                                                                                                         0.0s
 => CACHED [ 6/40] RUN if [[ 1 = 0 ]]; then     cd /tmp     && install_deb_pkg() { wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64//$1 -O $1 && dpkg -i $1 && rm $1; }      0.0s
 => CACHED [ 7/40] RUN if [[ 1 = 0 ]]; then rm -rf /usr/local/lib/python3.8/dist-packages/tensorrt/; fi                                                                                                     0.0s
 => CACHED [ 8/40] RUN if [[ 1 = 0 ]]; then if [[ 0 = 0 ]]; then     cd /tmp     && install_deb_pkg() { wget -q http://cuda-repo/release-candidates/Libraries/TensorRT/v9.3/9.3.0.1-d6cbd29d/12.2-r535/Ubu  0.0s
 => CACHED [ 9/40] RUN ln -sf /usr/bin/python3 /usr/bin/python                                                                                                                                              0.0s
 => CACHED [10/40] RUN ln -sf /usr/bin/pip3 /usr/bin/pip                                                                                                                                                    0.0s
 => CACHED [11/40] RUN apt install -y sox                                                                                                                                                                   0.0s
 => CACHED [12/40] RUN apt install -y --no-install-recommends libsndfile1-dev                                                                                                                               0.0s
 => CACHED [13/40] RUN apt install -y rapidjson-dev                                                                                                                                                         0.0s
 => CACHED [14/40] RUN apt install -y libb64-dev                                                                                                                                                            0.0s
 => CACHED [15/40] RUN apt install -y libgtest-dev                                                                                                                                                          0.0s
 => CACHED [16/40] COPY requirements.x86_64.1.txt requirements.x86_64.2.txt nvmitten-0.1.3b0-cp38-cp38-linux_x86_64.whl*     faster-transformer-bert-fp8-weights-scales.tar.gz* /tmp                        0.0s
 => CACHED [17/40] RUN if [[ 1 = 1 ]]; then     mv /opt/nvmitten-0.1.3b0-cp38-cp38-linux_x86_64.whl /tmp     && mv /opt/faster-transformer-bert-fp8-weights-scales.tar.gz /tmp; fi                          0.0s
 => CACHED [18/40] WORKDIR /tmp                                                                                                                                                                             0.0s
 => CACHED [19/40] RUN python3 -m pip install --upgrade pip  && python3 -m pip install --upgrade setuptools wheel virtualenv                                                                                0.0s
 => CACHED [20/40] RUN python3 -m pip install -r requirements.x86_64.1.txt  && python3 -m pip install -r requirements.x86_64.2.txt                                                                          0.0s
 => CACHED [21/40] WORKDIR /tmp                                                                                                                                                                             0.0s
 => CACHED [22/40] RUN if [[ 1 != 0 ]]; then     wget https://repo.anaconda.com/miniconda/Miniconda3-py38_23.5.2-0-Linux-x86_64.sh     && bash Miniconda3-py38_23.5.2-0-Linux-x86_64.sh -b -p /opt/minicon  0.0s
 => ERROR [23/40] RUN if [[ 1 != 0 ]]; then     conda config --env --set always_yes true     && conda install cmake ninja     && cd /tmp     && export CUDA_NVCC_EXECUTABLE="/usr/local/cuda/bin/nvcc"   1778.8s




All the models except Stable diffusion and LLama2 from Nvidia 4.0 submissions can be reproduced by using CM. You can see the run commands here:

https://docs.mlcommons.org/inference/benchmarks/image_classification/resnet50/#__tabbed_29_1

Thank you Arjunsuresh!

If using CM that means I should pull down the mlcommons@cm4mlops instead of the below closed/NVIDIA? why not use the NVIDIA distribution for NV GPU based inference?

https://github.com/mlcommons/inference_results_v4.0/tree/main/closed/NVIDIA 

Actually you just need to install `cmind` and then pull the CM scripts repository given [here](https://docs.mlcommons.org/inference/install/)

CM is doing just workflow automation. Underneath, CM is using Nvidia docker container and the same Nvidia code albeit the way the code is run is different as CM unify the run across all the MLPerf implementations. CM also does some code changes to support more Nvidia GPUs out of the box.

ok, I will try the cmind from the link you provided. Wish everything better!
Thanks guy, good evening!

Hi Arjunsuresh,

I tried the CM script from your [here](https://docs.mlcommons.org/inference/install/) and run the resnet50 following the guide in https://docs.mlcommons.org/inference/benchmarks/image_classification/resnet50/. But the CM script failed after my poor network failed to access www.dropbox.com for dataset downloading.

I have downloaded the dataset for www.dropbox.com manually and save it in the local storage.  How can I ask CM script to use the dataset stored in my local folder instead of downloading it from www.dropbox.com each time?

[Untitled Document 1.txt](https://github.com/user-attachments/files/15599130/Untitled.Document.1.txt)