Repository: mlcommons/inference
Issue: MLPerf-loadgen error: Read-only file system
Root cause: Configuration Error

I get a read-only file error when running the resnet-50 benchmark. I'm running on a shared hpc system without sudo access, using ngc tensorflow container (`tensorflow-24.05-tf2-py3`) with Singularity (Apptainer). I use `pip install --user` for most of the packages, and set the appropriate python environments. 

I can see that the MLPerf loadgen is installed correctly in my `$PYTHONUSERBASE` location, but the automation script is still searching for it in `/usr/local/lib/python3.10`, which is a read-only directory. 

I was able to run this same benchmark fine a few weeks ago, and I'm also able it to run it inside a python virtual environment (when the python is not /usr/bin/python), but doesnt work for this current scenario.

I have attached the `cm-repro` log 
[cm-run-script-info.json](https://github.com/user-attachments/files/16036291/cm-run-script-info.json)
[cm-run-script-input.json](https://github.com/user-attachments/files/16036292/cm-run-script-input.json)
files.

<details>

<summary>Here is the (partial) terminal output: </summary>

```sh
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:24:28_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

Compiling program ...


Running program ...

/oscar/data/datasci/pthakur8/myProjects/test3
           ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/customize.py
GPU Device ID: 0
GPU Name: NVIDIA GH200 480GB MIG 2g.24gb
GPU compute capability: 9.0
CUDA driver version: 12.4
CUDA runtime version: 12.4
Global memory: 24696061952
Max clock rate: 1980.000000 MHz
Total amount of shared memory per block: 49152
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor:  2048
Maximum number of threads per block: 1024
Max dimension size of a thread block X: 1024
Max dimension size of a thread block Y: 1024
Max dimension size of a thread block Z: 64
Max dimension size of a grid size X: 2147483647
Max dimension size of a grid size Y: 65535
Max dimension size of a grid size Z: 65535


    * cm run script "get generic-python-lib _package.dmiparser"

      * cm run script "detect os"
             ! cd /tmp/myenv/CM/repos/local/cache/d0a7f2c848894f76
             ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py

      * cm run script "detect cpu"

        * cm run script "detect os"
               ! cd /tmp/myenv/CM/repos/local/cache/d0a7f2c848894f76
               ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
               ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
             ! cd /tmp/myenv/CM/repos/local/cache/d0a7f2c848894f76
             ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py

      * cm run script "get python3"
           ! load /tmp/myenv/CM/repos/local/cache/675e702ab8ab49f7/cm-cached-state.json

Path to Python: /usr/bin/python3
Python version: 3.10.12


      * cm run script "get generic-python-lib _pip"
           ! load /tmp/myenv/CM/repos/local/cache/1a5d7303b29e4fca/cm-cached-state.json
               ! cd /tmp/myenv/CM/repos/local/cache/d0a7f2c848894f76
               ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
               ! call "detect_version" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py
          Detected version: 5.1
           ! cd /tmp/myenv/CM/repos/local/cache/d0a7f2c848894f76
           ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
           ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py

    * cm run script "get cache dir _name.mlperf-inference-sut-descriptions"
           ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-cache-dir/customize.py
Generating SUT description file for default-tf
HW description file for default not found. Copying from default!!!
         ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-sut-description/customize.py

  * cm run script "get mlperf inference results dir"
         ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-results-dir/customize.py

  * cm run script "install pip-package for-cmind-python _package.tabulate"
Requirement already satisfied: tabulate in /tmp/myenv/python/lib/python3.10/site-packages (0.9.0)
         ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/install-pip-package-for-cmind-python/customize.py

  * cm run script "get mlperf inference utils"

    * cm run script "get mlperf inference src"
         ! load /tmp/myenv/CM/repos/local/cache/e031a752477c4494/cm-cached-state.json
         ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-utils/customize.py
Using MLCommons Inference source from /tmp/myenv/CM/repos/local/cache/a1ce846fe5bc406f/inference

Running loadgen scenario: Offline and mode: performance

* cm run script "app mlperf inference generic _reference _resnet50 _tf _cuda _test _offline"

  * cm run script "detect os"
         ! cd /oscar/data/datasci/pthakur8/myProjects/test3
         ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
         ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py

  * cm run script "get sys-utils-cm"
=================================================
WARNINGS:

  This CM script will install extra OS system utils required for CM automation workflows!
=================================================

    * cm run script "detect os"
           ! cd /tmp/myenv/CM/repos/local/cache/9c3b6b5cdb4340a2
           ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
           ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py

  * cm run script "get python"
       ! load /tmp/myenv/CM/repos/local/cache/675e702ab8ab49f7/cm-cached-state.json

Path to Python: /usr/bin/python3
Python version: 3.10.12


  * cm run script "get mlcommons inference src"
       ! load /tmp/myenv/CM/repos/local/cache/e031a752477c4494/cm-cached-state.json

  * cm run script "get mlperf inference utils"

    * cm run script "get mlperf inference src"
         ! load /tmp/myenv/CM/repos/local/cache/e031a752477c4494/cm-cached-state.json
         ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-utils/customize.py

  * cm run script "get dataset-aux imagenet-aux"
         ! cd /tmp/myenv/CM/repos/local/cache/f278167ab96f499c
         ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-dataset-imagenet-aux/run.sh from tmp-run.sh

--2024-06-28 15:31:32--  https://www.dropbox.com/s/92n2fyej3lzy3s3/caffe_ilsvrc12.tar.gz
Resolving www.dropbox.com (www.dropbox.com)... 162.125.4.18
Connecting to www.dropbox.com (www.dropbox.com)|162.125.4.18|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://www.dropbox.com/scl/fi/pdhe77hzkuix2tf4rwc9v/caffe_ilsvrc12.tar.gz?rlkey=6p4gu3s6ajewrlwcrx7mp4re8 [following]
--2024-06-28 15:31:32--  https://www.dropbox.com/scl/fi/pdhe77hzkuix2tf4rwc9v/caffe_ilsvrc12.tar.gz?rlkey=6p4gu3s6ajewrlwcrx7mp4re8
Reusing existing connection to www.dropbox.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://uc852c3ed0ce820b60c7d996c584.dl.dropboxusercontent.com/cd/0/inline/CVtrhpMcdq8K-_8xYMjFMRDufPYCFGZOomlt8qgiy0mfkyZwGQQxhwAucnS36UBFaIMcv7QpXkhuB_19R0nWCpe7VJoXAHEe5v4YUIah2O5A8tL8ZI-wRQ7IRWpg8ImS3OI/file# [following]
--2024-06-28 15:31:33--  https://uc852c3ed0ce820b60c7d996c584.dl.dropboxusercontent.com/cd/0/inline/CVtrhpMcdq8K-_8xYMjFMRDufPYCFGZOomlt8qgiy0mfkyZwGQQxhwAucnS36UBFaIMcv7QpXkhuB_19R0nWCpe7VJoXAHEe5v4YUIah2O5A8tL8ZI-wRQ7IRWpg8ImS3OI/file
Resolving uc852c3ed0ce820b60c7d996c584.dl.dropboxusercontent.com (uc852c3ed0ce820b60c7d996c584.dl.dropboxusercontent.com)... 162.125.4.15
Connecting to uc852c3ed0ce820b60c7d996c584.dl.dropboxusercontent.com (uc852c3ed0ce820b60c7d996c584.dl.dropboxusercontent.com)|162.125.4.15|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /cd/0/inline2/CVtluJecUDp-dWYsYPW8bgHGg08AZ_egKCc2IAEUUXbywHpwJloyJBu3rpqHz7ocrrpizjND_QW_4vaQQSAVDoavJGbrWmKDG1UHErb3dRICKq7TYyqZzQ5EA0J_TCWUu4PeoNcKvWNn_GM-Lkl3K62yLuz2jGqPVhA7BbBoVKnI5pxt7-ldBBWcEd6Mh_Z-j69rNm-ebUiD5JTozQscq-ob3lm0phgNY9t2tyPo9DiTmSwq-OEzxKzl3Z4hHk-UpdTWQ1_YjIXwfNiJ2xspQcWfyPTKCIR7i9F-hO3MHWnQOHATtxT-9lbX33SuJ2nJtF-F-iSCS7sk-NyS4KDR32OvH41S-_uyR9mwiUE0QphubQ/file [following]
--2024-06-28 15:31:33--  https://uc852c3ed0ce820b60c7d996c584.dl.dropboxusercontent.com/cd/0/inline2/CVtluJecUDp-dWYsYPW8bgHGg08AZ_egKCc2IAEUUXbywHpwJloyJBu3rpqHz7ocrrpizjND_QW_4vaQQSAVDoavJGbrWmKDG1UHErb3dRICKq7TYyqZzQ5EA0J_TCWUu4PeoNcKvWNn_GM-Lkl3K62yLuz2jGqPVhA7BbBoVKnI5pxt7-ldBBWcEd6Mh_Z-j69rNm-ebUiD5JTozQscq-ob3lm0phgNY9t2tyPo9DiTmSwq-OEzxKzl3Z4hHk-UpdTWQ1_YjIXwfNiJ2xspQcWfyPTKCIR7i9F-hO3MHWnQOHATtxT-9lbX33SuJ2nJtF-F-iSCS7sk-NyS4KDR32OvH41S-_uyR9mwiUE0QphubQ/file
Reusing existing connection to uc852c3ed0ce820b60c7d996c584.dl.dropboxusercontent.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 17852691 (17M) [application/octet-stream]
Saving to: ‘caffe_ilsvrc12.tar.gz’

caffe_ilsvrc12.tar.gz    100%[===============================>]  17.03M  44.3MB/s    in 0.4s

2024-06-28 15:31:34 (44.3 MB/s) - ‘caffe_ilsvrc12.tar.gz’ saved [17852691/17852691]

det_synset_words.txt
imagenet.bet.pickle
imagenet_mean.binaryproto
synsets.txt
synset_words.txt
test.txt
train.txt
val.txt

  * cm run script "get cuda-devices"

    * cm run script "get cuda _toolkit"
         ! load /tmp/myenv/CM/repos/local/cache/e14674588c7d4a48/cm-cached-state.json

ENV[CM_CUDA_PATH_LIB_CUDNN_EXISTS]: no
ENV[CM_CUDA_VERSION]: 12.4
ENV[CM_CUDA_VERSION_STRING]: cu124
ENV[CM_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
ENV[CUDA_HOME]: /usr/local/cuda

         ! cd /tmp/myenv/CM/repos/local/cache/9c3b6b5cdb4340a2
         ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/run.sh from tmp-run.sh
rm: cannot remove 'a.out': No such file or directory

Checking compiler version ...

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:24:28_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

Compiling program ...


Running program ...

/tmp/myenv/CM/repos/local/cache/9c3b6b5cdb4340a2
         ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/customize.py
GPU Device ID: 0
GPU Name: NVIDIA GH200 480GB MIG 2g.24gb
GPU compute capability: 9.0
CUDA driver version: 12.4
CUDA runtime version: 12.4
Global memory: 24696061952
Max clock rate: 1980.000000 MHz
Total amount of shared memory per block: 49152
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor:  2048
Maximum number of threads per block: 1024
Max dimension size of a thread block X: 1024
Max dimension size of a thread block Y: 1024
Max dimension size of a thread block Z: 64
Max dimension size of a grid size X: 2147483647
Max dimension size of a grid size Y: 65535
Max dimension size of a grid size Z: 65535



  * cm run script "app mlperf reference inference _cuda _tf _offline _resnet50 _fp32"

    * cm run script "detect os"
           ! cd /tmp/myenv/CM/repos/local/cache/9c3b6b5cdb4340a2
           ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
           ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py

    * cm run script "detect cpu"

      * cm run script "detect os"
             ! cd /tmp/myenv/CM/repos/local/cache/9c3b6b5cdb4340a2
             ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
           ! cd /tmp/myenv/CM/repos/local/cache/9c3b6b5cdb4340a2
           ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
           ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py

    * cm run script "get sys-utils-cm"
=================================================
WARNINGS:

  This CM script will install extra OS system utils required for CM automation workflows!
=================================================

      * cm run script "detect os"
             ! cd /tmp/myenv/CM/repos/local/cache/9c3b6b5cdb4340a2
             ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py

    * cm run script "get python"
         ! load /tmp/myenv/CM/repos/local/cache/675e702ab8ab49f7/cm-cached-state.json

Path to Python: /usr/bin/python3
Python version: 3.10.12


    * cm run script "get cuda _cudnn"

      * cm run script "detect os"
             ! cd /tmp/myenv/CM/repos/local/cache/97ef52676ff04380
             ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
        # Requested paths: /usr/local/nvm/versions/node/v16.20.2/bin:/tmp/myenv/python/bin:/usr/local/mpi/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ucx/bin:/opt/tensorrt/bin:/usr/local/cuda/bin:/usr/cuda/bin:/usr/local/cuda-11/bin:/usr/cuda-11/bin:/usr/local/cuda-12/bin:/usr/cuda-12/bin:/usr/local/packages/cuda

        * /usr/local/cuda/bin/nvcc
               ! cd /tmp/myenv/CM/repos/local/cache/97ef52676ff04380
               ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-cuda/run.sh from tmp-run.sh
               ! call "detect_version" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-cuda/customize.py
        Detected version: 12.4

        # Found artifact in /usr/local/cuda/bin/nvcc
           ! cd /tmp/myenv/CM/repos/local/cache/97ef52676ff04380
           ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-cuda/run.sh from tmp-run.sh
           ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-cuda/customize.py
        Detected version: 12.4

    * cm run script "get nvidia cudnn"

      * cm run script "detect os"
             ! cd /tmp/myenv/CM/repos/local/cache/29e836b22b33414d
             ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
        # Requested paths: /usr/local/nvm/versions/node/v16.20.2/bin:/tmp/myenv/python/bin:/usr/local/mpi/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ucx/bin:/opt/tensorrt/bin:/usr/local/cuda/bin:/usr/cuda/bin:/usr/local/cuda-11/bin:/usr/cuda-11/bin:/usr/local/cuda-12/bin:/usr/cuda-12/bin:/usr/local/packages/cuda:/usr/local/cuda/lib64:/usr/cuda/lib64:/usr/local/cuda/lib:/usr/cuda/lib:/usr/local/cuda-11/lib64:/usr/cuda-11/lib:/usr/local/cuda-12/lib:/usr/cuda-12/lib:/usr/local/packages/cuda/lib:/lib/aarch64-linux-gnu:/usr/lib/aarch64-linux-gnu:/usr/local/lib:/lib:/usr/lib

        # Found artifact in /lib/aarch64-linux-gnu/libcudnn.so
           ! cd /tmp/myenv/CM/repos/local/cache/29e836b22b33414d
           ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-cudnn/run.sh from tmp-run.sh
           ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-cudnn/customize.py

ENV[CM_CUDA_PATH_INCLUDE_CUDNN]: /usr/include
ENV[CM_CUDA_PATH_LIB_CUDNN]: /lib/aarch64-linux-gnu
ENV[CM_CUDNN_VERSION]: 9.1.0


ENV[CM_CUDA_PATH_LIB_CUDNN_EXISTS]: yes
ENV[CM_CUDA_VERSION]: 12.4
ENV[CM_CUDA_VERSION_STRING]: cu124
ENV[CM_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
ENV[CUDA_HOME]: /usr/local/cuda


    * cm run script "get generic-python-lib _tensorflow"

      * cm run script "detect os"
             ! cd /tmp/myenv/CM/repos/local/cache/595a3199ec274a8d
             ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py

      * cm run script "detect cpu"

        * cm run script "detect os"
               ! cd /tmp/myenv/CM/repos/local/cache/595a3199ec274a8d
               ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
               ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
             ! cd /tmp/myenv/CM/repos/local/cache/595a3199ec274a8d
             ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py

      * cm run script "get python3"
           ! load /tmp/myenv/CM/repos/local/cache/675e702ab8ab49f7/cm-cached-state.json

Path to Python: /usr/bin/python3
Python version: 3.10.12


      * cm run script "get generic-python-lib _pip"
           ! load /tmp/myenv/CM/repos/local/cache/1a5d7303b29e4fca/cm-cached-state.json
               ! cd /tmp/myenv/CM/repos/local/cache/595a3199ec274a8d
               ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
               ! call "detect_version" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py
          Detected version: 2.15.0
           ! cd /tmp/myenv/CM/repos/local/cache/595a3199ec274a8d
           ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
           ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py

    * cm run script "get ml-model image-classification resnet50 raw _fp32 _tf"

      * cm run script "download-and-extract _url.https://zenodo.org/record/2535873/files/resnet50_v1.pb"

        * cm run script "download file _cmutil _url.https://zenodo.org/record/2535873/files/resnet50_v1.pb"

          * cm run script "detect os"
                 ! cd /tmp/myenv/CM/repos/local/cache/562e94e7fa5b4af9
                 ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
                 ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py

Downloading from https://zenodo.org/record/2535873/files/resnet50_v1.pb

Downloading to /tmp/myenv/CM/repos/local/cache/562e94e7fa5b4af9/resnet50_v1.pb

Downloaded: 100%
               ! cd /tmp/myenv/CM/repos/local/cache/562e94e7fa5b4af9
               ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/download-file/run.sh from tmp-run.sh
               ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/download-file/customize.py
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/download-and-extract/customize.py
           ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-ml-model-resnet50/customize.py

Path to the ML model: /tmp/myenv/CM/repos/local/cache/562e94e7fa5b4af9/resnet50_v1.pb


    * cm run script "get dataset image-classification imagenet preprocessed _full _NHWC"

      * cm run script "get python3"
           ! load /tmp/myenv/CM/repos/local/cache/675e702ab8ab49f7/cm-cached-state.json

Path to Python: /usr/bin/python3
Python version: 3.10.12


      * cm run script "get dataset image-classification original _full"
           ! load /tmp/myenv/CM/repos/local/cache/9a972955132c459c/cm-cached-state.json

      * cm run script "get dataset-aux image-classification imagenet-aux"
           ! load /tmp/myenv/CM/repos/local/cache/f278167ab96f499c/cm-cached-state.json

      * cm run script "get generic-python-lib _package.opencv-python-headless"

        * cm run script "detect os"
               ! cd /tmp/myenv/CM/repos/local/cache/e9d2e63c7fc64f00
               ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
               ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py

        * cm run script "detect cpu"

          * cm run script "detect os"
                 ! cd /tmp/myenv/CM/repos/local/cache/e9d2e63c7fc64f00
                 ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
                 ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
               ! cd /tmp/myenv/CM/repos/local/cache/e9d2e63c7fc64f00
               ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
               ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py

        * cm run script "get python3"
             ! load /tmp/myenv/CM/repos/local/cache/675e702ab8ab49f7/cm-cached-state.json

Path to Python: /usr/bin/python3
Python version: 3.10.12


        * cm run script "get generic-python-lib _pip"
             ! load /tmp/myenv/CM/repos/local/cache/1a5d7303b29e4fca/cm-cached-state.json
                 ! cd /tmp/myenv/CM/repos/local/cache/e9d2e63c7fc64f00
                 ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
                 ! call "detect_version" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py
            Detected version: 4.10.0.84
             ! cd /tmp/myenv/CM/repos/local/cache/e9d2e63c7fc64f00
             ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py

      * cm run script "get generic-python-lib _pillow"

        * cm run script "detect os"
               ! cd /tmp/myenv/CM/repos/local/cache/72ef7807d5fe406b
               ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
               ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py

        * cm run script "detect cpu"

          * cm run script "detect os"
                 ! cd /tmp/myenv/CM/repos/local/cache/72ef7807d5fe406b
                 ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
                 ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
               ! cd /tmp/myenv/CM/repos/local/cache/72ef7807d5fe406b
               ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
               ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py

        * cm run script "get python3"
             ! load /tmp/myenv/CM/repos/local/cache/675e702ab8ab49f7/cm-cached-state.json

Path to Python: /usr/bin/python3
Python version: 3.10.12


        * cm run script "get generic-python-lib _pip"
             ! load /tmp/myenv/CM/repos/local/cache/1a5d7303b29e4fca/cm-cached-state.json
                 ! cd /tmp/myenv/CM/repos/local/cache/72ef7807d5fe406b
                 ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
                 ! call "detect_version" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py
            Detected version: 10.3.0
             ! cd /tmp/myenv/CM/repos/local/cache/72ef7807d5fe406b
             ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/run.sh from tmp-run.sh
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-generic-python-lib/customize.py

      * cm run script "mlperf mlcommons inference source src"
           ! load /tmp/myenv/CM/repos/local/cache/e031a752477c4494/cm-cached-state.json
Using MLCommons Inference source from '/tmp/myenv/CM/repos/local/cache/a1ce846fe5bc406f/inference'
           ! cd /tmp/myenv/CM/repos/local/cache/dc38363043f84c37
           ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-preprocessed-dataset-imagenet/run.sh from tmp-run.sh
INFO:imagenet:Preprocessing 50000 images using 72 threads
INFO:imagenet:loaded 50000 images, cache=True, already_preprocessed=False, took=148.4sec
           ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-preprocessed-dataset-imagenet/customize.py

    * cm run script "get dataset-aux image-classification imagenet-aux"
         ! load /tmp/myenv/CM/repos/local/cache/f278167ab96f499c/cm-cached-state.json

    * cm run script "generate user-conf mlperf inference"

      * cm run script "detect os"
             ! cd /tmp/myenv/CM/repos/local/cache/9c3b6b5cdb4340a2
             ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py

      * cm run script "detect cpu"

        * cm run script "detect os"
               ! cd /tmp/myenv/CM/repos/local/cache/9c3b6b5cdb4340a2
               ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
               ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
             ! cd /tmp/myenv/CM/repos/local/cache/9c3b6b5cdb4340a2
             ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py

      * cm run script "get python"
           ! load /tmp/myenv/CM/repos/local/cache/675e702ab8ab49f7/cm-cached-state.json

Path to Python: /usr/bin/python3
Python version: 3.10.12


      * cm run script "get mlcommons inference src"
           ! load /tmp/myenv/CM/repos/local/cache/e031a752477c4494/cm-cached-state.json

      * cm run script "get sut configs"

        * cm run script "get cache dir _name.mlperf-inference-sut-configs"
               ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-cache-dir/customize.py
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-sut-configs/customize.py
Config file missing for given hw_name: 'default', implementation: 'reference', device: 'gpu,  backend: 'tf', copying from default
Using MLCommons Inference source from '/tmp/myenv/CM/repos/local/cache/a1ce846fe5bc406f/inference'
Output Dir: '/tmp/myenv/CM/repos/local/cache/047cc2094dd74a96/test_results/default-reference-gpu-tf-v2.15.0-default_config/resnet50/offline/performance/run_1'
resnet50.Offline.target_qps = 1.0
resnet50.Offline.max_query_count = 10
resnet50.Offline.min_query_count = 10
resnet50.Offline.min_duration = 0
resnet50.Offline.sample_concatenate_permutation = 0


    * cm run script "get loadgen"

      * cm run script "detect os"
             ! cd /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe
             ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py

      * cm run script "get python3"
           ! load /tmp/myenv/CM/repos/local/cache/675e702ab8ab49f7/cm-cached-state.json

Path to Python: /usr/bin/python3
Python version: 3.10.12


      * cm run script "get mlcommons inference src"

        * cm run script "detect os"
               ! cd /tmp/myenv/CM/repos/local/cache/c53ff10f0a874ae8
               ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
               ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py

        * cm run script "get python3"
             ! load /tmp/myenv/CM/repos/local/cache/675e702ab8ab49f7/cm-cached-state.json

Path to Python: /usr/bin/python3
Python version: 3.10.12


        * cm run script "get git repo _branch.master _repo.https://github.com/mlcommons/inference"
             ! load /tmp/myenv/CM/repos/local/cache/a1ce846fe5bc406f/cm-cached-state.json

CM cache path to the Git repo: /tmp/myenv/CM/repos/local/cache/a1ce846fe5bc406f/inference

             ! call "postprocess" from /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-src/customize.py

      * cm run script "get compiler gcc"
           ! load /tmp/myenv/CM/repos/local/cache/3942c2c0ce3f4dca/cm-cached-state.json

      * cm run script "get cmake"
           ! load /tmp/myenv/CM/repos/local/cache/a5a1e70da94f4cb1/cm-cached-state.json

Path to the tool: /usr/bin/cmake


      * cm run script "get generic-python-lib _package.wheel"
           ! load /tmp/myenv/CM/repos/local/cache/d23d01ab1f5142f0/cm-cached-state.json

      * cm run script "get generic-python-lib _pip"
           ! load /tmp/myenv/CM/repos/local/cache/1a5d7303b29e4fca/cm-cached-state.json

      * cm run script "get generic-python-lib _package.pybind11"
           ! load /tmp/myenv/CM/repos/local/cache/f55f55ea3fdf456c/cm-cached-state.json

      * cm run script "get generic-python-lib _package.setuptools"
           ! load /tmp/myenv/CM/repos/local/cache/a69cd3c5a61c449c/cm-cached-state.json
           ! cd /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe
           ! call /tmp/myenv/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-loadgen/run.sh from tmp-run.sh
******************************************************
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
mlperf_loadgen v4.0
-- Using C++ compiler flags: -O2 -O3 -W -Wall
-- Using C++ standard: 14
-- Using static linker flags:
-- Using shared linker flags: -O2
-- Using output path: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/build
-- Found PythonInterp: /usr/bin/python3 (found version "3.10.12")
-- Using Python interpreter: /usr/bin/python3
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/build

------------------------------------------------------------------------------------------------------
<<<<<<<< Here there is a bunch of CMake build warnings that I omitted for brevity >>>>>>>>>>>>>>
------------------------------------------------------------------------------------------------------
[100%] Linking CXX executable benchmark
[100%] Built target benchmark
Install the project...
-- Install configuration: ""
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/lib/libmlperf_loadgen.a
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/benchmark
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/bindings
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/bindings/c_api.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/demos
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/demos/lon
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/demos/token_metrics
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/docs
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/docs/src
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/early_stopping.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/issue_query_controller.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/loadgen.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/logging.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/query_dispatch_library.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/query_sample.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/query_sample_library.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/results.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/system_under_test.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/test_settings.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/test_settings_internal.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/tests
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/tests/loadgen_test.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/tools
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/utils.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/version.h
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/__pycache__
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/generated
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/mlcommons_loadgen.egg-info
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/build
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/build/temp.linux-aarch64-cpython-310
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/build/temp.linux-aarch64-cpython-310/bindings
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/build/temp.linux-aarch64-cpython-310/generated
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/build/lib.linux-aarch64-cpython-310
-- Installing: /tmp/myenv/CM/repos/local/cache/3dda0037bdb147fe/install/include/build/bdist.linux
-aarch64
Processing /tmp/myenv/CM/repos/local/cache/a1ce846fe5bc406f/inference/loadgen
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: mlcommons_loadgen
  Building wheel for mlcommons_loadgen (pyproject.toml) ... done
  Created wheel for mlcommons_loadgen: filename=mlcommons_loadgen-4.1-cp310-cp310-linux_aarch64.whl size=401113 sha256=29ea13bce052146a5c2e07744b00481b98968000b30eb47215d136cc7bdb9179
  Stored in directory: /tmp/pip-ephem-wheel-cache-x2pzcpwt/wheels/1b/46/ef/46d4eb6545c72e516e97d16b4293c8965b7dd9d61866190c41
Successfully built mlcommons_loadgen
Installing collected packages: mlcommons_loadgen
  Attempting uninstall: mlcommons_loadgen
    Found existing installation: mlcommons_loadgen 4.0.1
    Uninstalling mlcommons_loadgen-4.0.1:
      Successfully uninstalled mlcommons_loadgen-4.0.1
  Rolling back uninstall of mlcommons_loadgen
  Moving to /tmp/myenv/python/lib/python3.10/site-packages/mlcommons_loadgen-4.0.1.dist-info/
   from /tmp/myenv/python/lib/python3.10/site-packages/~lcommons_loadgen-4.0.1.dist-info
  Moving to /tmp/myenv/python/lib/python3.10/site-packages/mlperf_loadgen.cpython-310-aarch64-linux-gnu.so
   from /tmp/pip-uninstall-8lo1tzl3/mlperf_loadgen.cpython-310-aarch64-linux-gnu.so
ERROR: Could not install packages due to an OSError: [Errno 30] Read-only file system: '/usr/local/lib/python3.10/dist-packages/mlperf_loadgen.cpython-310-aarch64-linux-gnu.so'


CM error: Portable CM script failed (name = get-mlperf-inference-loadgen, return code = 256)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Note that it is often a portability issue of a third-party tool or a native script
wrapped and unified by this CM script (automation recipe). Please re-run
this script with --repro flag and report this issue with the original
command line, cm-repro directory and full log here:

https://github.com/mlcommons/cm4mlops/issues

The CM concept is to collaboratively fix such issues inside portable CM scripts
to make existing tools and native scripts more portable, interoperable
and deterministic. Thank you!
```
</details>

Hi @thehalfspace there was a change in the loadgen build to go to the pip default location which is reverted now. Can you please do `cm pull repo` and retry?  Please let us know if any other python package installation fails. 

Thanks @arjunsuresh , I tried that and now I get a new (related?) error. I'm sharing the end of the output, the earlier output lines are pretty much same as above:

```sh
[100%] Linking CXX executable benchmark
[100%] Built target benchmark
Install the project...
-- Install configuration: ""
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/lib/libmlperf_loadgen.a
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/benchmark
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/bindings
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/bindings/c_api.h
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/demos
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/demos/lon
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/demos/token_metrics
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/docs
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/docs/src
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/early_stopping.h
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/issue_query_controller.h
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/loadgen.h
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/logging.h
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/query_dispatch_library.h
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/query_sample.h
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/query_sample_library.h
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/results.h
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/system_under_test.h
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/test_settings.h
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/test_settings_internal.h
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/tests
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/tests/loadgen_test.h
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/tools
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/utils.h
-- Installing: /tmp/CM/repos/local/cache/3ad6b72242464ef1/install/include/version.h
Processing /tmp/CM/repos/local/cache/b2571fca81814d5a/inference/loadgen
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: mlcommons_loadgen
  Building wheel for mlcommons_loadgen (pyproject.toml) ... done
  Created wheel for mlcommons_loadgen: filename=mlcommons_loadgen-4.1-cp310-cp310-linux_aarch64.whl size=401121 sha256=351551a30f8891abee57a9ec6656991edf082e30000c248a5f9d54f493bfdee9
  Stored in directory: /tmp/pip-ephem-wheel-cache-o5ifqcrq/wheels/82/33/d5/c60f92357a32e1d9de6158361561a0e06da437ae9741a3b5a4
Successfully built mlcommons_loadgen
Installing collected packages: mlcommons_loadgen
  Attempting uninstall: mlcommons_loadgen
    Found existing installation: mlcommons_loadgen 4.0.1
    Uninstalling mlcommons_loadgen-4.0.1:
ERROR: Could not install packages due to an OSError: [Errno 30] Read-only file system: 'top_level.txt'


CM error: Portable CM script failed (name = get-mlperf-inference-loadgen, return code = 256)


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Note that it is often a portability issue of a third-party tool or a native script
wrapped and unified by this CM script (automation recipe). Please re-run
this script with --repro flag and report this issue with the original
command line, cm-repro directory and full log here:

https://github.com/mlcommons/cm4mlops/issues

The CM concept is to collaboratively fix such issues inside portable CM scripts
to make existing tools and native scripts more portable, interoperable
and deterministic. Thank you!
```



Does creating/updating `$HOME/.config/pip/pip.conf` file with the below config help?
```
[global]
user = true
```

Okay yes that worked perfectly. Thanks @arjunsuresh