Repository: mlcommons/inference
Issue: Llama2-70b fails to build model 
Root cause: Documentation Deficiency

When running Llama2-70b  offline test with 1 GPU in a NVIDIA container. Fails to build model. 
With this error:
CM error: Quantised model absent - did not detect config.json in path /root/CM/repos/local/cache/daa1b0f5f8d74ccf/models/Llama2/fp8-quantized-ammo/llama2-70b-chat-hf-tp2pp1-fp8!

CM command line. 
cm run script  --tags=run-mlperf,inference,_find-performance,_full,_r4.1-dev    --model=llama2-70b-99    --implementation=nvidia    --framework=tensorrt    --category=datacenter    --scenario=Offline    --execution_mode=test    --device=cuda     --gpu_name=l4 --quiet    --test_query_count=50    --tp_size=2    --nvidia_llama2_dataset_file_path=/root/llamadataset/open_orca/open_orca_gpt4_tokenized_llama.sampled_24576.pkl --docker_cache=no 

I have no idea what will fix this.
Steve 

Screen output from run. 
root@4aa21844667d:~# cm run script  --tags=run-mlperf,inference,_find-performance,_full,_r4.1-dev    --model=llama2-70b-99    --implementation=nvidia    --framework=tensorrt    --category=datacenter    --scenario=Offline    --execution_mode=test    --device=cuda     --gpu_name=l4 --quiet    --test_query_count=50    --tp_size=2    --nvidia_llama2_dataset_file_path=/root/llamadataset/open_orca/open_orca_gpt4_tokenized_llama.sampled_24576.pkl --docker_cache=no 
INFO:root:* cm run script "run-mlperf inference _find-performance _full _r4.1-dev"
INFO:root:  * cm run script "detect os"
INFO:root:         ! cd /root
INFO:root:         ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:         ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:  * cm run script "detect cpu"
INFO:root:    * cm run script "detect os"
INFO:root:           ! cd /root
INFO:root:           ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:         ! cd /root
INFO:root:         ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:         ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:  * cm run script "get python3"
INFO:root:       ! load /root/CM/repos/local/cache/e64cc0d91f024ccb/cm-cached-state.json
INFO:root:Path to Python: /usr/bin/python3
INFO:root:Python version: 3.8.10
INFO:root:  * cm run script "get mlcommons inference src"
INFO:root:       ! load /root/CM/repos/local/cache/025c3b1288384fc3/cm-cached-state.json
INFO:root:  * cm run script "get sut description"
INFO:root:    * cm run script "detect os"
INFO:root:           ! cd /root
INFO:root:           ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:    * cm run script "detect cpu"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /root
INFO:root:             ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:           ! cd /root
INFO:root:           ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:    * cm run script "get python3"
INFO:root:         ! load /root/CM/repos/local/cache/e64cc0d91f024ccb/cm-cached-state.json
INFO:root:Path to Python: /usr/bin/python3
INFO:root:Python version: 3.8.10
INFO:root:    * cm run script "get compiler"
INFO:root:         ! load /root/CM/repos/local/cache/77d822e2d2b74a0d/cm-cached-state.json
INFO:root:    * cm run script "get cuda-devices _with-pycuda"
INFO:root:      * cm run script "get cuda _toolkit"
INFO:root:           ! load /root/CM/repos/local/cache/55ce6a990ddb44c8/cm-cached-state.json
INFO:root:ENV[CM_CUDA_PATH_LIB_CUDNN_EXISTS]: no
INFO:root:ENV[CM_CUDA_VERSION]: 12.2
INFO:root:ENV[CM_CUDA_VERSION_STRING]: cu122
INFO:root:ENV[CM_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
INFO:root:ENV[CUDA_HOME]: /usr/local/cuda
INFO:root:      * cm run script "get python3"
INFO:root:           ! load /root/CM/repos/local/cache/e64cc0d91f024ccb/cm-cached-state.json
INFO:root:Path to Python: /usr/bin/python3
INFO:root:Python version: 3.8.10
INFO:root:      * cm run script "get generic-python-lib _package.pycuda"
INFO:root:           ! load /root/CM/repos/local/cache/a077048189d34c6a/cm-cached-state.json
INFO:root:      * cm run script "get generic-python-lib _package.numpy"
INFO:root:           ! load /root/CM/repos/local/cache/2ad4e07c3bbb467a/cm-cached-state.json
INFO:root:           ! cd /root
INFO:root:           ! call /root/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/detect.sh from tmp-run.sh
GPU 0:
INFO:root:           ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/customize.py
INFO:root:    * cm run script "get generic-python-lib _package.dmiparser"
INFO:root:         ! load /root/CM/repos/local/cache/54074d7823ff4258/cm-cached-state.json
INFO:root:    * cm run script "get cache dir _name.mlperf-inference-sut-descriptions"
INFO:root:         ! load /root/CM/repos/local/cache/18e34c80f1bf4b85/cm-cached-state.json
Generating SUT description file for 4aa21844667d-tensorrt
INFO:root:         ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-sut-description/customize.py
INFO:root:  * cm run script "get mlperf inference results dir _version.r4_1-dev"
INFO:root:       ! load /root/CM/repos/local/cache/8ea83cc1c2374edf/cm-cached-state.json
INFO:root:  * cm run script "install pip-package for-cmind-python _package.tabulate"
INFO:root:       ! load /root/CM/repos/local/cache/89f618a68d114b33/cm-cached-state.json
INFO:root:  * cm run script "get mlperf inference utils"
INFO:root:    * cm run script "get mlperf inference src"
INFO:root:         ! load /root/CM/repos/local/cache/025c3b1288384fc3/cm-cached-state.json
INFO:root:         ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-utils/customize.py
Using MLCommons Inference source from /root/CM/repos/local/cache/9a3d7bad549341f7/inference

Running loadgen scenario: Offline and mode: performance
INFO:root:* cm run script "app mlperf inference generic _nvidia _llama2-70b-99 _tensorrt _cuda _test _r4.1-dev_default _offline"
INFO:root:  * cm run script "detect os"
INFO:root:         ! cd /root
INFO:root:         ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:         ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:  * cm run script "get sys-utils-cm"
WARNING:root:=================================================
WARNING:root:WARNINGS:
WARNING:root:  This CM script will install extra OS system utils required for CM automation workflows!
WARNING:root:=================================================
INFO:root:    * cm run script "detect os"
INFO:root:           ! cd /root/CM/repos/local/cache/4bfdb57fb4664aff
INFO:root:           ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:  * cm run script "get python"
INFO:root:       ! load /root/CM/repos/local/cache/e64cc0d91f024ccb/cm-cached-state.json
INFO:root:Path to Python: /usr/bin/python3
INFO:root:Python version: 3.8.10
INFO:root:  * cm run script "get mlcommons inference src"
INFO:root:       ! load /root/CM/repos/local/cache/025c3b1288384fc3/cm-cached-state.json
INFO:root:  * cm run script "get mlperf inference utils"
INFO:root:    * cm run script "get mlperf inference src"
INFO:root:         ! load /root/CM/repos/local/cache/025c3b1288384fc3/cm-cached-state.json
INFO:root:         ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-utils/customize.py
INFO:root:  * cm run script "get cuda-devices _with-pycuda"
INFO:root:    * cm run script "get cuda _toolkit"
INFO:root:         ! load /root/CM/repos/local/cache/55ce6a990ddb44c8/cm-cached-state.json
INFO:root:ENV[CM_CUDA_PATH_LIB_CUDNN_EXISTS]: no
INFO:root:ENV[CM_CUDA_VERSION]: 12.2
INFO:root:ENV[CM_CUDA_VERSION_STRING]: cu122
INFO:root:ENV[CM_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
INFO:root:ENV[CUDA_HOME]: /usr/local/cuda
INFO:root:    * cm run script "get python3"
INFO:root:         ! load /root/CM/repos/local/cache/e64cc0d91f024ccb/cm-cached-state.json
INFO:root:Path to Python: /usr/bin/python3
INFO:root:Python version: 3.8.10
INFO:root:    * cm run script "get generic-python-lib _package.pycuda"
INFO:root:         ! load /root/CM/repos/local/cache/a077048189d34c6a/cm-cached-state.json
INFO:root:    * cm run script "get generic-python-lib _package.numpy"
INFO:root:         ! load /root/CM/repos/local/cache/2ad4e07c3bbb467a/cm-cached-state.json
INFO:root:         ! cd /root/CM/repos/local/cache/4bfdb57fb4664aff
INFO:root:         ! call /root/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/detect.sh from tmp-run.sh
GPU 0:
INFO:root:         ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/customize.py
INFO:root:  * cm run script "reproduce mlperf nvidia inference _run_harness _tensorrt _llama2-70b-99 _offline _cuda _l4"
INFO:root:    * cm run script "detect os"
INFO:root:           ! cd /root/CM/repos/local/cache/4bfdb57fb4664aff
INFO:root:           ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:    * cm run script "detect cpu"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /root/CM/repos/local/cache/4bfdb57fb4664aff
INFO:root:             ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:           ! cd /root/CM/repos/local/cache/4bfdb57fb4664aff
INFO:root:           ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:           ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:    * cm run script "get sys-utils-cm"
WARNING:root:=================================================
WARNING:root:WARNINGS:
WARNING:root:  This CM script will install extra OS system utils required for CM automation workflows!
WARNING:root:=================================================
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /root/CM/repos/local/cache/4bfdb57fb4664aff
INFO:root:             ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:    * cm run script "get mlperf inference nvidia scratch space"
INFO:root:         ! load /root/CM/repos/local/cache/8086ce6efbbf4c69/cm-cached-state.json
INFO:root:    * cm run script "get generic-python-lib _mlperf_logging"
INFO:root:         ! load /root/CM/repos/local/cache/1778d5dfdcdf4f10/cm-cached-state.json
INFO:root:    * cm run script "get mlcommons inference src"
INFO:root:         ! load /root/CM/repos/local/cache/025c3b1288384fc3/cm-cached-state.json
INFO:root:    * cm run script "get nvidia mlperf inference common-code _go"
INFO:root:         ! load /root/CM/repos/local/cache/7628df8d8a534114/cm-cached-state.json
INFO:root:    * cm run script "generate user-conf mlperf inference"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /root/CM/repos/local/cache/4bfdb57fb4664aff
INFO:root:             ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:      * cm run script "detect cpu"
INFO:root:        * cm run script "detect os"
INFO:root:               ! cd /root/CM/repos/local/cache/4bfdb57fb4664aff
INFO:root:               ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:             ! cd /root/CM/repos/local/cache/4bfdb57fb4664aff
INFO:root:             ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:      * cm run script "get python"
INFO:root:           ! load /root/CM/repos/local/cache/e64cc0d91f024ccb/cm-cached-state.json
INFO:root:Path to Python: /usr/bin/python3
INFO:root:Python version: 3.8.10
INFO:root:      * cm run script "get mlcommons inference src"
INFO:root:           ! load /root/CM/repos/local/cache/025c3b1288384fc3/cm-cached-state.json
INFO:root:      * cm run script "get sut configs"
INFO:root:        * cm run script "get cache dir _name.mlperf-inference-sut-configs"
INFO:root:             ! load /root/CM/repos/local/cache/70b1e0fcf3434ca2/cm-cached-state.json
INFO:root:             ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/get-mlperf-inference-sut-configs/customize.py
Using MLCommons Inference source from '/root/CM/repos/local/cache/9a3d7bad549341f7/inference'
Original configuration value 0.1 target_qps
Adjusted configuration value 0.101 target_qps
Output Dir: '/root/CM/repos/local/cache/8ea83cc1c2374edf/test_results/4aa21844667d-nvidia_original-gpu-tensorrt-vdefault-default_config/llama2-70b-99/offline/performance/run_1'
llama2-70b.Offline.target_qps = 0.01
llama2-70b.Offline.max_query_count = 50
llama2-70b.Offline.min_query_count = 50
llama2-70b.Offline.min_duration = 0
llama2-70b.Offline.sample_concatenate_permutation = 0

INFO:root:    * cm run script "get generic-python-lib _package.pycuda"
INFO:root:         ! load /root/CM/repos/local/cache/a077048189d34c6a/cm-cached-state.json
INFO:root:    * cm run script "get nvidia mitten"
INFO:root:         ! load /root/CM/repos/local/cache/2b121552ec104878/cm-cached-state.json
INFO:root:    * cm run script "get cuda _cudnn"
INFO:root:         ! load /root/CM/repos/local/cache/a89e2d2ba46d4cc3/cm-cached-state.json
INFO:root:ENV[CM_CUDA_PATH_LIB_CUDNN_EXISTS]: yes
INFO:root:ENV[CM_CUDA_VERSION]: 12.2
INFO:root:ENV[CM_CUDA_VERSION_STRING]: cu122
INFO:root:ENV[CM_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
INFO:root:ENV[CUDA_HOME]: /usr/local/cuda
INFO:root:    * cm run script "get tensorrt"
INFO:root:         ! load /root/CM/repos/local/cache/d7f87b114fea4908/cm-cached-state.json
INFO:root:    * cm run script "build nvidia inference server _go"
INFO:root:         ! load /root/CM/repos/local/cache/833bc01b228a4794/cm-cached-state.json
INFO:root:    * cm run script "reproduce mlperf inference nvidia harness _build_engine _tensorrt _llama2-70b-99 _offline _cuda _llama2-70b_ _v4.1-dev"
INFO:root:      * cm run script "detect os"
INFO:root:             ! cd /root/CM/repos/local/cache/c166ceeefd6d4ca8
INFO:root:             ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:      * cm run script "detect cpu"
INFO:root:        * cm run script "detect os"
INFO:root:               ! cd /root/CM/repos/local/cache/c166ceeefd6d4ca8
INFO:root:               ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:             ! cd /root/CM/repos/local/cache/c166ceeefd6d4ca8
INFO:root:             ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:             ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:      * cm run script "get sys-utils-cm"
WARNING:root:=================================================
WARNING:root:WARNINGS:
WARNING:root:  This CM script will install extra OS system utils required for CM automation workflows!
WARNING:root:=================================================
INFO:root:        * cm run script "detect os"
INFO:root:               ! cd /root/CM/repos/local/cache/4bfdb57fb4664aff
INFO:root:               ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:      * cm run script "get mlperf inference nvidia scratch space _version.4_0"
INFO:root:           ! load /root/CM/repos/local/cache/8086ce6efbbf4c69/cm-cached-state.json
INFO:root:      * cm run script "get generic-python-lib _mlperf_logging"
INFO:root:           ! load /root/CM/repos/local/cache/1778d5dfdcdf4f10/cm-cached-state.json
INFO:root:      * cm run script "get mlcommons inference src"
INFO:root:           ! load /root/CM/repos/local/cache/025c3b1288384fc3/cm-cached-state.json
INFO:root:      * cm run script "get nvidia mlperf inference common-code _go"
INFO:root:           ! load /root/CM/repos/local/cache/7628df8d8a534114/cm-cached-state.json
INFO:root:      * cm run script "get generic-python-lib _package.pycuda"
INFO:root:           ! load /root/CM/repos/local/cache/a077048189d34c6a/cm-cached-state.json
INFO:root:      * cm run script "get nvidia mitten"
INFO:root:           ! load /root/CM/repos/local/cache/2b121552ec104878/cm-cached-state.json
INFO:root:      * cm run script "get cuda _cudnn"
INFO:root:           ! load /root/CM/repos/local/cache/a89e2d2ba46d4cc3/cm-cached-state.json
INFO:root:ENV[CM_CUDA_PATH_LIB_CUDNN_EXISTS]: yes
INFO:root:ENV[CM_CUDA_VERSION]: 12.2
INFO:root:ENV[CM_CUDA_VERSION_STRING]: cu122
INFO:root:ENV[CM_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
INFO:root:ENV[CUDA_HOME]: /usr/local/cuda
INFO:root:      * cm run script "get tensorrt"
INFO:root:           ! load /root/CM/repos/local/cache/d7f87b114fea4908/cm-cached-state.json
INFO:root:      * cm run script "build nvidia inference server _go"
INFO:root:           ! load /root/CM/repos/local/cache/833bc01b228a4794/cm-cached-state.json
INFO:root:      * cm run script "reproduce mlperf inference nvidia harness _preprocess_data _tensorrt _llama2-70b-99 _cuda _llama2-70b_ _v4.1-dev"
INFO:root:           ! load /root/CM/repos/local/cache/bd717f6af2984966/cm-cached-state.json
INFO:root:      * cm run script "reproduce mlperf inference nvidia harness _download_model _tensorrt _llama2-70b-99 _cuda _llama2-70b_ _v4.1-dev"
INFO:root:        * cm run script "detect os"
INFO:root:               ! cd /root/CM/repos/local/cache/30c3fda00c3243a3
INFO:root:               ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:        * cm run script "detect cpu"
INFO:root:          * cm run script "detect os"
INFO:root:                 ! cd /root/CM/repos/local/cache/30c3fda00c3243a3
INFO:root:                 ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:                 ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:               ! cd /root/CM/repos/local/cache/30c3fda00c3243a3
INFO:root:               ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-cpu/run.sh from tmp-run.sh
INFO:root:               ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-cpu/customize.py
INFO:root:        * cm run script "get sys-utils-cm"
WARNING:root:=================================================
WARNING:root:WARNINGS:
WARNING:root:  This CM script will install extra OS system utils required for CM automation workflows!
WARNING:root:=================================================
INFO:root:          * cm run script "detect os"
INFO:root:                 ! cd /root/CM/repos/local/cache/4bfdb57fb4664aff
INFO:root:                 ! call /root/CM/repos/mlcommons@cm4mlops/script/detect-os/run.sh from tmp-run.sh
INFO:root:                 ! call "postprocess" from /root/CM/repos/mlcommons@cm4mlops/script/detect-os/customize.py
INFO:root:        * cm run script "get mlperf inference nvidia scratch space _version.4_0"
INFO:root:             ! load /root/CM/repos/local/cache/8086ce6efbbf4c69/cm-cached-state.json
INFO:root:        * cm run script "get generic-python-lib _mlperf_logging"
INFO:root:             ! load /root/CM/repos/local/cache/1778d5dfdcdf4f10/cm-cached-state.json
INFO:root:        * cm run script "get mlcommons inference src"
INFO:root:             ! load /root/CM/repos/local/cache/025c3b1288384fc3/cm-cached-state.json
INFO:root:        * cm run script "get nvidia mlperf inference common-code _go"
INFO:root:             ! load /root/CM/repos/local/cache/7628df8d8a534114/cm-cached-state.json
INFO:root:        * cm run script "get generic-python-lib _package.pycuda"
INFO:root:             ! load /root/CM/repos/local/cache/a077048189d34c6a/cm-cached-state.json
INFO:root:        * cm run script "get nvidia mitten"
INFO:root:             ! load /root/CM/repos/local/cache/2b121552ec104878/cm-cached-state.json
INFO:root:        * cm run script "get generic-python-lib _package.transformers"
INFO:root:             ! load /root/CM/repos/local/cache/3731fd0aa4524247/cm-cached-state.json
INFO:root:        * cm run script "get generic-python-lib _package.datasets"
INFO:root:             ! load /root/CM/repos/local/cache/c2b10fbfe9ac46b1/cm-cached-state.json
INFO:root:        * cm run script "get generic-python-lib _package.sentencepiece"
INFO:root:             ! load /root/CM/repos/local/cache/b366bc1409894ebc/cm-cached-state.json
INFO:root:        * cm run script "get generic-python-lib _package.protobuf"
INFO:root:             ! load /root/CM/repos/local/cache/7317384fb3dc4360/cm-cached-state.json
INFO:root:        * cm run script "get generic-python-lib _package.accelerate"
INFO:root:             ! load /root/CM/repos/local/cache/fc903fbf44a74b55/cm-cached-state.json
INFO:root:        * cm run script "get generic-python-lib _package.absl-py"
INFO:root:             ! load /root/CM/repos/local/cache/2721adfbfcac43a1/cm-cached-state.json
INFO:root:        * cm run script "get generic-python-lib _package.evaluate"
INFO:root:             ! load /root/CM/repos/local/cache/3012fe849a994507/cm-cached-state.json
INFO:root:        * cm run script "get generic-python-lib _package.nltk"
INFO:root:             ! load /root/CM/repos/local/cache/7986067456414fd3/cm-cached-state.json
INFO:root:        * cm run script "get generic-python-lib _package.numpy"
INFO:root:             ! load /root/CM/repos/local/cache/2ad4e07c3bbb467a/cm-cached-state.json
INFO:root:        * cm run script "get generic-python-lib _package.rouge-score"
INFO:root:             ! load /root/CM/repos/local/cache/be1d3ec5b3334832/cm-cached-state.json

CM error: Quantised model absent - did not detect config.json in path /root/CM/repos/local/cache/daa1b0f5f8d74ccf/models/Llama2/fp8-quantized-ammo/llama2-70b-chat-hf-tp2pp1-fp8!


@stbailey001 Nvidia llama2 run command needs to use `--docker` and must be run on host as it needs to launch a separate docker container to do quantization. Also, it'll need minimum 80 GB of GPU memory to run and to quantize, itll need about 150GB of memory. 

It is being run in a docker container. The GPU is an L4 so there is enough memory on the card and the VM has 290GB of memory.  So there need to be a separate container to run quantization? So then you don't run this in a container? 


yes, the llama2 run needs to start from the host as the run will launch 2 docker containers. 
I don't think a single L4 GPU can run llama2-70b - it is a 70 billion parameter model. 

Okay  thank you I'll run it from the VM host. I'll up the number of GPUs  that can be used.  Do you know what the minimum number of L4 that is needed?
Steve 

You're welcome. Unfortunately we don't have any reference point for llama2 on L4. [Nvidia configs](https://github.com/mlcommons/inference_results_v4.0/blob/main/closed/NVIDIA/configs/llama2-70b/Offline/__init__.py) are given for H100 and better GPUs only. We know that it can run on 8xL40S. So, I think at least 8 L4 GPUs will be needed and even then it may or may not run. 

The info you gave is very helpful! Thanks!