Repository: mlcommons/inference
Issue: CM error: no scripts were found with above tags and variations   can't find _for-nvidia-mlperf-inference-v4.1
Root cause: Configuration Error

When setting up to run Bert-99 I get the no scripts for the version v4.1 the pytorch install.

INFO:root:      * cm run script "get cuda _cudnn"
INFO:root:           ! load /root/CM/repos/local/cache/07724331991f4b22/cm-cached-state.json
INFO:root:ENV[CM_CUDA_PATH_LIB_CUDNN_EXISTS]: yes
INFO:root:ENV[CM_CUDA_VERSION]: 12.2
INFO:root:ENV[CM_CUDA_VERSION_STRING]: cu122
INFO:root:ENV[CM_NVCC_BIN_WITH_PATH]: /usr/local/cuda/bin/nvcc
INFO:root:ENV[CUDA_HOME]: /usr/local/cuda
INFO:root:      * cm run script "get tensorrt _dev"
INFO:root:           ! load /root/CM/repos/local/cache/6c6b78a9e502496c/cm-cached-state.json
INFO:root:      * cm run script "get gcc"
INFO:root:           ! load /root/CM/repos/local/cache/179374cd4edb4616/cm-cached-state.json
INFO:root:      * cm run script "get cmake"
INFO:root:           ! load /root/CM/repos/local/cache/09a3a03d41234597/cm-cached-state.json
INFO:root:Path to the tool: /root/CM/repos/local/cache/efc474aeb5b5474a/bin/cmake
INFO:root:      * cm run script "get generic sys-util _glog-dev"
INFO:root:           ! load /root/CM/repos/local/cache/f3902b9f4c954a9d/cm-cached-state.json
INFO:root:      * cm run script "get generic sys-util _gflags-dev"
INFO:root:           ! load /root/CM/repos/local/cache/a89b78a832d5450d/cm-cached-state.json
INFO:root:      * cm run script "get generic sys-util _libgmock-dev"
INFO:root:           ! load /root/CM/repos/local/cache/cf8adf8630ab4184/cm-cached-state.json
INFO:root:      * cm run script "get generic sys-util _libre2-dev"
INFO:root:           ! load /root/CM/repos/local/cache/9696878287f94075/cm-cached-state.json
INFO:root:      * cm run script "get generic sys-util _libnuma-dev"
INFO:root:           ! load /root/CM/repos/local/cache/e3c18f5190ac43c0/cm-cached-state.json
INFO:root:      * cm run script "get generic sys-util _libboost-all-dev"
INFO:root:           ! load /root/CM/repos/local/cache/af7d3a796839442f/cm-cached-state.json
INFO:root:      * cm run script "get generic sys-util _rapidjson-dev"
INFO:root:           ! load /root/CM/repos/local/cache/340d989cec914b49/cm-cached-state.json
INFO:root:      * cm run script "get nvidia mlperf inference common-code _go"
INFO:root:           ! load /root/CM/repos/local/cache/d15e087a93334b39/cm-cached-state.json
INFO:root:      * cm run script "get generic-python-lib _package.pybind11"
INFO:root:           ! load /root/CM/repos/local/cache/01fa12e55bd24432/cm-cached-state.json
INFO:root:      * cm run script "get generic-python-lib _pycuda"
INFO:root:           ! load /root/CM/repos/local/cache/c470d2f9cf4b40f6/cm-cached-state.json
INFO:root:      * cm run script "get generic-python-lib _opencv-python"
INFO:root:           ! load /root/CM/repos/local/cache/fcc71d49f7c94699/cm-cached-state.json
INFO:root:      * cm run script "get generic-python-lib _nvidia-dali"
INFO:root:           ! load /root/CM/repos/local/cache/57d39ad66d8848f5/cm-cached-state.json
INFO:root:      * cm run script "get mlperf inference nvidia scratch space _version.4_0"
INFO:root:           ! load /root/CM/repos/local/cache/48b7ceb222fd44b4/cm-cached-state.json
INFO:root:      * cm run script "get generic sys-util _nlohmann-json3-dev"
INFO:root:           ! load /root/CM/repos/local/cache/8d1df060ac5d4717/cm-cached-state.json
INFO:root:      * cm run script "get generic sys-util _git-lfs"
INFO:root:           ! load /root/CM/repos/local/cache/76aa2e66b2fa49d8/cm-cached-state.json
INFO:root:      * cm run script "install pytorch from.src _for-nvidia-mlperf-inference-v4.0 _for-nvidia-mlperf-inference-v4.1"

CM error: no scripts were found with above tags and variations

variation tags ['for-nvidia-mlperf-inference-v4.0', 'for-nvidia-mlperf-inference-v4.1'] are not matching for the found script install-pytorch-from-src with variations dict_keys(['branch.#', 'repo.https://github.com/pytorch/pytorch', 'repo.#', 'sha.#', 'tag.#', 'pr-to-apply.#', 'cherrypicks.#', 'for-intel-mlperf-inference-resnet50', 'for-intel-mlperf-inference-retinanet', 'for-intel-mlperf-inference-v3.1-bert', 'for-nvidia-mlperf-inference-v4.0', 'for-nvidia-mlperf-inference-v3.1', 'cuda'])
Steve 

Can you please do `cm pull repo` and retry the command with `--docker_cache=no` option?

Same error as before.

Can you please share the command you had tried?

cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1-dev    --model=bert-99    --implementation=nvidia    --framework=tensorrt    --category=datacenter    --scenario=Offline    --execution_mode=test    --device=cuda     --quiet    --test_query_count=100 --docker_cache=no

Looks fine. Did you install `cm4mlops` via `pip install cm4mlops`? 

Yes


What's the output of
```
cd $HOME/CM/repos/mlcommons@cm4mlops && git log
```
?

commit 1825237231600569e5598941009f3ee8d03c667f (HEAD -> main, origin/main, origin/HEAD)
Merge: ad11c9047 73bbc252f
Author: Admin account for cTuning.org <166590745+ctuning-admin@users.noreply.github.com>
Date:   Mon Sep 16 10:34:55 2024 +0200

    Merge pull request #218 from mlcommons/dev
    
    Merge pull request #217 from mlcommons/main

commit 73bbc252f0d1be74e0c68ae956ef88db8fd5b0ee (origin/dev)
Merge: dfbe4063f ad11c9047
Author: Grigori Fursin <gfursin@users.noreply.github.com>
Date:   Mon Sep 16 10:24:20 2024 +0200

    Merge pull request #217 from mlcommons/main
    
    regular sync

commit ad11c9047774eab9642181d8cd9c60f5167a0025
Merge: 5b208204e 0cf5b7e1a
Author: Arjun Suresh <arjunsuresh1987@gmail.com>
Date:   Fri Sep 13 19:16:04 2024 +0100

:



You are actually on the `main` branch - probably you had done manual pull sometime back. I have just synced the branches - so this should be fine if you do `cm pull repo` once more.

But more importantly just noticed that the run command you had shared is not having `--docker` option which is mandatory for `nvidia` implementation.

```
cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1-dev --model=bert-99 --implementation=nvidia --framework=tensorrt --category=datacenter --scenario=Offline --execution_mode=test --device=cuda --quiet --test_query_count=100 --docker_cache=no --docker
```

I'm running this inside a docker container so I didn't think I needed the --docker  option. 


Thanks the sync helped get it running!