zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Undefined symbol: __nvJitLinkAddData_12_1, version libnvJitLink.so.12
Root cause: Environment Incompatibility

![image](https://github.com/mlcommons/inference/assets/71482562/817d5dce-9687-4d54-930b-32dec76157c6)


Occured while running: `cm run script --tags=generate-run-cmds,inference --model=bert-99 --backend=pytorch  --mode=performance --device=cuda --quiet --test_query_count=1000 --network=sut`

Found same issue created at: https://github.com/pytorch/pytorch/issues/111469

I have tried to export env variable: `export LD_LIBRARY_PATH=$HOME/.local/lib/python3.12/sitepackages/nvidia/nvjitlink:$LD_LIBRARY_PATH`  but did not work

Another method mentioned: 

1. Downgrade torch to 2.0.1
2. Installing nightly build version of torch  

This error should be because pytorch was compiled for a later version of cuda. `--adr.cuda.version=12.4.1` should fix it. 

WG agrees this is resolved

Closing

Hi, I am facing the same issue with the gpt-j benchmark. 
Here are the output:
```bash
torch.distributed not initialized, assuming single world_size.
Quantized model exported to /mnt/models/GPTJ-6B/fp8-quantized-ammo/GPTJ-FP8-quantized
Total time used 14.93 s.
make: Leaving directory '/home/usr/CM/repos/local/cache/e6f880f23ece4993/repo/docker'
Traceback (most recent call last):
  File "/home/usr/CM/repos/local/cache/c69993974e204ca1/repo/closed/NVIDIA/code/gptj/tensorrt/onnx_tune.py", line 15, in <module>
    import torch
  File "/home/usr/Benchmark/cm/lib/python3.12/site-packages/torch/__init__.py", line 367, in <module>
    from torch._C import *  # noqa: F403
    ^^^^^^^^^^^^^^^^^^^^^^
ImportError: /home/usr/Benchmark/cm/lib/python3.12/site-packages/torch/lib/../../nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkComplete_12_4, version libnvJitLink.so.12
CM error: Portable CM script failed (name = get-ml-model-gptj, return code = 256)
```

with the command: 
`
cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1-dev    --model=gptj-99    --implementation=nvidia    --framework=tensorrt    --category=datacenter    --scenario=Offline    --execution_mode=test    --device=cuda     --docker --quiet    --test_query_count=50
`

I have tried to add `--adr.cuda.version=12.4.1` at the end of the command but it does not solve the issue. Should I reinstall torch with the proper version? My torch version is currently in 2.5.1.
Repository: mlcommons/inference
Issue: Does it support mobilenetv3

Does it support mobilenetv3?

yes, even though it is not an official MLPerf model it is compatible with `ResNet50` and can be submitted under the `open` division. This is an example [readme](https://github.com/mlcommons/inference_results_v4.0/tree/main/open/CTuning/code/mobilenet-v3-precision_uint8-large-minimalistic-224) file from one of the CTuning submission. 

I find it support mobilenetv1 not mobilenetv3 from readme file

No, it supports v1, v2 and v3 versions. The README I gave is specifically for the v3 one but it is linking to a global one that runs all three versions. 

Could you give the README that is specifically for v3 one

@sunpian1 it is not there currently but let me create one and share you by tomorrow. 

could you share me the README file that is specifically for v3 one?

@sunpian1 Does this work for you? https://gateoverflow.github.io/inference/benchmarks/image_classification/mobilenets/
Repository: mlcommons/inference
Issue: Do you  support a local path to the model?
Root cause: Documentation Deficiency

Do you  support a local path to the model?

`local path to the model` is supported in some cases depending on the implementation being used. For example, the bert implementation using deepsparse backend supports it. Which model would you like to import locally?

resnet50 mobilenetv3 and so on

In CM, we download or load a model using a separate CM script like [here](https://github.com/GATEOverflow/cm4mlops/tree/mlperf-inference/script/get-ml-model-mobilenet). After this we get the path to the model in an `ENV` variable which we use in the inference implementation script.

If you try this, you can either extend the CM script to plug in new models or populate the ENV variable and prevent the run of the CM script to get the model. 

[This](https://docs.mlcommons.org/cm4mlops/) is a WIP but can be useful to understand how CM scripts work.
Repository: mlcommons/inference
Issue: Terminal disppeared after rnnt inference running without any error (attach log)
Root cause: Unimplemented Feature Gap

I'm trying to run the rnnt inference on mlperf4.0 with the below commands and after a long time the whole terminal will disppear without any error prompt. I'm not sure whether is my basic environment is something missing. 
[log_rnnt_cuda_tensorrt.txt](https://github.com/mlcommons/inference/files/15442128/log_rnnt_cuda_tensorrt.txt)


cm run script --tags=run-mlperf,inference,_r4.0,_submission,_short  \
--division=closed \
--category=edge \
--device=cuda \
--model=rnnt \
--precision=float32 \
--implementation=mlcommons-python \
--backend=pytorch \
--scenario=Offline \
--execution_mode=test \
--power=no \
--adr.python.version_min=3.8 \
--clean \
--compliance=no \
--quiet \
--time

Hi @Bob123Yang we are no longer supporting rnnt as its removed from the inference list of benchmarks. 

Wo! but rnnt is still displayed in the page (https://github.com/mlcommons/inference) below:

![image](https://github.com/mlcommons/inference/assets/150206661/60b68e75-55d9-443a-842f-7978e654206f)


yes, that was for the last round. We have removed it for the upcoming round in July but the final list of benchmarks will be known only in 2 weeks. After that, the list will be updated in the README. 

Meanwhile is the log file shared for the command given? Because the log file shows `tensorrt` with Nvidia implementation whereas the given command is using `pytorch` with the reference implementation. If you add `--docker` to the command used for Nvidia implementation, `rnnt` should still run using the older code.
Repository: mlcommons/inference
Issue: Should the tokenizer be on the loadgen node(client) or SUT node?
Root cause: Documentation Deficiency

Hi @nv-jinhosuh  @arjunsuresh , I have a doubt regarding implementation of GPT-J loadgen over the network. Should the tokenization of texts occur within the SUT node subsequent to receiving the request from the loadgen node, or should it be integrated within the loadgen node itself before sending the request to SUT node? 

Presently, in reference to https://github.com/mlcommons/inference/pull/1702, The loadgen node transmits requests to the server in textual form, and tokenization is performed subsequently within the SUT node.

@pgmpablo157321 @nvzhihanj I believe we use 'already tokenized' input tensors (after preprocessing) in closed division for GPT-J right? 
If that is the case, text input tokenization is untimed, and NetDiv(LON) can also use the preprocessed input tensor. Or you can choose to tokenize the text inputs in LoadGen @anandhu-eng 

@nv-jinhosuh yes, that's correct. 

Thank you @nv-jinhosuh @arjunsuresh
Repository: mlcommons/inference
Issue: GPT-J Implementation not following the set limit for execution mode "fast"
Root cause: Configuration Error

```cm run script --tags=run-mlperf,inference,_performance-only --model=gptj-99 --backend=pytorch   --device=cuda --test_query_count=80 --beam_size=2 --precision=bfloat16  --rerun --quiet --env.GPTJ_BEAM_SIZE=2 --scenario=Offline --adr.compiler.tags=gcc --adr.mlperf-implementation.tags=_repo.https://github.com/anandhu-eng/inference/,_branch.GPT-J --adr.cuda.version=12.4.1 --execution-mode=fast --offline_target_qps=0.01```

Executing this command, the GPT-J model still continues to process samples more than 2 minutes(Specified execution time for execution mode `fast`).

Output:

```Loading PyTorch model...
BF16 autocast
Loading checkpoint shards: 100%|██████████| 3/3 [00:01<00:00,  2.23it/s]
/home/anandhu/.local/lib/python3.12/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use 
force_download=True.
  warnings.warn(
Casting models to GPU...
100%|██████████| 285/285 [00:00<00:00, 2729170.41it/s]
Constructing QSL
Encoding Samples
Finished constructing QSL.
Constructing QSL
Encoding Samples
Finished constructing QSL.
Finished constructing QDL!
Running LoadGen test...
Number of Samples in query_samples :  13368
  4%|▎         | 472/13368 [17:55<9:14:42,  2.58s/it]^C
CM error: Portable CM script failed (name = benchmark-program, return code = 2)```

@arjunsuresh  to fix

This problem was there for all the models with equal issue mode enabled - gptj, 3d-unet and llama2. [This commit](https://github.com/GATEOverflow/cm4mlops/commit/9da822a9a6ed05990c135e0ca3c8c7f48f65019d) should fix this.

@anandhu-eng This can be closed now right?

Hi @arjunsuresh , we can investigate later

This issue is now resolved.
Repository: mlcommons/inference
Issue: Performance improvement- GPT-J and BERT Offline scenario
Root cause: Algorithmic Error

The current implementation of GPT-J and BERT carries out the prediction in sequential manner. Could the performance of GPT-J and BERT be improved by implementing parallel processing through threads rather than sequential processing?

GPT-J ref: https://github.com/mlcommons/inference/blob/fa4fe53e53379dee27a216695a2b710d122154c7/language/gpt-j/backend.py#L72

BERT ref: https://github.com/mlcommons/inference/blob/fa4fe53e53379dee27a216695a2b710d122154c7/language/bert/pytorch_SUT.py#L68

@anandhu-eng : Please open PR
Repository: mlcommons/inference
Issue: Tokens per sample upper limit for GPTJ

Is there any reason why we have an [accuracy upper limit for LLAMA2 Tokens per sample](https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py#L109) but not for GPT-J? It's good to document this reason for users.

Apparently mixtral too: https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py#L381

Just my two cents but might it be because these models tend to keep repeating: [issue on TRT for llama](https://github.com/NVIDIA/TensorRT-LLM/issues/1788), and for [mixtral](https://github.com/NVIDIA/TensorRT-LLM/issues/1305) or maybe I have just confirmation bias.
Repository: mlcommons/inference
Issue: NPU Models On MLPerf Edge
Root cause: Documentation Deficiency

I am writing to inquire about the availability of quantized TFLite models optimized for NPUs (Neural Processing Units) that can be used for benchmarking our device in the MLPerf Edge framework.

Our device is equipped with an NPU, and we are keen to participate in the MLPerf Edge benchmarks. However, to accurately assess our device's performance and contribute to the MLCOMMONS benchmarks, we require models that are specifically tailored for NPUs and quantized for efficient inference.

Could you please provide information on where we can obtain such quantized TFLite models or if there are resources available to assist us in converting existing models to the required format? This would greatly facilitate our ability to conduct inferences and generate benchmark metrics aligned with the MLPerf Edge guidelines.

Thank you for your attention to this matter. I look forward to your guidance and assistance in accessing the necessary resources for our NPU submission to MLPerf.

There are quantized tflite models for mobilenets and efficientnets. [This readme](https://docs.mlcommons.org/inference/benchmarks/image_classification/mobilenets/) can guide in running them. 

@arjunsuresh  Thanx pal, but can we use these models to benchmark and submit in the MLCOMMONS suite?

These can be submitted to the open division, not the closed ones as these are not official models. But resnet50 does have a tflite version which can be submitted to the closed division. Which NPU are you targeting? 

@arjunsuresh  So I have a doubt in the new version of MLPerf Edge there are few models and I am focusing on Resnet50, unet and bert model. The models provided in the link are not quantized for NPU right? If not we can download it from other sources and make the benchmarks and submit in the open ones right? 

They include both quantized and non-quantized models. You can see the last round result on Nvidia Jetson Orin (CPU) [here](https://github.com/mlcommons/inference_results_v4.0/tree/main/open/CTuning/results/Nvidia_Jetson_Orin_AGX-tflite_armnn_cpp-cpu-armnn_tflite-v23.05-using_neon). 

R50 is the only official tflite model I know and I don't think it has a ready quantized version available. Which company are you representing? Are you trying to submit for 4.1 inference round which is in July?

No just a college student.
Repository: mlcommons/inference
Issue: ResNet50 ONNX opset-11 or opset-12?
Root cause: Documentation Deficiency

The [classification_and_detection README](https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/README.md) has several links for ResNet50 ONNX model:
- opset-8: https://zenodo.org/record/2592612/files/resnet50_v1.onnx
- opset-11: https://zenodo.org/record/4735647/files/resnet50_v1.onnx

The latter https://zenodo.org/record/4735647 takes you to a new location https://zenodo.org/records/4313974 but confusingly both mention opset-12:

> How this model was created: ``` download the frozen tensorflow model from https://zenodo.org/deposit/2535873 pip install -U tf2onnx python -m tf2onnx.convert --input resnet50_v1.pb --inputs input_tensor:0 --inputs-as-nchw input_tensor:0 --outputs ArgMax:0 --opset 12 --output resnet50_v1.onnx ```

Shall we assume opset-12 is correct?

@psyhtest I assume 8 and 11 are correct. Is there any reason for 12? 

If one page says "opset-11" and another one "opset-12", which one should we trust? Is there an easy way to query the binary itself?

Let me try that. Meanwhile does it make sense to release newer version of the models compatible with the latest tool chains? We can take up this task for v5.0. 

By importing and printing the opset versions, 8 and 11 are the correct ones. 

Excellent, good to know @arjunsuresh! I don't think we need to update this benchmark, as it's past its glory days anyway :).
Repository: mlcommons/inference
Issue: Huggingface.co How to clone llama-2-70b-chat-hf login fails and how to add token or ssh key?
Root cause: Documentation Deficiency

When trying to run llama-2 model fail to clone model from huggingface.com.
Current directory: /root/CM/repos/local/cache/73003f17f61e4bfd

Cloning Llama-2-70b-chat-hf from https://huggingface.co/meta-llama/Llama-2-70b-chat-hf

git clone  --recurse-submodules https://huggingface.co/meta-llama/Llama-2-70b-chat-hf --depth 5 repo

Cloning into 'repo'...
Username for 'https://huggingface.co': stbailey001
Password for 'https://stbailey001@huggingface.co': 
remote: Password authentication in git is no longer supported. You must use a user access token or an SSH key instead. See https://huggingface.co/blog/password-git-deprecation
fatal: Authentication failed for 'https://huggingface.co/meta-llama/Llama-2-70b-chat-hf/'

CM error: Portable CM script failed (name = get-git-repo, return code = 256)

cm command:
cm run script --tags=run-mlperf,inference,_find-performance,_full    --model=llama2-70b-99    --implementation=reference    --framework=pytorch    --category=datacenter    --scenario=Offline    --execution_mode=test    --device=cpu      --quiet    --test_query_count=50






When password is asked you can enter your API token.

Thanks that does work. But now hit some other issues.
Repository: mlcommons/inference
Issue: Mixtral reference implementation fails on CPU
Root cause: Algorithmic Error

```
WARNING:root:Some parameters are on the meta device device because they were offloaded to the disk and cpu.
Loaded model
Traceback (most recent call last):
  File "/home/arjun/CM/repos/local/cache/bf6b71d3765949a9/inference/language/mixtral-8x7b/main.py", line 168, in <module>
    main()
  File "/home/arjun/CM/repos/local/cache/bf6b71d3765949a9/inference/language/mixtral-8x7b/main.py", line 135, in main
    sut = sut_cls(
          ^^^^^^^^
  File "/home/arjun/CM/repos/local/cache/bf6b71d3765949a9/inference/language/mixtral-8x7b/SUT.py", line 159, in __init__
    self.load_model()
  File "/home/arjun/CM/repos/local/cache/bf6b71d3765949a9/inference/language/mixtral-8x7b/SUT.py", line 304, in load_model
    self.model = self.model.to(memory_format=torch.channels_last)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arjun/cm/lib/python3.12/site-packages/accelerate/big_modeling.py", line 455, in wrapper
    raise RuntimeError("You can't move a model that has some modules offloaded to cpu or disk.")
RuntimeError: You can't move a model that has some modules offloaded to cpu or disk.
```
Repository: mlcommons/inference
Issue: Mixtral mlperf log file is having a new line which fails when loaded with the mlperf logger
Root cause: Validation Gap

When the Mixtral server latency constraints are not met, the submission checker is breaking with the below error.
```
 File "/home/arjun/CM/repos/local/cache/f2ac2b26439f49be/inference/tools/submission/log_parser.py", line 44, in __init__
    raise RuntimeError("Encountered invalid line: {:}".format(line))
RuntimeError: Encountered invalid line: :::MLLOG {"key": "result_invalid_reason", "value": "TTFT constrain not met: Reduce target QPS to improve latency.
```

@pgmpablo157321: Please have a look
Repository: mlcommons/inference
Issue: Llama2 and Mixtral reference implementations are failing on systems where swap space is in use
Root cause: Validation Gap

```
/usr/bin/python3 main.py  --scenario Offline --dataset-path /home/cmuser/CM/repos/local/cache/75c4d7183a544cf0/processed-openorca/open_orca_gpt4_tokenized_llama.sampled_24576.pkl --device cpu   --mlperf-conf '/home/cmuser/CM/repos/local/cache/7d864d0a183a4313/inference/mlperf.conf' --user-conf '/home/cmuser/CM/repos/gateoverflow@cm4mlops/script/generate-mlperf-inference-user-conf/tmp/74e53467428b4bfa9cdd28ecbf4a3adf.conf' --output-log-dir /home/cmuser/CM/repos/local/cache/f76efe857f9441f4/test_results/56b2cf6f11b8-reference-cpu-pytorch-v2.3.1-default_config/llama2-70b-99/offline/performance/run_1 --dtype bfloat16 --model-path /home/cmuser/CM/repos/local/cache/abfbb9b5f2e7453d/repo 2>&1 ; echo $? > exitstatus | tee '/home/cmuser/CM/repos/local/cache/f76efe857f9441f4/test_results/56b2cf6f11b8-reference-cpu-pytorch-v2.3.1-default_config/llama2-70b-99/offline/performance/run_1/console.out'
Loading dataset...
Finished loading dataset.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:33<00:00,  2.22s/it]
WARNING:root:Some parameters are on the meta device device because they were offloaded to the disk and cpu.
Loaded model
Traceback (most recent call last):
  File "/home/cmuser/CM/repos/local/cache/7d864d0a183a4313/inference/language/llama2-70b/main.py", line 98, in <module>
    main()
  File "/home/cmuser/CM/repos/local/cache/7d864d0a183a4313/inference/language/llama2-70b/main.py", line 70, in main
    sut = sut_cls(
  File "/home/cmuser/CM/repos/local/cache/7d864d0a183a4313/inference/language/llama2-70b/SUT.py", line 123, in __init__
    self.load_model()
  File "/home/cmuser/CM/repos/local/cache/7d864d0a183a4313/inference/language/llama2-70b/SUT.py", line 244, in load_model
    self.model = self.model.to(memory_format=torch.channels_last)
  File "/home/cmuser/.local/lib/python3.10/site-packages/accelerate/big_modeling.py", line 455, in wrapper
    raise RuntimeError("You can't move a model that has some modules offloaded to cpu or disk.")
RuntimeError: You can't move a model that has some modules offloaded to cpu or disk.
```
Repository: mlcommons/inference
Issue: mixtral: accuracy check output contains `np.float64(...)` which doesnt suit metric regex
Root cause: Validation Gap

The output of [mixtral-8x7b/evaluate-accuracy.py](https://github.com/mlcommons/inference/blob/master/language/mixtral-8x7b/evaluate-accuracy.py):

```
xx.xx% pass@1
{'typescript': x, 'ruby': x, 'python': x, 'javascript': x, 'php': x, 'cpp': x}
{'typescript': x, 'ruby': x, 'python': x, 'javascript': x, 'php': x, 'cpp': x}
Results
{
    'rouge1': np.float64(x),
    'rouge2': np.float64(x),
    'rougeL': np.float64(x),
    'rougeLsum': np.float64(x),
    'gsm8k': x,
    'mbxp': x,
    'gen_len': np.int64(x),
    'gen_num': x,
    'gen_tok_len': x,
    'tokens_per_sample': x
}
```
conatins `np.float64(...)` text wrapped around fp value for given fields:
- rogue1, rogue2, rogueL, rogueLsum

contains `np.int64` text around long value for given fields:
- gen_len

this fails the regexes we have defined:
https://github.com/mlcommons/inference/blob/9e2c9f642e6e12b74e7c08d2e099c8af0e542873/tools/submission/submission_checker.py#L533-L537

im not entirely sure why this behaves differently from our llama script:
https://github.com/mlcommons/inference/blob/29edbb049081698fa07ebfe903a02404edb81e79/language/llama2-70b/evaluate-accuracy.py#L95

Verified Fix: 
1. wrap here with built-in `float(round(...))` https://github.com/mlcommons/inference/blob/29edbb049081698fa07ebfe903a02404edb81e79/language/mixtral-8x7b/evaluate-accuracy.py#L180
2. wrap here with built-in int `int(np.sum(...))` https://github.com/mlcommons/inference/blob/29edbb049081698fa07ebfe903a02404edb81e79/language/mixtral-8x7b/evaluate-accuracy.py#L215
Repository: mlcommons/inference
Issue: SDXL one of the ten reference images' caption generates a disturbing image

Hi, caption id `1303` generates an image that looks a bit disturbing. Since in 4.0, we disclosed the images to the public, e.g. https://github.com/mlcommons/inference_results_v4.0/tree/main/closed/NVIDIA/results/DGX-H100_H100-SXM-80GBx8_TRT/stable-diffusion-xl/Offline/accuracy/images, I would like to ask if we can replace 1303 with a different caption or just avoid showing the images in the results directory? We can remove the images after the review period if the latter is preferred. Thanks!

Also I think the taskforce agreed to not disclose the compliance images to the public.

@pgmpablo157321 
- Add option in submission checker to remove all the stable diffusion images folder, and skip the checks done for the images folder
- Remove the images from the inference_results_v4.0 repository
Repository: mlcommons/inference
Issue: Resnet50 Tflite Model For Multistream Scenario
Root cause: Validation Gap

When I ran the Resnet50 Model (tflite model - converted from the given repective onnx model to tflite) using the tflite backend for the MultiStream scenario it shows index out of bounds error. This is the error I see it while running it in the Offline Scenario too. Can anyone help me with it?
Thanks !
Repository: mlcommons/inference
Issue: Unify the version into a single file
Root cause: Architectural Constraint

Right now when we update the loadgen version, we need to change 3 different places:
1. CMakefiles: https://github.com/mlcommons/inference/blob/master/loadgen/CMakeLists.txt
2. Setup.py: https://github.com/mlcommons/inference/blob/master/loadgen/setup.py
3. version_generator.py: https://github.com/mlcommons/inference/blob/master/loadgen/version_generator.py

Unify them into a single VERSION.txt in the directory (or parent directory) and directly read from the file for these variables so we don't miss them.
@pgmpablo157321