zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Issues running object detection (ssd-resnet34 )
Root cause: Configuration Error

Hi all, I am running object detection ssd-resnet34 model using different frameworks, it looks like the script `run_and_time.sh` is not passing the right parameters and therefore throwing errors, see below:

`./run_and_time.sh tf ssd-resnet34 gpu`
output:
INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format='NCHW', dataset='coco-1200-tf', dataset_list=None, dataset_path='/home/CK-TOOLS/dataset-coco-2017-val/', inputs=['0:0'], max_batchsize=128, max_latency=[0.01, 0.05, 0.1], mode='Performance', model='/home/mlperf_models//ssd_resnet-34_from_onnx.pb', output='/output/results.json', outputs=['concat_63:0', 'concat_64:0'], profile='ssd-resnet34-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.SingleStream, TestScenario.Server, TestScenario.Offline, TestScenario.MultiStream], threads=112, time=None)

model=ssd_resnet-34_from_onnx.pb instead of ssd_resnet34_mAP_20.2.pb

`./run_and_time.sh onnxruntime ssd-resnet34 gpu`
Error:
```
INFO:main:Namespace(accuracy=False, backend='onnxruntime', cache=0, count=None, data_format='NCHW', dataset='coco-1200-onnx', dataset_list=None, dataset_path='/home/CK-TOOLS/dataset-coco-2017-val/', inputs=['image'], max_batchsize=128, max_latency=[0.01, 0.05, 0.1], mode='Performance', model='/home/mlperf_models//resnet34-ssd1200.onnx', output='/output/results.json', outputs=['bboxes', 'labels', 'scores'], profile='ssd-resnet34-onnxruntime', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.SingleStream, TestScenario.Server, TestScenario.Offline, TestScenario.MultiStream], threads=112, time=None)
/opt/anaconda3/lib/python3.7/site-packages/onnxruntime/capi/_pybind_state.py:12: UserWarning: Cannot load onnxruntime.capi. Error: 'libcublas.so.9.1: cannot open shared object file: No such file or directory'
  warnings.warn("Cannot load onnxruntime.capi. Error: '{0}'".format(str(e)))
Traceback (most recent call last):
  File "python/main.py", line 532, in <module>
    main()
  File "python/main.py", line 398, in main
    backend = get_backend(args.backend)
  File "python/main.py", line 228, in get_backend
    from backend_onnxruntime import BackendOnnxruntime
  File "/mlperf/python/backend_onnxruntime.py", line 7, in <module>
    import onnxruntime as rt
  File "/opt/anaconda3/lib/python3.7/site-packages/onnxruntime/__init__.py", line 21, in <module>
    from onnxruntime.capi._pybind_state import RunOptions, SessionOptions, get_device, NodeArg, ModelMetadata
```







Closing, It looks like the [README.md](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/README.md) needs to be updated with the instructions to run object detection through LoadGen directly without accesing the instructions in `inference/others`
Repository: mlcommons/inference
Issue: The documents are outdated
Root cause: Documentation Deficiency

https://github.com/mlcommons/inference/blob/master/language/llama2-70b/README.md

If we follow instruction in above doc it doesn't work.

For e.g.
Below index doesn't have version 2.2.0,
 https://download.pytorch.org/whl/nightly/cpu

Also if we use latest version
then there is conflict in below step

# Need to fetch Pablo's changes
git fetch origin pull/1523/head:llm-server
git merge llm-server

As per my understanding below is way how doc should be updated
Solution : 
Remove steps below from README.
git fetch origin pull/1523/head:llm-server
git merge llm-server
Repository: mlcommons/inference
Issue: Default behaviour of torch.load changed in v2.6 breaking some of the reference pytorch implementations
Root cause: External Dependency Breakage

```
 File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/torch/serialization.py", line 1441, in load
    raise RuntimeError(
RuntimeError: Cannot use ``weights_only=True`` with TorchScript archives passed to ``torch.load``. In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source.
Traceback (most recent call last):
```

@anandhu-eng @pgmpablo157321
Repository: mlcommons/inference
Issue: [405b-SUT] Max number of output tokens
Root cause: Configuration Error

For 405B the sampling parameter config sets the [max output tokens](https://github.com/mlcommons/inference/blob/master/language/llama3.1-405b/SUT_VLLM.py#L75) to be 20k.
However, given the reference output distribution with max output length of 1.7k, I don't think we should set this parameter in the sampler that high.
@nvzhihanj @arjunsuresh @mrmhodak 


max_new_tokens should be 2000 (max input length is 20000), this looks like a typo. Can you help submit a PR to patch it?
Repository: mlcommons/inference
Issue: Getting 401 unauthorised error white using interface of api for text to image generator
Root cause: Documentation Deficiency

I used interface api code in js and also created new token multiple times but still I am getting 401 error in api.any suggestion?

Which API are you referring to?

> Which API are you referring to?
this api for text-to-image
https://api-inference.huggingface.co/models/ZB-Tech/Text-to-Image

In MLCommons inference we are not using that API, right?
Repository: mlcommons/inference
Issue: Need to fix TEST01 performance check
Root cause: Algorithmic Error

This [performance checker file for TEST01]( https://github.com/mlcommons/inference/blob/master/compliance/nvidia/TEST01/verify_performance.py) has two main issues
1. It has a lot of redundant code
2. Regular expressions are used to extract performance value from the summary log instead of using mlperf log parser to extract the official performance value from the mlperf log detail like done by the submission checker. This also means, early stopping is not respected by the TEST01 check.

Since, the code freeze is in place, this can be a v5.0 postmortem item.  

PR exists but needs cleaned up. Arjun to do so this next week.

PR is here: https://github.com/mlcommons/inference/pull/2073/files 

Closing this issue since the PR has been merged.
Repository: mlcommons/inference
Issue: Release mlcommons inference submission tools as a Python package
Root cause: Unimplemented Feature Gap

The Python code inside the "tools/submission" directory like `log_parser` and `submission_checker` are useful to be imported into other Python codes and this process can be simplified if they are released as a Python package. I'm not sure if we should do a separate package just for the submission tools or one common package for the inference-src which will include the accuracy related tools too for all the current inference benchmarks.
Repository: mlcommons/inference
Issue: Official Readme files are having Huggingface Download Links for the Llama models
Root cause: Documentation Deficiency

The model links in the READMEs need to be replaced with the MLCommons hosted links for the ones where currently the Huggingface downloads are used, for reproducibility. 

https://github.com/mlcommons/inference/tree/master/language/llama3.1-405b#mlcommons-members-download
Repository: mlcommons/inference
Issue: Create MLCommons Croissant Metadata for the Datasets created by MLCommons Inference WG
Root cause: Unimplemented Feature Gap

We have some datasets like the one for Mixtral-8x7b where the dataset is created by the respective taskforce. It'll be good to have a Croissant metadata format for these. 

https://huggingface.co/spaces/MLCommons/croissant-editor?project=20250202184959940380
Repository: mlcommons/inference
Issue: Inference [4.1.5] Inference nvidia-smi command exits but failed to execute
Root cause: Environment Incompatibility

Hello,

I ran the following command: nohup make generate_engines RUN_ARGS="--benchmarks=stable-diffusion-xl --scenarios=offline" 2>&1 | tee -a /Inference_Store/mlperf_4.1.5_inference_data/mlperf-logs/generate_engines_offline

And ran into the following errors.
[2025-02-06 15:29:21,626 systems.py:174 WARNING] nvidia-smi command exists but failed to execute - Ignoring NUMA detection.
[2025-02-06 15:29:21,870 main.py:227 INFO] Detected system did not match any known systems. Exiting. SystemConfiguration(host_cpu_conf=CPUConfiguration(layout={CPU(name='Cortex-A78AE', architecture=<CPUArchitecture.aarch64: AliasedName(name='aarch64', aliases=(), patterns=())>, core_count=4, threads_per_core=1): 1}), host_mem_conf=MemoryConfiguration(host_memory_capacity=Memory(quantity=64.348844, byte_suffix=<ByteSuffix.GB: (1000, 3)>, _num_bytes=64348844000), comparison_tolerance=0.05), accelerator_conf=AcceleratorConfiguration(layout=defaultdict(<class 'int'>, {GPU(name='CTI Forge + Orin AGX', accelerator_type=<AcceleratorType.Integrated: AliasedName(name='Integrated', aliases=(), patterns=())>, vram=None, max_power_limit=None, pci_id=None, compute_sm=87): 1})), numa_conf=None, system_id=None)
(mlperf) nvidia@mlperf-inference-nvidia-aarch64-1018:/closed/NVIDIA$ nvidia-smi
Thu Feb  6 15:34:15 2025

Any advice on how to resolve this issue? How can I ensure my system is detected and how can I check what my known systems are.

NVIDIA-SMI version: 540.4.0
Cuda version: 12.6
Driver version 540.4.0
Repository: mlcommons/inference
Issue: Is beam search strategy mandatory in gptj-6b benchmark?

hi, I found that vllm will drop beam search feature in future releases due to complicated implementation and beam search is not so popular in real use.
So could I know if gptj-6b benchmark test will drop beam search strategy too?

Yes, it is mandatory for the Closed division.

However, for the reasons that you outlined, GPTJ might be dropped from MLPerf Inference too. (Normally, a benchmark need to survive 4 submission rounds. Conveniently, GPTJ will make its 4th appearance in the upcoming v5.0 round.)
Repository: mlcommons/inference
Issue: fail to run Resnet50 v1.5 on Offline mode
Root cause: Configuration Error

environment:
 GPU: T4
GPU driver:  410.104
model:TensorFlow Resnet50 v1.5 
when running inference v0.5  classification_and_detection on Offline mode, I get the error "out of memory". I don't know how to set the parameters to adjust the batchsize.
The cmd: CUDA_VISIBLE_DEVICES=3  python python/main.py --dataset  imagenet --dataset-path /data  --model  /workspace/inference/v0.5/classification_and_detection/resnet50_v1.pb --output /workspace/inference/v0.5/classification_and_detection/output/results_test.json --backend tensorflow --profile resnet50-tf --scenario  Offline   2>&1 | tee offline.log

error info:
![image](https://user-images.githubusercontent.com/19197677/60004349-0509b700-969f-11e9-9162-ff78f010447a.png)




The default batch size is 128 and it passes for tensorflow/1080Ti.
You can set the batch size with by adding ```--max-batchsize bs``` to the command line. Maybe the default is a little aggressive since we
will queue up parallel requests - if the runtime takes them that might explain high memory usage. You could use ```--threads N``` to limit this. By default it is set to the number of cores and if you have a fat machine it might queue up a lot.


It is actually does NOT affect the outcome. I believe mlperf_log_summary.txt still gives the right outcome. My understanding is that these are warning messages. 

Maybe not the message what you'd want to see when running benchmarks :)

closing this issue due to lack of activities. Please re-open if needed.

Hi @christ1ne / @guschmue, I am facing errors also running resnet50_v1.pb on offline mode (both cpu as well as Gpu T4 option). will be this issue fixed?

Hi @inspurasc, did you complete your test successfully?, if, so what configuration did you use?
Repository: mlcommons/inference
Issue: Llama3.1-405B non-zero temperature
Root cause: Configuration Error

In the reference implementation of Llama3.1-405B, temperature is [set to 1](https://github.com/mlcommons/inference/blob/7bf59976b5f4eb7c5b8f30a88af832e028028446/language/llama3.1-405b/SUT_VLLM.py#L71).  Is this intentional?

Normally, temperature should be set to zero for outputs for be more deterministic. Can an optimized implementation use a different temperature?

**N.B:** We are aware of an open [vLLM issue](https://github.com/vllm-project/vllm/pull/12802), due to which setting temperature to zero still results in non-determinism. Maybe we will need to recalibrate reference accuracy for the next round (v5.1).

Inference WG 18/Feb/2025: multiple parties run the reference implementation and obtained identical results. Maybe it's a by-product of `topk=1`. Optimized submissions should use the same parameters.

> ## [Does a temperature of 0 result in non-determinism?](https://www.vellum.ai/llm-parameters/temperature)
> A common case of confusion is if a temperature of 0 generates non-deterministic replies. In theory, yes. In practice, no.
>
> As noted by this [OpenAI Forum Thread](https://community.openai.com/t/clarifications-on-setting-temperature-0/886447/4), achieving non-determinism is impossible. A temperature of 0 does force the SoftMax function to choose the most likely response—which is the definition of greedy sampling and is non-deterministic. However, LLMs are not run in a vacuum; race conditions of multi-threaded code impacts the established likelihoods of tokens. Consequently, while temperature reduces randomness to a minimum, it doesn’t eliminate them.
>
> However, the randomness is minimized to the extent that developers can expect near non-determinism. For most queries that specify the structure of the expected output, this reduction in randomness is sufficient.

Am I alone in thinking that by "non-determinism" the authors of this article actually mean "determinism"? Surely, "the randomness is minimized" should mean "near determinism"?

It's like [flammable vs inflammable](https://www.merriam-webster.com/grammar/flammable-or-inflammable).
Repository: mlcommons/inference
Issue: [GNN] Acc eval script change: incompatibility or corrupted file?
Root cause: External Dependency Breakage

After the change to avoid using memmap when loading the labels #2081, we're encountering pickling errors via numpy.
We're unable to load the labels with `numpy.load`. The previous loading via `np.memmap` (https://github.com/mlcommons/inference/commit/be6ff52235b74a2f4ef85bf86fa3785045229fa8) did work on our environments without issues

Steps to reproduce:
1. `wget -c https://igb-public.s3.us-east-2.amazonaws.com/IGBH/processed/paper/node_label_2K.npy`
2. `python -c "import numpy as np; labels = np.load('node_label_2K.npy', mmap_mode=None)"`
3. Error: `ValueError: Cannot load file containing pickled data when allow_pickle=False`

After setting `allow_pickle=True`, we face another error: `_pickle.UnpicklingError: invalid load key, '\x00'`

Packages used: `numpy==1.26.4`, `torch==2.1.0+cpu`

Curious if there's a dependency issue on my side.
cc: @arjunsuresh @nv-alicheng

Was the previous mode of loading the labels incorrect leading to incorrect labels? Or the current modification fixes a previously missed bug? If the latest change doesn't concern correctness of loaded labels, can we bring back the old way so as the default, and switch to `np.load` if `--no-memmap` is passed?

Hi @attafosu I can confirm that this issue is there. @ashwin @nvzhihanj can you please confirm from Nvidia side?

Not sure if this is the best fix, but is an option to get the script working. 

https://github.com/mlcommons/inference/pull/2123/files

The fix looks good to me
Repository: mlcommons/inference
Issue: `GettingStarted` link in v0.5/classification_and_detection/README.md is broken

The link to GettingStarted points to "https://github.com/mlperf/inference/blob/master/cloud/image_classification/GettingStarted.ipynb"

It should point to "https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/GettingStarted.ipynb"

I'll change it, need to update the links in the tutorial as well.

not an issue anymore.
Repository: mlcommons/inference
Issue: [URGENT] Submission UI error while running generate_final_report.py
Root cause: Configuration Error

Hi @pgmpablo157321,

 https://github.com/mlcommons/inference/blob/master/tools/submission/generate_final_report.py#L22 has a default of **4.1** so the uploaded .tar fails with the error as below, could it be changed to **5.0** to allow for the filter_scenarios to have the correct like so that the csv is to be parsed correctly?

```
Traceback (most recent call last):
  File "/tmp/inference/tools/submission/generate_final_report.py", line 452, in <module>
    sys.exit(main())
  File "/tmp/inference/tools/submission/generate_final_report.py", line 328, in main
    MakeWorksheet(
  File "/tmp/inference/tools/submission/generate_final_report.py", line 249, in MakeWorksheet
    df = df[value(df[key])]
  File "/tmp/inference/tools/submission/generate_final_report.py", line 275, in <lambda>
    return lambda x: f(x, *args)
  File "/tmp/inference/tools/submission/generate_final_report.py", line 278, in FilterScenario
    return x.apply(
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/pandas/core/frame.py", line 8833, in apply
    return op.apply().__finalize__(self, method="apply")
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/pandas/core/apply.py", line 727, in apply
    return self.apply_standard()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/pandas/core/apply.py", line 851, in apply_standard
    results, res_index = self.apply_series_generator()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/pandas/core/apply.py", line 867, in apply_series_generator
    results[i] = self.f(v)
  File "/tmp/inference/tools/submission/generate_final_report.py", line 279, in <lambda>
    lambda y: y["Scenario"] in filter_scenarios[suite][y["Model"]], axis=1
KeyError: 'llama3.1-405b'
```
Repository: mlcommons/inference
Issue: Question: Benchmark results.
Root cause: Documentation Deficiency

Hello.
I know that you can see the results right after the test, but aren't they stored anywhere? I lost access to the test terminals several times, but the tests themselves were completed successfully. And I would really like to see the results

yes. The `find cache` command give in the submission link can help you with that.
https://docs.mlcommons.org/inference/submission/

If you want the result summary, simply do a generate submission command.
Repository: mlcommons/inference
Issue: RGAT minor issues , listed in description
Root cause: Configuration Error

1. audit file names audit.conf instead of audit.config causing issue with compliance test.I believe loadgen is looking for a file named audit.config.
   https://github.com/mlcommons/inference/blob/master/compliance/nvidia/TEST01/rgat/audit.conf
   Not a blocker , help appreciated.
2. v5.0 not listed in submission_checker [here](https://github.com/mlcommons/inference/blob/cc9a89e94553e0a071fb4a5980e76aa7d0cd2615/tools/submission/submission_checker.py#L1020), resulting in equal_issue_used_check failure.


1. The default expectation from loadgen is correct but we can use any filename and pass the file explicitly to the loadgen as in the reference implementation [here](https://github.com/mlcommons/inference/blob/master/graph/R-GAT/main.py#L486). To avoid confusion I have created a PR [here](https://github.com/mlcommons/inference/pull/2127) 
2. That's correct right? For v5.0 equal issue mode check is enabled. Previously it was not enabled due to a bug in the loadgen. 

Thank you @arjunsuresh for clear explanation. We can close this issue now
Repository: mlcommons/inference
Issue: Error in RsNet50 server run
Root cause: Validation Gap

when i used the command  mlcr run-mlperf,inference,_full,_r5.0-dev,_all-scenarios --model=resnet50 --implementation=reference --framework=onnxruntime --category=datacenter --server_target_qps=<SERVER_TARGET_QPS> --execution_mode=valid --device=cpu --quiet --env.MLC_GET_PLATFORM_DETAILS=no --adr.loadgen.tags=_from-pip --pip_loadgen=yes

I got this error 

![Image](https://github.com/user-attachments/assets/525a74e1-d4f9-47b2-b28b-078dad83671e)

attaching log file also please help with possible solution

[error.txt](https://github.com/user-attachments/files/18897670/error.txt)
Repository: mlcommons/inference
Issue: Correct licensing details for upscale_coco

The file [v0.5/tools/upscale_coco/coco.py](https://github.com/mlperf/inference/blob/master/v0.5/tools/upscale_coco/coco.py) is only marginally different from the original [coco.py](https://raw.githubusercontent.com/cocodataset/cocoapi/master/PythonAPI/pycocotools/coco.py):
```
$ cd ~/projects/mlperf/inference/v0.5/tools/upscale_coco
$ ls -la
total 40
drwxr-xr-x 2 anton anton  4096 Jun 25 10:55 .
drwxr-xr-x 3 anton anton  4096 Jun 25 10:55 ..
-rwxr-xr-x 1 anton anton 18399 Jun 25 10:55 coco.py
-rwxr-xr-x 1 anton anton  1423 Jun 25 10:55 README.md
-rwxr-xr-x 1 anton anton  3447 Jun 25 10:55 upscale_coco.py
-rwxr-xr-x 1 anton anton   107 Jun 25 10:55 upscale.sh
$ wget https://raw.githubusercontent.com/cocodataset/cocoapi/master/PythonAPI/pycocotools/coco.py
diff coco.py{,.1}
55c55
< from pycocotools import mask as maskUtils
---
> from . import mask as maskUtils
308c308
<         if type(resFile) == str: #or type(resFile) == unicode:
---
>         if type(resFile) == str or type(resFile) == unicode:
433c433
<         return m
---
>         return m
\ No newline at end of file
```
Therefore, it should be under the [2-clause BSD-style license](https://github.com/cocodataset/cocoapi/blob/master/license.txt) license, as otherwise the licensing terms for the COCO tools are unfortunately violated.

I believe the other files are from NVIDIA, so they should provide their own licensing terms.

We should correct it by providing the correct license text on top of each of the files. 

@DilipSequeira please follow up

@psyhtest and @DilipSequeira  is this still an issue or can we close it out?

2 things related to this:
- I added the resize/upscale to the pre-processing in the reference app ... that was easier than to document the process :)
- the scripts still use PIL ... we'd should to change them to opencv

@guschmue hope you can handle the script update? 

yes, can do.

@guschmue gentle ping, it will be great if we can close this out this week.

yeap
Repository: mlcommons/inference
Issue: Refactor TEST01 message so it's less confusing
Root cause: Documentation Deficiency

Right now if the part II of TEST01 fails, it will show up as "TEST FAIL" in the log which is confusing for the submitter.
We should refactor the message such that the user knows there's a fallback that will be run if the current test fails, also why the part II fails and part III should be run.

@pgmpablo157321 FYI
Repository: mlcommons/inference
Issue: Inference 5.0 submission failure for Llama3.1-405b caused by hardcode in submission checker
Root cause: Architectural Constraint

In the submission checker, this is a hardocded value here:
[inference/tools/submission/submission_checker.py at master · mlcommons/inference](https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py#L598)

and this is in the official user.conf in llama3, which is lower than that number
[inference/language/llama3.1-405b/user.conf at master · mlcommons/inference](https://github.com/mlcommons/inference/blob/master/language/llama3.1-405b/user.conf#L9)
and the packaging will fail

Please look into this.

Thanks,
Zhong
AMD

Hi @zhonglidaniel since llama3.1 has equal issue mode, this should not be an issue.