zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Fail to run mobilenet_v1 on imagenet 2012
Root cause: Validation Gap

run the cmd:
 CUDA_VISIBLE_DEVICES=0  python python/main.py --dataset  imagenet --dataset-path /data  --model  /workspace/inference/mobilenet_v1_1.0_224_frozen.pb --output /workspace/inference/v0.5/classification_and_detection/output/results_test_mobilenet.json --backend tensorflow --profile mobilenet-tf  --scenario  SingleStream 2>&1 | tee error.log
get the error info 👍 
![image](https://user-images.githubusercontent.com/19197677/60232257-a165d080-98ce-11e9-99b1-834e2fa76ef5.png)

Please help to resolve this problem.

Another, when I try to run ssd-resnet34 using coco 2017 , I get the error " No such file or directory: '/data/val2017/val_map.txt' ". Can you tell me where to get the file? 
cmd:
CUDA_VISIBLE_DEVICES=0  python python/main.py --dataset  imagenet --dataset-pa                      th /data/val2017/  --model  /workspace/inference/v0.5/classification_and_detection/ssd_mobilenet_v1_coco_2018_01_28/frozen_inference_graph.pb   --output                       /workspace/inference/v0.5/classification_and_detection/output/results_test_ssd.json --backend tensorflow --profile  ssd-mobilenet-tf  --scenario  Singl                      eStream

error:
![image](https://user-images.githubusercontent.com/19197677/60232378-0b7e7580-98cf-11e9-8aaa-b32230a4fd9f.png)





1. what's the TF, cuda version and python version? Can you share the outputs for the params like:
$ ./run_local.sh tf resnet50 gpu --scenario SingleStream
INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/home/t/CK-TOOLS/dataset-imagenet-ilsvrc2012-val/', inputs=['input_tensor:0'], max_batchsize=128, max_latency=[0.01, 0.05, 0.1], model='/home/t/GitHub/inference/v0.5/classification_and_detection//resnet50_v1.pb', output='/home/t/GitHub/inference/v0.5/classification_and_detection/output/resnet50-tf-gpu/results.json', outputs=['ArgMax:0'], profile='resnet50-tf', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.SingleStream], threads=12, time=None)
INFO:imagenet:loaded 200 images, cache=0, took=0.0sec

2. you should change imagenet to coco in your command. Imagenet are for classification tasks. COCO is for detection tasks.
Repository: mlcommons/inference
Issue: mlcr can't find Google Drive root (Llama2-70b)
Root cause: Environment Incompatibility

When trying to get model with MLCR:

2025/06/30 17:56:12 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2025/06/30 17:56:28 ERROR : Google drive root 'Llama-2-70b-chat-hf': error reading source root directory: directory not found

Results in not downloading the model at all. Error pops up multiple times before failing

Hi @djmatusz-lambda 
I am facing same error. How did you resolve this ?
Repository: mlcommons/inference
Issue: Assertion error
Root cause: Validation Gap

python3 preprocess.py \
  --raw_data_dir ~/3D\ UNET/kits19/data \
  --results_dir ~/3D\ UNET/build/preprocessed_data \
  --num_proc 4
/home/sahaas/3D UNET/mlc-inference/vision/medical_imaging/3d-unet-kits19/preprocess.py:29: DeprecationWarning: Please import `zoom` from the `scipy.ndimage` namespace; the `scipy.ndimage.interpolation` namespace is deprecated and will be removed in SciPy 2.0.0.
  from scipy.ndimage.interpolation import zoom
Preprocessing /home/sahaas/3D UNET/kits19/data...
Traceback (most recent call last):
  File "/home/sahaas/3D UNET/mlc-inference/vision/medical_imaging/3d-unet-kits19/preprocess.py", line 694, in <module>
    main()
  File "/home/sahaas/3D UNET/mlc-inference/vision/medical_imaging/3d-unet-kits19/preprocess.py", line 683, in main
    preprocess_with_multiproc(args)
  File "/home/sahaas/3D UNET/mlc-inference/vision/medical_imaging/3d-unet-kits19/preprocess.py", line 529, in preprocess_with_multiproc
    cases = preproc.collect_cases()
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sahaas/3D UNET/mlc-inference/vision/medical_imaging/3d-unet-kits19/preprocess.py", line 249, in collect_cases
    collected_set == target_set
AssertionError: Some of the target inference cases were found: {'case_00400'}


can anyone tell how to reslove this?


Hi @Sara2626-TechM , I'm assuming you are trying to get the preprocessed dataset for 3D-Unet.

I'm proposing an alternate method here which uses the MLCFlow automation. You can definitely use the automation to run the inference benchmarks also.

Install the automation tool:
```
pip install mlcflow && mlc pull repo mlcommons@mlperf-automations --checkout=dev
```

Run the script to get the preprocessed dataset:
```
mlcr get,dataset,kits19,preprocessed --outdirname=<PATH_TO_DOWNLOAD_KITS19_DATASET> --quiet -j
```

You can find more on how to use the automation tool for inference benchmarks [here](https://docs.mlcommons.org/inference/)
Repository: mlcommons/inference
Issue: RuntimeError: CUDA error: invalid configuration argument
Root cause: Environment Incompatibility

My enviroment：torch 2.6.0、torchrec 1.1.0、fbgemm_gpu 1.1.0
CUDA Version：12.4
报了下述错误：

<img width="1303" height="875" alt="Image" src="https://github.com/user-attachments/assets/6e897037-4c83-4a8f-9d09-e8e885c10bc0" />

<img width="1391" height="812" alt="Image" src="https://github.com/user-attachments/assets/f98aa8d0-f4b4-460f-9934-f1570130fac4" />

<img width="1388" height="675" alt="Image" src="https://github.com/user-attachments/assets/c44f5a89-5116-41a4-a5c7-00a7821cf5b2" />


Hi @artificialzjy , could you provide more information, such as:

- The benchmark you have tried to run
- The commands you have followed

This will help us better understand and assist you.

> Hi [@artificialzjy](https://github.com/artificialzjy) , could you provide more information, such as:
> 
> * The benchmark you have tried to run
> * The commands you have followed
> 
> This will help us better understand and assist you.
My benchmark is SingleStream and commands is CUDA_VISIBLE_DEVICES=5,6,7
./run_local.sh pytorch dlrm multihot-criteo gpu --accuracy in aA800 machine when i encounter above problem. Now I change the benchmark and commands to  SingleStream and commans is CUDA_VISIBLE_DEVICES=5,6,7 and
./run_local.sh pytorch dlrm multihot-criteo gpu --accuracy. Now I encounter a new problem：RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1] because the unspecified dimension size -1 can be any value and is ambiguous. I debug the code, maybe it caused by the dataset. But I use md5sum to valid the integrity of the dataset, its correct. So now I don't konw how to solve it.

<img width="1835" height="161" alt="Image" src="https://github.com/user-attachments/assets/ce2539c7-349d-44b2-b94a-20722afe18df" />
Repository: mlcommons/inference
Issue: ssd-resnet34 1200x1200 | onnx: TypeError: only size-1 arrays can be converted to Python scalars
Root cause: Interface Contract Mismatch

Hi all, I am running object detection as` ./run_and_time.sh onnxruntime ssd-resnet34 cpu` and getting the below errors:

```
2019-06-26 23:45:06.441716307 [W:onnxruntime:Default, execution_frame.cc:291 AllocateMLValueTensorSelfOwnBufferHelper] For mlvalue with index: 677, block in memory pattern size is: 4608 but the actually size is: 3008, fall back to default allocation behavior
2019-06-26 23:45:06.442116932 [W:onnxruntime:Default, execution_frame.cc:291 AllocateMLValueTensorSelfOwnBufferHelper] For mlvalue with index: 678, block in memory pattern size is: 4608 but the actually size is: 3008, fall back to default allocation behavior
2019-06-26 23:45:06.442507447 [W:onnxruntime:Default, execution_frame.cc:291 AllocateMLValueTensorSelfOwnBufferHelper] For mlvalue with index: 679, block in memory pattern size is: 2304 but the actually size is: 1536, fall back to default allocation behavior
Traceback (most recent call last):
  File "python/main.py", line 527, in <module>
    main()
  File "python/main.py", line 509, in main
    post_proc.finalize(result_dict, ds, output_dir=os.path.dirname(args.output))
  File "/mlperf/python/coco.py", line 173, in finalize
    image_id = int(detection[0])
TypeError: only size-1 arrays can be converted to Python scalars
```


should be fixed. But you might want to use onnxruntime-0.5 which is going to be released in a few days.
Repository: mlcommons/inference
Issue: Dataset preprocessing code is not shared for several models
Root cause: Documentation Deficiency

Hi,

https://github.com/mlcommons/inference/tree/master/language/llama2-70b provides `processorca.py` to generate the eval dataset, but several other models as https://github.com/mlcommons/inference/tree/master/language/deepseek-r1 and https://github.com/mlcommons/inference/tree/master/language/llama3.1-8b do not provide a reproduction script to generate the evaluation data.

This is an issue if e.g. one would like to apply the evaluation from mlperf on other models that may use other tokenizers.

Could the code used to generate this preprocessed data be shared?

Thank you

## Analysis and Solution for Issue #2245

Hi @mlcommons team,

I've analyzed the preprocessing documentation gap across language models in the inference repository. Here's what I found and a practical solution:

### Current State Analysis:

**✅ Llama2-70b**: Has comprehensive `processorca.py` showing complete preprocessing pipeline from Open Orca dataset  
**⚠️ Llama3.1-8b**: Has `download_cnndm.py` and `prepare-calibration.py` but no main preprocessing documentation  
**❌ DeepSeek-r1**: Uses preprocessed data from cloud storage with no reproduction methodology  

### Root Cause:
Newer models hide preprocessing behind preprocessed datasets, breaking the transparency established by Llama2-70b's approach.

### Proposed Maintenance Solution:

1. **Document existing preprocessing patterns** based on the successful `processorca.py` model
2. **Create PREPROCESSING.md files** for each model directory explaining:
   - Data sources and licensing
   - Tokenization approach  
   - Filtering and sampling steps
   - Adaptation guide for different tokenizers
3. **Establish documentation template** for future models

### Immediate Contribution:
I've prepared documentation templates and analysis that can be applied immediately to address this gap. This follows existing code patterns rather than introducing new architecture.

Would this maintenance-focused approach be helpful? I can start with documenting the preprocessing steps for the affected models.

Anivar
Repository: mlcommons/inference
Issue: RUN DLRMv2 on 8 GPUS, the Volatile GPU-Util is low
Root cause: Resource Mishandling

System is a  machine with 8x A100-80GB.
I run dlrmv2 on 8 GPUS with worldsize=8，the Volatile GPU-Util is low. Every GPU'sVolatile GPU-Util is occupied about 1%~5%. Besides, I run dlrmv2 on 1 GPU with worldsize=1, the Volatile GPU-Util is about 20%. At last, when it started to inference, it throwd an error：ERROR:main:thread: failed, unable to mmap 52 bytes from file </torch_248742_334510983_63890>: Cannot allocate memory (12). How to solve it?
Repository: mlcommons/inference
Issue: [Llama3.1-8B] Unable to downloaded preprocessed data with rclone
Root cause: Documentation Deficiency

Following the steps outlined [here](https://github.com/mlcommons/inference/tree/master/language/llama3.1-8b#preprocessed), rclone is unable to download the `cnn_eval.json`. 
```
rclone copy mlc-inference:mlcommons-inference-wg-public/llama3.1_8b/cnn_eval.json ./ -P
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.1s
```

Hi @attafosu , please find the links to download the dataset via `r2 downloader` [here](https://inference.mlcommons-storage.org/index.html). These will be updated in README's as a priority.

Thanks @anandhu-eng. I'm able to download
Repository: mlcommons/inference
Issue: [Whisper] Conflict of logic in accuracy_eval.py
Root cause: Algorithmic Error

Conflict of logic in accuracy_eval.py
2 steps are performed in the script (in order) for the text output of whisper.
1. [Filter](https://github.com/mlcommons/inference/blob/1bc3e998cb29a2ccb7635a5c74c875bf0c3b6432/speech2text/accuracy_eval.py#L31): only certain characters are accepted from the text output
e.g. only ‘a-z’, ‘ ’, ‘.’ is accepted

2. [Normalize](https://github.com/mlcommons/inference/blob/1bc3e998cb29a2ccb7635a5c74c875bf0c3b6432/speech2text/accuracy_eval.py#L89): convert certain phrases into another form
e.g. “one hundred dollar” -> “100$”.

Issues: Normalize will violate the filtering logic, in the example above
“100$” should be filtered out, but added because normalization comes 2nd

Fix
- Option 1: accept digits and dollar sign: PR filed https://github.com/mlcommons/inference/pull/2252 
  
- Option 2: do normalization before filtering
Risk: might change ref accuracy. 

@mrmhodak @hanyunfan FYI for post-mortem 5.1
@keithachorn-intel 

WG meeting: Resolve in 6.0, leave open for now 

This will be patched in Task force meeting.

PR 2252 to resolve has already been merged.  Can close issue.
Repository: mlcommons/inference
Issue: LLaMA2-70B SUT_API.py batching logic
Root cause: Validation Gap

When I try to run the llama2-70b performance benchmark with a vLLM backend server, SUT_API.py seems to overwrite the batch size with the total sample count. Is this intentional or do you expect users to change this setting? Also has the Server mode been implemented for GPU runs yet?
Repository: mlcommons/inference
Issue: Llama-8B missing SingleStream accuracy
Root cause: Unimplemented Feature Gap

@pgmpablo157321 discovered that we didn't have SS dataset and accuracy for the 5000 sub-samples athttps://github.com/mlcommons/inference/tree/master/language/llama3.1-8b#5000-samples-edge .
@attafosu I believe Taran and Tianmu mentioned this will be released but we couldn't find them. 

I can help with this missing SingleStream dataset issue. I've been working on preprocessing documentation improvements (PR #2270) and noticed similar gaps in dataset availability documentation.

For the 5000 samples edge case, it appears we need:
1. The specific SingleStream dataset split (5000 samples)
2. Corresponding accuracy metrics/ground truth
3. Clear documentation on where these files should be hosted

I'll investigate the dataset structure and work with the team to ensure these files are properly generated and documented. This aligns with my ongoing work to improve dataset preprocessing documentation across models.
Repository: mlcommons/inference
Issue: [Whisper] Do we have TEST01 in Whisper?

Do we have TEST01 in whisper?
I do not remember we discussed or promised this before in the TF meeting. 


@keithachorn-intel @attafosu 

PR: https://github.com/mlcommons/inference/pull/2277

I’ve emailed the task force chair to confirm this.

Given the time I would recommend that we waive TEST01 and postpone adding back audit to 6.0

Yes, TEST01 is enabled for Whisper.

> Given the time I would recommend that we waive TEST01 and postpone adding back audit to 6.0

I don't think this is appropriate.  TEST01 has been enabled (see https://github.com/mlcommons/inference/tree/master/compliance/nvidia/TEST01/whisper) and is already integrated into the submission checker. If it helps, I can confirm that it performs as expected on our end.

@keithachorn-intel I was digging through the taskforce meeting notes and don't recall we adding TEST01.
But we were able to get past it so don't mind including it. I assume only TEST01 should be needed right?

Hi, from what I've understood, yes TEST01 was not discussed and perhaps we can decide on that in the post-mortem, but it's normally present for all workloads except LLMs, but if there were not issues with it it can be kept for now, we have also been able to get past it so it should not be an issue

Sure, @wu6u3tw we can close this issue. We can further investigate next round.
Repository: mlcommons/inference
Issue: fail to run ssd-resnet34 1200x1200
Root cause: External Dependency Breakage

I want to run the inference v0.5 test case. The environment is : 
os: ubuntu 16.04  gpu: T4  cpu: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
I build the image using the given Dockerfile . And enter the container to run the test cmd.

1.
I try to run ssd_resnet34_mAP_20.2.pb using the below command, (using T4 )
 CUDA_VISIBLE_DEVICES=0  python python/main.py --dataset  coco-1200-tf    --dataset-path /data  --model  /workspace/inference/v0.5/classification_and_detection/ssd_resnet34_mAP_20.2.pb   --output /workspace/inference/v0.5/classification_and_detection/output/results_test_resebet.test.json --backend tensorflow --profile  ssd-resnet34-tf  --scenario  SingleStream  --count 100

but I met below error: . Please help me to resolve it .Thanks.

![image](https://user-images.githubusercontent.com/19197677/60324393-3fcd6100-99b7-11e9-9088-11a2d55d3e2f.png)

2. I am want to run the inference on cpu.  I use ssd_mobilenet_v1_coco_2018_01_28 model. The test running ok under SingleStream and Server scenario. But under MultiStream, I get this error:

![image](https://user-images.githubusercontent.com/19197677/60330085-444c4680-99c4-11e9-88d9-d1cff980be5e.png)

the cmd is :
 python python/main.py --dataset  coco-300    --dataset-path /da  --model  /workspace/inference/v0.5/classification_and_detection/ssd_mobilenet_v1_coco_2018_01_28/frozen_inference_graph.pb   --output /workspace/inference/v0.5/classification_and_detection/output/results_test_ssd.test.json --backend tensorflow --profile  ssd-mobilenet-tf  --scenario  MultiStream  --count  100 

3. And mobilenet-v1 can run ok under SingleStream and Server scenario. But under MultiStream, I get this error:
![image](https://user-images.githubusercontent.com/19197677/60330462-0f8cbf00-99c5-11e9-851d-987dc233ddc1.png)


For the first issue: I think we are waiting for this PR: https://github.com/mlperf/inference/pull/183

For issue 2 and 3 - there used to be a loadgen issue that looked like this but there was a fix maybe a week ago and I have not seen it since.

Can you try the latest commit 3f23f7c, rebuild loadgen and try TF 1.14? 
1. is resolved now.

can you share the errors for 2&3 with the latest? I don't see these issues with the latest commit @inspurasc




@christ1ne  
I have git clone the latest inference code.  And For 3 could run ok. But for 2 it's still could not run. 
I reproduce the test on GPU T4. Below is the error info:
![image](https://user-images.githubusercontent.com/19197677/60558420-cac5a700-9d7b-11e9-8aa1-4fd8938b084c.png)
Repository: mlcommons/inference
Issue: Submission checker wrongly insists on compliance TEST01 for LLMs
Root cause: Validation Gap

Manifested as:

```
[2025-07-18 15:33:31,229 submission_checker.py:3309 ERROR] Missing TEST01 in compliance dir closed/<submitter>/compliance/<system>/llama2-70b-99/Offline
[2025-07-18 15:33:31,229 submission_checker.py:3132 ERROR] closed/<submitter>/compliance/<system>/llama2-70b-99/Offline/TEST01/verify_accuracy.txt is missing in closed/<submitter>/compliance/<system>/llama2-70b-99/Offline/TEST01
[2025-07-18 15:33:31,229 submission_checker.py:2811 ERROR] compliance dir closed/<submitter>/compliance/<system>/llama2-70b-99/Offline has issues
```
Repository: mlcommons/inference
Issue: Interactive: still occasionally a separate workload rather than a fully fledged scenario?
Root cause: Documentation Deficiency

It's confusing that `mlperf.conf` still uses separate workloads and the Server scenario for the Interactive case.

<details><summary><code>grep -Ri interactive inference/loadgen/*</code></summary><pre>
docs/src/doxygen.cfg:# enable generation of interactive SVG images that allow zooming and panning.
docs/src/doxygen.cfg:INTERACTIVE_SVG        = YES
mlperf.conf:llama2-70b-interactive.*.performance_sample_count_override = 24576
mlperf.conf:llama3_1-405b-interactive.*.performance_sample_count_override = 8313
mlperf.conf:llama3_1-8b-interactive.*.performance_sample_count_override = 13368
mlperf.conf:llama2-70b-interactive.*.sample_concatenate_permutation = 1
mlperf.conf:llama3_1-405b-interactive.*.sample_concatenate_permutation = 1
mlperf.conf:llama3_1-8b-interactive.*.sample_concatenate_permutation = 1
mlperf.conf:llama2-70b-interactive.*.use_token_latencies = 1
mlperf.conf:llama3_1-405b-interactive.*.use_token_latencies = 1
mlperf.conf:llama3_1-8b-interactive.*.use_token_latencies = 1
mlperf.conf:# Target Latencies for interactive setting
mlperf.conf:llama2-70b-interactive.Server.target_latency = 0
mlperf.conf:llama2-70b-interactive.Server.ttft_latency = 450
mlperf.conf:llama2-70b-interactive.Server.tpot_latency = 40
mlperf.conf:# Target Latencies for interactive setting
mlperf.conf:llama3_1-405b-interactive.Server.target_latency = 0
mlperf.conf:llama3_1-405b-interactive.Server.ttft_latency = 4500
mlperf.conf:llama3_1-405b-interactive.Server.tpot_latency = 80
mlperf.conf:# Target Latencies for interactive setting
mlperf.conf:llama3_1-8b-interactive.Server.target_latency = 0
mlperf.conf:llama3_1-8b-interactive.Server.ttft_latency = 500
mlperf.conf:llama3_1-8b-interactive.Server.tpot_latency = 30
</pre></details>

Still, I assume that `user.conf` for Interactive is expected to look like:

```
*.Server.target_qps = <target QPS lower than for Server to meet more stringent latency constraints>
*.Server.min_duration = <minimum duration in milliseconds; at least 600,000>
```

(`*` here should cover both `llama2-70b-interactive` and `llama2-70b`, whichever is correct.)

I'm not sure whether PR #2281 was required. For us, just fixing the missing comma as in PR #2283 seemed sufficient.

@pgmpablo157321 Could you take a look
Repository: mlcommons/inference
Issue: Confusing mlcommon conf_type for the configuration file
Root cause: Validation Gap

Currently the conf_type is not enumed, and it's very confusing what 0,1,2 means
https://github.com/mlcommons/inference/blob/d82d7d42ae9bfaff9941a6698954918504d8f0eb/loadgen/test_settings_internal.cc#L532

The pybinding: https://github.com/mlcommons/inference/blob/d82d7d42ae9bfaff9941a6698954918504d8f0eb/loadgen/bindings/python_api.cc#L351
Specify that conf_type=1 means loading user.conf, but in the actual code there is a weird block to load conf_type=0 with conf_type=1? ([line540](https://github.com/mlcommons/inference/blob/d82d7d42ae9bfaff9941a6698954918504d8f0eb/loadgen/test_settings_internal.cc#L540)

I think we should:

1. enum the conf_type, and clean up the logic there
2. Report runtime error and exit immediately (for multiple conf file), instead of loadgen error that won't be caught until after the run

cc: @pgmpablo157321 @arjunsuresh 

@nvzhihanj 
1. Yes, that will be cleaner.
2. Yes, loadgen exiting early makes sense. I don't remember clearly but there was some concern though with this. If not we can do it for the next round.
Repository: mlcommons/inference
Issue: BERT-99 v5.0 startup failure, script error
Root cause: Unimplemented Feature Gap

Hello everyone. Today I tried to run BERT-99. I tried running version 5.0 and 5.1.
I got errors in both cases. I also tried to run the benchmark on the processor and it similarly failed.
It seems that now is the moment of transition from 5.0 to 5.1, perhaps the problem is related to this? I attach the log below

[bert99_v5.0_bad_run.txt](https://github.com/user-attachments/files/21470029/bert99_v5.0_bad_run.txt)

In v5.0 Nvidia didn't submit bert as it became edge only. So, this won't work as of now. 

@arjunsuresh 
Is there any information about whether this will work in v 5.1?

Hi @Agalakdak 

> Is there any information about whether this will work in v 5.1?

We can only know after the v5.1 results becomes public. We are happy to integrate if there is a submission for Bert from their side.