zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: RNN-T inference Benchmark
Root cause: Documentation Deficiency

Hi,
Few questions/issues regarding the RNN-T inference benchmark:

1. How come the dataset is loaded in memory? The comment says it speeds up execution, but it does not model actual inference systems in which data movement to the accelerator (e.g., via PCIe) is an actual property and can't just be ignored.
 https://github.com/mlcommons/inference/blob/master/speech_recognition/rnnt/QSL.py#L57

2. Correct me if I'm wrong, but seems like execution in the SUT is not in batches? I guess this was given as an example and could be changed in submission right?
https://github.com/mlcommons/inference/blob/master/speech_recognition/rnnt/pytorch_SUT.py#L88
Looking at the code: seems like the model supports batches, but preprocessing does not.

3. Correct me if I'm wrong, It seems like in RNN-T some of the preprocessing is part of the benchmark?
https://github.com/mlcommons/inference/blob/master/speech_recognition/rnnt/pytorch_SUT.py#L97

4. Continuing Q3, It seems like some of the preprocessing is part of the benchmark, however, I see some preprocessing when loading the file, which is not benchmarked:
https://github.com/mlcommons/inference/blob/master/speech_recognition/rnnt/pytorch/parts/segment.py#L104-L105
where `cls` calls
https://github.com/mlcommons/inference/blob/master/speech_recognition/rnnt/pytorch/parts/segment.py#L34-L43
maybe it should be benchmarked as well?

5. Readme lacks the recommended way to build the environment, I see several options in files -
CK, Docker, Conda. what is the intended way to set up env?





The WG will reply with more accurate answers, but here is my understanding:

1. In MLPerf-Inference benchmarks, the dataset starts at host memory instead of accelerator memory. The reason is as you mentioned. The `load_query_samples()` function you pointed out only loads the data to host memory, not to accelerator memory.
2. The reference implementation is just for your reference. You can submit your own code for best performance as long as they follow the MLPerf-Inference submission rules.
3. Yes, the raw waveform -> mel-spectrogram part is included in the benchmark.
4. Currently, the .wav file loading and decoding is not benchmarked because in some applications the input would be raw waveforms. For example, a robot whose accelerator reads raw sound wave directly from the microphone.
5. I don't think there is a recommended way per se. But @psyhtest has been maintaining the CK path, so it may be worth a try. I personally usually use docker.
Repository: mlcommons/inference
Issue: classification_and_detection readme 404s
Root cause: Documentation Deficiency

The readme for [vision/classification_and_detection](https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/README.md) has quite a few broken links:

- model=resnet50-v1.5, framework=tensorflow
    - model source: "tensorflow"
    - notes: "here"
- model=resnet50-v1.5, framework=onnx,pytorch
    - model source: "this script"
- model=mobilenet-v1, framework=onnx,pytorch
    - model source: "this script"
- model=ssd-mobilenet 300x300, framework=onnx
    - model source: "this script"
- model=ssd-resnet34 1200x1200, framework=tensorflow
    - model source: "from mlperf"
- model=ssd-resnet34 1200x1200, framework=pytorch
    - model source: "from mlperf"
- model=ssd-resnet34 1200x1200, framework=onnx
    - model source: "from mlperf"

model=resnet50-v1.5, framework=tensorflow, model source: "tensorflow", notes: "here"
fixed.

model=resnet50-v1.5, framework=onnx,pytorch, model source: "this script"
seems ok.

model=mobilenet-v1, framework=onnx,pytorch, model source: "this script"
seems ok.

model=ssd-mobilenet 300x300, framework=onnx, model source: "this script"
seems ok.

model=ssd-resnet34 1200x1200, framework=tensorflow, model source: "from mlperf"
fixed.

model=ssd-resnet34 1200x1200, framework=pytorch, model source: "from mlperf"
fixed.

model=ssd-resnet34 1200x1200, framework=onnx, model source: "from mlperf"
fixed.


merged.
Repository: mlcommons/inference
Issue: PyTorch Mobile

Hi,
I am working in PyTorch Mobile team.
We would like to add pytorch mobile benchmarking to MLCommond Mobile.
What will be the process of that integration?
Who can be a good POC for that to ask questions?

I think you want to join the mobile-wg to get this started.
@rnaidu02 can help.

@IvanKobzarev If you haven't joined Mobile WG, please join using the instructions provided at https://mlcommons.org/en/get-involved/#getting-started 

Mobile WG meets every Wednesday at 3 pm for 1 hr. Once you join the "Mobile WG", using the above mentioned link, you will receive the instructions to join the WG meetings, where we can discuss pytorch mobile benchmarking.

Pointers given to contact Mobile WG.
Repository: mlcommons/inference
Issue: update code in resnet main 
Root cause: Validation Gap

need change. cloud/image_classification/python/main.py:286. just want to follow up. @guschmue 

Here's a direct link: https://github.com/mlperf/inference/blob/master/cloud/image_classification/python/main.py#L286

https://github.com/mlperf/inference/pull/100
Repository: mlcommons/inference
Issue: Is the wrong TensorFlow model converted to ONNX for SSD-ResNet34?
Root cause: Configuration Error

This [line](https://github.com/mlcommons/inference/blob/819a825de093421eb1969a163e25f7abe72cd32e/vision/classification_and_detection/tools/convert-to-onnx.sh#L48) in the conversion script shows that the TF model `tf_ssd_resnet34_22.1.pb` is getting converted to  `ssd_resnet34_mAP_20.2.onnx`. I find it strange that `ssd_resnet34_mAP_20.2.pb` is not getting used. Is this intentional?

Note that the 22.1 model (on Zenodo here: https://zenodo.org/record/3345892) is different from the mAP_20.2 model (on Zenodo here: https://zenodo.org/record/3262269).

Also, note that the README claims that the TF->ONNX is using the "mAP 0.20" model.

@guschmue -- do you have insight on this? Either the code should be updated to reflect the README, or the README should be updated to reflect the code.
Repository: mlcommons/inference
Issue: What's model source for the pre-trained SSD-RN34?
Root cause: Documentation Deficiency

The [mlperf training repo](https://github.com/mlcommons/training/tree/master/single_stage_detector/ssd) only provides the SSD-RN34-300 training benchmark. However, in this inference repo, we are using pre-trained [SSD-RN34-1200 model](https://zenodo.org/record/3236545/files/resnet34-ssd1200.pytorch) for the inference. How is this SSD-RN34-1200 model get trained and where can we find the source of SSD-RN34-1200 training repo to get this pretrained model?

I think it comes from here:
https://github.com/mlcommons/inference/tree/v0.7/others/cloud/single_stage_detector

Hi @guschmue, thanks for the comment. From [your link](https://github.com/mlcommons/inference/tree/v0.7/others/cloud/single_stage_detector ), I only find the training script with tensorflow but not pytorch. Does this mean, pytorch SSD-RN34-1200 pre-trained model(`wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=13kWgEItsoxbVKUlkQz4ntjl1IZGk6_5Z'  -O 'resnet34-ssd1200.pth'`)  is trained by tensorflow and converted to pth format?

@itayhubara @ephrem-git Do you have any comments to the question from @leslie-fang-intel 

The pytorch training script seems to be here:
https://github.com/mlcommons/inference/tree/v0.7/others/cloud/single_stage_detector/pytorch

The official download link in the [readme](https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection) points to 

https://zenodo.org/record/3236545/files/resnet34-ssd1200.pytorch


The TF version is here: https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection. Look for "ssd-resnet34."

Thanks @guschmue 
Do you know the methods to run pytorch training script inside your link:
https://github.com/mlcommons/inference/tree/v0.7/others/cloud/single_stage_detector/pytorch ?
Repository: mlcommons/inference
Issue: Adding common optimizations to reference scripts
Root cause: Unimplemented Feature Gap

If we assume that there are common optimizations that are being performed across the vast majority of submitters, then perhaps we should just implement them in the reference scripts. For example in BERT offline: sorting the inputs according to their sequence length.

If for some reason an optimization does not benefit a hardware platform, then it could easily be disabled with a flag.

As of now, the reference scripts are not very useful besides showing the absolute bare minimum to run a model with loadgen. However, if we want to improve the baseline quality of our scripts, especially when using frameworks such as pytorch or tensorflow, then I think this effort will be worth it.
Repository: mlcommons/inference
Issue: Can not download dlrm-tf model 
Root cause: External Dependency Breakage

>gsutil cp gs://dmchen-data/dlrm_tf1_data/model_dir_0

INFO 0701 22:14:30.257003 retry_util.py] Retrying request, attempt #1...
INFO 0701 22:14:54.007968 retry_util.py] Retrying request, attempt #2...
INFO 0701 22:15:19.591622 retry_util.py] Retrying request, attempt #3...
INFO 0701 22:15:49.910099 retry_util.py] Retrying request, attempt #4...
INFO 0701 22:16:26.977019 retry_util.py] Retrying request, attempt #5...

Hi,cloud you update the new source of dlrm tensorflow v1 model? thanks 

Others have experienced this same problem: https://github.com/mlcommons/inference/issues/918

It's pretty shocking that the model for a key benchmark has apparently disappeared. Surely some others have their own copies of this model, even if it missing at that gs bucket. Anyone, anyone?

Updated the other issue.  We are actively working on re-uploading the data to a different location.

TF checkpoint for DLRM is available at https://drive.google.com/drive/folders/1B8n8YzkKSn5Khs2gqiu_Ql16_v-DxF3B?usp=sharing
Repository: mlcommons/inference
Issue:  raise JSONDecodeError("Expecting value", s, err.value)
Root cause: Validation Gap

Hi,
What could be the reason for below error?

> $  ./run_local.sh tflite --profile ssd-mobilenet-tf
> INFO:main:Namespace(accuracy=False, backend='tflite', cache=0, count=None, data_format=None, dataset='coco-300', dataset_list=None, dataset_path='/home/root/data/coco', debug=False, find_peak_performance=False, inputs=['image_tensor:0'], max_batchsize=32, max_latency=None, mlperf_conf='../../mlperf.conf', model='/home/root/model/resnet50_v1.tflite', model_name='ssd-mobilenet', output='/home/root/inference/vision/classification_and_detection/output/tflite-cpu/resnet50', outputs=['num_detections:0', 'detection_boxes:0', 'detection_scores:0', 'detection_classes:0'], profile='ssd-mobilenet-tf', qps=None, samples_per_query=None, scenario='SingleStream', threads=4, time=None, user_conf='user.conf')
> Traceback (most recent call last):
>   File "python/main.py", line 558, in <module>
>     main()
>   File "python/main.py", line 428, in main
>     ds = wanted_dataset(data_path=args.dataset_path,
>   File "/home/root/inference/vision/classification_and_detection/python/coco.py", line 59, in __init__
>     coco = json.load(f)
>   File "/usr/lib/python3.8/json/__init__.py", line 293, in load
>     return loads(fp.read(),
>   File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
>     return _default_decoder.decode(s)
>   File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
>     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
>   File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
>     raise JSONDecodeError("Expecting value", s, err.value) from None
> json.decoder.JSONDecodeError: Expecting value: line 7 column 1 (char 6)
> 

Thanks

We are trying to read annotations/instances_val2017.json from the coco directory. Something seems wrong with that file (like it is not json). Try to vi annotations/instances_val2017.json.

Do you want to run resnet50 or ssd-mobilenet ?
Repository: mlcommons/inference
Issue: SSD Mobilenet error using Torch 1.9
Root cause: External Dependency Breakage

padding="same" is causing error with pytorch 1.9

Traceback (most recent call last):
  File "models/ssd_mobilenet_v1.py", line 274, in <module>
    get_tf_pretrained_mobilenet_ssd(args.input_model)
  File "models/ssd_mobilenet_v1.py", line 254, in get_tf_pretrained_mobilenet_ssd
    model = create_mobilenetv1_ssd(91)
  File "models/ssd_mobilenet_v1.py", line 228, in create_mobilenetv1_ssd
    backbone = MobileNetV1Base()
  File "models/ssd_mobilenet_v1.py", line 53, in __init__
    conv_dw(32, 64, 1),
  File "models/ssd_mobilenet_v1.py", line 34, in conv_dw
    Conv2d_tf(
  File "/home/arjun/model-pytorch-mlperf-ssd-mobilenet/inference/vision/classification_and_detection/python/models/utils.py", line 46, in __init__
    super(Conv2d_tf, self).__init__(*args, **kwargs)
  File "/home/arjun/.local/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 430, in __init__
    super(Conv2d, self).__init__(
  File "/home/arjun/.local/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 90, in __init__
    raise ValueError(
ValueError: Invalid padding string 'SAME', should be one of {'same', 'valid'}

@guschmue Can you please check this issue?
Repository: mlcommons/inference
Issue: How do you ensure that the test can run on different hardware

Hi guys！

I am reading the code right now, I am a rookie and I have a quick question here. I think if you want to test different hardware, Then we have to support that hardware  at the bottom. So here is the question. If there is a totally new hardware coming in, is there any interfere that we could make our test support it. Or I am thinking in a wrong way? Thank you so much!

@AniFace  Please point to the code you are looking. Are you reading reference implementation code?
Reference code will provide a flow of benchmark implementation and it is up to the submitter to make changes to the code to work on new h/w.
Repository: mlcommons/inference
Issue: Benchmark in classification (i.e. resnet-50) didn't support pytorch backend?
Root cause: Configuration Error

Hello, I am running the test for resnet50 in classification and detection. Unfortunately, I found that the benchmark didn't support the PyTorch backend.  

1. In run_common.sh, the model for resnet50-PyTorch is resnet50_v1.onnx instead of resnet50-19c8e357.pth?
<img width="588" alt="截屏2021-07-12 13 59 49" src="https://user-images.githubusercontent.com/46394627/125238657-1cfe1300-e31a-11eb-93ce-0feef013b9b3.png">

2. There are no supported profiles for resnet50-pytorch.
<img width="455" alt="截屏2021-07-12 14 00 48" src="https://user-images.githubusercontent.com/46394627/125238818-5f275480-e31a-11eb-966e-87d50b3011d3.png">


@guschmue to look

changes are in place for ResNet50 + pytorch
Repository: mlcommons/inference
Issue: what does the parameter "--threads" do?

When I set the --threads to 1 rather than os.cpu_count() , i can get the best performance 

when i increase the --threads, the qps will low down.

It seems that this parameter will effect on the query in LoadGen, but  not to speed up the inference ? 


@guschmue to comment

Hi @guschmue.

If I understood you correctly at the last Inference WG meeting, you mentioned that the current implementation of QueueRunner is quite naïve and can be rewritten to improve parallelization and support dynamic batching.  
I assume you refer to this code https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/python/main.py .

We are hitting the same problem as reported in this issue with our TVM backend.

Can we provide our own implementation of this QueueRunner class with our v1.1 submission in the closed division to better utilize hardware resources?

Thank you!

CC @tmoreau89


Yes, you can provide improvements

Thank you!
Repository: mlcommons/inference
Issue: ERROR:main:thread: failed on contentid=['/data/dataset-imagenet-ilsvrc2012-val/ILSVRC2012_val_00000001.JPEG'], index 1 is out of bounds for axis 0 with size 1
Root cause: Validation Gap

Facing this error while running the SingleStream scenario. Any idea on how to fix this?

@samhithaaaa  which benchmark you ran? 

> @samhithaaaa which benchmark you ran?

@rnaidu02 MLPerf benchmark.

Out of MLPerf benchmarks, which model did you use? i.e ResNet50, MobileNet, ...
Based on the error, it looks like you may not have the dataset at the location as shown in the error or the image data got corrupted.
Repository: mlcommons/inference
Issue: Start_from_device isn't recorded in the results files
Root cause: Documentation Deficiency

When a system is able to meet the requirements, "Start_From_Device" can be enabled to allow inference workload starts from accelerators memory instead of host memory. But it is not documented in the final results file, which makes the validation very hard. 

Can we have it documented in the results files or logged somewhere?

@nvpohanh What is the best way to add this to the logs/results? We should probably also do this for end_on_device.

Maybe we can print it in detailed log? Let me file a PR to print that in LoadGen.

Actually, I just realized that I misunderstood the problem. **LoadGen does not know if QSL starts the input data from device because QSL is implemented by submitters.** The solution would be requesting NVIDIA-partners to submit the harness logs, but that won't be required by WG submission rules. Let us discuss this offline since this is not something that can be printed by LoadGen.

Thanks Po-han. @nvpohanh 

The RMDA feature was only implemented by NVIDIA in the last round - v1.0. I don't know if others will do the same for the new v1.1 submission. So just thinking it could be better if this can be covered in a more general way that covers everyone.

Thanks,
Frank
Repository: mlcommons/inference
Issue: [DLRM] add dataset licenses somewhere
Root cause: Documentation Deficiency

https://ailab.criteo.com/ressources/criteo-1tb-click-logs-dataset-for-mlperf/ is the license for DLRM. can we put this link in both training and inference so it's easily findable?


Now the DLRM model links to point to this license so as to download the model here:https://github.com/mlcommons/inference/tree/master/recommendation/dlrm/pytorch#supported-models

Is this sufficient?

LGTM

Let's close it in the WG.
Repository: mlcommons/inference
Issue: Random seeds added to wrong place in checker script
Root cause: Configuration Error

The seeds were updated in the wrong place in the checker script. [Commit 90243cb](https://github.com/mlcommons/inference/commit/90243c8bd745b808e5f0c2bd4593491a623cd595) has mistakenly changed the seeds for MLPerf v0.5. We need a new `v1.1` dictionary added to `MODEL_CONFIG` with the new seeds.

Sorry. The issue should be resolved by #988.
Repository: mlcommons/inference
Issue: [RNN-T] RuntimeError: Module 'LstmDrop' has no attribute 'inplace_dropout' :
Root cause: Validation Gap

Hi, i have a qustion about modify rnnt dropout.
when i modify 
-  rnnt.toml : dropout=0.0.
- pytorch_SUT.py:  model.load_state_dict(load_and_migrate_checkpoint(checkpoint_path),  strict=False) 

 Run RNNT MLPepf_inference. it show below error:
RuntimeError:
Module 'LstmDrop' has no attribute 'inplace_dropout' :
  File "/home/user/pim-workspace/MLPerf_inference/speech_recognition/rnnt/pytorch/rnn.py", line 84
        x, h = self.lstm(x, h)

        if self.inplace_dropout is not None:
           ~~~~~~~~~~~~~~~~~~~~ <--- HERE
            self.inplace_dropout(x.data)

Qustion: 

1. is it possible to modify RNNT dropout value=0.0?
2. how to solve this problem?
Repository: mlcommons/inference
Issue: Could you please provide a processed binary loader of the Terabyte dataset?
Root cause: Unimplemented Feature Gap

The Terabyte dataset is very large and very hard to preprocess, but for the inference task, we only need the last day's data, which is relatively small and affordable. Now many people including me are prohibited from playing with the full dataset because of the hardware limitation, so I am wondering if you could provide a saved binary loader for us? Thanks.

This is exactly what I have been wondering......Processing the whole 1.1 TB data only to get the last day as the test set is too expensive, and I had failed a few times and finally had to switch to the subsampled version. I hope the MLCommons community could help make this experiment more accessable.

@mnaumovfb Can you please take a look at the issue and comment?

We need Criteo to allow last day of logs to be preprocessed by mlcommons and shared with inference wg.

outdated
Repository: mlfoundations/evalchemy
Issue: Support CodeForces benchmark evaluation
Root cause: Unimplemented Feature Gap


Is this the test split you are talking about? 
https://huggingface.co/datasets/open-r1/codeforces/viewer/default/test

Or can you provide more details for the codeforces benchmark evaluation (is there one standard way of evaluation?) 

Or https://huggingface.co/datasets/Qwen/CodeElo? 

Hi @RyanMarten, thank you for your reply. I want to replicate the **Codeforces** results mentioned in the [DeepSeek-R1](https://arxiv.org/abs/2501.12948) paper. However, they don't provide too many details about it.

<img width="835" alt="Image" src="https://github.com/user-attachments/assets/9f274cd8-9135-401d-94b7-374c5138b050" />

<img width="784" alt="Image" src="https://github.com/user-attachments/assets/835febaa-8cef-4b1c-8839-0f172cdf5cd5" />

#106 solves that I think
Repository: mlfoundations/evalchemy
Issue: How many responses are sampled per query?

Thanks to the authors for their work！

I would like to ask how many responses are sampled per query. I don't seem to have found this hyperparameter. Can it be controlled?

Also, I found that running the GPQA task on 8*H20 takes several hours（deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B， maximum generation length is set to 32,768）. Is this normal?

Thank you!

For each task, you can define/change it, e.g for AIME25, you can find it [here](https://github.com/mlfoundations/evalchemy/blob/ce5cea94f9f0f61388d2234afb01d811ff4357f4/eval/chat_benchmarks/AIME25/eval_instruct.py#L48).
Repository: mlfoundations/evalchemy
Issue: Batch size
Root cause: Documentation Deficiency

I am evaluating  my model using this command, but the batch size is "1" for chat completions and not taking "5".
Error:
Chat completions does not support batching. Defaulting to batch size 1.
Any other way to evaluate my API based model using batching? I do not want to use curator.
python -m eval.eval \
  --model "openai-chat-completions" \
  --tasks "AIME24" \
  --model_name "Qwen/Qwen3-32B" \
  --apply_chat_template "True" \
  --model_args "tokenized_requests=False,batch_size=5" \  
  --output_path "logs" \
  --gen_kwargs "max_tokens=120000,temperature=0.6,top_p=0.95,top_k=20"

The parameters can be a bit confusing, batch size is a parameter of the evaluation not of the model. The basic usage example from the readme shows it:
```
python -m eval.eval \
    --model hf \
    --tasks HumanEval,mmlu \
    --model_args "pretrained=mistralai/Mistral-7B-Instruct-v0.3" \
    --batch_size 2 \
    --output_path logs
```