zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Documentation issue: git clone command needs to include recurse-submodules
Root cause: Documentation Deficiency

It should be noted that the git clone command needs to include recurse-submodules to get loadgen to compile, i.e.
git clone --recurse-submodules https://github.com/mlperf/inference.git mlperf_inference


I've stumbled upon the same with #230 :(. But shame on me, as I was following the [LoadGen "Quick Start" instructions](https://github.com/mlperf/inference/commit/1db1b6c418b7db365f17304fb0edef50c539a7c9) instructions, which mention this.

it is present in the LoadGen quick start guide.
closing this issue due to lack of activities. Please re-open if needed.
Repository: mlcommons/inference
Issue: Documentation issue: running code in container requires EXTRA_OPS for command line arguments
Root cause: Documentation Deficiency

It appears that the run_and_time.sh does not pass along command line arguments. Instead it expects these to be included in the EXTRA_OPS environment variable, eg.
export EXTRA_OPS="--scenario MultiStream"

This should be noted in the readme.

Can we make these options all command line based, or assign configuration file as input to avoid such confusion?

Typed on a small screen, excuse brevity and typos.
vj.

________________________________
From: Dave Jaffe <notifications@github.com>
Sent: Tuesday, June 25, 2019 5:33 PM
To: mlperf/inference
Cc: Subscribed
Subject: [mlperf/inference] Documentation issue: running code in container requires EXTRA_OPS for command line arguments (#216)


It appears that the run_and_time.sh does not pass along command line arguments. Instead it expects these to be included in the EXTRA_OPS environment variable, eg.
export EXTRA_OPS="--scenario MultiStream"

This should be noted in the readme.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://github.com/mlperf/inference/issues/216?email_source=notifications&email_token=ABT6DFEGIXZRN75XGKXIB53P4KFK3A5CNFSM4H3MPHKKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G3VA4PQ>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABT6DFC2TQ2OZZWZCXSHMBTP4KFK3ANCNFSM4H3MPHKA>.


yes, can do.

@christ1ne we can close this out, as this feature has already been added [here](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection#usage)

>     [--scenario list of SingleStream,MultiStream,Server,Offline]


ok. please reopen if needed.
Repository: mlcommons/inference
Issue: V5.0 results table: "download PowerPoint" not really working

https://mlcommons.org/benchmarks/inference-datacenter/
If you click the download Powerpoint on the website it will give you a slide with a screenshot:

![Image](https://github.com/user-attachments/assets/33737d25-ad84-4cae-9491-01a349c7f251)

Please remove it or fix it. @pgmpablo157321 FYI
Repository: mlcommons/inference
Issue: Multi-GPU run for Llama2-70b inference
Root cause: Documentation Deficiency

Hi,
Is it possible to run Llama2-70b inference on multiple GPUs? When I specify `--device "cuda:0,1"`, I get this error:
```
main.py: error: argument --device: invalid choice: 'cuda:0,1' (choose from 'cpu', 'cuda:0')
```
I even tried without quotes. It seems that only two options are available: cpu or 1 gpu device. Is that right?

@mahmoodn 
Just specify it like: 

`--device cuda:0`

This will be able to utilise all ur GPUs. Although I do find it weird that its labelled as cuda:0, seems misleading...
Repository: mlcommons/inference
Issue: Can we control CPU core usage in MLPerf on NVIDIA Orin?
Root cause: Unimplemented Feature Gap

# Can we control CPU core usage in MLPerf on NVIDIA Orin?

We are running MLPerf on an **NVIDIA Orin** platform with the following CPU configuration:

- 12 CPU cores  
- 1 thread per core  
- 3 clusters  

## Scenario

We tried running MLPerf with different core affinities using `taskset`. The goal is to limit the benchmark to specific cores and observe the expected core utilization.

Here are our observations:

| Command            | Scenario   | Observation             |
|--------------------|------------|--------------------------|
| `taskset -c 0-11`  | 12C 12T    | All 12 cores active      |
| `taskset -c 0-3`   | 4C 12T     | All 12 cores still active |
| `taskset -c 0`     | 1C 12T     | All 12 cores active (core switching observed) |

We expected only the specified cores (e.g., 0-3) to be active during execution, but all cores are being used regardless of the taskset configuration.

## Question

Is it possible to control or limit the CPU cores used by MLPerf (e.g., via `taskset` or other methods) on the **NVIDIA Orin** platform?

If **yes**, how can we configure MLPerf to respect the core affinity settings?

Thanks in advance!
Repository: mlcommons/inference
Issue: Better support from Loadgen to get optimal server QPS for the submitters
GitHub labels: postmortem 5.0

Options:

- [ ] Testing LoadGen's find peak performance mode
- [ ] Log additional advice related to possible increase of QPS

Pablo: The peak perf mode seems to add extra run time. Is that acceptable?

Arjun: Even if it takes 5-10x longer but the submitter can get peak performance without manual exploration, that would be useful.

Pablo: How do we want to solve this issue?

Arjun: Short documentation on how submitters can use this mode would be useful.

Anton: How is this implemented? Can set a target QPS, and if it passes, LoadGen may decide to raise it a bit. Some kind of search involved. Is there a range you set or steps? Just setting on a single parameter may be somewhat useful, but server tuning has multivariate impacts.

Pablo: As I understand it, only tunes for QPS. Need to see exactly how it's search. Can include it in the documentation.

I just passed the flag, and it needs a starting QPS.

Arjun: It does a binary search, so it needs an upper limit, too, correct?

Pablo: Not sure how it's calculated, but I didn't need another limit. I think one of the conditions is the starting QPS needs to pass the latency test. Might be using that one as an upper limit and zero, I guess? Not quite sure.

Anton: Suspect this was developed in the early days of LoadGen, so I won't know about LLM constraints. Will need to watch. Probably out of scope to develop another solution.

Arjun: We have seen submitters who only tune the target QPS manually. So this mode could have been useful for some submitters.

Anton: I'm somewhat skeptical. Remember, increasing the run time can improve perf. This is a second variable.

Arjun: Those run time opts are relatively small. This is not targeted to solve that problem. This is for submitters who failed to set the correct server QPS. 

Pablo to follow up with documentation.
Repository: mlcommons/inference
Issue: Review meetings automation discussion
GitHub labels: postmortem 5.0

This issue is meant to discuss ways to automate review meeting tasks. For example: Detect which organizations are represented in the meeting and automatically prioritize issues accordingly.

Post mortem task proposed by @psyhtest 

Anton: Many times during review meetings, we have chair call out companies about issues, and then we wait when no one is in the call. We have technology to detect who joins, have a dashboard of which companies are present, prioritize discussion to focus on those present. 

Another suggestion: Getting "points" for doing your peer review. If you participate well, you get your issues considered with a higher priority.

Scott to talk with systems team about possibilities for automation, new tools for managing these things. Unlikely to be possible without a change of tools, but we'll see.
Repository: mlcommons/inference
Issue: Consolidate dir structure
GitHub labels: postmortem 5.0

After v4.1 we removed legacy ‘calibration’ dir because it was no longer valued (usurped by code dir). There continue to be three largely overlapping dirs which are inconvenient to navigate and only separated due to legacy (not utility): compliance, measurements, results. Due to common structure, these can be easily merged as follows.  Simplifying the dir structure will reduce common difficulties, merge complications, and especially error debugging from submission checker (frequent vague errors due to misnaming and missing files):

<img width="624" alt="Image" src="https://github.com/user-attachments/assets/67ed184d-b533-4d3a-9ec1-018b18726320" />

Post mortem task proposed by @keithachorn-intel 

PR created to consolidate Results/Measurements/Compliance into 'Results' dir.

@pgmpablo157321 - The directories for compliance and measurements are set here and reused only as variables:
- [Compliance](https://github.com/mlcommons/inference/blob/cc7d6d8eabc5cc60f8b2e540f25f1edd2a9f6cd3/tools/submission/submission_checker.py#L2676)
- Measurements: [here](https://github.com/mlcommons/inference/blob/cc7d6d8eabc5cc60f8b2e540f25f1edd2a9f6cd3/tools/submission/submission_checker.py#L2448) and [here](https://github.com/mlcommons/inference/blob/cc7d6d8eabc5cc60f8b2e540f25f1edd2a9f6cd3/tools/submission/submission_checker.py#L2448)

Would updating these two to match the results_dir be sufficient, or do other tools need to be updated also?  Perhaps for a transitionary cycle, adding a check whether the original dirs exist first before defaulting to results_dir would cover that case.
Repository: mlcommons/inference
Issue: Must preview results be followed by a published result or is a submission + withdrawal acceptable?
GitHub labels: postmortem 5.0

- There have been cases where competitive results got changed and even though available submissions were ready, the company was not happy publishing them. 
- One potential solution here will be for the preview system submitter to show the performance to the review committee and make the code and hardware available in the subsequent round. Whether the preview system must now be audited is a question to be answered. 

Post mortem task proposed by @arjunsuresh 


Would be good to clarify before the next round. Need a rules proposal in order to move forward.

Arjun to draft a proposal.

Draft PR given here: https://github.com/mlcommons/policies/pull/194/files
Repository: mlcommons/inference
Issue: Incentivize cooperative behavior and punish uncooperative behavior
GitHub labels: postmortem 5.0

How do we [incentivize](https://github.com/mlcommons/submissions_inference_v5.0/issues/65#issuecomment-2741866193) cooperative behavior and punish uncooperative behavior e.g.:
- A peer review performed by the next meeting: +1 point?
- A peer review not performed by the next meeting: -1 point?

Post mortem task proposed by @psyhtest
Repository: mlcommons/inference
Issue: Cleanup/Alignment of submission structure
GitHub labels: postmortem 5.0

- The submission directory has five specified directories, and these remain sufficient to organize the necessary data (“A submission should take the form of a directory with the following structure. The structure must be followed…”).  For many rounds however, vendor harnesses have unnecessarily expanded far beyond this structure with vendor-specific scripts, config details, environment, and documentation.  Though the original intent may have been benign, the result is to make one vendor ubiquitously represented across nearly all submitter content (see image below) and to clutter the defined structure.
- There are several potential solutions to alleviate such concerns. However the most straight-forward are:
    - As the code is nearly identical across OEMs, the vendors need only reference the original vendor code.  This is the existing Intel approach.
    - A vendor can create a dir such as ‘{SUBMITTER}/code/nv_harness’ to contain all the extra content currently residing in the ‘{SUBMITTER} dir.
- In the case where harness-specific documentation is replicated in OEM ‘common’ locations (ex. [here](https://github.com/mlcommons/submissions_inference_v5.0/blob/main/closed/Dell/code/README.md) or [here](https://github.com/mlcommons/submissions_inference_v5.0/tree/main/closed/Dell/documentation)), name files in such a way that the scope is clear (ie. calibration.md -> calibration.nv.md or nv/calibration.md).
- This issue was initiated in v4.1 post-mortem, but tabled with plans of offline resolution. No further action has been taken.

<img width="623" alt="Image" src="https://github.com/user-attachments/assets/432b06cf-7d88-4ac8-99d9-5d4ab2346740" />

Post mortem task proposed by @keithachorn-intel 

PR created to highlight the intent of aligning to the outlined directory structure.

Closing since the discussion has moved to [policies#195](https://github.com/mlcommons/policies/pull/195)
Repository: mlcommons/inference
Issue: Merge LLM interactive scenarios into the benchmark (as a separate server scenario, instead of a separate benchmark)
Root cause: Unimplemented Feature Gap

As titled. The required changes might be:

- LoadGen to support server scenarios with more than 1 set of latency thresholds (TTFT/TPOT)
- User to select the latency scenario based on a flag (userSettings probably)?
- Accuracy checker to validate the result with the correct threshold
- Result table to distinguish the 2 server scenarios

@pgmpablo157321 

@pgmpablo157321 Could you take a look and help figure out whether these changes can be implemented before the v5.1 code freeze before next Tuesday? 

@nvzhihanj @hanyunfan I opened a PR with the changes needed for this. No LoadGen changes were needed for this change, we are just adding a scenario in the submission checker and making sure the run for this scenario is a server run with the correct latencies. The submission will look something like this:

- results
    - llama2-70b
        - Interactive (optional)
            - accuracy
            - performance
                - run1
                    - mlperf_log_accuracy.json
                    - mlperf_log_detail.txt (server run with interactive latencies)
                    - mlperf_log_summary.txt (server run with interactive latencies)
        - Server
        - Offline

Pablo will run more tests on it

NVIDIA’s proposal on 6/17: 
Choice 1: Datacenter submission has to  have Offline + at least one of (Server and Interactive)
Choice 2: Datacenter submission has to have Offline and Server (and Interactive is optional extra)
MLCommon, prefer Choice1, give 1 week for everyone to think about it.


@pgmpablo157321 I think all the interactive parameters and latencies for llama-405B and 8B are missing in the mlperf.conf: https://github.com/mlcommons/inference/blob/0a3570efb0309b5581f2831d84c05fe5483b5ef7/loadgen/mlperf.conf#L60

Can you help add them?

Also in the existing mlperf.conf, the llama2-interactive still seems like a separate benchmark. Not sure if we can change to llama2-70b.interactive.xxx this round: https://github.com/mlcommons/inference/blob/0a3570efb0309b5581f2831d84c05fe5483b5ef7/loadgen/mlperf.conf#L94

Linking this thread to the discussion in PR #2224
Repository: mlcommons/inference
Issue: Enabling vllm in llama2-70b
Root cause: Documentation Deficiency

I am trying to implement the vllm flag in this reference implementation, however as I understand I have to manually spin up vllm outside of running the benchmark. However, the dockerfile does not include any vllm package installation. I was wondering which version of vllm was used to test this. Thanks!

We didn't use vLLM when creating Llama2-70B - feel free to use any version that works

Thanks, I saw it is an available flag in the reference code but it doesn't seem to work. Maybe that should be removed?

`vllm` support via OpenAI compatible API was added by NeuralMagic. 
https://github.com/mlcommons/inference/blob/master/language/llama2-70b/SUT_API.py
Repository: mlcommons/inference
Issue:  Running object detection through LoadGen
Root cause: Documentation Deficiency

It looks like the README.md needs to be updated with the instructions to run object detection through LoadGen, it is missing the instruction to download the right model for tf (ssd_resnet-34_from_onnx.pb), I found the instructions in `inference/others/cloud/single_stage_detector/`

should be updated now. please re-open if needed.
Repository: mlcommons/inference
Issue: Selection criteria of models for benchmark

Hello,

I have a genuinely curious question. How did you select the models for the inference benchmark? There are many models available, and the MLPerf Datacenter inference benchmark is conducted on the following models:

1. resnet50-v1.5
2. retinanet 800x800
3. bert
4. dlrm-v2
5. 3d-unet
6. gpt-j
7. stable-diffusion-xl
8. llama2-70b
9. llama3.1-405b
10. mixtral-8x7b
11. rgat
12. pointpainting

Is there a particular reason why these models were chosen? Thank you for any pointers or guidance.

Thanks.



I think they were chosen according to their popularity at that point in time. + these models are widely known
They also affect various fields of application. Currently, there are tens of thousands of different neural tools, and adding all popular models would be too redundant (even in terms of support and launch in future versions)

@Agalakdak Thank you for your response.

I understand the reasoning, but I’m not fully convinced that popularity alone was the main criterion for model selection. It would be helpful if someone from the working committee could clarify the official selection process, especially since these benchmarks influence industry-wide tooling and optimization efforts.

Appreciate any further insights.

@arjunsuresh @guschmue @psyhtest @pgmpablo157321 

Model discussions take place during the regular weekly Working Group (WG) meetings and in the spin-off task force sessions. If you're interested in getting more involved, consider joining the Inference WG. Please note that the Inference WG is a members-only weekly group. You can apply to join here:

https://mlcommons.us21.list-manage.com/subscribe?u=8a4948c98e20b539fcbb3fb07&id=db1322ee57




> Model discussions take place during the regular weekly Working Group (WG) meetings and in the spin-off task force sessions. If you're interested in getting more involved, consider joining the Inference WG. Please note that the Inference WG is a members-only weekly group. You can apply to join here:
> 
> https://mlcommons.us21.list-manage.com/subscribe?u=8a4948c98e20b539fcbb3fb07&id=db1322ee57

Thank you, @hanyunfan. I
Repository: mlcommons/inference
Issue: TF Object Detection: Segmentation fault (accuracy mode) / Several errors (performance mode))
Root cause: External Dependency Breakage

Hi all, please see below the tracelog when running object detection with `./run_and_time.sh tf ssd-resnet34 gpu` in accuracy and performance mode:

**Accurracy Mode:**
Error:
```
INFO:main:starting TestScenario.SingleStream
Traceback (most recent call last):
  File "python/main.py", line 527, in <module>
    main()
  File "python/main.py", line 504, in main
    lg.StartTest(sut, qsl, settings)
  File "/mlperf/python/dataset.py", line 59, in load_query_samples
/mlperf/run_helper.sh: line 15: 10 Segmentation fault  (core dumped) python3 python/main.py --scenario SingleStream,Server,Offline,MultiStream --accuracy $opts --output /output/results.json
```

**Performance Mode - TestScenario.Offline :**
A lot warnings:
`17-val/val2017/000000460347.jpg', '/home/CK-TOOLS/dataset-coco-2017-val/val2017/000000386912.jpg', '/home/CK-TOOLS/dataset-coco-2017-val/val2017/000000017627.jpg', '/home/CK-TOOLS/dataset-coco-2017-val/val2017/000000153343.jpg'], Cannot feed value of shape (128, 3, 1200, 1200) for Tensor '0:0', which has shape '(1, 3, 1200, 1200)'`
Result:
`TestScenario.Offline qps=621.77, mean=18.6959, time=39.53, queries=24576, tiles=50.0:17.2981,80.0:30.5487,90.0:34.7427,95.0:37.1441,99.0:39.1970,99.9:39.4925`

**Performance mode - TestScenario.MultiStream:**
A lot of errors/warnings:
`ERROR:main:thread: failed on contentid=['/home/CK-TOOLS/dataset-coco-2017-val/val2017/000000153343.jpg', '/home/CK-TOOLS/dataset-coco-2017-val/val2017/000000080671.jpg', '/home/CK-TOOLS/dataset-coco-2017-val/val2017/000000017627.jpg', '/home/CK-TOOLS/dataset-coco-2017-val/val2017/000000342006.jpg'], Cannot feed value of shape (4, 3, 1200, 1200) for Tensor '0:0', which has shape '(1, 3, 1200, 1200)'`
Result:
`TestScenario.MultiStream qps=232.88, mean=0.0158, time=422.11, queries=98304, tiles=50.0:0.0150,80.0:0.0166,90.0:0.0196,95.0:0.0222,99.0:0.0301,99.9:0.0409`







@vilmara  can you try the latest code? The model has been updated in readme and it is working for me.

hi @christ1ne, I will try it thanks. Did you run it local or with docker  container?

I think this was the older tf model that only could handle batch size 1. If you run it with ```--max-batch-size 1``` it should work. But the new version of this model does not have that restriction anymore and should work in your case.
Repository: mlcommons/inference
Issue: 5.1 Update the compatibility table and compliance test pages

Hi @pgmpablo157321 Could you help update the:
 
1. Compatibility table page: https://github.com/mlcommons/policies/blob/master/MLPerf_Compatibility_Table.adoc 
2. Compliance test page: https://github.com/mlcommons/inference/tree/master/compliance/nvidia 

To have 5.1 new models included

also add new models in the sample definition table of the rules doc: https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#3-scenarios

And the bandwidth requirement table:
https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#3-scenarios


> also add new models in the sample definition table of the rules doc: https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#3-scenarios
> 
> And the bandwidth requirement table: https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#3-scenarios

Is this PR #312

- [https://github.com/mlcommons/policies/pull/202 [github.com]](https://urldefense.com/v3/__https:/github.com/mlcommons/policies/pull/202__;!!LpKI!lhmRd92oBmzGTULWrUtvbzpeKE4I5l22isz8pKGW3aATbk9cyByOa-zSH1DCY5o6yattfR8P6QFd04Gup3HkJOTLSzs$)
- [https://github.com/mlcommons/policies/pull/203 [github.com]](https://urldefense.com/v3/__https:/github.com/mlcommons/policies/pull/203__;!!LpKI!lhmRd92oBmzGTULWrUtvbzpeKE4I5l22isz8pKGW3aATbk9cyByOa-zSH1DCY5o6yattfR8P6QFd04Gup3Hkll1Cvjo$)

closing as it was done.
Repository: mlcommons/inference
Issue: Feedback Capture: Standardizing the request routing algorithm

Suggestion received as outlined below. I've opened an issue to gather feedback on interest in this idea. Please leave a comment if you'd like to see it included in a future MLPerf Inference release.



_Submitter implemented request routing within the audited system based on input sequence length. While this approach is effective (compared to naive round-robin approach), it may be possible to improve efficiency by incorporating node load or system utilization as additional factors in routing decisions. Given the increasing complexity of modern, multinode inference systems, there is an opportunity for MLCommons to consider standardizing this aspect of the benchmarking environment. 
Establishing a standardized interface or set of guidelines for request routing would allow all participants to contribute and benchmark innovative algorithms in this area. This could foster joint innovation and more accurately reflect real-world deployment scenarios in MLPerf benchmarking. As multinode, high-throughput inference becomes the norm, evolving the benchmarking framework in this direction may prove beneficial for the community._


This will be closed after 9/9 should there be no interest.