zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Proposal: Change all module/folder name from hyphen to underscore
Root cause: Architectural Constraint

Python module disallows usage of hyphen ('-') in module name, and it makes importing and module run very complicated. We should change the naming of folder and module (e.g. llama2-70b) to use underscore.
@pgmpablo157321 FYI


Affects 3D-UNet, Llama2-70B, DLRM-v2
Leaving open for now, can affect workflow
@arjunsuresh and @gfursin : Any comments?

@nvzhihanj and @mrmhodak - we should be able to support such changes in CM. Please ping us if/when this change is done so that we could test all our workflows. Thanks!
Repository: mlcommons/inference
Issue: SDXL model download link broken
Root cause: External Dependency Breakage

https://cloud.mlcommons.org/index.php/s/LCdW5RM6wgGWbxC 
https://cloud.mlcommons.org/index.php/s/DjnCSGyNBkWA4Ro
Pages Not Found error for SDXL model links.

Not only SDXL but all MLCommons hosted links like GPTJ, DLRMv2 are also broken via wget. MLCommons admins have suggested to use the rclone link which are wrapped in the CM comment [here](https://github.com/mlcommons/inference/issues/1600#issuecomment-1924812135)

`cm pull repo ctuning@mlcommons-ck`

Does every submitter need to use the ctuning account or they can use their own accounts?

@nvyihengz Please use `cm pull repo mlcommons@ck` - the ctuning one was a temporal link until the PR was merged.

`mlcommons@ck` is a short hand for `https://github.com/mlcommons/ck`  

Known affected workloads @mrmhodak @pgmpablo157321 
SDXL https://github.com/mlcommons/inference/tree/master/text_to_image 
GPTJ https://github.com/mlcommons/inference/tree/master/language/gpt-j
DLRMv2 https://github.com/mlcommons/inference/tree/master/recommendation/dlrm_v2/pytorch#downloading-model-weights

@nvyihengz We do have the cloudflare rclone commands for all. (wget links are still broken though)

```
python3 -m pip install cmind
cm pull repo mlcommons@ck
cm run script --tags=get,ml-model,gptj,_pytorch,_rclone -j
cm run script --tags=get,ml-model,dlrm,_pytorch,_weight_sharded,_rclone -j
cm run script --tags=get,ml-model,sdxl,_fp16,_rclone -j
cm run script --tags=get,ml-model,sdxl,_fp32,_rclone -j
```

Running `cm` command will print the underlying rclone config but it is better to use `cm` commands as the configs can change later but cm commands won't.
Repository: mlcommons/inference
Issue: Proposal: run clang-format and autopep8 on the existing code base; potentially add pre-commit hook for code style checking

The current C++ code base follows C and fortran coding style, which is a bit stale given that C++14 is used.  We would like to clang-format all the C++ code base and run autopep8 on all the python code base to adapt to the latest industry standards and modern C++ coding.

This can be done in v4.1. 
@pgmpablo157321 FYI. I can help if needed.

As mentioned by Pablo, pre-commit hook is not necessarily installed by the developer. Automatically apply the format change in the PR automation is more reasonable.
Note: if automatic reformat is enabled, clang-format config needs to include `SortIncludes: false` so the functionality of the original code is not affected. 

@pgmpablo157321 to handle

clang-format has many [style options](https://clang.llvm.org/docs/ClangFormatStyleOptions.html) and even predefined options (LLVM, Google, Chromium, Mozilla, WebKit, Microsoft). Any thoughts on which one we should adopt?

As a recycled compiler person :), I'd vote for LLVM ;).

Closing this issue.
Repository: mlcommons/inference
Issue: Stable Diffusion Negative Prompt required?
Root cause: Documentation Deficiency

Hi there,

May I ask if the stable diffusion inference benchmark requires the `negative_prompt` as an input `"normal quality, low quality, worst quality, low res, blurry, nsfw, nude"` showing here: 

https://github.com/mlcommons/inference/blob/840435a1abe8be823e0a5a18ad3173bc520aa24f/text_to_image/backend_pytorch.py#L23C9-L23C24

Yes it is required by the rules.

thanks @nvyihengz
Repository: mlcommons/inference
Issue: Inference v3.1 results table has a wrong name for GPT-J columns 
Root cause: Validation Gap

Edge doesn't contain "Server", and the metric should be latency, not Queries/s. Could you correct this?

![image](https://github.com/mlcommons/inference/assets/6924448/02c88f64-2073-495a-bcc2-f6d37ba21938)


@pgmpablo157321

Many of the CTuning systems are shown as "Available Cloud" - not sure how
Repository: mlcommons/inference
Issue: Llama2-70b Loadgen Issue in Offline Scenario
Root cause: Validation Gap

Since the fixes for token measuring in the server scenario have been introduced, there is now a new issue in the offline scenario:
```
:::MLLOG {"key": "error_runtime", "value": "n_tokens argument missing or attempted to record 0 as number of tokens", "time_ms": 0.151550, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": true, "is_warning": false, "file": "logging.cc", "line_no": 442, "pid": 394, "tid": 419}}
```

Every time we call
```
                response = [lg.QuerySampleResponse(qitem[i].id, bi[0], bi[1])]
                lg.QuerySamplesComplete(response)
```
in offline, we are missing the new n_tokens field
Repository: mlcommons/inference
Issue: Understanding the Benchmarking Scenarios for DLRmv2
Root cause: Documentation Deficiency

Hi!
I've been delving into the DLRMv2 benchmark, and I want to confirm my understanding of the scenarios. For the Server scenario, my understanding is that it runs this command
```
./run_local.sh pytorch dlrm multihot-criteo cpu --scenario Server --max-ind-range=40000000 --samples-to-aggregate-quantile-file=./tools/dist_quantile.txt --max-batchsize=2048
```

Each query contains one sample, and the sample has B user-item pairs. The number of user-item pairs, B, is variable and determined based on a distribution sampled from dist_quantile.txt, ranging between 100-700. Is it corrrect to say that the reported throughput in Queries Per Second (QPS) reflects the samples processed per second ([like these ones](https://mlcommons.org/benchmarks/inference-datacenter/)), and the throughput in terms of the number of user-item pairs processed per second is greater than QPS (around B * QPS) because each sample contains multiple user-item pairs? 

Also, for the Offline scenario, it seems like the number of user-item pairs per sample is also drawn from the dist_quantile.txt. I was wondering what number is used for the number of samples per query. 

Thanks!
Repository: mlcommons/inference
Issue: Reintroducing min_query_count for SingleStream (64) and Server (662)
Root cause: Validation Gap

The minimum query count (`min_query_count`) was [removed](https://github.com/mlcommons/inference/commit/995ffee37682a0870f359bb00d5f4672d78d6424) from `mlperf.conf` a while ago. I believe the thinking was that submitters could choose how many samples to process. As long as the minimum run duration (`min_duration`) constraint of 10 minutes was met, early stopping would take care of estimating the 90th percentile for SingleStream, and the 99th percentile for MultiStream and Server.

However, for SingleStream early stopping still requires at least 64 samples to estimate the 90th percentile. Similarly, for MultiStream early stopping requires at least 662 samples to estimate the 99th percentile. So trying to process less than 64 samples for SingleStream and 662 queries for Server will result in `INVALID` runs.

Perhaps it would be better to reintroduce these constraints to match [the one for MultiStream](https://github.com/mlcommons/inference/blob/486a629ea4d5c5150f452d0b0a196bf71fd2021e/mlperf.conf#L37):
```
*.MultiStream.min_query_count = 662
```

yes, it'll be good to make loadgen generate minimum that many queries, right?
Repository: mlcommons/inference
Issue: host_processor_vcpu_count in <system_desc_id>.json
Root cause: Validation Gap

According to https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#57-system_desc_idjson-metadata it's required to provide field host_processor_vcpu_count.

I see that relevant checks are missing in https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py so the submitter get 
`submission_checker.py:3240 WARNING] <redacted>, field host_processor_vcpu_count is unknown`.

My proposal is to remove `host_processor_vcpu_count` from the submission rules because this rule is not enforced anyway. In inference_results_v3.1 this field is present only in `closed/Intel-HabanaLabs/systems/HLS-Gaudi2-PT.json`.

If we provide only host_processor_vcpu_count instead of host_processor_core_count then submission checker fails:
```
submission_checker.py:3220 ERROR] <REDACTED>, field host_processor_core_count is missing
submission_checker.py:3240 WARNING] <REDACTED>, field host_processor_vcpu_count is unknown
```

A related issue: https://github.com/mlcommons/inference/issues/1351
Repository: mlcommons/inference
Issue: SDXL: Are latents considered inputs of the model?
Root cause: Documentation Deficiency

I got this question yesterday and I am not 100% of the answer. My guess is no, since they behave more like a random seed than an input. But, I wanted to make sure of that since there might be some implications (E.g Having to send them throughout the Network for the network division).

WG: The latent is constant and it is not a user input. It shouldn't be consider a model input.
Repository: mlcommons/inference
Issue: Slightly different target qps for TEST05 server scenario

Is it valid to run TEST05 for the server scenario with a slightly different target qps?

TEST05 test that the performance falls within a range of 5% from the original performance when varying the random seed. For the server scenario the performance is highly dependent on `target_qps`. If this parameter is slightly reduced for TEST05 and the performance still falls in that range, does that count as a valid run?

yes, we can do that. It is already done in many previous submissions. In our CM automation we reduce it by 3% and for TEST01 by 6% to minimize any reruns
Repository: mlcommons/inference
Issue: How to handle failed requests(5XX) of an online inference service?
Root cause: Documentation Deficiency

When I want to test an online inference service by POST requests, it is necessary to record all response, because of [this](https://github.com/mlcommons/inference/blob/268bc9dc8a3c0a96bbb7d38482c0ce5016507633/loadgen/logging.h#L398)

However, if the response's status code is not 200, recording this response and calculating latency based on it would be inaccurate, because of [this](https://github.com/mlcommons/inference/blob/268bc9dc8a3c0a96bbb7d38482c0ce5016507633/loadgen/results.cc#L406). Moreover, using this approach to find peak performance would not yield the optimal QPS. 

Is there any support or recommended approach for handling this scenario?
Repository: mlcommons/inference
Issue: Processed orca dataset for LLaMA is not published
Root cause: Documentation Deficiency

Hi,

It was promised that the processed dataset will be published in the MLCommons cloud.

However, the instruction in https://github.com/mlcommons/inference/tree/master/language/llama2-70b still explains how to generate the dataset on our own (with processorca.py).

md5sum of the generated file is also not present ( 5fe8be0a7ce5c3c9a028674fd24b00d5 ) in the README which may potentially cause problems during review or audit process (what if a submitter generates slightly different dataset which slightly impacts the results). IMHO at least md5sum should be present in the readme.

CC @pgmpablo157321 @nv-alicheng @ashwin @attafosu 

Hi Michal, I do think the link to sign the consent form (which directs the user to the gdrive link) should be posted: https://docs.google.com/forms/d/e/1FAIpQLSc_8VIvRmXM3I8KQaYnKf7gy27Z63BBoI_I1u02f4lw6rBp3g/viewform


@pgmpablo157321 can you help add that?

Hi @nvzhihanj ,

Thanks. Yes, I agree that the link to the consent form should be also in the readme. However, the issue is about the dataset (24576 samples), not weights.

We would like to make sure that:
* submitters do use the same dataset for their submission
* it's easy to to reproduce the results without generating the dataset by an organization which is not MLCommons Member (weights and tokenizer can be obtained directly from HF after signing the Meta licence)

@szutenberg the dataset is also in the gdrive.
For reproducing the dataset, I think https://github.com/mlcommons/inference/tree/master/language/llama2-70b#get-dataset provides the guidance to generate the pkl file. Are you suggesting that the MD5SUM of the pickle file be added to the readme?

@szutenberg Inside the link of the MLCommons Llama 2 (obtained after filling out the form), there is a copy of the preprocessed dataset. Is it enough to post the link of the form and the instructions to get the preprocessed dataset? Or do we ideally want to share the dataset directly? The confidentiality restriction is for the Llama 2 weights, so this should be possible. I just want to have very clear what do I have to do and to ask for.

@pgmpablo157321 ideally if we share the dataset directly.

IMHO md5sum in README is must have since it's our own, custom dataset. I'm just not sure if 2 years later these repro steps will generate exactly the same output...

Solved by #1638
Repository: mlcommons/inference
Issue: RetinaNet TEST05 is not passing for a Singlestream run
Root cause: Configuration Error

We are trying to reproduce a qaic submission but for retinanet SS run never passes TEST05. The performance numbers are consistent across the runs and so reruns or even longer runs are not helping 

## TEST05 log
```
================================================
MLPerf Results Summary
================================================
SUT name : KILT_SERVER
Scenario : SingleStream
Mode     : PerformanceOnly
90th percentile latency (ns) : 19641101
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes
Early Stopping Result:
 * Processed at least 64 queries (56356).
 * Would discard 5469 highest latency queries.
 * Early stopping 90th percentile estimate: 19666047
 * Early stopping 99th percentile estimate: 21004798

```

## Actual performance run
```
================================================
MLPerf Results Summary
================================================
SUT name : KILT_SERVER
Scenario : SingleStream
Mode     : PerformanceOnly
90th percentile latency (ns) : 18503321
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes
Early Stopping Result:
 * Processed at least 64 queries (35227).
 * Would discard 3390 highest latency queries.
 * Early stopping 90th percentile estimate: 18526238
 * Early stopping 99th percentile estimate: 20238324
 ```
```

@arjunsuresh Why is the number of samples different between the two runs? (10-11 minutes vs 16-17 minutes?)  Maybe the system you are running on (AWS?) has insufficient cooling, so longer runs become slower?

@arjunsuresh Please provide full summary logs including settings like `performance_sample_count`. 

Thank you @psyhtest for replying. The number of samples are different because on a repeated run we are automatically making the run longer. But we tried to match the duration for the TEST05 and performance runs - no difference. We even ran for 20 minutes each - then also no difference. But `performance_sample_count = 128` worked. I think we should revisit TEST05 for retinanet considering the variable performance on different inputs. 

Recommended to use a high performance_sample_count for RetinaNet. Will discuss this issue further for 4.1. Closing for now.
Repository: mlcommons/inference
Issue: Broadcom is having trouble submitting the submission package through UI
Root cause: Validation Gap

Broadcom has completed compliance tests for bert, bert-99, 3d-unet, retinanet, rnnt and stable-diffusion-xl. The compliance tests pass. We are having these two issues.

1. We are having trouble creating sa compliant submission tar ball.

2. As suggested by Miro and Victor, I tried to submit an empty file as a place holder. The UI is not letting us upload. After choosing the almost empty tar file, the upload button does not seem to do anything.

Thanks,

-Uday

We were able to submit the tar ball for two systems. The online submission checker is still reporting errors that we are trying to resolve.

I have re-located this issue to issue 30 in the submission repo. Please move all discussion there.
Repository: mlcommons/inference
Issue: LlaMa2-70b run_accuracy.sh issue with consolidate_results.py
Root cause: Documentation Deficiency

LlaMa2-70b has a run_accuracy.py script that as a last step that calls another script, consolidate_results.py. That last step does not work, it expects to read output pkl files, but those are not being created in previous steps. 

The README marks this step as optional - so we should either remove if not needed, or fix.

@nvzhihanj , please help. 


I believe the consolidate_results.py is not needed if the pickle input file already has all the samples (24576). That script is a by-product of preprocessing that @nv-alicheng uses IIRC.

consolidate_results.py is an optional step that was used to generate a pickle file for manual viewing / data analysis. It is not required to run the accuracy script. The pkl files that it consumes are generated here (https://github.com/mlcommons/inference/blob/master/language/llama2-70b/dataset.py#L90) during the accuracy inference.
Repository: mlcommons/inference
Issue: Question: LLama-70B - Calibration and Quantization
Root cause: Documentation Deficiency

1. In the Orca dataset, there is a Calibration pkl file which is generated after the preprocessing step.
Can this be used as the calibration dataset or should we use the cnn-daily dataset which has been used in other language benchmarks.

2. The current README file(https://github.com/mlcommons/inference/blob/e39003a9c4c89a2215db0ca57ad7a57b16f9a785/language/llama2-70b/README.md) doesn't give any pointers towards calibration or quantization. We are submitting for the first time and would appreciate any help we can get.
Repository: mlcommons/inference
Issue: Reduce the dataset size for LLAMA2
Root cause: Configuration Error

I believe there is no need to run LLAMA2-70B model for 24576 samples. GPT-J-6B is run for 13368 samples, Stable diffusion is run for 5000 samples, 3d-unet is run for 43 samples and all other MLPerf inference models are way faster. In fact, LLAMA2-70B is at least an order of magnitude slower than any other MLPerf inference model. For coming rounds, we should reduce the LLAMA2-70B dataset size to some 4-digit number. 

Another idea would be to create a representative subset for performance runs, i.e. with sample distribution similar to the original dataset, but only 1/10th the size.

@psyhtest Thats a good option. But in LLAMA2 case the dataset is already a subset selection. So, it might be easier to reduce the dataset size itself though this means a new accuracy threshold.
If we use a different dataset for performance runs, TEST01 might be complicated when it gets introduced for LLAMA2.
Repository: mlcommons/inference
Issue: LLama2-70B - Accuracy scores lower than expected using the reference implementation 
Root cause: Documentation Deficiency

We use the Llama reference code.
The Accuracy scores from the evaluation are lower than the ones mentioned in the README.md file.

Any insights are greatly appreciated!



At Fp16, the accuracies are expected to be lower. But by what percentage? 

There is about a 25% difference we are observing @arjunsuresh 

@rgandikota I have only run llama2-7b model and so can't really comment on this. But this is the accuracy we got on bfloat16 and llama2-7b
`(42.0595, 19.853, 26.7729, 1194.4)`

@arjunsuresh Thank you for the quick response! Do the submission rules mandate that we must meet the accuracy scores published in the reference implementation? 

@rgandikota  yes, llama2 has 2 variants- one requires 99% of the reference accuracy to be met and another require 99.9% of the reference accuracy to be met. If either is not met, the option is to submit to the "open" division where no accuracy constraint is there. For example, the accuracy I shared will go to the open division. 

Of the 4 accuracy metrics for llama2, the final one is the tokens per sample and for this the accuracy threshold is 90% and not 99/99.9 as can be seen [here](https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py#L1127)

Also, to do a closed division llama2 submission, both offline and server scenario must be run and both meet the accuracy constraints. But in the open division, we can submit individual scenario results. 

@arjunsuresh Thank you so much for the insights. Really appreciate them. 
Any pointers on what might cause the low scores in your experience? We are trying to debug if it has something to do with our code

@rgandikota You're welcome. Unfortunately we haven't tried to run LLAMA2 using TensorRT-LLM like you are trying to do. So, I can't really help here. @nv-alicheng can you please help?

Hi @rgandikota , you need to become a partner of NVIDIA for TRT-LLM related support for MLPerf. Please reach out to NVIDIA internal customer representative or SAs to get help regarding this.

@nvzhihanj  Thank you, we are looking into the option for the next rounds. This issue was meant to be a more generic question around how we can go about debugging accuracy issues if we use the reference implementation. Didn't mean to make it TRT-LLM specific.

To debug accuracy issue, you would want to make sure the reference implementation works, and compare per-sequence, or even per-step/per-layer output of the model.
Repository: mlcommons/inference
Issue: Loadgen built with uncommitted changes - needs revisit
Root cause: Validation Gap

`Loadgen built with uncommitted changes` error should be thrown only when loadgen-related files are modified. Currently this is thrown when any of the files in the inference repository is modified.

Agree. We've had this triggered even when only file permissions changed (e.g. `g+w`).

All it needed was a `.` and I'm sure it has already wasted hours of experiment runs :(
Repository: mlcommons/inference
Issue: Tips please - personal project + idea for inference 

Hi there, just saw the video from the ai engineer
Inference looks really awesome

I've got this mini project in mind for years now,
Getting a model to tell how full is a glass/bowl/cooking pot/other container in (from a camera stream)

0%-100% from empty to full/overflowing 
Commercially, if this can be achieved from a small model, it could be added to home water units, smart coffee makers, etc,
But for me personally, I want my dog's water bowl to always be full.

Yes this can be done in a bunch of different ways from Arduino ultrasound and others but, a camera+ai approach means that the solution could be generic, and work for all types of glasses and containers, transparent or opaque, small glass or big pot for the dogs. It will only fill if the bowl is positioned mostly center (with accepted margin) taking the bowl/glass away stops the flow. 
Simple metrics, simple rules, operate 1 iot valve,  work generically.

So basically I wanted to ask if anyone could recommend which to use for this application?
OpenCV + clip (+ kalman filter)? Or are there models out there that might be better? Is there a chance to do it from a weak espcam or rpi 0/cm, or will I have to do the inference on a server away from the sensor? I'll appreciate any wisdom you might have 

Thanks a lot and have a good one!

P.s - a bit meta, it would be awesome when there'll be an llm that is always up-to-date with inference, and we could ask it these sort of questions, get best matching models, hone in and choose models that are suited for the hw of choice, and even generate code which pulls from inference to match our mission
Like an inference-assisted issue2repo ai ftw

All the best!