zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: MLPerf 4.0 Inference Results on Nvidia GH200 issue 
Root cause: Configuration Error

Hi,

https://github.com/mlcommons/inference_results_v4.0/tree/main/closed/NVIDIA
https://github.com/mlcommons/inference_results_v4.0/tree/main/closed/NVIDIA/code/gptj-99/tensorrt

I am trying to reproduce the gptj Offline scenario benchmark on a Nvidia gh200 system. I have followed the steps and created a quantized and fine tuned version.

I have setup a Docker Container using make rebuild and downloaded the gptj model, data and preprocessed it.

I did not make any changes as per this script since the system is already a known system:
**python3 -m scripts.custom_systems.add_custom_system**

I created **configs/gptj/Offline/custom.py** with this content:

__from . import *
import os
import sys
sys.path.insert(0, os.getcwd())_

_from code.common.constants import Benchmark, Scenario
from code.common.systems.system_list import KnownSystem
from configs.configuration import *
from configs.gptj import GPUBaseConfig_

_class OfflineGPUBaseConfig(GPUBaseConfig):
    scenario = Scenario.Offline
    gpu_copy_streams = 1
    gpu_inference_streams = 1
    tensor_parallelism = 1
    precision = "fp16"
    enable_sort = False
    num_sort_segments = 2_

_@ConfigRegistry.register(HarnessType.Custom, AccuracyTarget.k_99, PowerSetting.MaxP)
class GH200_96GB_aarch64x1(OfflineGPUBaseConfig):
    system = KnownSystem.gh200
    gpu_batch_size = 480
    use_fp8 = True
    enable_sort = False
    offline_expected_qps = 30_


Then I executed "**BUILD_TRTLLM=1** make build", there were some warnings but it did complete.

**fp8_quant_model_path** in **code/gptj/tensorrt/builder.py** is pointing to the correct quantized model folder

When I run this command, I get a warning:
_make generate_engines RUN_ARGS="--benchmarks=gptj --scenarios=Offline --config_ver=high_accuracy --test_mode=AccuracyOnly --fast"_
[2024-07-12 03:30:12,363 main.py:229 INFO] Detected system ID: KnownSystem.gh200
**[2024-07-12 03:30:12,581 main.py:134 WARNING] No registered config for gptj.Offline.gh200 for WorkloadSetting(custom_k_99_9_MaxP)**

When I run this command, there is no output, no accuracy number anywhere:
_make run_harness RUN_ARGS="--benchmarks=gptj --scenarios=Offline --config_ver=high_accuracy --test_mode=AccuracyOnly --fast"_
**[2024-07-12 03:30:56,822 main.py:229 INFO] Detected system ID: KnownSystem.gh200
[2024-07-12 03:30:57,040 main.py:134 WARNING] No registered config for gptj.Offline.gh200 for WorkloadSetting(custom_k_99_9_MaxP)**

======================== Result summaries: ========================

**No results**

Any ideas what is wrong here?

Thanks
Repository: mlcommons/inference
Issue: Zenodo models
Root cause: Documentation Deficiency

The MLPerf inference models are residing in https://zenodo.org/communities/mlperf and there are duplicates in there. We need to clean-up the repo, along with GitHub.

I reviewed the page, see other issues, but don't see duplicates. 

TODOs for me on the MobileNet quantized model: Apache 2.0 licensing, add an explanation of how the weights were defined, tag with MLPerf inference 0.5. Will close by 7/3.

I assume this is done @DilipSequeira ?

Yes.
Repository: mlcommons/inference
Issue: Key Error while running NVIDIA LLAMA2 70b
Root cause: Validation Gap

The system specified here: https://github.com/mlcommons/inference_results_v4.0/blob/62d45067ad3cab5b9a48520a405850e3101050d3/closed/NVIDIA/configs/llama2-70b/Offline/__init__.py#L41

Does not appear to be in the `system_lists` file:
https://github.com/mlcommons/inference_results_v4.0/blob/main/closed/NVIDIA/code/common/systems/system_list.py
Repository: mlcommons/inference
Issue: SDXL TEST01: Need clarification

Hi,

I have a question about TEST01 compliance check for SDXL
https://github.com/mlcommons/inference/tree/master/compliance/nvidia/TEST01
In case Part II fails, we proceed to Part III.
Then, the difference between baseline and test accuracy should be within x%.

I was wondering what the value x is for SDXL as the inference rules only specify the acceptable range of FID and CLIP scores.
(https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc)

Thanks.

Hi @hans-intel the accuracy % used for SDXL is 98% (2% tolerance) for the FID score and 99.8% (0.2% tolerance) for the CLIP score.

Thank you @arjunsuresh!
Repository: mlcommons/inference
Issue: MoE Server ignores setting TTFT/TPOT latencies from user.conf
Root cause: Configuration Error

For an Open submission, we have tried setting TTFT/TPOT latencies in `user.conf` e.g.:
<pre>
mixtral-8x7b.Server.ttft_latency = 3000
mixtral-8x7b.Server.tpot_latency = 300
</pre>

However, in the resulting `mlperf_log_summary.txt` we would still see:
<pre>
ttft_latency (ns): 2000000000
tpot_latency (ns): 200000000
</pre>

Only manually modifying `mlperf.conf` would get us the right values.

This may not be specific to MoE, but to all LLM benchmarks.

@psyhtest Are we allowed to change the server scenario latency for the open division?

I can't see why not?

I don't recall seeing that in the rules 😇. 

As usual, "anything goes" in Open.

I was wrong - this is allowed [in the rules](https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#412-relaxed-constraints-for-the-open-division). We should get the `user.conf` working. 

@psyhtest You also need to place the use_token_latencies flag
```
mixtral-8x7b.*.use_token_latencies = 1
mixtral-8x7b.Server.ttft_latency = 3000
mixtral-8x7b.Server.tpot_latency = 300
```
Repository: mlcommons/inference
Issue: MLPerf server scenario summary for LLM models are confusing
Root cause: Validation Gap

In the below summary, `Performance constraints` is shown as satisfied. Not sure what exactly this means as TPOT Early Stopping Result has failed. 

```
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Server
Mode     : PerformanceOnly
Completed samples per second    : 0.11
Completed tokens per second: 43.45
Result is : INVALID
  Performance constraints satisfied : Yes
  Min duration satisfied : Yes
  Min queries satisfied : NO
  Early stopping satisfied: NO
Recommendations:
 * The test exited early, before enough queries were issued.
   See the detailed log for why this may have occurred.
TTFT Early Stopping Result:

TPOT Early Stopping Result:
 * Run unsuccessful.
 * Processed 33 queries.
 * Would need to run at least 805 more queries,
 with the run being successful if every additional
 query were under latency.

```

```
Completed tokens per second                 : 43.08
Min First Token latency (ns)                : 0
Max First Token latency (ns)                : 0
Mean First Token latency (ns)               : 0
50.00 percentile first token latency (ns)   : 0
90.00 percentile first token latency (ns)   : 0
95.00 percentile first token latency (ns)   : 0
97.00 percentile first token latency (ns)   : 0
99.00 percentile first token latency (ns)   : 0
99.90 percentile first token latency (ns)   : 0

Min Time to Output Token (ns)                : 0
Max Time to Output Token (ns)                : 0
Mean Time to Output Token (ns)               : 0
50.00 percentile time to output token (ns)   : 0
90.00 percentile time to output token (ns)   : 0
95.00 percentile time to output token (ns)   : 0
97.00 percentile time to output token (ns)   : 0
99.00 percentile time to output token (ns)   : 0
99.90 percentile time to output token (ns)   : 0

```

@psyhtest Are you seeing this issue?

It's due to an issue in the implementation.
Repository: mlcommons/inference
Issue: dlrm or dlrm-v2 is the official model name?

In the [mlperf.conf](https://github.com/mlcommons/inference/blob/master/mlperf.conf#L12) we have both `dlrm` and `dlrm-v2` and this is confusing to the submitters as to which one to use. Even though `dlrm-v2` is the expected one, we have submissions where `dlrm` is used. Currently, both `dlrm` and `dlrm-v2` are having non-conflicting values, but it is good to clarify this. 

Before mlperf 3.1 only dlrm model was supported. After release of dlrmv2 model, from mlperf 3.1 onwards both dlrm is replace by dlrm-v2 model.
Repository: mlcommons/inference
Issue: Incorrect values for accuracy delta percentage
Root cause: Algorithmic Error

When checking for the allowed delta percentage between test01 and baseline accuracy, the submission checker sets it to 1 or 0.1 based on the name of the benchmark

https://github.com/mlcommons/inference/blob/28144996310f5303a09355d55f238786737dc346/tools/submission/submission_checker.py#L2474-L2478

I think this behavior is incorrect for `stable-diffusion-xl`, where the `CLIP_SCORE` metric has a tolerance of `0.2%` and the `FID_SCORE` of `2%` based on the allowed ranges.

The allowed delta percentage for those cases should be `0.2%` and `2%` respectively 
@arjunsuresh Can you confirm this is the correct behavior?

@pgmpablo157321 yes, that's correct. We need to use two different deltas for `CLIP` and `FID` scores.

WG comments:
Only change FID in TEST01 - that increases the range 
All existing results passing current checker are valid
Repository: mlcommons/inference
Issue: Conda intel channel disappeared breaking Intel MLPerf inference reproducibility
Root cause: External Dependency Breakage

```
Collecting package metadata (current_repodata.json): failed

UnavailableInvalidChannel: HTTP 403 FORBIDDEN for channel intel <https://conda.anaconda.org/intel>

The channel is not accessible or is invalid.
```

More details [here](https://community.intel.com/t5/Intel-Integrated-Performance/Problems-installing-with-conda-HTTP-403-FORBIDDEN/td-p/1611876)
Repository: mlcommons/inference
Issue: Why not support multi-gpu?
Root cause: Unimplemented Feature Gap

I checked the reference implementation and found that the code didn't support multi-gpu.
So why not support multi-gpu?
![image](https://github.com/user-attachments/assets/9ef24165-3c0c-405d-ac67-69aa25a58b27)


I think you are looking at the reference model where only the bare minimum was implemented to demonstrate the functionality. Optimized implementation is submitter's job.
Repository: mlcommons/inference
Issue: Make MLPerf inference results easier to reproduce and reduce the reproducibility burden of submitters 
Root cause: Unimplemented Feature Gap

This proposal is to make MLPerf inference results easier to reproduce as well as to reduce the burden on the submitter to support reproducibility.

Currently MLPerf inference submitters follow custom READMEs/automations making it harder for external users to reproduce them and each submitter has to take care to maintain the reproducibility of their submissions. The below proposal is aimed at solving these problems. 

During the review period ensure that the submission works via CM with expected performance and documentation is in the [Inference docs site](https://docs.mlcommons.org/inference/benchmarks/).
After this is done, CM team can ensure reproducibility and answer queries from the public for the following one year (reproducibility support ends after one year).
CM for MLPerf inference can easily support multiple code versions for reproducibility. We are supporting the latest Inference code release (from Nvidia, Intel, Qualcomm, Neural Magic and MLCommons) and the one just preceding it.
We currently have all the Nvidia and Intel MLPerf inference benchmarks from v4.0 and all v3.1 Qualcomm benchmarks in CM - so supporting v4.1 code during the review period is not difficult.

The timeframe needed to support a submitter code depends on the complexity in the submission code (calibration, compilation, run configuration etc.) For submitters using the supported code bases of Nvidia and Intel, we expect to support the submissions in maximum 1-2 days once they share the code with us. For other submitters it can take up to a week.
Repository: mlcommons/inference
Issue: [Urgent] Submission checker error while processing SDXL TEST01 result
Root cause: Validation Gap

Hi All,

I just ran the latest submission checker and it fails on the part that recently changed for SDXL TEST01 accuracy delta.

```python
Traceback (most recent call last):
  File "submission_checker.py", line 786, in get_delta_perc
    if metric in self.accuracy_delta_perc[model]:
TypeError: unhashable type: 'list'
```
`metric` is a list: `[('CLIP_SCORE', 31.68631873), ('FID_SCORE', 23.01085758)]`
This is caused by a typo in line 2615

```python
required_delta_perc = config.get_delta_perc(model, acc_types)
```
I think it should be `acc_type` instead of `acc_types`.

This is a blocking issue for submission so please respond as soon as possible.


@pgmpablo157321 @mrmhodak @mrasquinha-g 

@hans-intel Seems like a typo, can you quickly check with PR #1804

@pgmpablo157321 I checked, it works for me. Please merge it to the main branch.
Repository: mlcommons/inference
Issue: Some questions regarding the maintainability of MLPerf inference benchmarks

The below questions are not documented anywhere AFAIK. It'll be good to clarify these.

| Benchmark  | Responsible Maintainer  | Run support duration  | Code improvements welcome? |
|---|---|---| --- |
| Submitted Results  | Submitter or Inference WG? |  ? | No? |
| Reference Implementations  |  Inference WG ? |  Until a benchmark is retired or some months after the retirement?  | Yes? |
Repository: mlcommons/inference
Issue: Submission checker complaints on min_query_count
Root cause: Validation Gap

The submission checker strictly checks for config parameters such as the 24.5k min_query_count for offline even when that's not relevant to the benchmark, e.g gptj. 
In principle this min_query_count check is irrelevant to benchmarks that have equal-issue mode requirement, but the submission checker would still throw an error in this regard, if a submitter used a user.conf that modified the min_query_count (which has no impact on the benchmark run)
@pgmpablo157321 @ashwin @mrmhodak 

https://github.com/mlcommons/inference/pull/1810
Repository: mlcommons/inference
Issue: vision model accuracy updates in readme

I see higher accuracy than reported in readme.
export DATA_DIR=/../CK-TOOLS/dataset-imagenet-ilsvrc2012-val/

$ ./run_local.sh tf resnet50 gpu --accuracy
Accuracy qps=129.15, mean=0.006666, time=387.14, acc=**73.87,** queries=50000, tiles=50.0:0.0060,80.0:0.0065,90.0:0.0089,95.0:0.0118,99.0:0.0120,99.9:0.0122

yes, I see exactly the same number. Will update it.

Is the accuracy code final, with LoadGen etc.?

yes, just running mobilenet as well.

[updated] I got the following for mobilenet

[TestScenario.SingleStream qps=9315.74, mean=0.002564, time=5.37, acc=**68.29,** queries=50000, tiles=50.0:0.0017,80.0:0.0041,90.0:0.0054,95.0:0.0055,99.0:0.0056,99.9:0.0057

same number here.

SSD-mobilenet: 
TestScenario.SingleStream qps=186.31, mean=0.017249, time=26.84, acc=93.33, mAP=**0.23**, queries=5000, tiles=50.0:0.0172,80.0:0.0177,90.0:0.0180,95.0:0.0182,99.0:0.0189,99.9:0.0202

Please verify and update readme if needed @guschmue . Thanks!

reminder to update the readme of mobilenet and ssd-mobilenet @guschmue Thanks!

fixed.
Repository: mlcommons/inference
Issue: Submission checker results for SingleStream using 97-percentile results instead of 90-percentile results
Root cause: Algorithmic Error

@pgmpablo157321 
One of our submission results for singlestream was having wrong result showing in the final table. It should be showing 90-perc latency, but actually showing 97-perc latency.
![image](https://github.com/user-attachments/assets/bcdc1d35-7cfb-46d1-a274-86b65fbcb95c)
![image](https://github.com/user-attachments/assets/350add63-3787-4067-b686-32f2c487cb4d)



@nvzhihanj Its an unexpected error. Is it happening for all the models? Also we have early stopping result right?
Repository: mlcommons/inference
Issue: Llama2-70b TEST06 is broken for Reference implementation
Root cause: Validation Gap

TEST06 for llama2 is not working with the reference implementation as the first token data is saved as float32 here: https://github.com/mlcommons/inference/blob/master/language/llama2-70b/SUT.py#L321 while the rest of the data is in int32. This issue should be affecting anyone relying on the reference code for the submission like happened [here in v4.0](https://github.com/mlcommons/inference_results_v4.0/blob/main/open/RedHat-Supermicro/results/AS-4125GS-TNRT-RedHat-OpenShift-AI/llama2-70b-99/Server/accuracy/mlperf_log_accuracy.json#L2)

A quick fix can be to modify the TEST06 verification to support different dtypes just for the token data (submitters can do this). Proper fix is to fix this in the reference implementation - as it should not be misleading the submitters. 

https://github.com/mlcommons/inference/pull/1814

Fixed now
Repository: mlcommons/inference
Issue: quesiton about Llama2-70b

Hello,
when downloading the processed dataset for llama2-70b with rclone: as specified on the file "language/llama2-70b/README.md" on the "get dataset section" I noticed the file "mlperf_log_accuracy.json" within the folder. Is it the accuracy file, linked to the execution of the  benchmark on DGX-H100 with FP32 accuracy ?
If not, is this file publicly accessible?
Thank you,
Regards,

Hi @amine-ammor , the accuracy refers to the accuracy run. 

From https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc:
>For each benchmark, MLPerf will provide pointers to:
>An accuracy data set, to be used to determine whether a submission meets the quality target, and used as a validation set
>A speed/performance data set that is a subset of the accuracy data set to be used to measure performance

If you would like to change the precision of your model, you can change it in the `custom.py` config file if you are running the NVIDIA implementation at https://github.com/mlcommons/inference_results_v4.0/tree/main/closed/NVIDIA

If this resolves your question, you can close the issue.
Repository: mlcommons/inference
Issue: disabling logging in loadgen

how does one go about disabling logging in the loadgen. Is there a compiler flag for that?

thanks 

Hi. I've added an enable_trace setting that still needs to be hooked up. This will be a runtime flag for easy experimentation to determine whether we will require traces for submission.

In the short term, I do not plan to have a simple flag to disable other types of logging. However, there are plans to make the results formatting client programmable where the client could provide a null formatter. Would that be acceptable?
Repository: mlcommons/inference
Issue: how to set llama offline query size
Root cause: Documentation Deficiency

How to set `user.conf` to have specific query size(for experiments) for offline scenario in both performance and accuracy mode?
I've been trying below config but still got 24576.
```
*.Offline.samples_per_query = 10
*.Offline.max_query_count = 10
*.Offline.performance_sample_count_override = 10
```
Repository: mlcommons/inference
Issue: Report Total Number of Accelerators for multi-host submissions
Root cause: Unimplemented Feature Gap

During the submission process, the summary CSV that is generated from the https://github.com/mlcommons/inference/blob/master/tools/submission/generate_final_report.py script reports `Nodes` and `a#`, where `Nodes` comes from `number_of_nodes` field (https://github.com/mlcommons/inference/blob/master/tools/submission/generate_final_report.py#L35) and `a#` comes from the `accelerators_per_node` field (https://github.com/mlcommons/inference/blob/master/tools/submission/generate_final_report.py#L38). 

The intent of the summary script is to show the total number of accelerators. The current logic is fine with single-node submissions, but can be confusing from multi-node submissions. For example, a submission that uses 4 VMs/Nodes, each with 4 accelerator chips, will report `Nodes` as 4 and `a#` as 4, which is not wrong, but confusing. There should be somewhere reference of 16.

Suggestion: We can have another column with `total_a` to , or rename `a#` to report 16 accelerators total, where the script calculates 16 by multiplying `number_of_nodes` * `accelerators_per_node`.
Repository: mlcommons/inference
Issue: Merge conflicts when attempting to run CPU-only LLAMA2 inference
Root cause: Documentation Deficiency

Hi,

The [documentation](https://github.com/mlcommons/inference/blob/master/language/llama2-70b/README.md#prepare-environment) states the following about @pgmpablo157321 's changes

```bash
# Need to fetch Pablo's changes
git fetch origin pull/1523/head:llm-server
git merge llm-server
```

But doing so on the latest branch results in merge failure due to conflicts. Could you please let me know what these changes are for? It just seems bad practice to not include context in the documentation such as mentioning the specific commit this patch applies to

The reference READMEs are mostly meant for working only at the time of release of a benchmark. After that they are usually not maintained as we are doing it via CM automation. 

The CM automation method requires `sudo` access which may not be available on all clusters. So I had to use these instructions to create my own Singularity image. I think it would help for a reproducible benchmark suite like MLPerf to include flexible options. 

@rajesh-s `sudo` is not mandatory for CM scripts but as of now it is used in many scripts so that there are no workflow failures. If you can share the no sudo error we can help you run even without sudo access (as long as the necessary packages are already installed). 

Meanwhile we'll also give a ready to use `no-sudo` option soon. 

Thank you. The `no-sudo` option will be convenient in the future.

The first no sudo error I encountered in the current CM-Llama2-70B-Inference is the check for git-lfs

`sudo dnf install -y git-lfs`

Is there a place in the CM flow where I could override this and other similar sudo commands? I already have git-lfs installed. 

For later, I think it would be best to put these pre-requisites in the [Benchmark install](https://docs.mlcommons.org/inference/install/) step

Sure. I suggest to just comment out this line
https://github.com/mlcommons/cm4mlops/blob/main/script/get-generic-sys-util/run.sh#L5

The local path to this will be $HOME/CM/repos/mlcommons@ck/script/get-generic-sys-util.

Yes, we'll add the additional requirements per benchmark. Thank you for the feedback. 

Thank you!

That worked but the local path was slightly different than above `~/CM/repos/mlcommons@cm4mlops/script/get-git-repo/run.sh` in case someone else finds this hack useful. 

@arjunsuresh I encountered a new situation where sudo is needed, when I moved over to the GPU-based runs

`CM error: Please envoke cm run script "get cudnn" --tar_file={full path to the cuDNN tar file}!`

Would it be possible to extract CUDNN to a local path instead?

Hi @rajesh-s for running on Nvidia GPUs, we recommend using `--docker` option which will use a docker with all requirements inside it.

Alternatively cudnn needs to be installed in the cuda directory. So, both can be installed in the user space but it won't be straightforward.