zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Pytorch BERT failed to load state_dict
Root cause: Interface Contract Mismatch

Issue Description: 
I manually download the model.pytorch and vocab.txt to the designated folder and run `/home/user/cm/bin/python3 run.py --backend=pytorch --scenario=Offline   --max_examples 10 --mlperf_conf '/home/user/CM/repos/local/cache/b737554800c84148/inference/mlperf.conf' --user_conf '/home/user/CM/repos/mlcommons@ck/cm-mlops/script/generate-mlperf-inference-user-conf/tmp/c8eb2a31fd70402a93daee688bf391fa.conf' --accuracy 2>&1 | tee /home/user/CM/repos/local/cache/454869b45fbf4f67/test_results/default-reference-gpu-pytorch-v2.2.1-default_config/bert-99/offline/accuracy/console.out` command triggering the BERT benchmarking. 

**Error Message:**
```
Loading BERT configs...
Loading PyTorch model...
Traceback (most recent call last):
  File "/home/user/CM/repos/local/cache/b737554800c84148/inference/language/bert/run.py", line 150, in <module>
    main()
  File "/home/user/CM/repos/local/cache/b737554800c84148/inference/language/bert/run.py", line 75, in main
    sut = get_pytorch_sut(args)
  File "/home/user/CM/repos/local/cache/b737554800c84148/inference/language/bert/pytorch_SUT.py", line 111, in get_pytorch_sut
    return BERT_PyTorch_SUT(args)
  File "/home/user/CM/repos/local/cache/b737554800c84148/inference/language/bert/pytorch_SUT.py", line 60, in __init__
    self.model.load_state_dict(torch.load(model_file))
 

>  File "/home/user/cm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2153, in load_state_dict
>     raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
> RuntimeError: Error(s) in loading state_dict for BertForQuestionAnswering:
>         Unexpected key(s) in state_dict: "bert.pooler.dense.weight", "bert.pooler.dense.bias"

```
Repository: mlcommons/inference
Issue: Performance requirement for GPT-J
Root cause: Documentation Deficiency

Dear mlcommons,
I have a question regarding the performance requirement for GPT-J. According to [inference_policies](https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc), the performance requirement is specified as "99% of FP32 and 99.9% of FP32 (rouge1=42.9865, rouge2=20.1235, rougeL=29.9881)". If the submission were to satisfy 99.9% of FP32, should the submission achieve equal to greater than 99% in all the three metrics? For example, is the submission with the metric "rouge1=42.95, rouge2=20.11, rougeL=29.93" considered invalid as it failed to meet the 99.9% performance requirement for rougeL despite scoring above 99.9% in rouge1 and rouge2? 

Thanks in advance,

Sincerely,

Sung Hyuck Hong 

That's true except for the final parameter `GEN_LEN` for which 90% is fine. You can see the exact requirements [here](https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py#L1126)

Thank you for your prompt response!
Repository: mlcommons/inference
Issue: Units in the summary file for llama offline results are wrong
Root cause: Configuration Error

For llama benchmarks, the submission checker uses tokens per second for Offline, but samples per second for Server. https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py#L1385

However, the summary.csv still [use](https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py#L2543-L2544) samples/second as the header to report Offline results for all models. This can cause confusion and mislead readers since the units here are wrong. Orders of magnitude different between offline and server.

Should this be changed to tokens/sec just for llama? Thanks

For example, if mlperf_log_summary.txt reports 1 sample/sec and 100 tokens/sec for Server, and reports 1.1 sample/sec and 110 tokens/sec for Offline, the submission checker will produce summary.csv with:

- Server (queries/s): 1
- Offline (samples/s): 110

-Benchmark comprehensibility is crucial to MLCommons' success. We have to explain and justify the results and metrics to a broader audience than just ML experts. We will engage with the press, who will write up the results for their readership. Also, MLPerf results are often used for marketing purposes. Both of these channels engage a wider audience of interested parties seeking to understand the results.

-Token/s is widely used and understood in the industry for LLM performance characterization beyond MLPerf. Reporting results in that metric makes the outcome more readily understandable to more people.

-Reporting one set of results in tokens/s and another related set in samples/s will likely lead to confusion on the part of many observers. 

-The initial rationale for using different metrics for Offline and Server has been eroded with the move to reporting completed rather than scheduled work. Per Zhihan Jiang: 

"In v4.0, we used different metrics for Offline and Server because Server used  "scheduled samples per second” initially which could not be directly converted to “token per second” (you don’t know the token/sample ahead of time in loadgen). 

But later we applied “result samples per second” for Llama2 (and this round for all the benchmarks), I second that we should use “results token per second” for server to align with Offline."

-There has been some discussion of sticking with the current plan for 4.0 and then changing the metrics later in 4.1. However, the Llama 2 benchmark is brand new. Any change made later will compromise comparability with this round's results. We would be better off making a change now and avoiding further complications down the road.
Repository: mlcommons/inference
Issue: Compliance checker logs not checked by the submission UI
Root cause: Validation Gap

[compliance_checker_log.txt](https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#563-inference) inside the results directory is mentioned as a requirement by the submission rules but is not enforced by the submission UI. 

Remove as a requirement, need to submit a PR to Policies. Pablo to handle the PR
Repository: mlcommons/inference
Issue: MLPerf-Inference Server: change metrics from scheduled to completed samples per second for all benchmarks
Root cause: Unimplemented Feature Gap

As presented in https://docs.google.com/presentation/d/1Y_AKEJ6h1g5k3ntrL7nTazWw3xVDzJ_tjOGkLQ6VDMI/edit?usp=sharing the completed sample per second is a better representation of the throughput than scheduled QPS. 
@pgmpablo157321 to help implement after the conclusion of v4.0

Required code changes for this:
- Change results.cc to report completed samples per second
- Change submission checker to extract completed samples per second

Not necessary to change metrics in the compliance tests, they already use completed samples per second: [test01](https://github.com/mlcommons/inference/blob/87ba8cb8a6a4f6525f26255fa513d902b17ab060/compliance/nvidia/TEST01/verify_performance.py#L64-L69), [test04](https://github.com/mlcommons/inference/blob/87ba8cb8a6a4f6525f26255fa513d902b17ab060/compliance/nvidia/TEST04/verify_performance.py#L64-L70), [test05](https://github.com/mlcommons/inference/blob/87ba8cb8a6a4f6525f26255fa513d902b17ab060/compliance/nvidia/TEST05/verify_performance.py#L63-L69)
Repository: mlcommons/inference
Issue: Llama2 LoadGen server mode: TPS not reported properly
Root cause: Algorithmic Error

In v4.0 submission, we found in the **server** log that "result_token_throughput" is not reported properly, and most of them are at the e-09 scale (@pgmpablo157321 feel free to to check [this link](https://github.com/mlcommons/submissions_inference_v4.0/blob/957fccae6fa56ca80175899768e144c86b9ee7ee/closed/Intel-HabanaLabs/results/HLS-Gaudi2-PT/llama2-70b-99/Server/performance/run_1/mlperf_log_detail.txt#L109) and [this link](https://github.com/mlcommons/submissions_inference_v4.0/blob/957fccae6fa56ca80175899768e144c86b9ee7ee/closed/NVIDIA/results/DGX-H100_H100-SXM-80GBx2_TRT/llama2-70b-99.9/Server/performance/run_1/mlperf_log_detail.txt#L109). There is another metric "result_token_throughput_with_loadgen_overhead", but I am not sure if it shows the right metric.

Since we are shifting from QPS to TPS, we need to fix and report the "result_completed_token_per_second". 

Seems like the result_token_per_second is in the summary.txt. Not sure why it's not in the details.txt

Closing since this is fixed.
Repository: mlcommons/inference
Issue: PM 4.0: Better Error Handling in Submission UI
Root cause: Validation Gap

(Arjun) Modify the submission checker so that it reports any error and fails instead of relying on “grep ERROR” to find any errors in the submission - this way submitters will be in sync with the submission checker run behind the submission UI
https://github.com/mlcommons/inference/pull/1663 


This is done now. @pgmpablo157321 is there a way we can verify if we can remove the grep ERROR part?
Repository: mlcommons/inference
Issue: v4.0 release
GitHub labels: inference v4.0

postmortem 4.0

Hi,
Is it possible to make a v4.0 snapshot (release) of this repository [here](https://github.com/mlcommons/inference/releases), please?
We need to add the stable v4.0 inference version to our CM automation workflows for MLPerf.
Thanks!


That's a great idea, @gfursin! We also need it for our automated reproducible [X](https://github.com/krai/axs2mlperf) workflows.

Also we need tag v4.0.

@pgmpablo157321 @mrmhodak - any update here?

@pgmpablo157321 can you please release the 4.0 version based on [this commit](https://github.com/mlcommons/inference/commit/e39003a9c4c89a2215db0ca57ad7a57b16f9a785)? 

Ping.

@pgmpablo157321 to fix
Repository: mlcommons/inference
Issue: repo not found while executing main.py
Root cause: Environment Incompatibility

Hi all,

while i am trying to run the benchmark for text to image, its facing an error of repo not found.
![Screenshot from 2024-04-03 18-58-48](https://github.com/mlcommons/inference/assets/52161957/051e18ac-1f4e-43bd-82f7-79a7e6befb0c)


please let me know for any suggestions. Thanks in advance.
Repository: mlcommons/inference
Issue: v4.0 Inference auditing procedure postmortem

Specifically for the nominated auditing selection procedure. To prevent misleading information during the nominated auditing selection procedure, it's essential to prioritize transparency and accuracy in the data provided. An issue arose where a significant performance difference of over 20% for L40S was mentioned without specifying the comparison context. This lack of specificity resulted in a mistake during the auditing process, where the comparison inadvertently paired top results with outlier low results. Consequently, this led to voters making decisions based on incorrect information.

To address this, we propose using a table format similar to the one below to provide relevant data along with the nomination:

<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">


<body link="#0563C1" vlink="#954F72">


  |   |   |   | resnet |   |  
-- | -- | -- | -- | -- | -- | --
  |   |   |   | Server | Offline |  
ID | Submitter | System | Availability | Queries/s | Samples/s | Reason for nomination
4.0-0028 | Dell | Dell PowerEdge R760xa (4x L40S, TensorRT) | available | 179,615.00 | 175,746.00 | Pre-GPU performance is XYZ faster than 4.0-0029
4.0-0029 | Dell | Dell PowerEdge R7615 (2x L40S, TensorRT) | available | 90,571.10 | 88,893.10 |  
… |   |   |   |   |   |  



</body>

</html>



This will help avoid mistakes, which can result in misleading voters. It will also contribute to making the correct auditing selection, ensuring that resources are allocated appropriately. 

Thanks,
Frank
 

@pgmpablo157321 @mrmhodak @bitfort 

Also, Could you help to label it as v4.0 postmortem. Thanks.

@hanyunfan : Can you present at WG Meeting 5/21
Repository: mlcommons/inference
Issue: Memory leak while benchmarking Bert Pytorch_SUT 
Root cause: Resource Mishandling

System: AWS CPU gravitton3.
OS: Linux RHEL
Backend: Pytorch 1.13, Python 3.7
MLC model: Bert-99
Scenario: Offline.

I encountered an issue when benchmarking Bert's Pytorch_SUT using MLPerf reference implementation. 
A memory leak is observed when increasing the min_query_count progressively. 
The parameter *.Offline. min_duration is set to 10 to control the batch size (min_query_count).

Each query sample keeps increasing the memory usage without releasing activations is my guess.

Throughput is also very low compared to tensorflow backend. 
P.S : I didn't use CM automation tool.
Repository: mlcommons/inference
Issue: 【GPT-J】checkpoint.zip download Incompletely
Root cause: Documentation Deficiency

i download checkpoint.zip :
wget https://cloud.mlcommons.org/index.php/s/QAZ2oM94MkFtbQx/download --output-document checkpoint.zip
but with error pytorch checkpoint file cannot load
when i unzip checkpoint.zip,i found it lacks pytorch_model-00002-of-00003.bin this file.
So i download pytorch_model.bin form Huggingface;
according pytorch_model.bin.index.json,i split pytorch_model.bin to get this file: pytorch_model-00002-of-00003.bin
but i run:
python3 main.py --scenario=Offline --model-path=./model/ --dataset-path=./data/cnn_eval.json --max_examples=16
it raise ValueError: Trying to set a tensor of shape torch.Size([50400, 4096]) in "weight" (which has shape torch.Size([50401, 4096])), this look incorrect.
can @mlcommons-bot help me?

It sounds like you're using old download instructions.
Try the new download instructions [here](https://github.com/mlcommons/inference/tree/master/language/gpt-j#download-gpt-j-model).
Repository: mlcommons/inference
Issue: Submission checker forces to have “gptj-99” and “gpt-99.9” in the code dir
Root cause: Architectural Constraint

It ends up that we have even three identical directories (see NVIDIA submission) which does not help with the review and understanding the repo by someone not involved in the WG.

Solution:
there should be one directory per model

if needed, the submitter can create 99 & 99.9 directories

Now:
- gptj-99.9
- gptj-99
- gptj

After:
- gptj

After - optionally, directories per accuracy category:
- gptj/99.9
- gptj/99

I think we can discuss further on this. The problem with the proposal is that we no longer have the same model names inside the results/measurements directory (like [this](https://github.com/mlcommons/inference_results_v4.0/tree/main/closed/NVIDIA/measurements/DGX-H100_H100-SXM-80GBx1_TRT) ) and the ones inside the code directory. Also to avoid duplicates in the code directory we can have softlinks as done [here](https://github.com/mlcommons/inference_results_v4.0/tree/main/closed/CTuning/code) though this can be tricky on some filesystems.
Repository: mlcommons/inference
Issue: model_max_length for GPT-J tokenizer
Root cause: Configuration Error

Dear mlcommons,
I found in issue #1384 that "399 examples are longer than 1919 tokens", where 1919 is set as the model_max_length for GPT-J tokenizer. However, I don't see any change in the code for backend.py. Is it allowed to change [this parameter ](https://github.com/mlcommons/inference/blob/master/language/gpt-j/backend.py#L60) as 2048, which is the maximum length that GPT-J is allowed to use? (self.model.config.n_positions == 2048 is True)

Thanks in advance,

Sincerely,

Sung Hyuck Hong

assigned to @attafosu 

@sunghyuckhong I think that tokenizer is redundant and not used in the code, and that parameter has no effect. But you're right. I think we should change it to 2048 for completeness

Thank you for the answer. While it seems that the parameter I mentioned above won't have any effect, it seems that the truncation with max_length = 1919 is performed in [here](https://github.com/furiosa-ai/inference-compression/blob/master/language/gpt-j/dataset.py#L75). Given that the max_new_length is 128, it looks like that the truncation is used to ensure that the total length (input_prompt + generated tokens) does not exceed n_positions. Is my speculation correct? Thank you in advance :) 

> Thank you for the answer. While it seems that the parameter I mentioned above won't have any effect, it seems that the truncation with max_length = 1919 is performed in [here](https://github.com/furiosa-ai/inference-compression/blob/master/language/gpt-j/dataset.py#L75). Given that the max_new_length is 128, it looks like that the truncation is used to ensure that the total length (input_prompt + generated tokens) does not exceed n_positions. Is my speculation correct? Thank you in advance :)

Yes, that's correct.
Repository: mlcommons/inference
Issue: README link cannot open

In code  "https://github.com/mlcommons/inference/tree/master/loadgen/docs/src/README.md"

these two links 404

![image](https://github.com/mlcommons/inference/assets/61659904/64bfaa29-55c7-481b-a12c-b3d3a57a2dcf)

assigned to @pgmpablo157321
Repository: mlcommons/inference
Issue: Could not identify license file for opentelemetry-cpp
Root cause: External Dependency Breakage

![image](https://github.com/mlcommons/inference/assets/71482562/402386d5-fc98-4e37-bca2-ff8be1517769)

The above error occurs during installation after build is complete (100%) when executing the command "cm docker script --tags=build,nvidia,inference,server" from https://github.com/mlcommons/ck/blob/master/docs/mlperf/inference/resnet50/README_nvidia.md.

@anandhu-eng This issue with pytorch build is reproducible. Let me check that. 

Sure @arjunsuresh. I just tried to clone the opentelemetry-cpp repo using `git clone https://github.com/open-telemetry/opentelemetry-cpp.git` and found out that vcpkg folder (`opentelemetry-cpp\tools\vcpkg`) is empty. 

![image](https://github.com/mlcommons/inference/assets/71482562/e75d7e6f-3cfb-42ab-90fe-da8ff4ccc109)


@anandhu-eng Can you try this now?

```
cm rm repo mlcommons@ck
cm pull repo gateoverflow@cm4mlops
cm docker script --tags=build,nvidia,inference,server --docker_cache=no --docker_cm_repo=gateoverflow@cm4mlops
```

Thank you @arjunsuresh , the docker build was successful but there is some error with Nvidia runtime.
![Screenshot 2024-05-07 153338](https://github.com/mlcommons/inference/assets/71482562/0b685d73-9b04-4fa0-85f4-7fc0eb0ff2a3)

I have put it as a new issue: https://github.com/mlcommons/inference/issues/1695

WG agrees to close
Repository: mlcommons/inference
Issue: Skip scenarios in submission checker
Root cause: Unimplemented Feature Gap

It is sometimes useful to skip a scenario when validating the correctness of the submission package when using [submission_checker.py](https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py). For example, if we only have Offline results, but not any Server results, we would like the submission checker to still run through all of its validation checks, but not throw an error because either 1) the Server results are not populated in the directory, or 2) the Server results are there, but are INVALID for any reason.

A feature request is to add logic to skip scenario(s).

Closing
Repository: mlcommons/inference
Issue: Confused about quality target
Root cause: Documentation Deficiency

Hey guys,

I have some questions about the quality target. Based on my understanding, taking GPT-J text summarization as an example, the result of fp32 is `(rouge1=42.9865, rouge2=20.1235, rougeL=29.9881)`, while other precisions including fp16 and int8 must achieve 99% of fp32? Why is it 99%? Is there any reference basis for this standard? 

Looking forward to your response.

This `99%` is used as the same value is used for the older benchmarks. Also, there is a higher 99.9% variant and both targets were achieved with lower precision models.

Do you see any problem with this target requirement for GPT-J?

I just cannot figure out why 99% and 99.9% of FP32 are the quality target. 😂

AFAIK they are just a chosen metric and supposed to be representative of a "good accuracy". But in case of some models like `stable diffusion for text to image generation` this accuracy metric [is different](https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py#L1129)

WG agrees to close
Repository: mlcommons/inference
Issue: Invalid runtime name: nvidia
Root cause: Environment Incompatibility

The below error occurs during running the docker container when executing the command "cm docker script --tags=build,nvidia,inference,server" from https://github.com/mlcommons/ck/blob/master/docs/mlperf/inference/resnet50/README_nvidia.md.

Platform where docker is run: Windows

![Screenshot 2024-05-07 153338](https://github.com/mlcommons/inference/assets/71482562/1b531115-8140-4cdc-9dd3-d82984c7abe4)


@anandhu-eng Can you please do `cm pull repo` and retry the command?

WG agrees to close
Repository: mlcommons/inference
Issue: llama3 support?

Hi,

When can we expect llama3 support in these benchmarks?

Thanks!

llama3 is not an official MLPerf inference model and I don't think there is a plan to add it either. But it is compatible with llama2-70b official model and you can run it using the below CM command. (We'll be adding all these commands in a easy to use documentation by end of this month.)
```
cm run script  --tags=generate-run-cmds,_performance-only  --model=llama2-70b-99   --device=cuda  \
--quiet --precision=bfloat16 --execution-mode=valid  \
--adr.llama2-model.tags=_stub.meta-llama/Meta-Llama-3-8B \
--adr.inference-src.tags=_repo.https://github.com/GATEOverflow/inference
```

WG agrees to close this
Repository: mlcommons/inference
Issue: Package 'networkx' requires a different python: 3.8.10 not in '>=3.9'
Root cause: Environment Incompatibility

![Screenshot 2024-05-11 233703](https://github.com/mlcommons/inference/assets/71482562/ddabf583-9e6f-434e-8fad-e7aeedc8b144)

Issue arised upon executing script: 
cm run script --tags=generate-run-cmds,inference --model=bert-99 --backend=pytorch  --mode=performance --device=cpu --quiet --test_query_count=1000 --network=sut

Executed in Ubuntu 20.04 through docker in windows.

Problem is with networkx requiring python3.9+. This has been fixed in CM now. Can you try `cm pull repo` and retry the command?

Thanks @arjunsuresh the server script is working now.

WG agrees this is resolved