zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: Loadgen version number in v2.1 branch is listed as 2.0 
Root cause: Configuration Error

https://github.com/mlcommons/inference/blob/042f210a50f3f7bda97f6a2746d3c38497ef480b/loadgen/version_generator.py#L96

Same as https://github.com/mlcommons/inference/pull/1173. Cannot merge it due to CLA issue. 

How about these occurences ("1.1"):
- https://github.com/mlcommons/inference/blob/8823573324b253e784edf29330c248c9373472e4/loadgen/setup.py#L71
- https://github.com/mlcommons/inference/blob/8823573324b253e784edf29330c248c9373472e4/loadgen/setup.py#L77

Included both updates in #1175
Repository: mlcommons/inference
Issue: Broken CLA sign up link

The CLA bot says:

MLCommons CLA bot: 
Thank you for your submission, we really appreciate it. We ask that you sign our [MLCommons CLA](https://github.com/mlcommons/systems/blob/main/mlcommons_cla.txt) and be a member before we can accept your contribution. If you are interested in membership, please contact [membership@mlcommons.org](mailto:membership@mlcommons.org) .
0 out of 1 committers have signed the MLCommons CLA.

However, the link "https://github.com/mlcommons/systems/blob/main/mlcommons_cla.txt" is broken.

@weimingzha0  Please use this link to get github id authorized https://docs.google.com/forms/d/e/1FAIpQLSd4gItNaxbm9okFroxJEUmkmPjAJ6YegNxmHRAq8f6Uv_SDyw/viewform


Addressed the broken link issue in this PR https://github.com/mlcommons/inference/pull/1180
Repository: mlcommons/inference
Issue: ssd-mobilenet 300x300 onnx model converted from tensorflow can be optimized.
Root cause: Unimplemented Feature Gap

[`ssd-mobilenet 300x300`](https://zenodo.org/record/4735652/files/ssd_mobilenet_v1_coco_2018_01_28.onnx) from https://github.com/mlcommons/inference/tree/r1.1/vision/classification_and_detection. 

The `ZerosLike inserts Casts before Mul` issue can be optimized refer: https://github.com/onnx/tensorflow-onnx/issues/1997
which can improve the inference performance.

Optimized model `opset 11` and `opset 15`.
[ssd_mobilenet_v1_coco_2018_01_28_opset11.zip](https://github.com/mlcommons/inference/files/9149255/ssd_mobilenet_v1_coco_2018_01_28_opset11.zip)
[ssd_mobilenet_v1_coco_2018_01_28_opset15.zip](https://github.com/mlcommons/inference/files/9149257/ssd_mobilenet_v1_coco_2018_01_28_opset15.zip)




Hi @hwangdeyu , thank you for the new model files. Do you have the performance comparison of the models? ssd-mobilenet is removed from inference in closed division but we can [still use](https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#412-relaxed-constraints-for-the-open-division) it to submit results to open division.


Not relevant anymore, closing issue
Repository: mlcommons/inference
Issue: Exception in check_performance_dir: float() argument must be a string or a number, not 'NoneType'
Root cause: Validation Gap

Hello, I am trying to make a submission that passes the `tools/submission/submission-checker.py`. 
I am getting an exception in `check_performance_dir()` seemingly from parsing the mlperf_log_detail.txt in my sample result.

```
python ../../tools/submission/submission-checker.py --input /home/mgoin/slow/sub                                                                                        
[2022-07-20 11:49:05,606 log_parser.py:50 INFO] Sucessfully loaded MLPerf log from open/NeuralMagic/results/nmcpuserver2-deepsparse/bert-99/SingleStream/accuracy/mlperf_log_detail.txt.
[2022-07-20 11:49:05,608 log_parser.py:50 INFO] Sucessfully loaded MLPerf log from open/NeuralMagic/results/nmcpuserver2-deepsparse/bert-99/SingleStream/performance/run_1/mlperf_log_detail.txt.
[2022-07-20 11:49:05,608 submission-checker.py:1640 ERROR] open/NeuralMagic/results/nmcpuserver2-deepsparse/bert-99/SingleStream/performance/run_1 caused exception in check_performance_dir: float() argument must be a string or a number, not 'NoneType'
[2022-07-20 11:49:05,608 submission-checker.py:1660 ERROR] open/NeuralMagic/results/nmcpuserver2-deepsparse/bert-99/SingleStream/performance/run_1 has issues
[2022-07-20 11:49:05,608 submission-checker.py:1885 INFO] ---
[2022-07-20 11:49:05,608 submission-checker.py:1891 INFO] ---
[2022-07-20 11:49:05,608 submission-checker.py:1894 ERROR] NoResults open/NeuralMagic/results/nmcpuserver2-deepsparse/bert-99/SingleStream
[2022-07-20 11:49:05,608 submission-checker.py:1897 INFO] ---
[2022-07-20 11:49:05,608 submission-checker.py:1898 INFO] Results=0, NoResults=1
[2022-07-20 11:49:05,608 submission-checker.py:1900 ERROR] SUMMARY: submission has errors
```

Here is the file that is failing the checker:
[mlperf_log_detail.txt](https://github.com/mlcommons/inference/files/9152499/mlperf_log_detail.txt)

Has anyone run into this type error before?

Which version of LoadGen and the submission checker are you using? I see in the logs you provided that you use the random seeds from v2.0.

```
"key": "effective_qsl_rng_seed", "value": 6655344265603136530
"key": "effective_sample_index_rng_seed", "value": 15863379492028895792
"key": "effective_schedule_rng_seed", "value": 12662793979680847247
```

I think the error is that the submission checker is requesting a field you don't have in the logs:
https://github.com/mlcommons/inference/blob/fc02e9b19d1a3dc1b18cc425d6c2d7b25407ea96/tools/submission/submission-checker.py#L1076
For v2.0 this should be:
https://github.com/mlcommons/inference/blob/fc02e9b19d1a3dc1b18cc425d6c2d7b25407ea96/tools/submission/submission-checker.py#L756-L762

But it seems the LoadGen version you are using doesn’t output this value.

Thank you @pgmpablo157321 , I think you are right about the version being incorrect. I was using the dockerfile approach in language/bert folder and it doesn't specifically checkout any branch so I believe it got mixed up. Putting in an explicit checkout seemed to fix the problem.
Repository: mlcommons/inference
Issue: Submission checker requires the name of the scenario to be capitalised in compliance_dir
Root cause: Algorithmic Error

When running the submission checker for `r2.1`, we noticed that the submission checker requires the name of the scenario to be capitalised in compliance_dir. In `tools/submission/submission-checker.py` [line 1667](https://github.com/mlcommons/inference/blob/r2.1/tools/submission/submission-checker.py#L1667):
```
compliance_dir = os.path.join(division, submitter, "compliance",
                              system_desc, model_name, scenario_fixed)
```
where `scenario_fixed` are one of the followings:
```
["Offline", "Server", "SingleStream", "MultiStream"]
```

In previous rounds, the `scenario` in `compliance_dir` has always been non-capitalised, for instance, it would be something like this [`closed/Krai/compliance/odroid-armnn-v21.11-neon/resnet50/offline/`](https://github.com/mlcommons/inference_results_v2.0/tree/master/closed/Krai/compliance/odroid-armnn-v21.11-neon/resnet50/offline), but currently, the checker is looking for something like this: `closed/Krai/compliance/odroid-armnn-v21.11-neon/resnet50/Offline/`
Repository: mlcommons/inference
Issue: Comparing against the latency constraint should be done in nanoseconds
Root cause: Algorithmic Error

I have logs from running BERT-99 Server power experiments and compliance tests on the same machine.

```
$ grep "99.00 percentile latency (ns)\"" $CK_EXPERIMENT_DIR/*bert-99-server*2575*/*.0001.json -R | cut -c 142-
bert-99-server-performance-target_qps.2575-compliance.TEST01-vc.17/ckp-63257ca27e1c48a0.0001.json:          "99.00 percentile latency (ns)": "129891439", # 129.89 ms
bert-99-server-performance-target_qps.2575-compliance.TEST05-vc.17/ckp-9e3ec383bf4e4ba3.0001.json:          "99.00 percentile latency (ns)": "129980943", # 129.99 ms
bert-99-server-performance-target_qps.2575-vc.17-power.workload/ckp-4e02ab6e000a5e96.0001.json:          "99.00 percentile latency (ns)": "129558629", # 129.56 ms
bert-99-server-performance-target_qps.2575-vc.17-power.workload/ckp-84d28b14c156432f.0001.json:          "99.00 percentile latency (ns)": "129613744", # 129.61 ms
```
As can be seen, the 99th percentile latency is under the BERT latency constraint of 130 ms (130,000,000 ns) in all the cases.

Astonishingly, the tests are marked `INVALID`:
```
$ grep "Result is : INVALID" $CK_EXPERIMENT_DIR/*bert-99-server*2575*/*.0001.json -R | grep -v MLLOG | cut -c 142-
bert-99-server-performance-target_qps.2575-compliance.TEST01-vc.17/ckp-63257ca27e1c48a0.0001.json:            "Result is : INVALID\n",
bert-99-server-performance-target_qps.2575-compliance.TEST05-vc.17/ckp-9e3ec383bf4e4ba3.0001.json:            "Result is : INVALID\n",
```
I guess this is due to rounding and then comparing against the latency constraint.
Repository: mlcommons/inference
Issue: URGENT: need to update the power-dev submodule
Root cause: Configuration Error

At the moment, it points to r2.0, whereas we have changes on r2.1 that changed the checksums.

We need something like [this](https://github.com/mlcommons/inference/pull/989/files).

@nvpohanh once sent me these instructions for the uninitiated:

> The normal way to update a submodule:
```
git fetch
git checkout r1.1
git pull origin r1.1
git checkout -b dev-branch
cd tools/submission/power-dev
git fetch
git checkout r1.1
git pull origin r1.1
cd ../../..
git add tools/submission/power-dev
git commit -m "Update power-dev hash"
git push fork dev-branch
```
> Basically, to the git submodule is a black box to its parent git repo. You can just do the normal git operations inside the submodule, and at the end go back to the parent git repo and add the submodule to staged changes and commit the change.

Filed https://github.com/mlcommons/inference/pull/1197

Thanks a lot @nvpohanh!

In the future, we should create a checklist of things to do before every submission round e.g.:
- generate random seeds
- create a new `inference` branch
- create and new `power-dev` branch
- update the `power-dev` submodule on the `inference branch
- etc.

/cc @rnaidu02 @tjablin
Repository: mlcommons/inference
Issue: Please bump power-dev hash to the latest
Root cause: External Dependency Breakage

Similar to https://github.com/mlcommons/inference/pull/1197, we must get the latest `power-dev` fix to r2.1 for the submission checker not to reject valid submissions for a [trivial reason](https://github.com/mlcommons/power-dev/pull/268#issuecomment-1198635012).

@nvpohanh @pgmpablo157321 

Many thanks Pablo!
Repository: mlcommons/inference
Issue: The group accuracy for different groups in shufflenet turns out the same,what causes this problem?
Root cause: Configuration Error

I run shufflenet network by executing ./run.sh imagenet group1 or ./run.sh imagenet group2. The accuracy of different groups is the same, why did it turn out this result?
For group1(./run.sh <imagenet dir> group1):
NET: shufflenet	METRIC: accuracy	ID: 0
PLATFORM: Linux-4.18.0-17-generic-x86_64-with-Ubuntu-18.04-bionic--IntelR-CoreTM-i5-8250U-CPU--1.60GHz	HASH: ubuntu
FRAMEWORK: caffe2	COMMIT: 0ebe252c9c81f64a3604b7b340550353c0537566	TIME: 2019-05-08 15:58:19
NET latency: value median 33488.19351  MAD: 1049.27826
image_preprocess convert: value median 2865.79500  MAD: 463.04500
preprocess data_pack: value median 74817.60000  MAD: 1008.20000
shufflenet total_number_of_top1_corrects: value median 33234.00000  MAD: 0.00000
shufflenet total_percent_of_top1_corrects: value median 66.46800  MAD: 0.00000
INFO 20:09:58 harness.py: 235:  ======= success =======
INFO 20:09:58 repo_driver.py: 360: One benchmark run successful for 0ebe252c9c81f64a3604b7b340550353c0537566

For group8: (./run.sh <imagenet dir> group8)
NET: shufflenet	METRIC: accuracy	ID: 0
PLATFORM: Linux-4.18.0-17-generic-x86_64-with-Ubuntu-18.04-bionic--IntelR-CoreTM-i5-8250U-CPU--1.60GHz	HASH: ubuntu
FRAMEWORK: caffe2	COMMIT: 0ebe252c9c81f64a3604b7b340550353c0537566	TIME: 2019-05-08 15:58:19
NET latency: value median 14934.82733  MAD: 212.27360
image_preprocess convert: value median 1397.55500  MAD: 229.61500
preprocess data_pack: value median 31351.20000  MAD: 324.00000
shufflenet total_number_of_top1_corrects: value median 33234.00000  MAD: 0.00000
shufflenet total_percent_of_top1_corrects: value median 66.46800  MAD: 0.00000
INFO 11:20:34 harness.py: 235:  ======= success =======
INFO 11:20:34 repo_driver.py: 360: One benchmark run successful for 0ebe252c9c81f64a3604b7b340550353c0537566
Repository: mlcommons/inference
Issue: Compliance TEST01 failure reasoning
Root cause: Unimplemented Feature Gap

Compliance TEST01 can fail for some systems due to non-determinism and an additional [non-determinism test](https://github.com/mlcommons/inference/tree/master/compliance/nvidia/TEST01#non-determinism) can let this pass. In case of such a scenario, it would be good to add "accuracy_failure.txt" as a mandatory file along with the current mandatory ones- `baseline_accuracy.txt` and `compliance_accuracy.txt`, describing the reason of failure.
Repository: mlcommons/inference
Issue:  How to use ssd-mobilenet 300x300 (torch version) provided by mlcommons for inference process
Root cause: Unimplemented Feature Gap

I would like to ask, how to use the ssd-mobilenet model provided by MLcommons on pytorch and convert it into the corresponding jit model, can you provide a demo or link?
![image](https://user-images.githubusercontent.com/17673134/190618838-dfb8f1a4-f4a6-41ce-a754-931d3773d02b.png)


@lixiaolx Checking the reference implementation, I can see that unfortunately this model is not implemented. Since it was removed in version 2.1, there is no incentive to fix it. There a few steps you can follow to run this benchmark, but you'll have to modify some code.

1. Clone the repository with its submodules: 
```
git clone --recurse-submodules https://github.com/mlcommons/inference.git --depth 1
```
2. Install LoadGen
```
cd inference/loadgen
CFLAGS="-std=c++14" python setup.py develop
cd ..
```
3. Go to the `vision/classification_and_detection` folder and download the model and the dataset there. The dataset in its own folder. Download model:
```
cd vision/classification_and_detection
wget https://zenodo.org/record/3239977/files/ssd_mobilenet_v1.pytorch
```
Download dataset:
```
mkdir data
cd data
wget http://images.cocodataset.org/zips/val2017.zip
wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip
unzip val2017.zip
unzip annotations_trainval2017.zip
```
4. Open the file `vision/classification_and_detection/run_common.sh` and add the following code in the pytorch section:
```Bash
if [ $name == "ssd-mobilenet-pytorch" ] ; then
    model_path="$MODEL_DIR/ssd_mobilenet_v1.pytorch"
    profile=ssd-mobilenet-pytorch
fi
```
5. With this setup you should be able to run the benchmark from a jupyter notebook. The notebook `vision/classification_and_detection/GettingStarted.ipynb` contains most of the code for this, but the examples it has are very different. You can run the benchmark from a notebook in the folder `vision/classification_and_detection` with the following cells:
```Python
import os
root = os.getcwd()
os.environ['MODEL_DIR'] = root
os.environ['DATA_DIR'] = os.path.join(root, "data")
os.environ['EXTRA_OPS'] ="--time 600"
```
Inside of the jupyter notebook, you can run the performance benchmark:
```
!./run_local.sh pytorch ssd-mobilenet [device] --scenario [Scenario]
```
Inside of the jupyter notebook, you can compute the accuracy:
```
!./run_local.sh pytorch ssd-mobilenet [device] --accuracy
```
Where device is cpu or gpu

Not relevant anymore, closing issue
Repository: mlcommons/inference
Issue: question about speech to text benchmark 

Hi, I'm looking for a speech to text benchmark for datacenter.
I found that MLPerf still use RNN-T for the speech to text benchmark.
However, there are many other better network(in terms of WER) using on transformer-like block.
I'm curious why MLPerf keep using RNNT even for a v2.1.

Thank you in advance.

RNNT was dropped few versions ago, whisper was added as a speech to text model, closing issue
Repository: mlcommons/inference
Issue: TEST04 Evaluation

TEST04 is only applicable to ResNet50. Is it still worthwhile?

TEST04 is a good compliance test as it indeed ensures that no significant benefit from caching happened for a submission. But it does not make sense to have it just for `resnet50`. The current problem is that due to variable length inputs we cannot apply this to other models. Is it possible to increase the input size (currently just 1) to say 8 or 16 (which still should ensure caching) in Test-04 and make it applicable to other models?

We agree that it would be desirable if this test was more widely applicable. It applying only to Resnet50 currently does not invalidate it, in our opinion. Especially since almost all submitters submit this benchmark. For some, it represents 50%-100% of the benchmarks they submit.
We welcome more input from the WG on how this test can be improved.

For v3.0 there is no change for TEST 04. For v3.1, TEST 04 will support beyond ResNet50.

Pablo will create an issue to address the v3.1 requirements, work with other folks to create the requirements and implement it for v3.1 submission.


Not relevant anymore, closing issue
Repository: mlcommons/inference
Issue: TEST05 Evaluation

Is TEST05 still worthwhile?

[This test](https://github.com/mlcommons/inference/tree/master/compliance/nvidia/TEST05) should ensure that the SUT does not favour any particular set of random seeds. However, its random seeds never change. Therefore, a rogue submitter could make their SUT favour both this round's (updated) random seeds and the TEST05's (perennial) random seeds. In fact, the latter would be even easier. A better approach would be to vary both sets of random seeds in each round.

I do not see too much value in this test either way, to be honest.

The random seeds was changed once after either the first or second round, I don't quite remember. Agree that the seeds should change every round along with the official ones, and the change be reflected in the submission checker. Can Pablo take this on?

Keep the TEST 05

Tasks:
1. Create 1 random seeds for TEST 05
2. Add the new seed (**.test05_rng_seed)  to mlperf.conf
3. TEST 05 to read the seed info from mlperf.conf
4. Pablo to make the changes to TEST 05
Repository: mlcommons/inference
Issue: Bert-onnxruntime reference implementation fails to run with onnxruntime_gpu
Root cause: Interface Contract Mismatch

If onnxruntime is supporting more than one execution provider, bert-onnxruntime reference implementation is failing with below error.
```
Loading ONNX model...
Traceback (most recent call last):
  File "/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/abdb11453edc4744/inference/language/bert/run.py", line 120, in <module>
    main()
  File "/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/abdb11453edc4744/inference/language/bert/run.py", line 76, in main
    sut = get_onnxruntime_sut(args)
  File "/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/abdb11453edc4744/inference/language/bert/onnxruntime_SUT.py", line 84, in get_onnxruntime_sut
    return BERT_ONNXRuntime_SUT(args)
  File "/media/arjun/T7_Touch/hp-envy/CM/repos/local/cache/abdb11453edc4744/inference/language/bert/onnxruntime_SUT.py", line 44, in __init__
    self.sess = onnxruntime.InferenceSession(model_path, self.options)
  File "/home/arjun/.local/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 347, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/home/arjun/.local/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 375, in _create_inference_session
    raise ValueError(
ValueError: This ORT build has ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'] enabled. Since ORT 1.9, you are required to explicitly set the providers parameter when instantiating InferenceSession. For example, onnxruntime.InferenceSession(..., providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'], ...)
Finished destroying SUT.

```
Repository: mlcommons/inference
Issue: Query: Is running two instances of MLPerf valid for Offline and Server Submission

Hi,

To get the best performance from system(X86arch), I am running two instances of MLPerf simultaneously through shell script (Each with their own Loadgen and SUT), each instance is bind to particular socket.     
This way we will be getting **_two mlperf_log_summary.txt_**  and the final QPS will be added from both the output files. 

Is this a valid submission request?

Thanks in Advance!


Currently - this is not valid.
However, we have been working on a multiple-clients-over-the-network PR and will soon open it to review. This shall be able to allow a similar submission over the network while taking care of statistics, files, tests and submission checking correctly.

@avinashcpandey As @saar-eliad mentioned, as of now only a single instance of loadgen is allowed even for 3.0 round. But for your case you should get the exact same performance running on a dual socket with loadgen running on one of the socket by ensuring that the loadgen buffer is local to each socket. That is, we have to replicate the loadgen buffer at least one per socket. [This paper](https://accml.dcs.gla.ac.uk/papers/2022/4thAccML_paper_7(14).pdf) talks about the same and the performance achieved on an AMD Milan/Rome server using dual sockets (128 physical cores) and 18 Qualcomm accelerators went over 400,000 infs/s for ResNet50 (linear scaling from a single device performance).  This shows that for scalable performance we do not need to run multiple loadgen instances. If you face any issue in performance please comment here or you can also join our [task force](https://docs.google.com/forms/u/1/d/e/1FAIpQLSdZ7tlkIminKxzLlPF9bCkIqXyPmFty99tsufKGyQWPQifmQw/viewform?usp=send_form) where we'll be discussing such common problems affecting all submitters.  

Thanks @arjunsuresh and @saar-eliad for the prompt response. I will explore as you suggested.