zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: target_latency_percentile is uninitialized for offline mode
Root cause: Validation Gap

In loadgen/test_settings_internal.cc, funciton TestSettingsInternal::TestSettingsInternal(
    const TestSettings &requested_settings, size_t qsl_performance_sample_count) () , on the path of offline mode, target_latency_percentile is not initialized, as a result we may got "nan" in mlperf_log_detail.txt for "effective_target_latency_percentile" and the submission checker will report error. 

We may fix it with this patch:
diff --git a/loadgen/test_settings_internal.cc b/loadgen/test_settings_internal.cc
index 6cccae8..6ab9d47 100644
--- a/loadgen/test_settings_internal.cc
+++ b/loadgen/test_settings_internal.cc
@@ -48,6 +48,7 @@ TestSettingsInternal::TestSettingsInternal(
       performance_issue_same(requested.performance_issue_same),
       performance_issue_same_index(requested.performance_issue_same_index),
       performance_sample_count(0) {
+  target_latency_percentile = 0;
   // Target QPS, target latency, and max_async_queries.
   switch (requested.scenario) {
     case TestScenario::SingleStream:


This is fixed in: https://github.com/mlcommons/inference/pull/837
Please use the latest r1.0 branch.

closing if no further comment.
Repository: mlcommons/inference
Issue: submission checklist needs updates and definition in rules
Root cause: Documentation Deficiency

Submission checklist needs to be updated for v1.0:
https://github.com/mlcommons/inference_policies/blob/master/submission_checklist.md

The rules regarding the submission checklist are still not defined:
https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#submission-checklist

This is a carry-over issue that existed for v0.7 as well.
https://github.com/mlcommons/inference_policies/issues/170

It seemed like the checklist wasn't very useful last time. Do we want to update it or cut it?

Short answer: I like the idea of cutting it.

Long answer:
I think the point you made in the v0.7 issue thread is a good one to consider: that "many of the questions are no longer necessary with the new submission checker scripts". To add to that, I think the current checklist is only valuable as a double-check for the submitter themselves to make sure they are following important rules that they might not have realized existed. But that's more of what `submission-checker.py` is meant to do. One problem is that a submitter goes through this checklist only right at the end, after all work is already done. That doesn't give the submitter a chance to fix anything that they might have done wrong.

It would be great if we only relied on `submission-checker.py`, and augment it with more checks to cover what would go unchecked (and only if deemed important) if we get rid of this submission checklist. Submitters generally run `submission-checklist.py` with plenty of time left to change whatever errors / violations they might have encountered.

If the WG decides that some of the questions in this checklist are critical and aren't already covered by `submission-checker.py`, then perhaps new fields should be introduced in an existing JSON spec file (e.g., in the system description JSON file, or the measurements dataset / training JSON files). In fact, the measurements JSON file ([link](https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#system_desc_id_implementation_id_scenario-json-metadata)) already has input / retraining fields that overlap with the checklist's "numerics" question and could be augmented with additional fields that could cover several of the other questions if needed.

+1 for cutting

+1 for bidding it goodbye!

let's officialize it tomorrow in the WG.

WG: agree to remove the requirement on submitting https://github.com/mlcommons/inference_policies/blob/master/submission_checklist.md
We have the submission checker now to perform a similar function
@guschmue
Repository: mlcommons/inference
Issue: [submission checker] Single Stream result is still the 90th percentile
Root cause: Algorithmic Error

One of my experimental results is 78 ms:

```
================================================
MLPerf Results Summary
================================================
SUT name : TFLite_SUT
Scenario : SingleStream
Mode     : PerformanceOnly
90th percentile latency (ns) : 77975401
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes

================================================
Additional Stats
================================================
QPS w/ loadgen overhead         : 12.94
QPS w/o loadgen overhead        : 12.94

Min latency (ns)                : 73726788
Max latency (ns)                : 162143685
Mean latency (ns)               : 77278132
50.00 percentile latency (ns)   : 75739770
90.00 percentile latency (ns)   : 77975401
95.00 percentile latency (ns)   : 88838337
97.00 percentile latency (ns)   : 90892308
99.00 percentile latency (ns)   : 100776825
99.90 percentile latency (ns)   : 142315902
```

However, the submission checker thinks it is 101 ms:
```
INFO    Results closed/Krai/results/xavier-tflite-v2.4.1-ruy/resnet50/offline 12.940300 with power = 28.172995
INFO    Results closed/Krai/results/xavier-tflite-v2.4.1-ruy/resnet50/singlestream 100.776825 with power = 28.172995
```

That is, it takes the 99th percentile latency not the 90th percentile latency as the Single Stream result. This disagrees with LoadGen.

Looks like a bug in submission checker. I can fix it...

Typo here: https://github.com/mlcommons/inference/blob/r1.0/tools/submission/submission-checker.py#L335 :)

Fix: #872
Repository: mlcommons/inference
Issue: 404 error on loadgen documentation
GitHub labels: inference v2.1 and backlog

The loadgen [READMe.md](https://github.com/mlcommons/inference/blob/master/loadgen/README.md) has a broken link to the pre-compiled documentation:
https://mlperf.github.io/inference/loadgen/index.html

@briandersn  do you know if we have a new URL somewhere after we migrated to mlcommons organization?
Repository: mlcommons/inference
Issue: Migrate DLRM to TF v2
Root cause: External Dependency Breakage

Hello, 

I've been trying to run dlrm with the 'out of the box setup'. But it looks like it's not working with tensorflow v2. Are there any plans to move to v2 or are there any workarounds ?

Thanks.

Adding error logs for reference:

```
$ python dlrm_main.py
Traceback (most recent call last):
  File "dlrm_main.py", line 13, in <module>
    import dlrm
  File "/home/ubuntu/inference/recommendation/dlrm/tf/dlrm.py", line 24, in <module>
    from tensorflow.contrib import  layers as contrib_layers
ModuleNotFoundError: No module named 'tensorflow.contrib' 
```

(btw I assigned @guschmue by mistake!).  

@guschmue @rnaidu02, There is also this related issue for TF v1 DLRM.  Why does the code in TFV1 DLRM include training scripts?
As a result the code contains optimize_loss function that is used for training. This loss function comes from a depreciated module  (tf.contrib) in TF v2.
Repository: mlcommons/inference
Issue: got errors while running submission-checker in SubmissionExample.ipynb
Root cause: Documentation Deficiency

Hi,
I attempted to run master branch's SubmissionExample.ipynb
According to issue #738, I switched to branch r0.7 (but I still used master branch's SubmissionExample.ipynb since I wanted to test resnet50) 
```
git clone https://github.com/mlperf/inference.git --recurse-submodules
git checkout r0.7
```
Since I only wanted to test the edge system type, so I commented the Server and MultiStream case in the shell script provided by SubmissionExample.ipynb
```
function one_model {
    # args: framework device model ...
    one_run SingleStream 1 $@ --max-latency 0.0005
    # one_run Server 5 $@
    one_run Offline 1 $@ --qps 1000
    # one_run MultiStream 1 $@
}
```
Everything went well until  I tried to run submission-checker.py
```
python submission-checker.py --input ${SUBMISSION_ROOT} > ${SUBMISSION_DIR}/submission-checker.log 2>&1;cat ${SUBMISSION_DIR}/submission-checker.log
```
Below was the error messages I got
```
ERROR:main:closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy not hash value for mlperf_log_accuracy.json
ERROR:main:closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy/mlperf_log_accuracy.json is not truncated
ERROR:main:closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy, accuracy not valid
ERROR:main:no compliance dir for closed/mlperf-org/results/tf-gpu/resnet/SingleStream
ERROR:main:closed/mlperf-org/results/tf-gpu/resnet/SingleStream is OK but accuracy has issues
ERROR:main:closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy not hash value for mlperf_log_accuracy.json
ERROR:main:closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy/mlperf_log_accuracy.json is not truncated
ERROR:main:closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy, accuracy not valid
ERROR:main:no compliance dir for closed/mlperf-org/results/tf-gpu/resnet/Offline
ERROR:main:closed/mlperf-org/results/tf-gpu/resnet/Offline is OK but accuracy has issues
INFO:main:---
INFO:main:---
ERROR:main:NoResults closed/mlperf-org/results/tf-gpu/resnet/SingleStream
ERROR:main:NoResults closed/mlperf-org/results/tf-gpu/resnet/Offline
INFO:main:---
INFO:main:Results=0, NoResults=2
ERROR:main:SUMMARY: submission has errors
```

Did you follow the compliance test steps here? https://github.com/mlcommons/inference/tree/r1.0/compliance/nvidia

Thanks

No I haven't. I'll check that out. Thanks

1. Setup
If I use the master branch's  SubmissionExample.ipynb to run the benchmark,  which address should I place audit.config?
2. Since there are four tests in NVIDIA's compliance test directory, should I copy audit.config and re-run the shell scripts in SubmissionExample.ipynb  four time to complete all the compliance test? 

@psyhtest Do you know where `audit.config` should be put when running compliance test if SubmissionExample.ipynb is used?
Repository: mlcommons/inference
Issue: git submodule issue caused by power-dev
Root cause: Configuration Error

On branch r1.0, when we run "git submodule sync && git submodule update --init --recursive" we will meet error as  https://github.com/mlperf/power-dev does not exist. 

git diff 06f07e3f6c488037eefbaba15ee72aa9676ff01c .gitmodules
diff --git a/.gitmodules b/.gitmodules
index 246722f..12a50d5 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -17,3 +17,6 @@
 [submodule "vision/medical_imaging/3d-unet/nnUnet"]
        path = vision/medical_imaging/3d-unet/nnUnet
        url = https://github.com/MIC-DKFZ/nnUNet.git
+[submodule "tools/submission/power-dev"]
+       path = tools/submission/power-dev
+       url = ../power-dev


Please git clone the inference repo using `https://github.com/mlcommons/inference` instead of `https://github.com/mlperf/inference`. The repo has been permanently moved.

Thanks for the quick response. Let me have a try.

Things are tricky when you update your github username... I encountered the same problem and spent a hard time finding this issue. MLCommons is a good project, but currently I find it bad in the documentation: invalid links, outdated versions, wrong commands, etc. Do you have any plan to update those READMEs? Or would you welcome a PR to fix this? @naghshineh
Repository: mlcommons/inference
Issue: ModuleNotFoundError: No module named 'mlperf_loadgen'
Root cause: Environment Incompatibility

i use git commond `git clone --recurse-submodules https://github.com/mlcommons/inference.git`
then in the inference/vision/classificaiton
when I use `./run_local.sh tf resnet50 gpu`

terminal raise
`
Traceback (most recent call last):
  File "python/main.py", line 20, in <module>
    import mlperf_loadgen as lg
ModuleNotFoundError: No module named 'mlperf_loadgen'
`

Hi @Heyuanhong ,
I am facing the same problem.
How did you solve this problem?
Repository: mlcommons/inference
Issue: [submission checker] add a function to check for results directory if measurements or compliance are present
Root cause: Unimplemented Feature Gap

Can we add a section of code to the submission_checker that returns an error if measurements or compliance directories for a given system_name are present, but the results folder is missing?

Currently submission_checker only returns an error if the reverse is true, aka if a results directory is present but there is no compliance/measurements folder.  

tricky because the checker walks the result directory and than checks everything needed for that result is there.
Always starts with a result.
We'd need to add another walk starting with the other directories and than compare that to the results directory.


could we get away with not doing a full walk, but perhaps comparing if a top directory within compliance/measurements does not have a corresponding results directory? You'd miss individual results submissions that way, but it would prevent wholescale absence of an entire system's results
Repository: mlcommons/inference
Issue: Handle timezones for power logs in submission-checker
Root cause: Validation Gap

Right now the submission-checker checks timestamps of power client and server logs here https://github.com/mlcommons/inference/blob/r1.0/tools/submission/submission-checker.py#L732
But those logs has time in localtime that can depend on timezone. The offset for timezones are located in client.json and server.json files and should be used to calculate real timestamps.

I will take the action to take timezone into account in submission checker. ETA next Monday or Tuesday

Will also try DST if the check fails in the first pass

@ykurlaev Done in #896 .

I realized that DST handling is not needed here since both server and client would be in DST when that happens.
Repository: mlcommons/inference
Issue: Using 3D-Unet with Tensorflow
Root cause: Unimplemented Feature Gap

I tried to run 3D-Unet with a tensorflow build but it seems that it depends on nnUNET which is PyTorch only.
"make preprocess_data" or "python3 run.py --backend=tf --scenario=SingleStream" leads to import torch errors.

@nvpohanh are there any tf support for 3d-unet ?

If you install pytorch in the tf container, does it solve the issue?

I run into this issue when trying to use both pytorch and tf
```
python3 run.py --backend=pytorch --scenario=SingleStream --max_examples=1
Traceback (most recent call last):
  File "run.py", line 107, in <module>
    main()
  File "run.py", line 53, in main
    from pytorch_SUT import get_pytorch_sut
  File "/home/ubuntu/inference/language/bert/pytorch_SUT.py", line 25, in <module>
    import torch
  File "/home/ubuntu/python3-venv/lib/python3.7/site-packages/torch/__init__.py", line 195, in <module>
    _load_global_deps()
  File "/home/ubuntu/python3-venv/lib/python3.7/site-packages/torch/__init__.py", line 148, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/usr/local/lib/python3.7/ctypes/__init__.py", line 356, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/ubuntu/python3-venv/lib/python3.7/site-packages/torch/lib/libgomp-d22c30c5.so.1: cannot allocate memory in static TLS block
```

@nvpohanhm, ideally we are interested in testing TensorFlow and PyTorch stacks separately for Aarch64 .
Repository: mlcommons/inference
Issue: mismatched version in top-level loadgen CMakeLists.txt for tags v0.7, v0.7.1 and branch r0.7
Root cause: Configuration Error

Not sure if this is intended or not, but the top-level CMakeLists.txt file causes artifacts to be generated with version 0.5 instead of 0.7 as I would expect.

~/project/sources/mlperf_inference/loadgen/build$ git branch
  | * r0.7
~/project/sources/mlperf_inference/loadgen/build$ cat ../CMakeLists.txt | head
  | # The mlperf_loadgen version.
  | set(mlperf_loadgen_VERSION_MAJOR 0)
  | set(mlperf_loadgen_VERSION_MINOR 5)
  | message("mlperf_loadgen v${mlperf_loadgen_VERSION_MAJOR}.${mlperf_loadgen_VERSION_MINOR}")

Same situation on tags v0.7 and v0.7.1, and on master branch.

I think that was a bug back then, but no one fixed it. :)

I fixed it recently in CMakeLists and https://github.com/mlcommons/inference/pull/960 fixes it in setup.py as well.
Repository: mlcommons/inference
Issue: Wrong links in README

It seems the links in this section (v1.0): https://github.com/mlcommons/inference#mlperf-inference-v10-submission-03122020 for the "reference app" are pointing to the r0.7 branch instead of r1.0


I saw there's a PR (https://github.com/mlcommons/inference/pull/901/files) for it. I close this issue.
Repository: mlcommons/inference
Issue: PR missing in moving from r0.7 to r1.0

I don't know if that's really important but it seems the change in this PR: https://github.com/mlcommons/inference/pull/774 was merged to the r0.7 branch (at the time) but it disappeared in the r1.0. 
It was labelled as "in-progress v1.0" and closed: https://github.com/mlcommons/inference/issues?q=label%3A%22in-progress+v1.0%22+is%3Aclosed but it was not added to the branch r1.0: https://github.com/mlcommons/inference/compare/r1.0...r0.7

The r0.7 branch is frozen in time after the submission. If we make fixes during the submission we'll merge them 1 time back to master after the submission.
Once the submission is done, the release branches should no longer be touched since they are suppose to make the submission reproducible (hence we can't modify them),

I guess maybe I put the question in the wrong way. I'm not asking for modification.
I was wondering why a change that was part of branch r0.7 is not part of branch r1.0, is that on purpose?
By the way if you're aware that's fine, not a big issue.

Because the change went into the r0.7 branch way after the submission round and we stop merging the release branch right after the submission.
Guess this change should have gone to master, than it would carry forward.

I will create a new PR and turn in the patch to the master branch shortly.
Repository: mlcommons/inference
Issue: Steps to build inference
Root cause: Documentation Deficiency

Hi,
I am newbie to ML & inference. Could anyone please help me with building steps for inference?

Thanks

Each benchmark has its own README with build instructions: 
- [BERT](https://github.com/mlcommons/inference/tree/master/language/bert)
- [DLRM TF](https://github.com/mlcommons/inference/tree/master/recommendation/dlrm/tf)
- [DLRM PyTorch](https://github.com/mlcommons/inference/blob/master/recommendation/dlrm/pytorch)
- [RNNT](https://github.com/mlcommons/inference/tree/master/speech_recognition/rnnt)
- [Resnet, SSD-Large, SSD-Small, and MobileNet](https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection)
- [3D-UNET](https://github.com/mlcommons/inference/tree/master/vision/medical_imaging/3d-unet)

and there is some tutorial
here:
https://github.com/mlcommons/inference/blob/master/SubmissionExample.ipynb
and here:
https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/GettingStarted.ipynb
Repository: mlcommons/inference
Issue: Process not getting Completed for Multi-instance code(Intel MLPERF0.7-SUB)
Root cause: Validation Gap

Hi,

         I Have checked out MLPERF Reference code and ran it for Server and offline. Since numbers, I am going through Intel's Submission code. I can able to setup following their guidelines and able to run with my model. But, when I am running it for Performance mode , since accuracy verification is not needed in that mode I am keeping **--include_accuracy** as _false_ .The process is not at all completing, seems it is hanging. Can anyone help me on this! It would be a great help.

Thanks
Repository: mlcommons/inference
Issue: BERT Pytorch INT8 reference model
GitHub labels: inference v1.1

Currently the only INT8 model is in ONNX, can NVIDIA share the Pytorch version where the ONNX INT8 is converted from? Thanks! @DilipSequeira 

WG: @DilipSequeira  will look into the schedule to release a PyTorch version

WG: @DilipSequeira said they can do, expected ~ 19/05

I added a BERT int8 PyTorch reference model in https://github.com/mlcommons/inference/pull/937. Although the PyTorch model isn't the exact one the ONNX was converted from (PyTorch F1 score is slightly higher than ONNX F1 score).

https://github.com/mlcommons/inference/pull/937
Repository: mlcommons/inference
Issue: DLRM inference README out of date
Root cause: Documentation Deficiency

The instructions reference v0.5 and don't seem to be compatible with the v1.0 branch.

It looks like there was previously a link to the following repo which is no longer included, but required -

https://github.com/facebookresearch/dlrm

I've cloned this and experimented with both the mlperf and master branches but each has its own problems.

Possibly there's a reliance on files in the MLCommons training repo too?

Would be good to have a working example. :)

Some issues here: 
https://github.com/mlcommons/inference/issues/604

@mnaumovfb 

Can you be more specific? What exactly is not working and what is the error log? CC @dkorchevgithub 

@dkorchevgithub Would you care to create a PR with your changes please?

@psyhtest I created PR https://github.com/mnaumovfb/inference/pull/10 with this and other changes. Meanwhile, I am looking into more updates to readme. Thank you for your suggestions on docker, I incorporated them in such a way that the code and data can be shared among different containers. Please let us know how it works for you.

Thanks @dkorchevgithub! Perhaps @mnaumovfb will need to accept your PR and then create a PR from his fork to mlcommons/inference?
Repository: mlcommons/inference
Issue: DLRM tf model download problem
Root cause: Environment Incompatibility

hi，@christ1ne  @mnaumovfb @guschmue ,sorry to disturb, for some reason,I can't access google cloud and  download checkpoint and data through "gs://dmchen-data/dlrm_tf1_data/model_dir_0".
Would you do me a fovor, transfer the model and data to google drive, thanks for help.

First, I wanted to mention that DLRM TF backend is experimental, so we did not fully test it. @petermattson and @tayo would probably be the right people to talk about the weights for it.

> First, I wanted to mention that DLRM TF backend is experimental, so we did not fully test it. @petermattson and @tayo would probably be the right people to talk about the weights for it.

Ok,thanks for reply. I am eager to test the TF implementation for DLRM.

Hi，@petermattson and @tayo，I have signed in google cloud，but raises that not have access to the storage bucket while downloading. Can you tell me how to fix it?Thanks for help.

I did not work on the inference weights for DLRM.  @davidmochen , any ideas?

I'm having the same issue. It appears that the model still exists at this GCS location but is no longer publicly accessible. @davidmochen 

Unfortunately I don't have much context on the model checkpoint for inference.  I will attempt to contact David via a different method.


All -- We are actively working to find a new home for the data.  The previous location is stale.  I will update here once the data has been uploaded.

Checkpoint and data are available here:  [DLRM inference data](https://drive.google.com/drive/folders/1B8n8YzkKSn5Khs2gqiu_Ql16_v-DxF3B?usp=sharing).

It seems like the link above only contain checkpoint, not the actual data. 

Screenshot?  From my view, the directory includes a 90GB data file.
Repository: mlcommons/inference
Issue: Prerequisite to build inference
Root cause: Documentation Deficiency

Hi,
What are the packages/modules required to build inference benchmarks on Linux(RISCV64)? 

Thanks



I think there is nothing special you'd need to do for RISCV64 assuming you have the compiler toolchain.
You could just try building loadgen via cmake and than run one of the python vision benchmarks.
We know that is fine rpi but I don't know if anybody tried riskv yet.
If you run into issues - send a PR or file an issue so we can fix it.