zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: about code directory in the submission
Root cause: Algorithmic Error

In v0.5/tools/submission/submission-checker.py

        impl = system_file[len(system_desc) + 1:-5]
        code_dir = os.path.join(parts[0], parts[1], "code", model, impl)

are used to get the code directory for conformance validation.
system_file are an array of filenames as 
<system_desc_id>_<implementation_id>_{scenario}.json

so impl will be assigned as <implementation_id>_{scenario}

thus resulting in the code directory name as

{division}/{submitter}/code/{model}/<implementation_id>_{scenario}

which means submission-checker.py assume a seperate copy of code implementation for each scenario. 

scenario detection in the implementation code is assumed against the mlperf rule. Having a seperate code implementation is similar to scenario detection.

Thus I proposed to only check once whether such directory 
{division}/{submitter}/code/{model}/<implementation_id> exist, instead of repeating checking of the code directory for each scenario with different name.




you are correct, I'll fix it.

Thanks


------------------------------------------------------------------
From:Guenther Schmuelling <notifications@github.com>
Sent At:2019 Oct. 7 (Mon.) 19:39
To:mlperf/inference <inference@noreply.github.com>
Cc:Aaron <aaron.zhong@alibaba-inc.com>; Author <author@noreply.github.com>
Subject:Re: [mlperf/inference] about code directory in the submission (#490)

you are correct, I'll fix it.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
Repository: mlcommons/inference
Issue: Add data type support in audit test 01 verify_accuracy.py
Root cause: Unimplemented Feature Gap

I got the test01 from here `https://drive.google.com/file/d/1oAOb_I1RfGei3wr-t5mTNLmMkagbiugR/view`

The data type support shall be added, just like mlperf accuracy check script at here `https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/tools/accuracy-imagenet.py`.
So that user can have freedom to utilize data space after the end of label field.

The modification is very little, the attachment for reference is the one I changed following the style of mlpef accuracy-imagenet.py. The new command line is sth. like below,
python3 ./verify_accuracy.type.added.py --accuracy_log ./mlperf_log_accuracy.acc.mobile.server.json --performance_log ./mlperf_log_accuracy.audit.mobile.server.json **--dtype int32**

BTW, changing to python3 will make the script much faster.

@christ1ne @georgelyuan 

[verify_accuracy.type.added.py.zip](https://github.com/mlperf/inference/files/3702864/verify_accuracy.type.added.py.zip)



See #395 for example

@georgelyuan will fix this in the next release per discussion

@georgelyuan is this issue still relavant?

Already fixed per WG discussion today.
Repository: mlcommons/inference
Issue: submission checker - checking for unneeded (according to the submission rules) files
Root cause: Validation Gap

One issue, when I make a submission run that supposes to be both accuracy and performance run I get 3 logs. we submit both files summary and details log under the performance folder and accuracy log under the accuracy folder. So the checker can check all files. What the checker asks is to copy the same files twice, ones under the performance and ones under accuracy. This is against the rules and seems unneeded. The fix is very simple:
line 26-27 should be replaced with
REQUIRED_PERF_FILES = ["mlperf_log_summary.txt", "mlperf_log_detail.txt"]
REQUIRED_ACC_FILES =  ["mlperf_log_accuracy.json", "accuracy.txt"]



@georgelyuan do we need this for v0.7?

Po-Han claims this is fixed. If he's bluffing, he will find a solution :)
Repository: mlcommons/inference
Issue: LoadGen conf file improvements for v0.7
Root cause: Validation Gap

This is a tracker for LoadGen conf file improvements for next version.

### LoadGen conf files use different benchmark naming convention
In [mlperf.conf](https://github.com/mlperf/inference/blob/master/v0.5/mlperf.conf), it uses `ssd-resnet34` and `resnet50` instead of the official naming convention, namely `ssd-large` and `resnet`. We should unify the naming convention.

### LoadGen conf files use different LoadGen setting names
For example, conf file uses `*.SingleStream.target_latency` while it corresponds to the setting which is named `single_stream_expected_latency_ns`. We should unify these names.

### LoadGen conf files use different units than LoadGen settings
For example, conf file uses "ms" for `*.SingleStream.target_latency` but the setting it self uses "ns". The scale of 1,000,000 is implicit and is not clear to the user. It also caused issues like #471. We should unify the units.

### LoadGen conf files should fail loudly when there is typo
Currently, if `FromConfig()` fails silently if there is a typo in the conf file, or if there is a typo in the input benchmark/scenario name. This leads to unexpected bugs or performance degradation because of wrong settings. Instead, `FromConfig()` should shout loudly if anything does not look right.

Some other thoughts:

### We should mark clearly which fields are supposed to be overridden by user.conf

### We should make [main.py](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/main.py) able to load both mlperf.conf and user.conf


yes, can do.
Repository: mlcommons/inference
Issue: MultiStream runtime is >100x longer than other scenarios
Root cause: Unimplemented Feature Gap

This is a suggestion for next version.

Under the rules for v0.5, we require that the MultiStream scenario runs for at least 270366 queries for image benchmarks. At 20fps (and 15fps for ssd-large), this means MultiStream will take 270366/20/60 = **225mins** to run (**300mins** for ssd-large), regardless of how many samples per query or how fast the system runs. This results in runtime which is **more than two orders of magnitude longer than the other scenarios**. From benchmarking perspective, it doesn't make sense that we have such large runtime differences across scenarios.

In Server scenario, we have similar requirements on min_query_count, but since there is only one sample per query, it does not result in such long runtime (and to be honest, I believe that most server product can process >>1 sample/s).

In Offline scenario, we also have similar requirements on minimum samples per query (=24576). This indeed will cause smaller devices to suffer, but at least it is still one order of magnitude shorter than the MultiStream requirements.
Repository: mlcommons/inference
Issue: LoadGen does not check if samples_per_query >= 24576 in Offline scenario
Root cause: Validation Gap

This is a friendly reminder to all of us, and is related to #457 and #475.

Currently, it seems that LoadGen does not check if samples_per_query is not less than 24576, as required by [the rules](https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#scenarios). It does [checks against `min_sample_count`](https://github.com/mlperf/inference/blob/master/loadgen/loadgen.cc#L948), but `min_sample_count` is [set to](https://github.com/mlperf/inference/blob/master/loadgen/test_settings_internal.cc#L136) `std::max<int>(min_query_count, target_sample_count)` in Offline scenario.

And in [mlperf.conf](https://github.com/mlperf/inference/blob/master/v0.5/mlperf.conf), `min_query_count` is set to 1. @psyhtest Do you think we should set `min_query_count` to 24576? I know it does not make sense from the appearance, but in fact LoadGen implicitly assign this value to `min_sample_count` and set `min_query_count` to 1 [here](https://github.com/mlperf/inference/blob/master/loadgen/test_settings_internal.cc#L132).

To workaround this issue, there are things we can do:
- Set `min_query_count` to 24576.
- Set a higher `expected_qps` than the actual to get more samples.
- Set a higher `min_duration` to get more samples.

This may be best fixed by a rules update saying that the total number of samples needs to be at least 24576. That text was written when it was assumed that putting 24k samples in a single query was viable on all systems...
Repository: mlcommons/inference
Issue: TEST04 doesn't generate sufficient queries to make result valid
Root cause: Configuration Error

After apply TEST04A audit.config, the loadgen will only generate limited number of queries,
Before
"pid": 51578, "tid": 51578, "ts": 141398ns : Effective Settings:
"pid": 51578, "tid": 51578, "ts": 141398ns : Scenario : Offline
"pid": 51578, "tid": 51578, "ts": 141398ns : Test mode : Performance
"pid": 51578, "tid": 51578, "ts": 141398ns : samples_per_query : **1980000**

After
"pid": 53865, "tid": 53865, "ts": 712282ns : Effective Settings:
"pid": 53865, "tid": 53865, "ts": 712282ns : Scenario : Offline
"pid": 53865, "tid": 53865, "ts": 712282ns : Test mode : Performance
"pid": 53865, "tid": 53865, "ts": 712282ns : samples_per_query : **50000**

For faster model, it makes the result invalid due to execution time shorter than min duration, and will fail below check in verify_test4_performance.py
        if re.match("Result is", line):
            valid = line.split(": ",1)[1].strip()
            if valid == 'INVALID':
                sys.exit("TEST FAIL: TEST04-A results are invalid")

The TEST04B may have same issue.

I think either having the loadgen to generate more queries, or removing these checks shall be good.

@christ1ne @georgelyuan




Thanks for bringing this up. Since Test04 runs through the performance-sample only once, it will end early. I will remove the check for INVALID results.

> Thanks for bringing this up. Since Test04 runs through the performance-sample only once, it will end early. I will remove the check for INVALID results.

short time also means the performance result might not be as stable as longer runs. Thus using threshold to validate pass or fail is not a valid way.

The threshold has been increased to +-20% for Single stream tests which have very short duration - 0.2ms and is +-10% for other scenarios. The check is in both directions (+ and - ) to accommodate the noise that can be seen for runs of very short times.
Is there a bigger % drift being noticed on the systems?  

@georgelyuan what's the latest on this? Is this fixed?

This is not a problem, it is expected behavior. This audit run does not have a minimum run time requirement.
Repository: mlcommons/inference
Issue: TEST05 script doesn't get score from summary log
Root cause: Validation Gap

In below code of TEST05 verify_performance.py,
        if ref_mode == "Offline":
            if re.match("Samples per second ", line):
                ref_score = line.split(": ",1)[1].strip()

The space after **Samples per second** will cause the script cannot find score, so it always fail. Because in loadgen offline log, there’s no space between ‘d’ and colon, like below.

SUT name : CSUT
Scenario : Offline
Mode     : Performance
Samples per secon**d:** 1666.57

@christ1ne @georgelyuan


Thanks. Thought I fixed this on my end but there may have been some merge issues.
Feel free to update it on your end to see if there are further issues, while I prepare the next release.

Alex, I doublechecked the release and I see that line has already been fixed in both TEST01 and TEST05. Can you doublecheck on your end?

Just pulled again and it works. I think this issue can be closed.
Repository: mlcommons/inference
Issue: 2 false errors from submission-checker.py
Root cause: Validation Gap

Just tried latest repo checker, and there’re 2 potential false errors.

1.**ERROR:main:XXX/mobilenet/Offline/accuracy has no accuracy.txt.** Generate it with accuracy-imagenet.py or accuracy-coco.py or process_accuracy.py
The rule spec doesn’t requires this accuracy.txt (only a json file is listed). The spec is here `https://github.com/mlperf/policies/blob/master/submission_rules.adoc#inference`
    
It seems the scrip expect we redirect the output of accuracy-imagenet.py to a txt file, guessing from here `https://github.com/mlperf/inference/blob/master/SubmissionExample.ipynb`
   ```
 # accuracy run
    ./run_local.sh $@ --scenario $scenario --accuracy --output $output_dir/$scenario/accuracy
    python $cmd --mlperf-accuracy-file $output_dir/$scenario/accuracy/mlperf_log_accuracy.json \
            >  $output_dir/$scenario/accuracy/accuracy.txt
    cat $output_dir/$scenario/accuracy/accuracy.txt
```

So which one is correct, the rules or the script?

2.**ERROR:main:XXX/resnet/Server/accuracy has file list mismatch ([u'mlperf_log_detail.txt', u'mlperf_log_summary.txt', u'accuracy.txt'])**
It seems the script is looking for  mlperf_log_detail.txt and mlperf_log_summary.txt, but per the rules, those 2 files shall be under /performance, like below,
```
performance/
•	run_x/ # 1 run for single stream and offline, 5 otherwise
o	mlperf_log_summary.txt
o	mlperf_log_detail.txt
o	mlperf_log_trace.json
```
I think the script shall be modified to follow rule spec.

And one more question, the rule requires compliance_checker_log.txt, is this just redirecting the output of submission-checker.py to a txt file?

@christ1ne  @schmuell @guschmue 


I raised the same question before as in https://github.com/mlperf/inference/pull/473#commitcomment-35373935

But I think these are minor file adjustment issues. And Vijay has asked Peter to change the rule accordingly in last meeting.

Yes, I know. Bunch of files that we'd want are not in the doc. I field a
issue for it here:
https://github.com/mlperf/policies/issues/6

There is no other record of the accuracy so it would be helpful for the
reviewers to have accuracy.txt,

Its not the end of the world if it is not in the submission ... I could
bulk process this after the submission is closed.


On Thu, Oct 10, 2019 at 6:37 AM AARON ZHONG <notifications@github.com>
wrote:

> I raised the same question before as in #473 (comment)
> <https://github.com/mlperf/inference/pull/473#commitcomment-35373935>
>
> But I think these are minor file adjustment issues. And Vijay has asked
> Peter to change the rule accordingly in last meeting.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/504?email_source=notifications&email_token=AATDG7UGVHHKRYOR5KZ23BDQN4VZRA5CNFSM4I7KKCWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA4LCYA#issuecomment-540586336>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AATDG7U4CKPEIRAFBE7EX2LQN4VZRANCNFSM4I7KKCWA>
> .
>
Repository: mlcommons/inference
Issue: gnmt process_accuracy script
Root cause: Unimplemented Feature Gap

In process accuracy script it seems that the mlperf_log_accuracy.json output should contain bpe32000 decoded sentences as the reference sentences are decoded in line 52:
ref = [e_utils._clean(s, "bpe").split(" ") for s in ref_sentences].
My understanding is that the model itself outputs bpe32000 coded sentences.
If thats the case, should benchmark's code  post processing include this BPE32000 decoding?
Can we implement this decoding in AccuracyOnly mode?
Repository: mlcommons/inference
Issue: CLA issue when submitting pull request

Hi all,

The pull request checker claims that I need to sign the google, CLA, while I'v already done so (using ohada@hailo.ai, and hailo-mlperf@googlegroups.com).
Need your urgent help, so we could submit tomorrow.

Thanks,
Ohad

My advice is to check your email by running: git config --list
If the email you see doesn't much the email you gave when you signed the CLA (probably your Gmail account) then you should change it by, git config --global user.email "your_email@aaa.com"
Hope it helps 

I tried and unfortunately it didn't work

The commit seems to be ok.
@petermattson might be able to check on the CLA.
Repository: mlcommons/inference
Issue: TEST01 not sure what to run in verify_performance
Root cause: Documentation Deficiency

Hi,

**I run this line:**
$ python verify_performance.py -r mobilenet_v1/accuracy/mlperf_log_detail.txt -t mobilenet_v1/performance/mlperf_log_detail.txt 

**And get this response:**
Verifying performance.
reference score = 0
test score = 0
TEST FAIL: Test score invalid

I am not sure if I'm giving the right files to the test. If I'm wrong, what files should I use?

Thanks.

The summary file is mlperf_log_summary.txt
I'll update the documentation. Thanks!
Repository: mlcommons/inference
Issue: ssd-resnet34 cannot run on tensorflow
Root cause: Environment Incompatibility

tensorflow cpu version did not support NCHW version pb.
 
get error like bellow:
InvalidArgumentError (see above for traceback): Default MaxPoolingOp only supports NHWC on device type CPU
         [[node ssd1200/pool1/MaxPool (defined at launch_benchmark.py:105) ]]

This is a duplicate of https://github.com/mlperf/inference/issues/219

@guschmue Thanks. Do you have the tool to convert the NCHW version model to NHWC?

Not that I know of.  In theory you can just use transpose() around ops that are nchw but than you take a performance hit. We have some code in tf2onnx that deals with the reverse here https://github.com/onnx/tensorflow-onnx/blob/master/tf2onnx/optimizer/transpose_optimizer.py but it is a bit more complicated that you'd expect.
Maybe take a look at https://github.com/onnx/onnx-tensorflow ... it might deal with it.

Is there a Graph definition file available for the published TF model that can be shared? (".pbtxt" or ".pb"). Or anyone knows of an easy way to generate it?

Cpu only supports channel_last format model , I update channel_last pb model on https://zenodo.org/record/3708755#.XmszXqgzY2w.
Repository: mlcommons/inference
Issue: Syntax error when running TEST04-A/verify_test4_performance.py
Root cause: Validation Gap

  File "nvidia/TEST04-A/verify_test4_performance.py", line 124
    elif (float(test_score) > float(ref_score) and test_mode == "Single Stream":
                                                                               ^
SyntaxError: invalid syntax

it seems a '(' is forget to delete

@DilipSequeira can you please assign this to George, oddly I can't find him in  the list of assignees. Peter said he added him, maybe you know his id?

Fixed by #516
Repository: mlcommons/inference
Issue: Audit test --- TEST01, unreachable code.
Root cause: Validation Gap

In TEST01/verify_accuracy.py,

(At Line 79~81)
```
if data_perf.size == 0 or data.size == 0:
                if data_perf.size != data_perf.size:
                    num_perf_log_data_mismatch += 1
```

The increment is basically unreachable. The second if statement should be "if data_perf.size != data.size:"
Repository: mlcommons/inference
Issue: Audit test, TEST01 needs a bit different command in `truncate_log.sh` when running on Mac.
Root cause: Environment Incompatibility

At Line 9,
```
sed -i '$ s/,$/]/g' ${log}.new
```
should be replaced by 
```
sed -i "" '$ s/,$/]/g' ${log}.new
```
when running on Mac.

Regards,
Dongsun.
Repository: mlcommons/inference
Issue: loadgen build failed: test_settings_internal.h is missing
Root cause: Validation Gap

cloned the latest repo and trying to build loadgen. But get the error below:

➜  loadgen git:(master) CFLAGS="-std=c++14 -O3" python setup.py bdist_wheel  
running bdist_wheel
running build
running build_ext
building 'mlperf_loadgen' extension
gcc -pthread -B /home/guoheng/anaconda3/compiler_compat -Wl,--sysroot=/ -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c++14 -O3 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/home/guoheng/anaconda3/include/python3.7m -c loadgen.cc -o build/temp.linux-x86_64-3.7/loadgen.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
loadgen.cc:38:10: fatal error: test_settings_internal.h: No such file or directory
 #include "test_settings_internal.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1


Hi，What is your version of GCC，I use version 4.5.0 of GCC，Prompt me when I run this command
c++: error: unrecognized command line option ‘-std=c++14’
Repository: mlcommons/inference
Issue: TEST04-B WAR in main for Offline mode to provide queries which are not from contiguous memory locations
Root cause: Validation Gap

I think we are hitting this same bug. It's not 100% clear yet if it's the same bug that affects us, but it seems reasonable to assume it is related. We're looking into it further to see if it's the same.

For us, this bug isn't a question of different performance for the audit. Rather, TEST04-B makes our Offline tests never complete. Our code (also tensorflow) assumes batched indices in contiguous memory, and getting a non-batched same-index tensor is definitely going to break that.

Shouldn't the audit test not require changing the submitted test code?

>  That's why, after asking WG about this, we got a hint to create a big tensorflow tensor at LoadSamplesToRam and use tensor slice to get a new tensor for any contiguous index detected in the query.

_Originally posted by @mthmsn in https://github.com/mlperf/inference/issues/503#issuecomment-542417311_

Approved WAR from @mthmsn. Should include this fix in reference implementation for next round of MLPerf Inference.

Quoting the WAR proposed by @mthmsn 

> Narrowed down that the problem is with the reference implementation of main.py here: https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/main.py#L284-L303
It calls self.model.predict() once, but then tries to iterate though multiple results in the loop for idx, query_id in enumerate(qitem.query_id), which would normally cause an exception when trying to grab processed_results[idx] for any idx above 0, but it continues running the code since the exception occurred in the finally block of the previous code's exception checking. It wasn't even printing this exception at first.
As a workaround, I can loop the initial call to self.model.predict() with the same input to get the expected number of results. I think this follows the spirit of TEST04-B since it is running the same input multiple times and showing that caching is not happening. This workaround comically cripples the performance, since it essentially becomes much worse than even single-batch mode. Doing this the audit does produce a PASS, but with a warning about TEST04-B being significantly slower than TEST04-A.
I will assume this is a sufficient workaround unless anyone suggests otherwise.

This issue does not appear to have been resolved yet, even though we are now getting ready to submit for v0.7. I will assume my workaround from v0.5 is still sufficient to use for v0.7 unless anyone suggests otherwise.
Repository: mlcommons/inference
Issue: To Run on Windows OS
Root cause: Documentation Deficiency

I want to run MLPerf - Inference Benchmark on a hardware with Windows OS. Can anyone please suggest on building a Windows binary for this? 
If there already is one, Please direct me 
Thank you

The reference code is in Python. You should be able to run it easily. 

For windows build loadgen with
```
python setup.py develop
```
The only difference is that there is no equivalent of (this script)[https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/run_local.sh] 
so you need call main.py with its command line arguments.


Thanks

@aditvas - Did it work for you? If so, can you post the steps here?