zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: What's the difference btw many latency in the result?
Root cause: Documentation Deficiency

![image](https://user-images.githubusercontent.com/55476509/65573905-e6b99d00-dfa6-11e9-86a7-f15ac29b8157.png)

There are many latencies.
I know latency definition in the context of mlperf.

![image](https://user-images.githubusercontent.com/55476509/65573953-094bb600-dfa7-11e9-829e-4ccb8885bdbd.png)

What is the difference ? 

1) Min
2) Max
3) Mean
4) 50,90,95,99, 99.9 percentile?
*) What is  "QPS w/ and w/o loadgen overhead"?

Thank you !


Perhaps this will help: https://en.wikipedia.org/wiki/Percentile

Do you have any idea about  "QPS w/ and w/o loadgen overhead"?


There is max latency and min latency in summary.

Since latency is "The latency for a query is the difference between the time time when the LoadGen receives a reply and the time when the query was scheduled."

What is the meaning of those term?

Btw every query latency, most high latency is max and low is min latency ?
Can I understand like this ?

> Do you have any idea about "QPS w/ and w/o loadgen overhead"?
w/  = samples / duration
w/o = 1 / (average latency)

Where:
* Duration includes time spent in the loadgen.
* Average latency does not include time spent in the loadgen.

> Since latency is "The latency for a query is the difference between the time time when the LoadGen receives a reply and the time when the query was scheduled."
> What is the meaning of those term?

scheduled = just before loadgen calls SystemUnderTest::IssueQuery
receives a reply = just after SUT calls QuerySamplesComplete

> Btw every query latency, most high latency is max and low is min latency ?
> Can I understand like this ?

 I think that's correct.
Repository: mlcommons/inference
Issue: Offline results valid but has error in detailed log
Root cause: Validation Gap

I updated the loadgen to the latest commit 5a4803e37b and ran the mobilenet in Offline scenario. The result is VALID in the summary log, but there is error in the detailed log.

So is this a valid run? Or is this a bug in LoadGen that needs to be fixed? @briandersn @guschmue 

```
./run_local.sh tf mobilenet gpu --scenario Offline
```
```
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Offline
Mode     : Performance
Samples per second: 2095.89
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
...
================================================
Test Parameters Used
================================================
samples_per_query : 66000
target_qps : 1000
target_latency (ns): 0
max_async_queries : 1
min_duration (ms): 60000
max_duration (ms): 0
min_query_count : 1
max_query_count : 0
qsl_rng_seed : 0
sample_index_rng_seed : 0
schedule_rng_seed : 0
accuracy_log_rng_seed : 0
accuracy_log_probability : 0
print_timestamps : false
performance_issue_unique : false
performance_issue_same : false
performance_issue_same_index : 0
performance_sample_count : 500

No warnings encountered during test.

1 ERROR encountered. See detailed log.
```

The error information in the detailed log:
```
"pid": 442365, "tid": 442365, "ts": 8187ns : LoadgenVersionInfo:
"pid": 442365, "tid": 442365, "ts": 8187ns : version : .5a1 @ 5a4803e37b
"pid": 442365, "tid": 442365, "ts": 8187ns : build_date_local : 2019-09-25T16:47:37.313196
"pid": 442365, "tid": 442365, "ts": 8187ns : build_date_utc   : 2019-09-25T16:47:37.313219
"pid": 442365, "tid": 442365, "ts": 8187ns : git_commit_date  : 2019-09-24T08:29:25-07:00
"pid": 442365, "tid": 442365, "ts": 8187ns : git_log :
...
"pid": 442365, "tid": 442365, "ts": 424708285ns : GeneratedQueries: "queries" : 2, "samples per query" : 66000, "duration" : 2000000
124 "pid": 442365, "tid": 442365, "ts": 58630628555ns : ERROR : Ending early: Ran out of generated queries to issue before the minimum query count and test duration were reached.

```

> Samples per second: 2095.89

> target_qps : 1000

Increase your `*.SingleStream.target_qps` to 2100. If this happens again, increase it further.

If the actual duration was still over 60 seconds, then the result is still valid, but may be worse than what your SUT is actually capable of doing.

Thanks @riking. This issue is gone now.
Repository: mlcommons/inference
Issue: On recommendation, one said reduce target qps and one said increase target qps~ what should I do ?
Root cause: Algorithmic Error

![image](https://user-images.githubusercontent.com/55476509/65653316-bbd75380-e04f-11e9-89bd-a129c36b83fd.png)

On recommendation, one said reduce target qps and one said increase target qps~ what should I do ?

![image](https://user-images.githubusercontent.com/55476509/65653355-df020300-e04f-11e9-8ab2-40416df3bb97.png)


I think the `Min duration satisfied: NO` was caused by a bug which was fixed by #414. Could you try the latest master?

@kdk3776 : Does this bug still exist in the latest master?

@briandersn  I am not sure about your meaning of latest master~ you mean fork and pull latest version from git ?
Repository: mlcommons/inference
Issue: Is Ending early ERROR in offline scenario false?
Root cause: Algorithmic Error

Below error is just found in offline detail log,
`ERROR : Ending early: Ran out of generated queries to issue before the minimum query count and test duration were reached.
`
But in summary log, it says,
```
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
```

We actually ran 120 seconds and completed 660000 files, so the error in detail log seems false.

In loadgen.cc, there's a check to disable the error, but the duration here seems the query issue duration, not the query completion duration, is it the root cause?

```
    } else {
      if (queries_issued >= settings.min_query_count &&
          duration > settings.min_duration) {
        LogDetail([](AsyncDetail& detail) {
          detail(
              "Ending naturally: Minimum query count and test duration met.");
        });
        ran_out_of_generated_queries = false;
        break;
      }
    }
```
@briandersn @tjablin @guschmue @christ1ne @nvpohanh 

Could you check if this can be fixed by #461?

@nvpohanh Thanks for the prompt response. Just tried but it doesn't solve it, the error is still reported in log. 
The 2 queries becomes 1 now as #461 is supposed to do so.

The error in the detail log definitely appears to be false.

> In loadgen.cc, there's a check to disable the error, but the duration here seems the query issue duration, not the query completion duration, is it the root cause?

I agree that is the root cause. If you change the condition to "duration > settings.target_duration", I think that should fix it. Target duration corresponds to the minimum issue duration for all scenarios. We may even want to rename it min_issue_duration.

@xgu3 can you do a quick verification on if 'duration > settings.target_duration' fix it? If so, please issue an PR. If not, we can make this offline message an exception. 

From another bug, make sure your target_qps is at or above your achievable qps

It's not the root cause, the error happens even setting target_qps as double achievable qps. I think Brian's root cause analysis is correct.

any ETA to fix this issue as xgu3 proposed and Brian approved?

WG: This error is allowed for v0.5 submission Offline scenario. We will try to fix the error for v0.6.

A PR to fix this is here, if we want to pull it in to an allowed hash: #479
Repository: mlcommons/inference
Issue: CAS assertion error in loadgen
Root cause: Resource Mishandling


Sometimes loadgen will assert like below,
mlperf::logging::TlsLogger::Log(mlperf::logging::AsyncLogEntry&&): Assertion `cas_fail_count < 3' failed.

And sometimes it reports 2 CAS errors in log.

We tried comment out the assert and it can continue.

So there're 2 problems,
1. How to avoid the assertion?
2. Is the CAS error in loadgen log needed to be cleaned for submission?

@briandersn @tjablin @guschmue @christ1ne @nvpohanh


How often does this show up?

1) I wonder if you change this [memory_order_relaxed](https://github.com/mlperf/inference/blob/d31c18fbd9854a4f1c489ca1bc4cd818e48f2bc5/loadgen/logging.cc#L817) to memory_order_seq_cst, whether that will avoid hitting the assert. That is the memory order used in the CAS failure branch and is the only thing I can think of that would result in an inconsistent view of the memory access order between the producing and consuming threads. 

2) As long as it doesn't stall there forever, the program should be correct. It is worrying that it ever has to retry more than 3 times though, since that is a sign you could end up with "stuttery" logging overhead which could be bad for long tail performance metrics.

@xgu3  : Were you able to see if using memory_order_seq_cst helps?

@briandersn : I am seeing the same issue and using memory_order_seq_cst in the line you pointed out did not seem to help. I have disabled the assert for now to move forward. The assert happens quite frequently.

Huh. What if you change 3 to 4 or 5? @rachithayp @xgu3 

recommend this to be one of the accepted error in v0.5 submission.

@briandersn @christ1ne Changing value to 6 seems to work better for now. Else we will have to retry a number of times to get a Valid run. I can create a PR with ">=6" value for assert. Let me know if this is acceptable

WG agrees with change.
Repository: mlcommons/inference
Issue: resnetv1.5 onnx model equivalence
Root cause: Documentation Deficiency

In the [Zenodo link](https://zenodo.org/record/2592612#.XY2Es5NKj4M), we have multiple versions


Version 0.1 10.5281/zenodo.2592612 --> latest
Version 0.1 10.5281/zenodo.2541184 --> same model but without softmax. we will submit with this model since softmax is not needed according to the rules

Just want to make this explicit before the deadline. @guschmue already confirmed the model compliance for the 2 versions. 



WG: this is compliant.
Repository: mlcommons/inference
Issue: we should specify mobilenet v1 in readme

currently this link does not show steps for v1 explicitly.
https://github.com/mlperf/inference/blob/master/edge/object_classification/mobilenets/README.md

@christ1ne I actually did this on a branch back in December, but never submitted a PR. Let me rebase and update. 

@christ1ne Please review.

I've also started adding more comments about the "anatomy" of CK benchmarks, but so far only updated the TFLite section.

See further changes in https://github.com/mlperf/inference/pull/57 with instructions for both non-quantized and quantized (asymmetric, training-aware) models.

See many further improvements in #76. I believe this issue can be resolved.
Repository: mlcommons/inference
Issue: Which version of COCO for calibration?
Root cause: Documentation Deficiency

Just to clarify, from which version of COCO are these calibration images?
https://github.com/mlperf/inference/blob/master/calibration/COCO/coco_cal_images_list.txt

I could not find them in val2017, but val2017 is what is referenced here:
https://github.com/mlperf/inference/tree/master/others/cloud/single_stage_detector/pytorch

We use val2017. The calibration might come from train2017.

I've found similar image filenames in val2014 and train2017, and so far they are the same files (example hash match below for the last image in the list).  It would be nice to add clarity here though.

$ md5sum val2014/COCO_val2014_000000511117.jpg 
cd81784d6252c106a8857e595b4b409b  val2014/COCO_val2014_000000511117.jpg
$ md5sum train2017/000000511117.jpg
cd81784d6252c106a8857e595b4b409b  train2017/000000511117.jpg

What do you mean by "we use val2017"?  The provided calibration image list(s) are not the required images to use for calibration?

In general we use coco 2017 for everything.

I can confirm that all the images from coco_cal_images_list.txt are in both val2014 and train1027.

The confusion mainly came from COCO's calibration images are then from its training set while ImageNet calibration images are from its validation set.  Maybe calibration images should always come from the training set?

Its a perfectly valid issue. For some reason we have been specific with imagenet (like imagenet2012) everywhere but not with coco ... we should update the README's to say coco 2017 

IIRC, the WG decided that downloading the whole ImageNet training set just for a hundred  calibration images would be impractical hence using the validation set. For COCO, however, this was fine.
Repository: mlcommons/inference
Issue: user.conf cannot set SingleStream.target_latency < 1
Root cause: Configuration Error

The smallest *.SingleStream.target_latency in user.conf is 1. It means the smallest QPS is 1000, sometimes it makes loadgen cannot generate sufficient queries to meet 60s duration.

I tried to set below 50000 ns of loadgen directly, then loadgen will generate more queries. But setting 0.5 ms in user.conf seems doesn't take effect.

`      --single_stream_expected_latency_ns 500000 `

Is there a way to set <1 ms in user.conf?
Or shall it be marked as an exception of submission to not using user.conf?


I recommend modifying the loadgen with example like --single_stream_expected_latency_ns 500000 to be allowed for v0.5 given the tight timeline. 

That command line flag is interpreted by the loadgen integration, not LoadGen itself, correct? You just need your loadgen integration to apply a more precise latency after loading the standard config.

This should be a reference code only change

Yes, it's interpreted by the loadgen integration and pass the value to lodagen. So do you mean this is allowed for submission?

I think it would be soo much better to teach loadgen to re-use the buffer if it runs out of it.
For the reference app we could do a thing like this (we did not set the latency because we just did not expect that it needs to be set):
```
     if args.max_latency:
         settings.server_target_latency_ns = int(args.max_latency * NANO_SEC)
         settings.multi_stream_target_latency_ns = int(args.max_latency * NANO_SEC)
+       settings.single_stream_expected_latency_ns = int(args.max_latency * NANO_SEC)
```


WG: you can use '--single_stream_expected_latency_ns 500000' in the code you submit to make it v0.5 complaint. Please re-open the issue if needed.
Repository: mlcommons/inference
Issue: coco.py, out of array bound.
Root cause: Validation Gap

When running an `accuracy` mode of `ssd-resnet34` with this command `./run_local.sh tf ssd-resnet34 gpu --scenario SingleStream --accuracy`, loadgen throws an exception as follows:

```
Traceback (most recent call last):
  File "python/main.py", line 547, in <module>
    main()
  File "python/main.py", line 530, in main
    post_proc.finalize(result_dict, ds, output_dir=args.output)
  File "/home/ubuntu/git/mlperf_inference/v0.5/classification_and_detection/python/coco.py", line 189, in finalize
    image_indices.append(self.content_ids[batch])
IndexError: list index out of range
```


It seems that `len(self.results) > len(self.content_ids)` as this line throws the exception: `image_indices.append(self.content_ids[batch])`.

I have printed out the size of the two arrays:

```
self.results size: 5000
self.content_ids size: 0
```

It seems that the following lines (L:148~150) are never executed:
```
for idx in range(0, bs):
            # keep the content_id from loadgen to handle content_id's without results
            self.content_ids.append(ids[idx])
```


How was this resolved? I am facing a similar issue
Repository: mlcommons/inference
Issue: `--samples-per-query` does not increase "samples per query" in Offline mode.
Root cause: Configuration Error

To run an Offline mode, I have executed the following command:
```
./run_local.sh tf mobilenet gpu --scenario Offline --samples-per-query 27000
```

However, the result shows that this execution used only 66 samples per query regardless the number after `--samples-per-query`.

`--qps 2000` option actually can increase the number of samples per query.



@darkrsw You are right. Check out https://github.com/mlperf/inference/issues/457 please. At the moment I am unable to run the performance mode with the required minimum duration and the minimum number of samples.

I've tried `--qps 2000` which increased the number of queries according to the output:
```
$ export EXTRA_OPS="--scenario Offline --qps=2000" && ./run_local.sh tf mobilenet cpu
...
TestScenario.Offline qps=217.36, mean=290.3721, time=607.278, queries=132000, tiles=50.0:279.7469,80.0:473.0812,90.0:541.4182,95.0:575.8801,99.0:603.2422,99.9:607.2080
```
but `samples_per_query` according to `mlperf_log_summary.txt`:
```
================================================
Test Parameters Used
================================================
samples_per_query : 132000
target_qps : 2000
target_latency (ns): 0
max_async_queries : 1
min_duration (ms): 60000
max_duration (ms): 0
min_query_count : 1
max_query_count : 0
qsl_rng_seed : 0
sample_index_rng_seed : 0
schedule_rng_seed : 0
accuracy_log_rng_seed : 0
accuracy_log_probability : 0
print_timestamps : false
performance_issue_unique : false
performance_issue_same : false
performance_issue_same_index : 0
performance_sample_count : 500
```

@psyhtest Agreed.
By defintion, we can adjust only `--samples-per-query` for the submission rather than `--qps`.


The output is misleading:
```
TestScenario.Offline qps=217.36, mean=290.3721, time=607.278, queries=132000, tiles=50.0:279.7469,80.0:473.0812,90.0:541.4182,95.0:575.8801,99.0:603.2422,99.9:607.2080
```
It appears that `queries` should be `samples`, `qps` - `samples per second` (only the same thing when samples per query is 1)?

@guschmue
Repository: mlcommons/inference
Issue: Tag a LoadGen release for v0.5
Root cause: Unimplemented Feature Gap

The [inference_rules.adoc](https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc) states:

> The LoadGen must be compiled from a tagged approved revision of the mlperf/inference GitHub repository without alteration. [...]

Tag a release in this repository, now that we've announced code freeze. I suggest naming it v0.5.0 just in case any critical issues are found that the WG deems worth patching.

proposal:

tag 55c0ea4e772634107f3e67a6d0da61e6a2ca390d as v0.5.0
tag 61220457dec221ed1984c62bd9d382698bd71bc6 as v0.5.1

Approve both of these for submission
Repository: mlcommons/inference
Issue: StartTest input parameters
Root cause: Interface Contract Mismatch

Hi,
**the c_api (under the binfing direcrory) files StartTest has the following signature:**
void StartTest(void* sut, void* qsl, const TestSettings& settings)

**while the loadgen.cc**

void StartTest(SystemUnderTest* sut, QuerySampleLibrary* qsl,
               const TestSettings& requested_settings,
               const LogSettings& log_settings);

it would be better to use the same signature as the loadgen.cc this will enable to set the application to set log level without recompiling the the loadgen library. other wise since default log setting structure is been used  we need to recompile the loadgen every time we need to change the log settings.
Thanks,
Reuven

Hi @sheferke. Feel free to update the bindings and upstream. The bindings aren't officially part of the loadgen and can be modified however you wish. They also aren't subject to the freeze restrictions.
Repository: mlcommons/inference
Issue: Recipe for server mode submission
Root cause: Documentation Deficiency

@guschmue @christ1ne 

To ensure that we generate the correct logs in the right places, what is the expected recipe for server mode submission?

- output of five submission runs?
- output of one submission run and four additional perf runs?
- output of one accuracy run and five perf runs?
- any combination of runs that generates files with appropriate name and content?


My understanding from Brian is that the submission run functionality in LoadGen is incomplete. Therefore, you should submit the output of one accuracy run and five performance runs.

@briandersn Can you confirm?

one accuracy run and five perf runs 


Confirming.

TestSettings::mode == SubmissionRun is incomplete and not tested. Folks should run manually with AccuracyOnly (1x) and PerformanceOnly (5x).

"SubmissionRun" is documented with a TODO still, and I've seen reports of issues when using "SubmissionRun". In hindsight, it would have been better to remove SubmissionRun or make it fail without doing anything.

We should probably fix the submission rules, then.
Current:
```
    performance/
        run_x/ # 1 run for single stream and offline, 5 otherwise
            mlperf_log_summary.txt
            mlperf_log_detail.txt
            mlperf_log_trace.json
    accuracy/
        mlperf_log_accuracy.json
```
Should be:
```
    performance/
        run_x/ # 1 run for single stream and offline, 5 otherwise
            mlperf_log_summary.txt
            mlperf_log_detail.txt
    accuracy/
        mlperf_log_summary.txt
        mlperf_log_detail.txt
        mlperf_log_accuracy.json
```
Does this make sense?

ok will open a PR on this and review it in the next WG meeting
Repository: mlcommons/inference
Issue: can we use diffrenr precision(fp16 or fp32) for mlperf inference.
Root cause: Documentation Deficiency

Can we use different precision for mlperf inference. What precision is used in reference implementations. Does it uses any calibration to convert from default precision (fp32)  to desired precision. 

See https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#81-weight-definition-and-quantization

Thanks for your quick response.
Repository: mlcommons/inference
Issue: What is the file compliance_checker_log.txt used for?

In the submission rules, there is a mandatory file called compliance_checker_log.txt. What should be included in this file? Whether the result is valid or invalid is already in mlperf_log_*. So I don't know what logs should be in compliance_checker_log.txt.

Do we have an official compliance checker? 

Yes, +Guenther Schmuelling <guschmue@microsoft.com> is almost done with it.

On Sun, Oct 6, 2019 at 9:51 AM itay hubara <notifications@github.com> wrote:

> Do we have an official compliance checker?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/487?email_source=notifications&email_token=ABT6DFAOZ27KFKRDSGKQMTDQNHUOVA5CNFSM4I5YU7NKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAOKSQA#issuecomment-538749248>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABT6DFEJRKS3AYI72CYZAETQNHUOVANCNFSM4I5YU7NA>
> .
>


-- 
Vijay Janapa Reddi, Ph. D. | Associate Professor
33 Oxford St. | Maxwell Dworkin 147 | Cambridge, MA 02138
John A. Paulson School of Engineering and Applied Sciences | Harvard
University
Email: vj@eecs.harvard.edu | Web:
http://scholar.harvard.edu/vijay-janapa-reddi


O.K - we don't have a lot of time and there are still issues with this file. Including unneeded JSON files that it checks that they exist - for instance, mlperf_log_summary.json in accuracy folder and also another file that I am not sure how generates 'accuracy.txt'? Anyway, several members raised issues with this checker and since I have another 1-2 working days before the submission it is almost impossible for us to include this file... I think perhaps this file should be submitted later on (when we submit the audits) 

The PR for the checker is here and sitting there for a few days:
https://github.com/mlperf/inference/pull/473