Repository: mlcommons/inference
Issue: question about the rule for single stream
Root cause: Configuration Error

I am running mobilenet TF with imagenet dataset, and the result summary is as follows:

MLPerf Results Summary
SUT name : PySUT
Scenario : Single Stream
Mode     : Performance
90th percentile latency (ns) : xxx
Result is : INVALID
  Min duration satisfied : NO
  Min queries satisfied : Yes
Recommendations:
 * Decrease the expected latency so the loadgen pre-generates more queries.

The 90th percentile latency xxx is much smaller than the default 10000 ms Min duration time. What does the recommendation mean? Does it mean we can change the minimum duration time with the --time option in classification_and_detection main.py code? I tried to decrease the --time option value to a number lower than xxx, but then the Min queries will not be satisfied. 

Also in the inference rule, the duration for single stream is 1024 queries and 60 seconds. I think it means we should use the exact 1024 queries and 1 sample per query, right? But what does the 60 seconds mean? I didn't see it in the summary log or other codes as a constraint.



https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc#3-scenarios

10000ms min duration it the minimum time the test needs to run - the rules doc want it to be 60sec.
90th percentile latency is the time per request that mlperf wants in the submission.
```--time 60``` will run it for long enough to satisfy the rules, not giving ```--time``` will use the time from the rules doc (aka 60sec).

I think the recommendation text is not correct in this case.

@guschmue What I do not understand is the 1024 queries should be very fast, say 1 sec, then how can we make it run more than 10000 ms (i.e. 10 sec)? What is the logic here?

By the way, right now the reference code uses 10000 ms if not giving --time , instead of 60 sec. 

 

Ah, it is 'Min duration >=60sec' AND 'num queries >=1024' , else loadgen will say it was an invalid run (which you care about only for submission).
I'll check the reference code to make sure it is in sync with the rules doc.

Yes we may submit the results, so we care that the results are valid.

In the reference code main.py
```
  if args.time:
            # override the time we want to run
            settings.min_duration_ms = args.time * MILLI_SEC
            settings.max_duration_ms = args.time * MILLI_SEC
```
Here should we only override settings.min_duration_ms? Otherwise the result is invalid every time the test is run.


Yes. To make sure both mins are reached, we should avoid setting max_duration and max_queries. Otherwise the loadgen will exit early. The loadgen's recommendation is bogus in this case.

@briandersn  should we set the default single_stream_expected_latency_ns=60*1000000? 


> @briandersn should we set the default single_stream_expected_latency_ns=60*1000000?

Probably not. I think we should increase the default only if it's causing users to run out of memory because too many queries are being pre-generated.

The current default targets 1k QPS without loadgen overhead. If people aren't OOM-ing, maybe we can decrease the default latency instead.

Hi @briandersn @guschmue , I am testing my implementation for submission with resnet50 in single-stream scenario, and the performance mode result is INVALID because "Min duration satisfied : NO". I used the default configuration setting in which min_duration_ms=60000 and min_query_count=1024. The error message in the detailed log is "Ending early: Ran out of generated queries to issue before the minimum query count and test duration were reached. Please update the relevant expected latency or target qps in the TestSettings so they are more accurate."

This message is in IssueQueries() in Loadgen. Based on its implementation, it seems it's possible to run out generated queries before the duration was reached. So I tried to debug the code to stay in IssueQueries() for long enough time, and then my result became VALID. So is this a bug in Loadgen? If not, is there any suggestion to avoid this error? Thanks!

I just found that the submitter can change the default setting SingleStream.target_latency=10. I changed it to a smaller value to let Loadgen pre-generate more queries, and then my result is VALID now. 

So in theory there is no target latency for single stream but I guess loadgen needs it to prep the queries.
in mlperf.conf this should set it to 10ms:
```
*.SingleStream.target_latency = 10
```
but doesn't work for me either. I looked briefly over the code and it looks ok but something fishy here. I'll take a look.

By the way, the mlperf.config specified MultiStream.target_latency_percentile = 90 for multi-stream, but the rule says tail latency is "99% for image models and 97% for otherwise", so should we change 90 to 99 when running Resnet50?

let me check.

For SingleStream.target_latency:
There is only single_stream_expected_latency_ns which we are setting when reading mlperf.conf. I wonder if this PR is releated: https://github.com/mlperf/inference/pull/414