Repository: mlcommons/inference
Issue: Required minimum Query Count not met by user config
Root cause: Validation Gap

For Krai's customary submissions on Arm dev boards, the submission checker reports issues that appear to be related to early stopping e.g.:

```
ERROR   closed/Krai/results/xavier-power.0-fan.cool-tflite-v2.7.1-ruy/resnet50/multistream/performance/run_1/mlperf_log_detail.txt
Required minimum Query Count not met by user config, Expected=7914, Found=1024
```

The corresponding `mlperf_log_summary.txt` appears to be perfectly fine:
```
================================================
MLPerf Results Summary
================================================
SUT name : TFLite_SUT
Scenario : SingleStream
Mode     : PerformanceOnly
90th percentile latency (ns) : 76429022
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes
Early Stopping Result:
 * Processed at least 64 queries (7914).
 * Would discard 728 highest latency queries.
 * Early stopping 90th percentile estimate: 76735434
 * Early stopping 99th percentile estimate: 92576335

================================================
Additional Stats
================================================
QPS w/ loadgen overhead         : 13.19
QPS w/o loadgen overhead        : 13.19

Min latency (ns)                : 73076837
Max latency (ns)                : 133407325
Mean latency (ns)               : 75803143
50.00 percentile latency (ns)   : 74856925
90.00 percentile latency (ns)   : 76429022
95.00 percentile latency (ns)   : 85099968
97.00 percentile latency (ns)   : 87772700
99.00 percentile latency (ns)   : 91349359
99.90 percentile latency (ns)   : 100716330

================================================
Test Parameters Used
================================================
samples_per_query : 1
target_qps : 12.5
target_latency (ns): 0
max_async_queries : 1
min_duration (ms): 600000
max_duration (ms): 0
min_query_count : 1024
max_query_count : 0
qsl_rng_seed : 6655344265603136530
sample_index_rng_seed : 15863379492028895792
schedule_rng_seed : 12662793979680847247
accuracy_log_rng_seed : 0
accuracy_log_probability : 0
accuracy_log_sampling_target : 0
print_timestamps : 0
performance_issue_unique : 0
performance_issue_same : 0
performance_issue_same_index : 0
performance_sample_count : 1024

No warnings encountered during test.

No errors encountered during test.
```

Same for inferred Offline and MultiStream results:
```
$ grep ERROR $HOME/projects/mlcommons/inference_results_v2.0_Krai/closed/Krai/submission_checker.log | grep -v WARNING
ERROR   closed/Krai/results/xavier-power.0-fan.cool-tflite-v2.7.1-ruy/resnet50/multistream/performance/run_1/mlperf_log_detail.txt Required minimum Query Count not met by user config, Expected=7914, Found=1024
ERROR   closed/Krai/results/xavier-power.0-fan.cool-tflite-v2.7.1-ruy/resnet50/singlestream/performance/run_1/mlperf_log_detail.txt Required minimum Query Count not met by user config, Expected=7914, Found=1024
ERROR   closed/Krai/results/xavier-power.0-fan.cool-tflite-v2.7.1-ruy/resnet50/offline/performance/run_1/mlperf_log_detail.txt Required minimum Query Count not met by user config, Expected=7914, Found=1024
ERROR   closed/Krai/results/xavier-power.0-fan.cool-armnn-v21.11-neon/resnet50/multistream/performance/run_1/mlperf_log_detail.txt Required minimum Query Count not met by user config, Expected=10904, Found=1024
ERROR   closed/Krai/results/xavier-power.0-fan.cool-armnn-v21.11-neon/resnet50/singlestream/performance/run_1/mlperf_log_detail.txt Required minimum Query Count not met by user config, Expected=10904, Found=1024
ERROR   closed/Krai/results/xavier-power.0-fan.cool-armnn-v21.11-neon/resnet50/offline/performance/run_1/mlperf_log_detail.txt Required minimum Query Count not met by user config, Expected=10904, Found=1024
```

@ckstanton Would you agree that this ERROR is spurious?

@psyhtest I agree, this error shouldn't be happening.  I'll take a look at the submission checker.

@psyhtest @ckstanton it is because this comparison i'm making here https://github.com/mlcommons/inference/blob/ca3e3823834f0e7f7b900a8391b50cef32ad4180/tools/submission/submission-checker.py#L975-L982
Here, I am taking the `required_min_query_count` for the early stopping scenario from the logs (as the number of queries that where actually made). I though this will always pass when the logs were consistent. What configuration did you use here @psyhtest? Should we remove this comparison for the case of early stopping @ckstanton?

Also agree this shouldn't be happening

Since for early stopping we just care that minimum time duration is met and that early stopping says the run was successful, I think it doesn't make sense to check min_query_count in this case.  So yes @pgmpablo157321 , it makes sense to remove this comparison.  Thanks