Repository: mlcommons/inference
Issue: Can't run server scenario 
Root cause: Algorithmic Error

Trying to run the server scenario, currently it seems we have a problem in the test settings value,
load gen issues one  query request and then exits. also noticed the following printout:

max_async_queries : 2147483647

is this applicable or there is some problem in loadgen or in my benchmark

i have included the summary log and detailed log
[mlperf_log_summary.txt](https://github.com/mlperf/inference/files/3447292/mlperf_log_summary.txt)

[mlperf_log_detail.txt](https://github.com/mlperf/inference/files/3447293/mlperf_log_detail.txt)

Did someone stumbled on this issue?
thanks,
Reuven


The error message is:

```ERROR : Ending early: Too many outstanding queries."issued" : 1, "outstanding" : 1```

after about 200us.

which is calculated here: https://github.com/mlperf/inference/blob/bb820ae4f7ad61869a9e4d8fa9979006d6b19ab0/loadgen/loadgen.cc#L562-L568

Target QPS is 1, target latency is 100000000ns = 100ms

https://github.com/mlperf/inference/blob/bb820ae4f7ad61869a9e4d8fa9979006d6b19ab0/loadgen/loadgen.cc#L621-L623

In the Server scenario, I think that the correct equation of outstanding queries is `outstanding queries = (queries that should be already processed == issued queries - 1) - completed queries`.

i.e.
```
      size_t queries_outstanding =
          (queries_issued - 1) -
          response_logger.queries_completed.load(std::memory_order_relaxed);
```

@sheferke: Is this still an issue for you?

The max_async_queries isn't applicable in the server scenario, and @riking and @kstreee-furiosa have the correct root cause.

@kstreee-furiosa is working on binary search and part of their task will prevent this early out when it doesn't make sense, like in this case.

In the mean time, a workaround could be to increase the target_qps so it allows for more outstanding queries.
 

Hi all, I'm working to figure out the relation between the number of outstanding queries and target qps. As @briandersn said, If my work is done, the problem will be fixed (cause I will consider this issue too while I work).

Hi @kstreee-furiosa, is this issue fixed yet? Now I also have invalid result and the error is also "Ending early".

Here is my summary output:
```
SUT name : PySUT
Scenario : Server
Mode     : Performance
Scheduled samples per second : 200.84
Result is : INVALID
  Performance constraints satisfied : Yes
  Min duration satisfied : Yes
  Min queries satisfied : NO
Recommendations:
 * The test exited early, before enough queries were issued.
   See the detailed log for why this may have occurred.

================================================
Additional Stats
================================================
Completed samples per second    : 200.82

Min latency (ns)                : 2137088
Max latency (ns)                : 18961126
Mean latency (ns)               : 3722495
50.00 percentile latency (ns)   : 3130896
90.00 percentile latency (ns)   : 5635640
95.00 percentile latency (ns)   : 6608903
97.00 percentile latency (ns)   : 7296470
99.00 percentile latency (ns)   : 8881325
99.90 percentile latency (ns)   : 12789590

================================================
Test Parameters Used
================================================
samples_per_query : 1
target_qps : 200
target_latency (ns): 10000000
max_async_queries : 2147483647
min_duration (ms): 60000
max_duration (ms): 0
min_query_count : 270336
max_query_count : 0
qsl_rng_seed : 0
sample_index_rng_seed : 0
schedule_rng_seed : 0
accuracy_log_rng_seed : 0
accuracy_log_probability : 0
performance_issue_unique : false
performance_issue_same : false
performance_issue_same_index : 0
performance_sample_count : 500

No warnings encountered during test.

1 ERROR encountered. See detailed log.
```
The error message in the detailed log:
```
GeneratedQueries: "queries" : 270336, "samples per query" : 1, "duration" : 1348378898415
ERROR : Ending early: Too many outstanding queries."issued" : 25837, "outstanding" : 11
```



update: after updating LoadGen to commit hash 6fdf78cc61dd7eac6604ce5edc871db32e22b645, the error disappeared and the result is valid now.

@renganxu : Does this mean the bug can be closed?

@briandersn Yes I think the bug is fixed and this issue can be closed.