Repository: mlcommons/inference
Issue: multistream target latency is not loaded by loadgen from conf file
Root cause: Configuration Error

  // keys that apply to MultiStream
  lookupkv(model, "MultiStream", "target_latency_percentile", nullptr,
           &multi_stream_target_latency_percentile, 0.01);
  lookupkv(model, "MultiStream", "target_qps", nullptr,
           &multi_stream_target_qps);
  if (lookupkv(model, "MultiStream", "samples_per_query", &val, nullptr))
    multi_stream_samples_per_query = static_cast<int>(val);
  if (lookupkv(model, "MultiStream", "max_async_queries", &val, nullptr))
    multi_stream_max_async_queries = static_cast<int>(val);

but just don't see target_latency is loaded anywhere, so we found that the target latency is set as 150ms, luckily we have backup plan that can initialize the setting options, which is overwritten by conf loading, but as loadgen does not overwrite latency target for multistream, our backup plan just work.

@guschmue this looks urgent to me. thanks

> // keys that apply to MultiStream
> lookupkv(model, "MultiStream", "target_latency_percentile", nullptr,
> &multi_stream_target_latency_percentile, 0.01);
> lookupkv(model, "MultiStream", "target_qps", nullptr,
> &multi_stream_target_qps);
> if (lookupkv(model, "MultiStream", "samples_per_query", &val, nullptr))
> multi_stream_samples_per_query = static_cast(val);
> if (lookupkv(model, "MultiStream", "max_async_queries", &val, nullptr))
> multi_stream_max_async_queries = static_cast(val);
> 
> but just don't see target_latency is loaded anywhere, so we found that the target latency is set as 150ms, luckily we have backup plan that can initialize the setting options, which is overwritten by conf loading, but as loadgen does not overwrite latency target for multistream, our backup plan just work.

so I thought our backup plan work, but it turns out the target latency is calculated in loadgen as

      double target_latency_seconds =
          max_async_queries / requested.multi_stream_target_qps;
don't know why it's set this way, i will look into it.

so we change the max async query to 1 and everything works. So I will cease to track this issue. But there are several target latency config settings in mlperf.conf, people might just rely on that. So keep this issue open in case people need to address it.

@aaronzhongii You're right. I don't see this `multi_stream_target_latency_ns` being used in MultiStream scenario, so maybe not that urgent now. Thanks

Sorry this is a little confusing. multi_stream_target_latency_ns is only used by the unofficial MultiStreamFree scenario. MultiStream QPS and target latency are tightly coupled.

@guschmue : We should make sure multi_stream_max_async_queries is 1 for all submitters.

isn't 1 the default ? Don't think we change it anywhere?

I see it in the log files as 1:
```
closed/mlperf-org/results/tf-gpu/mobilenet/MultiStream/accuracy/mlperf_log_detail.txt
106:"pid": 19137, "tid": 19137, "ts": 27542ns : multi_stream_max_async_queries : 1
closed/mlperf-org/results/tf-gpu/mobilenet/MultiStream/performance/run_1/mlperf_log_detail.txt
106:"pid": 19795, "tid": 19795, "ts": 23224ns : multi_stream_max_async_queries : 1
```