Repository: mlcommons/inference
Issue: Submission checker infers MultiStream from SingleStream incorrectly
Root cause: Algorithmic Error

In the below, the SingleStream results are measured, while the Offline and MultiStream are inferred:
```
INFO    ---
INFO    Results closed/Krai/results/xavier-power.0-fan.cool-armnn-v21.11-neon/resnet50/multistream 78.487476 with power_metric = 1.249195
INFO    Results closed/Krai/results/xavier-power.0-fan.cool-armnn-v21.11-neon/resnet50/offline 18.176100 with power_metric = 22.698287
INFO    Results closed/Krai/results/xavier-power.0-fan.cool-armnn-v21.11-neon/resnet50/singlestream 65.907289 with power_metric = 1.249195
INFO    Results closed/Krai/results/xavier-power.0-fan.cool-tflite-v2.7.1-ruy/resnet50/multistream 92.576335 with power_metric = 2.143931
INFO    Results closed/Krai/results/xavier-power.0-fan.cool-tflite-v2.7.1-ruy/resnet50/offline 13.192100 with power_metric = 28.274067
INFO    Results closed/Krai/results/xavier-power.0-fan.cool-tflite-v2.7.1-ruy/resnet50/singlestream 76.735434 with power_metric = 2.143931
INFO    ---
INFO    ---
INFO    Results=6, NoResults=0
INFO    SUMMARY: submission looks OK

6 errors, 0 warnings, 45 infos
```

From a log, it is clear where the results are taken:
```
================================================
MLPerf Results Summary
================================================
SUT name : TFLite_SUT
Scenario : SingleStream
Mode     : PerformanceOnly
90th percentile latency (ns) : 76429022
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes
Early Stopping Result:
 * Processed at least 64 queries (7914).
 * Would discard 728 highest latency queries.
 * Early stopping 90th percentile estimate: 76735434
 * Early stopping 99th percentile estimate: 92576335 # basis for calculating MultiStream

================================================
Additional Stats
================================================
QPS w/ loadgen overhead         : 13.19
QPS w/o loadgen overhead        : 13.19

Min latency (ns)                : 73076837
Max latency (ns)                : 133407325
Mean latency (ns)               : 75803143 # basis for calculating Offline
50.00 percentile latency (ns)   : 74856925
90.00 percentile latency (ns)   : 76429022
95.00 percentile latency (ns)   : 85099968
97.00 percentile latency (ns)   : 87772700
99.00 percentile latency (ns)   : 91349359
99.90 percentile latency (ns)   : 100716330
```
The Offline result is OK. The MultiStream result should be multiplied by 8.

@psyhtest  Is this an error from running loadgen or the submission checker? What I mean is:

```
* Early stopping 99th percentile estimate: 92576335 # basis for calculating MultiStream
```
Is this result correct and the submission_checker is not inferring the correct result from this? Or is this output supposed to be 8 times higher?

I believe this estimate ("Early stopping 99th percentile estimate") is correct, but since it is for a single sample, it should be multiplied by 8 in the final table.

How was this done before? I see here basically the same computation and the variable `samples_per_query ` seems to be taken from the logs the same way as now. https://github.com/mlcommons/inference/blob/9e89b395ddf6d823c8804d9a00afff26c67a06b9/tools/submission/submission-checker.py#L934-L936




@pgmpablo157321 It's a new rule for v2.0 - never been done before. The formula looks correct to me. Can you please check that `samples_per_query=8`?

> @pgmpablo157321 It's a new rule for v2.0 - never been done before. The formula looks correct to me. Can you please check that `samples_per_query=8`?

`samples_per_query` is actually 1. It is taken from the logs, and since it is running `SingleStream` it becomes 1.

Ah, OK. I'd simply update this variable before the multiplication e.g.:
```
 if (scenario_fixed in ["MultiStream"] and not config.uses_legacy_multistream()) and scenario in ["SingleStream"]: 
     inferred = True 
     samples_per_query = 8
     res = (latency_99_percentile * samples_per_query) / MS_TO_NS 
```

Hi @pgmpablo157321,

The Performance metric is now inferred correctly, thanks!

However, we still have a similar issue for Power. After a discussion in the Power WG, we [concluded](https://github.com/mlcommons/inference/pull/1061#discussion_r798116301) that the MS power metric should be energy consumed per query. Therefore, the actual SS power metric should be multiplied by 8 to get the inferred MS power metric.

@s-idgunji 

> Hi @pgmpablo157321,
> 
> The Performance metric is now inferred correctly, thanks!
> 
> However, we still have a similar issue for Power. After a discussion in the Power WG, we [concluded](https://github.com/mlcommons/inference/pull/1061#discussion_r798116301) that the MS power metric should be energy consumed per query. Therefore, the actual SS power metric should be multiplied by 8 to get the inferred MS power metric.
> 
> @s-idgunji

If I understood correctly this change for the power metric (multiplying by 8) should only be done when inferring MS from SS?


Hi Pablo-  Acknowledging Anton's point .  The power/energy metric should match performance,  so if the latency is per query, the energy should be per query

> However, we still have a similar issue for Power. After a discussion in the Power WG, we [concluded](https://github.com/mlcommons/inference/pull/1061#discussion_r798116301) that the MS power metric should be energy consumed per query. Therefore, the actual SS power metric should be multiplied by 8 to get the inferred MS power metric.