Repository: mlcommons/inference
Issue: A way to get performance loadgen statistics in a program
Root cause: Documentation Deficiency

What is the intended way to get statistics from loadgen?

Currently in the [mlperf mobile app](https://github.com/mlcommons/mobile_app_open) we use the `ReportLatencyResults` function to calculate performance from latency.
However, recently this function was [removed](https://github.com/mlcommons/inference/commit/a8a43dcf2ce9f795be637f3126250c1cfa0d736e), so our app is not compatible with the `r2.1` release.

Here is the list of data that we currently gather:
* latency percentile values
* benchmark duration
* total number of finished queries
* result validity

As far as I can see, current loadgen API doesn't provide any way to get these values.
Am I missing something?

I see that percentile values and result validity are available in the `mlperf_log_detail.txt` and `mlperf_log_summary.txt`.
Summary logs are likely intended to be human readable but it seems like detailed log file uses JSON format, so it's probably intended to be parsed by other programs?

Questions:

1. Is the format of `mlperf_log_detail.txt` stable? Can I parse it without worrying that parser will break after updating loadgen to some future version?
2. How can I get total number of finished queries?
3. How can I get benchmark duration without loadgen overhead?

@TheKanter @rnaidu02 
Can one of you help prioritize this issue please. 

@d-uzlov @mkankana 
Sorry for the late reply.

1. `mlperf_log_detail.txt` should be stable for obtaining the following data:
    - latency percentile values
    - benchmark duration (maybe? see answer 3)
    - total number of finished queries
    - result validity
The data LoadGen reports changes from version to version, but the format has been stable since [v1.0](https://github.com/mlcommons/inference_results_v1.0) and these metrics should not change. We parse this file in the submission checker using the `MLPerfLog` class implemented [here](https://github.com/mlcommons/inference/blob/bd1af52c7f912e5fc0531c988a55dee160c431c0/tools/submission/log_parser.py#L26)
2. You can get this value by parsing the line with key `result_query_count` from the logs. For example, one of such lines would be:
```
:::MLLOG {"key": "result_query_count", "value": 12345, ... 
```
3. I am not 100% sure this is beeing reported directly in the `mlperf_log_detail.txt` file. I think a way to compute this is to take the difference between the `time_ms` of two specific rows. A way to compute this would be to take the following rows:
```
:::MLLOG {"key": "generic_message", "value": "Starting performance mode", "time_ms": 0.123456, ...
:::MLLOG {"key": "result_validity", "value": "VALID", "time_ms": 1234567, ...

```
And compute the difference between them. I am not 100% sure of this. Let me run some experiments and I'll update you on this question


@d-uzlov @mkankana 

> 3. How can I get benchmark duration without loadgen overhead?

How did you get this value before the `ReportLatencyResults` was removed?

> How did you get this value before the `ReportLatencyResults` was removed?

We calculated a sum of all values reported by `ReportLatencyResults`.

https://github.com/mlcommons/mobile_app_open/blob/469c2449a59d3195edcbbd42ebc13abda6dcfff8/flutter/cpp/mlperf_driver.h#L101-L112

Taking a look at the code I see you are computing the sum of the latencies. This is not being directly reported in the logs. I see two possible solutions for solving this:
1) You could take the following entries in the logs and compute the desired value:
```
:::MLLOG {"key": "result_query_count", "value": 167279 ... # Number of queries
:::MLLOG {"key": "result_mean_latency_ns", "value": 10742907 ... # Mean latency
```
Multiplying both values should give the sum of all latencies.

2) We could open an issue and request a LoadGen patch to report the sum of the latencies in the logs.

>1. You could take the following entries in the logs and compute the desired value:

This is sufficient for us.

I have tested parsing suggested values from logs, and everything seems to work fine for our use cases.
Thank you so much for your help.