Repository: mlcommons/inference
Issue: ReportLatencyResultsCallback callback implementation 
Root cause: Documentation Deficiency

hi,
 I am working on MLperf submission for a SW+HW solution, my benchmark implementation is C++ based.
I am trying to figure out how to implement the following callback: **ReportLatencyResultsCallback**.
From the name I deduced it is been called by loadgen to retrieve the timing data for all samples of the query sent to the SUT.  But form the callback function signature it seems that it is not the case (const int64_t *), can I get clarification 
for this function usage and implementation. The Python reference example is not using this callback at all, so if there is a C++ benchmark implementation that would be excellent.
,Thanks
Reuven


Hello!
I also have a related question. I am trying to run predictions by overriding `mlperf :: SystemUnderTest.IssueQuery (const std :: vector <mlperf :: QuerySample> & samples)` and  `QuerySampleLibrary.LoadSamplesToRam( const std::vector<mlperf::QuerySampleIndex>& samples)` . However, I do not understand how the indices I get in `LoadSamplesToRam ()` relate to the indices used in the `QuerySample` structure? I am not sure about the Accuracy mode, but in Performance mode one "sample" can be used for prediction multiple times

Hi Reuven, bellycat.

Regarding _ReportLatencyResultsCallback_: That is an optional hook that enables a client to process the raw latencies recorded by the loadgen. It is not necessary to implement it since the loadgen log files will include a summary of the latencies.

Regarding _QuerySampleIndex_: The index in the the QuerySample struct should be the same value that was in the _LoadSamplesToRam_ call; i.e. it is not an index into the |samples| vector argument of _LoadSamplesToRam_.


Hi Brian,
thanks for answering the question, i would like to better understand the mechanism behind time measurement applied loadgen, when and how does time measurement being executed and logged?  is this responsibility of my IssueQuery  callback implementation, i tried to follow the path set by the Python reference example but it seems to be a bit misleading.
in the python code it seems that the measurements start at the beginning of run_one_item() after generating a new item, and stopped after the post_process but i failed to understand how it is been recorded by loadgen.
Thanks,
Reuven

The reference python code has timing code of its own, but that will not be used for submissions. The official stats are tracked within the loadgen internally.

The end timestamp of every sample is taken here:
https://github.com/mlperf/inference/blob/939d5f91df8eb642ab0e530ee0289eb332a1db6c/loadgen/loadgen.cc#L155

That timestamp is submitted to the logging thread, which aggregates the results from all the threads calling QuerySamplesComplete. The latency is recorded on the logging thread here: https://github.com/mlperf/inference/blob/939d5f91df8eb642ab0e530ee0289eb332a1db6c/loadgen/loadgen.cc#L235

At the end of the test, PerformanceSummary::Log takes the latencies (and other timestamps) and calculates metrics applicable to the requested scenario:
https://github.com/mlperf/inference/blob/939d5f91df8eb642ab0e530ee0289eb332a1db6c/loadgen/loadgen.cc#L804

Hi Brian,
thanks for clarifying this point, i would like to check with you two more issues i faced during implementation:

-  in multi stream and offline scenarios i can process all samples delivered during the call IssueQurey callback, lets assume i get 8 samples, i can process this using one thread in a single calculation of my HW  and report the results in one call of  QuerySamplesComplete, how time measurement is calculated for such batch size processing (all calculation are completed within the same time instance)?

- are the good results counter and total result counter as calculate by the python version is needed in the c++ implementation or loadgen calculates this internally.
thanks,
Reuven 

> Hi Brian,
> thanks for clarifying this point, i would like to check with you two more issues i faced during implementation:
> 
> * in multi stream and offline scenarios i can process all samples delivered during the call IssueQurey callback, lets assume i get 8 samples, i can process this using one thread in a single calculation of my HW  and report the results in one call of  QuerySamplesComplete, how time measurement is calculated for such batch size processing (all calculation are completed within the same time instance)?

If multi_stream_samples_per_query is 8, all samples will have the same start and end timestamps.

> * are the good results counter and total result counter as calculate by the python version is needed in the c++ implementation or loadgen calculates this internally.

The loadgen calculates everything it needs for performance metrics internally.

>   thanks,
>   Reuven