Repository: mlcommons/inference
Issue: Is Loadgen C++ API usable? 
Root cause: Documentation Deficiency

Can anyone tell me: is Loadgen C++ API usable or not yet?
I saw code samples [here](https://github.com/mlperf/inference/tree/master/loadgen/tests) and [here](https://github.com/mlperf/inference/tree/master/loadgen/demos) but don't understand how `loadgen` should work.
As I understood, there is "dispatcher" function
```
void StartTest(SystemUnderTest* sut, QuerySampleLibrary* qsl,
               const TestSettings& requested_settings,
               const LogSettings& log_settings);
```
which manage process.
`SystemUnderTest` is class which loads model to memory and run inference (`IssueQuery` method).
`QuerySampleLibrary` is class which intended to load/unload data to/from memory (`LoadSamplesToRam` and `UnloadSamplesFromRam` methods).
When `IssueQuery` finish inference, then It call `mlperf::QuerySamplesComplete` to send results.
What I don't understand:
1. Who generates `std::vector<QuerySampleIndex>& samples` arguments for methods `LoadSamplesToRam`, `IssueQuery`, `UnloadSamplesFromRam`? (I guess what `loadgen` library can do that, but I'm not sure)
2. How data should be transferred between `QuerySampleLibrary` and `SystemUnderTest` classes? (No any ideas, because all samples have "empty" realizations for  `LoadSamplesToRam` and `UnloadSamplesFromRam` methods). 
3. Who and how should accept data after `mlperf::QuerySamplesComplete`?
4. Is any data-transfer diagram or workflow description may be? I've didn't find any documentation which could me explain this...

Or C++ API is still "under construction" and can't be used?

Thanks in advance for help and advice.

I took a look at [script](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/main.py) and as I have understood it should be some global object(class) which will serve requests from methods of classes `SystemUnderTest` and `QuerySampleLibrary`. And this global object will "to know all" about model, test samles and etc. 

Am I right?

@briandersn, can you comment please? Are we on the right track here? 

Apologies for the incomplete documentation and sample code.

> Can anyone tell me: is Loadgen C++ API usable or not yet?

The C++ API is ready for use. The python bindings are built on top of it and some folks have already integrated it with their systems.

> I saw code samples [here](https://github.com/mlperf/inference/tree/master/loadgen/tests) and [here](https://github.com/mlperf/inference/tree/master/loadgen/demos) but don't understand how `loadgen` should work.
> As I understood, there is "dispatcher" function
> 
> ```
> void StartTest(SystemUnderTest* sut, QuerySampleLibrary* qsl,
>                const TestSettings& requested_settings,
>                const LogSettings& log_settings);
> ```
> 
> which manage process.
> `SystemUnderTest` is class which loads model to memory and run inference (`IssueQuery` method).
> `QuerySampleLibrary` is class which intended to load/unload data to/from memory (`LoadSamplesToRam` and `UnloadSamplesFromRam` methods).
> When `IssueQuery` finish inference, then It call `mlperf::QuerySamplesComplete` to send results.
> What I don't understand:
> 
> 1. Who generates `std::vector<QuerySampleIndex>& samples` arguments for methods `LoadSamplesToRam`, `IssueQuery`, `UnloadSamplesFromRam`? (I guess what `loadgen` library can do that, but I'm not sure)

The loadgen logic does that. See "GenerateQueries" in loadgen.cc.

> 2. How data should be transferred between `QuerySampleLibrary` and `SystemUnderTest` classes? (No any ideas, because all samples have "empty" realizations for  `LoadSamplesToRam` and `UnloadSamplesFromRam` methods).

This is intentionally left undefined since different systems will look very different, depending on how they might preprocess the data. The loadgen only works with sample indeces (QuerySampleIndex), with will be consistent between the  QuerySampleLibrary and SystemUnderTest.

> 3. Who and how should accept data after `mlperf::QuerySamplesComplete`?

QuerySamplesComplete is a global function you can call from anywhere at any time. As long as you provide the correct ResponseId for each sample, the loadgen will know what to do.

> 4. Is any data-transfer diagram or workflow description may be? I've didn't find any documentation which could me explain this...

I'll try to get to this, but will probably focus on more complete example code first.

> 
> Or C++ API is still "under construction" and can't be used?

It's ready.


> 
> Thanks in advance for help and advice.



Thank you