Repository: mlcommons/inference
Issue: Decouple SUT and QSL in C++ interface
Root cause: Architectural Constraint

Hi,

Loadgen's `mlperf::StartTest` function - which accepts qsl and sut parameters - implicitly assumes that before calling `sut->IssueQuery` it first needs to call `qsl->LoadSamplesToRam` for those samples. I think it is a problematic assumption:
1) `mlperf::SystemUnderTest::IssueQuery` doesn't require calling `mlperf::QuerySampleLibrary::LoadSamplesToRam`, `mlperf::SystemUnderTest` class doesn't know anything about any `mlperf::QuerySampleLibrary` class.
2) `mlperf::QuerySampleLibrary::LoadSamplesToRam` function doesn't affect any `mlperf::SystemUnderTest`, again, it doesn't know about this class.

Basically, loadgen assumes a strong link between sut and qsl, while there is no such link in the interface definitions. In some sense, it is asking those inheriting from `mlperf::SystemUnderTest` and `mlperf::QuerySampleLibrary` to implement a backdoor in one of them (or both) for loadgen to function properly.

My proposal is to have QSL returning actual samples:
1) `mlperf::QuerySampleLibrary::LoadSamplesToRam`function should return `std::vector<std::shared_ptr<SampleData>>`
2) mlperf::QuerySample class needs to have `QuerySampleIndex index;` field replaced with `std::shared_ptr<SampleData> data;` field, this way SystemUnderTest::IssueQuery would get the complete info about samples to run inference for. Moreover, it will get this info only, significantly reducing the risk of inadvertent leak of data processing from timed part to untimed one.

SampleData is the new class, with data for the sample stored in the location where queries “arrive from the most logical source, which is usually the network or an I/O device such as a camera”, please see [Inference Rules](https://github.com/mlperf/inference_policies/blob/master/inference_rules.adoc)

We might have HostSampleData class inside loadgen for convenience as probably the vast majority of participants would have sample data in host memory:
```
class HostSampleData: public SampleData
{
public:
     std::vector<unsigned char> data;
}
```
But loadgen doesn't really need to know where is exactly the data located, it gets those samples from QSL and sends them to SUT, which downcasts smart pointers to the SampleData derived class it is capable of working with.

Decoupling QSL and SUT will allow constructing multiple QSLs and SUTs and run different tests with them (provided the samples are of the same type) passing different combinations of qsl and sut to `mlperf::StartTest`, a reasonable thing to do with C++ libraries.

I can work on the PR, but I wonder if timing is right for this? The changes would require those who already inherited from `mlperf::QuerySampleLibrary` and `mlperf::SystemUnderTest` make changes as well, which might be unreasonable burden with v0.5 deadline not that far away?

Thanks,
Max.


Hi Maxim. Thanks for the feedback!

You bring up good points about the coupling between the QSL and SUT. As you alluded to though, I think it will be better to target such a change for the next version of MLPerf Inference.

At this point, I think we'd only want to change the core API if it was blocking a submitter or unfairly handicapping them. If that were the case, I assume the working group would be on board.

I do think we should move towards something like you've proposed. Especially with some of the later rule changes regarding contiguous samples, the loadgen now asks the QSL to load the same sample twice in some cases but then uses the same index to refer to both. Something like your SampleData would allow the loadgen to differentiate between two instances. This isn't strictly necessary for correct functionality, but it is a bit confusing given the current implementation.

Hi Brian,

Sounds great! Let's return to this issue after v0.5 is out.

Thanks,
Max.

@christ1ne : Shall we tag this "WG future version"?

okay!