Repository: mlcommons/inference
Issue: URGENT: LoadGen API broken
Root cause: Interface Contract Mismatch

https://github.com/mlcommons/inference/pull/1184 committed both to master and r2.1 has broken the LoadGen API just 10 days before to the deadline. This must now be accommodated for in submitters workflows.

Moreover, the change is incomplete: `const` must be removed from this line as well
https://github.com/mlcommons/inference/blob/r2.1/loadgen/query_sample_library.h#L37

@psyhtest 
- Could you show any logs of the error?
- Would removing the `const` from the mentioned line solve the issue?
- Would rolling back this changes solve the issue?

@nv-jinhosuh 
- Are these changes necessary for you? 
- Are the currently working ok for you?

This is due to LoadGen Over the Network requirement where QDL which inherits the SUT to dynamically query the name to other node (SUT node) for SUT's name. So QDL->Name() call cannot be const; it cannot simply return const name string like before. Please note that QDL only inherits SUT in LoadGen node as LoadGen node won't have SUT at all. 

I think this change should have been included in the early LoadGen changes for supporting network division. I found it last week that QDL->Name() cannot be done without this change so made an MR.

@psyhtest I think QSL can stay as const as we don't need to change any content when QSL->Name() call happens.

@pgmpablo157321 reverting the MR will make everyone's code working as API stays the same before the MR. 
Anyone who wants to submit to network division may have hard times if the MR is reverted. There is a brutal way of making things mutable and stick with const and make some unusual approaches, but that I think is something we shouldn't do.

Here are the errors I got from our [QAIC](https://github.com/krai/ck-qaic) implementation:
```
g++-9 -c    -I../ -DCK_HOST_OS_NAME2_LINUX=1 -DCK_HOST_OS_NAME_LINUX=1 -DCK_TARGET_OS_NAME2_LINUX=1 -DCK_TARGET_OS_NAME_LINUX=1 -std=c++11 -O3 -ffast-math    -Wall -Wno-sign-compare -std=c++17 -I/opt/qti-aic/dev/inc/ -I/opt/qti-aic/dev/inc/qaic-api-common -I/home/anton/CK-TOOLS/lib-qaic-api-0.3-gcc-9.4.0-master-linux-64/install/ -I/home/anton/CK-TOOLS/lib-protobuf-host-for.qaic-linux-64/install/include/ -I/home/anton/CK-TOOLS/lib-qaic-api-0.3-gcc-9.4.0-master-linux-64/include -I/home/anton/CK-TOOLS/lib-rtl-xopenme-0.3-gcc-9.4.0-linux-64/include -I/home/anton/CK-TOOLS/lib-mlperf-loadgen-static-gcc-9.4.0-compiler.python-3.8.10-linux-64/include  ../harness.cpp  -o harness.o                                                                                                                                                         In file included from ../harness.cpp:46:                                                                                                                                                                        ../harness.h:181:22: error: ‘const string& SystemUnderTestQAIC::Name() const’ marked ‘override’, but does not override                                                                                            181 |   const std::string &Name() const override { return name_; }
      |                      ^~~~
../harness.cpp: In function ‘void TestQAIC(Program*)’:
../harness.cpp:610:23: error: cannot declare variable ‘sut’ to be of abstract type ‘SystemUnderTestQAIC’
  610 |   SystemUnderTestQAIC sut(prg, ts.scenario);
      |                       ^~~
In file included from ../harness.cpp:46:
../harness.h:175:7: note:   because the following virtual functions are pure within ‘SystemUnderTestQAIC’:
  175 | class SystemUnderTestQAIC : public mlperf::SystemUnderTest {
      |       ^~~~~~~~~~~~~~~~~~~
In file included from ../harness.cpp:42:
/home/anton/CK-TOOLS/lib-mlperf-loadgen-static-gcc-9.4.0-compiler.python-3.8.10-linux-64/include/system_under_test.h:41:30: note:       ‘virtual const string& mlperf::SystemUnderTest::Name()’
   41 |   virtual const std::string& Name() = 0;
      |                              ^~~~
In file included from ../benchmark.h:62,
                 from ../harness.cpp:45:
```
We can live with this change if `system_under_test.h` is also properly updated. Otherwise, we will freeze our code at the previous commit.

I believe https://github.com/mlcommons/inference/pull/1184  has all the SUT related calls and API handled. Again, I think QSL is not required for any change (maybe I am wrong, please correct me if that's the case).

[@liorkhe](https://github.com/liorkhe) FYI

@psyhtest @nv-jinhosuh Would making these changes solve the issue?

1. Remove the `const` in this line (in `system_under_test.h`):
https://github.com/mlcommons/inference/blob/1844c6879a60e69889cc67c7f91ff7fd4b6ed09a/loadgen/system_under_test.h#L41
2. Remove the `const` in this line (in `python_api.cc`) https://github.com/mlcommons/inference/blob/1844c6879a60e69889cc67c7f91ff7fd4b6ed09a/loadgen/bindings/python_api.cc#L50
3. Remove the `const` in this line (in `c_api.cc`) https://github.com/mlcommons/inference/blob/1844c6879a60e69889cc67c7f91ff7fd4b6ed09a/loadgen/bindings/c_api.cc#L39

This works for the python tests and the reference implementation. But I haven't been able to reproduce the error you get


@pgmpablo157321 No pablo, basically what's required is already in MR #1184.
Before #1184, it used to be like
```
virtual const std::string& Name() const = 0;
```
`const` for return value is fine. The issue is, what the function touches if it is being called. QDL in LON requires to inherit this SUT via system_under_test.h, and has to make a dynamic call to SUT node (different machine, different process), to get the name back and then it can return it. being `Name() const` there's no easy way of working it around.

And unfortunately this is an API change, as we cannot inherit `virtual const std::string& Name()` (changed API), with previously used `const std::string& Name() const`, likely in everyone's codebase. 
I am very sorry about this inconvenience. But since network division is open this round, this LoadGen change seemed/deemed necessary, in my opinion.

I'm not sure why removing `const` from the QSL has fixed things for me. But it's getting late here, so I'll take another look tomorrow.

@nv-jinhosuh Sure, but I don't mean rolling back the changes you made. For your case my question would be: Will making these changes (keeping #1184) screw something for LON?

And for @psyhtest, I see you are requesting changes in `system_under_test.h`. Is the line `virtual const string& mlperf::SystemUnderTest::Name()` the main issue? Will getting rid of the `const` here fix things? Is it in the QSL or SUT?

@psyhtest If that's the case, it's likely that QSL in your implementation is inheriting the SUT or vice versa, I guess. Theoretically, QSL and SUT is a separate identity where Name() call can co-exist. If that is easier path to everybody, I am all in for making QSL->Name() to be non-const as well, i.e. `virtual const QSL::Name() = 0` 

I am sorry for the inconvenience.

@pgmpablo157321 Keeping changes from #1184 will help anyone with LON and submitting to network division. But it may agitate many others from what Anton @psyhtest suffered. I don't know what's the correct way.

In other words, due to #1184, someone who doesn't submit to network division still needs a hotfix to their code, for the portion of any structure inheriting SUT for Name() call.

FWIW, this round, the network division submission checker requires SUT->Name() to be 'timed' to make sure the network is involved in the flow.

@psyhtest @nv-jinhosuh 
Were you able to solve this issue in your code? I was talking with Ramesh and Tom about this. Would it be of any help to make a commit reverting the changes and allow to use the current version LoadGen or the one with these changes? 

Reverting the changes should look something like [this](https://github.com/mlcommons/inference/compare/r2.1...pgmpablo157321:inference:r2.1_alternative)

If no other people sees issues and if Anton is okay, please do not revert it.

Changing [this](https://github.com/mlcommons/inference/blob/r2.1/loadgen/query_sample_library.h#L37):
```
  /// \brief A human readable name for the model.
  virtual const std::string& Name() const = 0;
```
to:
```
  /// \brief A human readable name for the model.
  virtual const std::string& Name() = 0;
```
should fix it.

To my curiosity, @psyhtest Anton, is there any reason QSL would/should inherit SUT? 

@nv-jinhosuh I'll let you know after the deadline :)

Thank you @psyhtest ! If that's the case, I think we can relax const to QSL as well. I am all for it. This is unfortunately due to QDL specification that requires inheriting SUT and we use Name() call for compliance to network division rule. I didn't see any reason QSL to inherit SUT, so I didn't make that change. 
@pgmpablo157321 if you apply the above change Anton suggested, there shall be a handful of tests and API hooks to be changed together. Just FYI.

Has this issue been solved?  Because I am currently have this error

```
Executing prepared batch file tmp-i7o5dqzr.sh ...

g++ -c    -I../ -DCK_HOST_OS_NAME2_LINUX=1 -DCK_HOST_OS_NAME_LINUX=1 -DCK_TARGET_OS_NAME2_LINUX=1 -DCK_TARGET_OS_NAME_LINUX=1 -std=c++11 -DTF_LITE_1_13 -Wall -Wno-sign-compare -Wno-psabi -I/root/CK-TOOLS/lib-tflite-src-static-gcc-7.5.0-v2.7.1-with.ruy-linux-64/src -I/root/CK-TOOLS/lib-tflite-src-static-gcc-7.5.0-v2.7.1-with.ruy-linux-64/build/flatbuffers/include -I/root/CK-TOOLS/lib-tflite-src-static-gcc-7.5.0-v2.7.1-with.ruy-linux-64/build/tensorflow/absl-cpp/absl -I/root/CK-TOOLS/lib-mlperf-loadgen-static-gcc-7.5.0-compiler.python-3.6.9-linux-64/include  ../classification.cpp  -o classification.o
../classification.cpp:193:22: error: ‘const string& SystemUnderTestSingleStream::Name() const’ marked ‘override’, but does not override
   const std::string& Name() const override { return name_; }
                      ^~~~
../classification.cpp:279:22: error: ‘const string& QuerySampleLibrarySingleStream::Name() const’ marked ‘override’, but does not override
   const std::string& Name() const override { return name_; }
                      ^~~~
../classification.cpp: In function ‘void TestSingleStream(Program*)’:
../classification.cpp:301:31: error: cannot declare variable ‘sut’ to be of abstract type ‘SystemUnderTestSingleStream’
   SystemUnderTestSingleStream sut(prg);
                               ^~~
../classification.cpp:184:7: note:   because the following virtual functions are pure within ‘SystemUnderTestSingleStream’:
 class SystemUnderTestSingleStream : public mlperf::SystemUnderTest {
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../classification.cpp:16:0:
/root/CK-TOOLS/lib-mlperf-loadgen-static-gcc-7.5.0-compiler.python-3.6.9-linux-64/include/system_under_test.h:41:30: note: 	virtual const string& mlperf::SystemUnderTest::Name()
   virtual const std::string& Name() = 0;
                              ^~~~
../classification.cpp:302:34: error: cannot declare variable ‘qsl’ to be of abstract type ‘QuerySampleLibrarySingleStream’
   QuerySampleLibrarySingleStream qsl(prg);
                                  ^~~
../classification.cpp:271:7: note:   because the following virtual functions are pure within ‘QuerySampleLibrarySingleStream’:
 class QuerySampleLibrarySingleStream : public mlperf::QuerySampleLibrary {
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../classification.cpp:15:0:
/root/CK-TOOLS/lib-mlperf-loadgen-static-gcc-7.5.0-compiler.python-3.6.9-linux-64/include/query_sample_library.h:37:30: note: 	virtual const string& mlperf::QuerySampleLibrary::Name()
   virtual const std::string& Name() = 0;
                              ^~~~
```

@xihajun Please provide the output from:
```
$ ck show env --tags=mlperf,inference,source
```
and:
```
$ cd ~/CK/ck-mlperf
$ git remote -v
```

Thanks @psyhtest 
```
Last login: Mon Oct  3 19:24:40 2022 from 86.16.54.227
root@ubuntu-s-1vcpu-1gb-amd-fra1-01:~# ck show env --tags=mlperf,inference,source
Env UID:         Target OS: Bits: Name:                                          Version:     Tags:

85342a416040d29b   linux-64    64 MLPerf Inference source (official v2.1 branch) r2.1-318cb13 64bits,host-os-linux-64,inference,mlcommons,mlperf,r2.1,source,src,target-os-linux-64,uninstalled,v0,v0.1,v0.1.0
root@ubuntu-s-1vcpu-1gb-amd-fra1-01:~# cd ~/CK/ck-mlperf
root@ubuntu-s-1vcpu-1gb-amd-fra1-01:~/CK/ck-mlperf# git remote -v
origin	https://github.com/krai/ck-mlperf (fetch)
origin	https://github.com/krai/ck-mlperf (push)
```
I tried v0.7 here and it worked :)