Repository: mlcommons/inference
Issue: offline mode queries are not contiguous
Root cause: Unimplemented Feature Gap

For offline, image IDs are not contiguous within the query. Is this intended?

This is not a requirement in the spec currently. However, other submitters have asked about contiguous samples in offline mode as well. (Daisy, who works with @christ1ne emailed the mlperf list previously.)

@DilipSequeira does Brian's comment answer your question?

Thanks - it does answer that question, and raises another: where can I find the loadgen spec? I have a google doc

https://docs.google.com/document/d/1iMyhtRfn0l47Mqveed4auAJvWbxSJXWLzK1tNJOBRPM/edit

Is there a copy in github to file issues against?

The goal of Offline is to allow the SUT to maximize throughput. For accelerators, it's somewhat important that the queries be contiguous, and in a typical use case the code loading and decompressing them would ensure that.

Similarly, my recollection of the discussion on Multistream is that we agreed that in a typical use case the marshalling code (from e.g. camera) would also ensure it.


The spec I was referring to was the mlperf spec / strawperson. Unfortunately, I don't have an up-to-date doc for the loadgen; the closest thing would be the comments in test_settings.h which details a lot of the loadgen's contract with the user as implemented.

The loadgen does only produce contiguous samples in the Multistream scenario, but I only added that after it was agreed upon and added to the mlperf spec.

I agree it makes sense to do contiguous for offline too. We should see if there's consensus at the next submitters meeting.






Thanks Brian

The rules are a policy document; it will be useful to have a separate spec for loadgen which is more explicit about the details. Then the process to put an issue on the WG agenda will be to file it against that. 

Something else to discuss in the next WG meeting.

1) WG would like to import loadgen spec into GitHub. Google Doc + LoadGen integration guide. 

2) We should have offline to have contiguous samples in Offline. 

@briandersn please help 1) and 2)

@DilipSequeira volunteered. Thanks! 

@briandersn, I opened PR #332, where I took your suggestion from #325 to run it through clang.

On your other suggestion in #325: I don't think a solution without any interruption in contiguity is possible for the large queries sent in offline mode.. At least not when we have to consider limited memory.

I do have an idea to improve the randomness, as right now, there is a possibility that submitters find patterns. 

Hi @briandersn ,

I was wondering whether it would make sense to make Offline+Accuracy mode contiguous as well? The reasoning is that the SUT should not be aware what mode it is in. Having one mode be contiguous and the other not, would expose this.

I do see several places in the code that explicitly tweak settings for Offline+Accuracy mode, which I don't fully understand. E.g., the following line (line 324) complicates this change:
```
size_t samples_per_query = settings.samples_per_query;
if (mode == TestMode::AccuracyOnly && scenario == TestScenario::Offline) {
    samples_per_query = loaded_sample_set.sample_distribution_end;
  }
```

I am not sure why number of samples per query changes for AccuracyOnly mode.

Yes, it would make sense for Offline+Accuracy mode to be contiguous as well.

The number of samples per query changes for accuracy mode to avoid issuing a ton of duplicate samples.

There are numerous ways for a client to infer accuracy vs. performance, so I don't think we need to worry about preventing that too much - especially since George (of NVIDIA) landed a feature to check accuracy in performance mode for auditing purposes. As long as we don't *require* the client to know what mode it's in, we should be okay.

This is done enough for v0.5