Repository: mlcommons/inference
Issue: TEST04-B break offline contiguous mode
Root cause: Validation Gap

have you tried your test on your own implementation? commit c36d1f3a0404c351de45217a9d8f33925ac99caa, is contributed by NVidia nvmbreughe to fix the non-contiguous indexes sent from loadgen in offline mode. But your test04-b will break the contiguous mode, as loadgen will repeatedly send down same index, instead of contiguous indexes as before. 

To fix this:
1) loadgen need to call LoadSamplesToRam with same indexes
2) or test04 is trying to see if caching is performed. Assume an implementation with caching trick employed, 04-B performance will be better than 04-A, thus change the python script to only check if the performance of 04-B is better than 04-A instead of checking both side as now.

And the best place to check if caching is done is the latency percentile. But latency might not be a stable value, thus the validation method shall not be as brutal as using a simple threshold. 

For MultiStream scenario, i still see unique index sent down instead of same index though.


@aaronzhongii we have tried the tests internally and have not seen problems.

The test does not break the contiguous mode for Multi-stream. In case of Multi-stream, when samples_per_query < perf_sample_count, same query is repeatedly issued - samples within the query are from contiguous memory locations. As the readme points out, if samples_per_query >perf_sample_count, Test04 is not required for multi-stream mode as both Test04-A and test04-B will be identical.

Since audit tests are meant to stress SUT in newer ways, it is understood that test04-B can cause degraded performance as samples are not being read from contiguous memory locations. The check of performance in both directions + or - 10% is for such reason as mentioned as well as to address the noise that is seen due to shorter runtimes of the test. 
Are you observing drifts larger than +-10% on your implementations? 

Regarding latency percentile, single stream is already checking 90th percentile latency which is also the metric used to score this scenario. For server, samples/second is a more stable metric, as you said and hence that is being used by the validation method.

We actually don't have issue with MultiStream mode, as contiguous index is somewhat maintained. 
But we have problem in Offline scenario, so can you explain a little bit in detail how contiguous index mode is maintained in Offline mode?
As for the performance drop, it's about 40% from A to B because we have to call tensorflow session run for each sample instead of sending down the whole batch as performance count of QSL. 
For Server scenario, the target_qps is selected by the submitter, thus the variation from A to B won't be as huge if any caching technic is employed. But, you can catch the cheating by studying the latency closely.

Aaron
------------------------------------------------------------------
From:Ritika Borkar <notifications@github.com>
Sent At:2019 Oct. 10 (Thu.) 16:59
To:mlperf/inference <inference@noreply.github.com>
Cc:Aaron <aaron.zhong@alibaba-inc.com>; Mention <mention@noreply.github.com>
Subject:Re: [mlperf/inference] TEST04-B break offline contiguous mode (#503)

@aaronzhongii we have tried the tests internally and have not seen problems.
The test does not break the contiguous mode for Multi-stream. In case of Multi-stream, when samples_per_query < perf_sample_count, same query is repeatedly issued - samples within the query are from contiguous memory locations. As the readme points out, if samples_per_query >perf_sample_count, Test04 is not required for multi-stream mode as both Test04-A and test04-B will be identical.
Since audit tests are meant to stress SUT in newer ways, it is understood that test04-B can cause degraded performance as samples are not being read from contiguous memory locations. The check of performance in both directions + or - 10% is for such reason as mentioned as well as to address the noise that is seen due to shorter runtimes of the test.
 Are you observing drifts larger than +-10% on your implementations?
Regarding latency percentile, single stream is already checking 90th percentile latency which is also the metric used to score this scenario. For server, samples/second is a more stable metric, as you said and hence that is being used by the validation method.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.



I read test-04 readme carefully again, here i quote

"(TEST04-B) Issue same sample P times: In Offline scenario, the same sample is repeated P times to fill the query."

which will break contiguous index mode for Offline. And I do open this issue for Offline mode, as in the title of the issue.

 Contiguous mode is NOT maintained in Offline scenario as I said earlier and it is understood that since this is an audit test to stress SUT, it is allowed to notice performance degradation that can occur due to reading samples from non-contiguous locations.


> Contiguous mode is NOT maintained in Offline scenario as I said earlier and it is understood that since this is an audit test to stress SUT, it is allowed to notice performance degradation that can occur due to reading samples from non-contiguous locations.

ok, nice to know about that. then there is no point to validate the percentage of the performance degradation, right?

> As for the performance drop, it's about 40% from A to B because we have to call tensorflow session run for each sample instead of sending down the whole batch as performance count of QSL.

Test4-B also forms a whole batch as performance count of QSL using the same sample repeatedly in GenerateQueries  as implemented [here](https://github.com/mlperf/inference/blob/master/loadgen/loadgen.cc#L392).

I will edit the validation check to waive Test04-B being slower than test04-A as pass. I think that should resolve this concern. 

> > As for the performance drop, it's about 40% from A to B because we have to call tensorflow session run for each sample instead of sending down the whole batch as performance count of QSL.
> 
> Test4-B also forms a whole batch as performance count of QSL using the same sample repeatedly in GenerateQueries as implemented [here](https://github.com/mlperf/inference/blob/master/loadgen/loadgen.cc#L392).

The query from mlperf is a collection of indices. And LoadSamplesToRam load all image data to memory with a specific sequence. When query comes, if implementation is using tensorflow as framework, it needs to convert the indices into tensor, as tensorflow only takes tensor instead of indices. So the primitive way to solve that is, as in reference implementation, to copy data from loaded memory to the new tensor created for this query by looking up the data in the memory through indices. This will introduce cpu overhead of moving data from one DRAM location to another DRAM, this is because tensorflow does not open memory allocator clearly for chips neither cpu or gpu. 

This cpu overhead is timed if implementation employs it, which will affect the final result and does not reflect the real performance of the accelerator in the real case. Because in the real case, non index will be sent down, only data will be. Reshuffling data is ridiculous in the real case.

That's why, after asking WG about this, we got a hint to create a big tensorflow tensor at LoadSamplesToRam and use tensor slice to get a new tensor for any contiguous index detected in the query. 

Thus for your offline mode batch, which does not match with the data layout in the memory for QSL, the big batch is actually a multiple of 1 batch, which we have to break them into. 

> I will edit the validation check to waive Test04-B being slower than test04-A as pass. I think that should resolve this concern.

thanks. much appreciated.

I think we are hitting this same bug. It's not 100% clear yet if it's the same bug that affects us, but it seems reasonable to assume it is related. We're looking into it further to see if it's the same.

For us, this bug isn't a question of different performance for the audit. Rather, TEST04-B makes our Offline tests never complete. Our code (also tensorflow) assumes batched indices in contiguous memory, and getting a non-batched same-index tensor is definitely going to break that.

Shouldn't the audit test not require changing the submitted test code?

>  That's why, after asking WG about this, we got a hint to create a big tensorflow tensor at LoadSamplesToRam and use tensor slice to get a new tensor for any contiguous index detected in the query.