Repository: mlcommons/inference
Issue: MultiStream TestScenario is always INVALID
Root cause: Validation Gap

![Screen Shot 2020-02-13 at 3 10 16 PM](https://user-images.githubusercontent.com/36496141/74487018-4855a780-4e73-11ea-9f02-bddcf8ceb26c.png)
![Screen Shot 2020-02-13 at 3 10 16 PM](https://user-images.githubusercontent.com/36496141/74487745-28bf7e80-4e75-11ea-8fb1-64e459f9fdb8.png)
![Screen Shot 2020-02-13 at 3 10 36 PM](https://user-images.githubusercontent.com/36496141/74487751-2eb55f80-4e75-11ea-80e4-ee5b0590873d.png)
![Screen Shot 2020-02-13 at 3 10 45 PM](https://user-images.githubusercontent.com/36496141/74487758-36750400-4e75-11ea-8454-3b99c2d93efc.png)


Hi @samhithaaaa, your SUT (system-under-test) takes 76.8 ms per query with 4 samples per query, which is why the result gets marked as INVALID. Can you try to reduce the number of samples per query please? (Are you using the v0.5 reference code? With which model?)

For example, for ResNet, each query should be returned within 50 ms.

> Hi @samhithaaaa, your SUT (system-under-test) takes 76.8 ms per query with 4 samples per query, which is why the result gets marked as INVALID. Can you try to reduce the number of samples per query please? (Are you using the v0.5 reference code? With which model?)

Hi,I tried reducing the samples per query to 1,2,3 respectively but doesn't help.

> > Hi @samhithaaaa, your SUT (system-under-test) takes 76.8 ms per query with 4 samples per query, which is why the result gets marked as INVALID. Can you try to reduce the number of samples per query please? (Are you using the v0.5 reference code? With which model?)
> 
> Hi,I tried reducing the samples per query to 1,2,3 respectively but doesn't help.

Yes,I am using v0.5 model and resnet model.

@samhithaaaa Can you share the sequence of instructions you used to run please? How did you reduce the samples per query?

> @samhithaaaa Can you share the sequence of instructions you used to run please? How did you reduce the samples per query?
./run_local.sh tf resnet50 cpu --count 100 --samples-per-query 1 --scenario MultiStream

> @samhithaaaa Can you share the sequence of instructions you used to run please? How did you reduce the samples per query?

This is the command I had given.

> > @samhithaaaa Can you share the sequence of instructions you used to run please? How did you reduce the samples per query?
> 
> This is the command I had given.

I have also tried by giving samples per query upto 4 in interval of 1 like exaple:--samples-per-query 1,--samples-per-query 2,--samples-per-query 2,saples-per-query 3 ,--samples-per-query 4

With those commands, did you see any changes in the log files please? 

No,its the same.I am attaching the screen shot for your
![Screen Shot 2020-02-14 at 10 39 32 AM](https://user-images.githubusercontent.com/36496141/74558107-76d88e80-4f16-11ea-828f-2106a7d4f092.png)
![Screen Shot 2020-02-14 at 10 39 44 AM](https://user-images.githubusercontent.com/36496141/74558120-7cce6f80-4f16-11ea-8caf-b7c19f142119.png)

 reference

I've had another look, and I miscounted the number of digits: your original result was 768 ms, not 76.8 ms. Reducing the number of samples per query works, but gives you 350 ms with 1 sample per query. This is 7 times greater than the required latency constraint for ResNet of 50 ms. Unless you optimise your SUT, the result will be INVALID with any number of samples per query. 

> I've had another look, and I miscounted the number of digits: your original result was 768 ms, not 76.8 ms. Reducing the number of samples per query works, but gives you 350 ms with 1 sample per query. This is 7 times greater than the required latency constraint for ResNet of 50 ms. Unless you optimise your SUT, the result will be INVALID with any number of samples per query.

Yes,it was 768ms.and how do I optimize the SUT?could you please help me on how to process on that?

> > I've had another look, and I miscounted the number of digits: your original result was 768 ms, not 76.8 ms. Reducing the number of samples per query works, but gives you 350 ms with 1 sample per query. This is 7 times greater than the required latency constraint for ResNet of 50 ms. Unless you optimise your SUT, the result will be INVALID with any number of samples per query.
> 
> Yes,it was 768ms.and how do I optimize the SUT?could you please help me on how to process on that?

I have also tried on x_86 architecture that does not work either

Also, with the 350 ms latency you need around 175 queries to satisfy the minimum duration of 60 s.

What's your SUT, if I may ask? My Raspberry Pi 4 is slightly faster with the properly configured TFLite v.1.15 software (which unfortunately wasn't the case for my v0.5 submission.) 

Oh, I know many optimisation techniques from my compiler background: parallelization, vectorization, locality improvement, software pipelining, instruction scheduling... 

> What's your SUT, if I may ask? My Raspberry Pi 4 is slightly faster with the properly configured TFLite v.1.15 software (which unfortunately wasn't the case for my v0.5 submission.)

I am currently using arm architecture using Loadgen  

> > What's your SUT, if I may ask? My Raspberry Pi 4 is slightly faster with the properly configured TFLite v.1.15 software (which unfortunately wasn't the case for my v0.5 submission.)
> 
> I am currently using arm architecture using Loadgen

with tensorflow 1.14 version 

Yes, but with what kind of Arm device?

You may want to take a look at our Collective Knowledge workflows for ArmNN which we used extensively for hundreds of MLPerf Inference v0.5 submissions: https://github.com/ARM-software/armnn-mlperf

The best ResNet result was on the Mali GPU in HiKey960 - just over 200 ms. Still 4 times higher than the MutiStream latency constraint.

> Yes, but with what kind of Arm device?
> 
> You may want to take a look at our Collective Knowledge workflows for ArmNN which we used extensively for hundreds of MLPerf Inference v0.5 submissions: https://github.com/ARM-software/armnn-mlperf

It is ARMV8 architecture customized for our own company purpose

Some processors are just not cut out to meet this constraint. The single-threaded code of ArmNN should be pretty well optimized. Maybe with 16+ Cortex-A-class cores the constraint can be met. 

I think [Ampere Computing](https://amperecomputing.com)'s server-class [products](https://amperecomputing.com/product/) with customized 32 and 64 cores running at up to 3.3 GHz would appear to be sufficiently equipped.