Repository: mlcommons/inference
Issue: Resnet50 multi-stream takes very long time
Root cause: Configuration Error

I tried to evaluate my multi-stream implementation with resnet50 model and found it took very long time. Then I pulled the latest inference code (commit d405e963d2f62af1ee24047ea870b66decf4e6c9) and tried to run the reference implementation. I ran the TensorFlow resnet50 multi-stream on V100 GPU and it has run for more than one day and it still has not finished yet. Is this expected? Has anyone else run resnet50 multi-stream on V100 and how long does it take?

yes, you are right - just tried and it seems to be way to slow. 
I see the same on the reference  app and on a c++ implementation.
Fyi @briandersn 

Can you attach example the log files and traces? I'm running the multi-stream demo, and it looks okay.

Is the problem that the frequency is too low? Or is it that the test runs way longer than the minimum requested duration and sample count?

Hi @briandersn , I attached the detail log and trace log. I am not sure what frequency you refer to, but the test runs way longer than the minimum duration. The trace file is very large, so I only included the trace up to 24th query. After running for about 1 hour, it reached 4471 query and it is still running. It didn't reach the minimum query count 270336.

So is this speed expected even for the reference implementation? I will run my implementation to see whether it's much faster.

[mlperf_log_detail.txt](https://github.com/mlperf/inference/files/3618711/mlperf_log_detail.txt)
[mlperf_log_trace.txt](https://github.com/mlperf/inference/files/3618715/mlperf_log_trace.txt)


The target_qps is 1 in both the "Requested Settings" (as requested by the loadgen client) and "Effective Settings" as used by the loadgen. It should be 20 to be spec-compliant.

The default value in test_settings.h is 10.
So something is external to the loadgen is setting it to 1.

@guschmue : I see that mlperf.conf sets all target_qps to 1:
https://github.com/mlperf/inference/blob/master/v0.5/mlperf.conf

Can we set it for multi-stream to be conditional based on the model used?

For multi-stream, the loadgen determines the latency constraint by calculating (max_async_queries / target_qps), ignoring target_latency.



I wasn't sure what to do with target qps since in theory we should not need to pass it (noticed issues if I don't touch it so I set it to 1).

Sure we can change it to whatever is needed.
So you want ```1/target_latency``` ?


1 / target_latency, yeah.

ok, I'll send a PR shortly and see if that fixes it.

Seems like the checks are failing :(

yeah, wrong github user. that is why I opened a new PR and closed the old one.

Hi @briandersn, thanks for the help! After setting  MultiStream.target_qps = 20.0, I could run the benchmark successfully and the result is valid with the default MultiStream.samples_per_query = 4. Although my system can support higher samples per query, the benchmark stopped after 270,336 queries for samples_per_query=4. But the rule says LoadGen will use a binary search to find a candidate value. My understanding is that LoadGen should help find the maximum samples per query. So the benchmark should not stop until the maximum samples per query is found, but now it stopped after only one valid result samples_per_query=4. Could you help clarify this?

   

i think there's a --find-peak-performance that you can use. but i am not sure if the result will be consistent enough. Another way to approach it is to manually increase the samples_per_query 

@willc2010 thanks for your suggestion. Yes right now I am manually trying different samples_per_query values. I will try the --find-peak-performance option.

it seems this issue has been resolved. please re-open if needed.

Its not resolved: this https://github.com/mlperf/inference/pull/432 needs to be merged.