Repository: mlcommons/inference
Issue: GNMT is missing server mode user input code
Root cause: Unimplemented Feature Gap

Hi Christine,
 
CC+ Brian A.
 
Thanks for bringing this up. I wanted to check with you this week what the expected user interface for this is. Right now, all loadgen settings are set to values that were convenient for testing.
 
Here are some questions that would help me improve this:
Is the default set of mlperf_loadgen.TestSettings() what we eventually want?
[Christine] not sure about this. Brian can provide the input here.
What settings do we want the user to provide? Just querries_per_sample and latency?
Wasn’t the plan to vary qps and latency until targets aren’t met anymore?
[Christine] We should match the vision app.
Example from the vision app readme:  ./run_local.sh tf resnet50 gpu --count 100 --time 60 --scenario Server --qps 200 --max-latency 0.2
The binary search is nice but I don’t think Brian can implement it anytime soon.
 
Shall I disable any output? The way I process latencies is really bare bones. The loadgen log files should be consulted to get insight in latencies.
[Christine] The minimum is to have the mlperf_log_summary to report the right latencies.
 
Do we need options to select input files/checkpoints for translation?
[Christine] I don’t think so as we don’t have any plan for multiple framework support.
 
Thanks,
Max


@nvmbreughe  @briandersn 

> Is the default set of mlperf_loadgen.TestSettings() what we eventually want?

No, the defaults are meant to help people get started with the loadgen easily. The final settings for submission will depend on the mlperf spec, which is still being finalized.

> The binary search is nice but I don’t think Brian can implement it anytime soon.

Still on my radar, but other issues keep cropping up.

@DilipSequeira please provide ETA.

Hi,

I'm currently OOO. I estimate ETA to be 07/31. Will that work?

@christ1ne, the title of this issue is slightly confusing. Is there something in addition to parameter fixing that needs to happen here? Server mode is already supported through "--scenario Server" option.

@nvmbreughe please feel free to change the title. What I meant is that right now you cannot use --QPS and --max_latency options from the user command line input.

I addressed this in PR #286. 

A few notes/questions:

1. --qps 
   a. Only applies to offline and server. I could easily extend this for the other scenarios, but that is not what the vision app is doing.
   b. The default setting of qps=10 is way to high for offline mode
2. --max-latency. Similar comment: this only applies to server mode, so that we follow vision apps. I could extend this to other scenarios as well
3. There are 3 more settings that vision apps have, but translation doesn't:
    a. --count: I believe this is irrelevant and should be left to 3003, i.e., number of sentences in our test. Instead of hard-coding this number, I could also calculate it by reading the input.
    b. Number of queries. This comes under different option names, depending on the scenario (e.g., "queries-single" for SingleStream). I did not set these, but given that the numbers are the same for all apps, shouldn't we have the number of queries for each scenario be figured out by loadgen?
   c. --time: This is the minimum runtime to run for. But similar comment as 3.b.: 60s seems to be the default for all.

@christ1ne and @briandersn , could you please take a look at these comments and provide your thoughts? Thanks!

looks good to me. Thanks @nvmbreughe 

closing out this issue as it was resolved.