Repository: mlcommons/inference
Issue: Is INVALID server result false after using mlperf.conf?
Root cause: Configuration Error


After using mlperf.conf, the server log will report invalid like following,
```
================================================
MLPerf Results Summary
================================================
SUT name : CSUT
Scenario : Server
Mode     : Performance
Scheduled samples per second : 666.66
Result is : INVALID
  Performance constraints satisfied : NO
  Min duration satisfied : Yes
  Min queries satisfied : Yes
Recommendations:
 * Reduce target QPS to improve latency.

================================================
Additional Stats
================================================
Completed samples per second    : 666

Min latency (ns)                : 10011203
Max latency (ns)                : 24560917
Mean latency (ns)               : 10751228
50.00 percentile latency (ns)   : 10532956
90.00 percentile latency (ns)   : 11468163
95.00 percentile latency (ns)   : 11934767
97.00 percentile latency (ns)   : 12281803
99.00 percentile latency (ns)   : 13032638
99.90 percentile latency (ns)   : 14451148
```

The problem is that the 99% latency is 13.02ms per log and less than 15ms, which meets mperf model requirements.

I tried set below in user.conf, and it pass.
`*.Server.target_latency = 15`

But seems the correct value source shall be the below line in the mlperf.conf, so that mobilenet can have another line to specify different value.
`resnet50.Server.target_latency = 15`

If this is correct, shall below settings.target_latency.count() be changed to use different line in mlperf.conf?
```
    case TestScenario::Server:
      ProcessLatencies();    
      if (target_latency_percentile.sample_latency >
          settings.target_latency.count()) {
        *recommendation = "Reduce target QPS to improve latency.";
        perf_constraints_met = false;
      }
```

@briandersn @tjablin @guschmue @christ1ne @nvpohanh


If we cannot resolve this issue by Friday (9/27) code freeze, I recommend we just implement the workaround in user.conf

---------------
I tried set below in user.conf, and it pass.
*.Server.target_latency = 15
---------------

This should be already set here:
https://github.com/mlperf/inference/blob/master/v0.5/mlperf.conf#L25


@guschmue Different model has different latency bound, so I think below line is designed to be used for resnet50 to check whether the result is valid or not.  
`resnet50.Server.target_latency = 15`

Another line is needed for mobilenet, 
`mobilenet.Server.target_latency = 10`

Below line is used for loadgen to generate queries.
`*.Server.target_latency = 15`

Or is there other usage of resnet50.Server.target_latency = 15?

The '*' matches all models but a more qualified path takes precedence.
Kind of reads as:
what ever is not in this list
```
resnet50.Server.target_latency = 15
ssd-resnet34.Server.target_latency = 100
gnmt.Server.target_latency = 250
```
will take this setting:
```
*.Server.target_latency = 10
```


@guschmue the problem is that resnet50.Server.target_latency = 15 doesn't take effect. As described in the very beginning. 

Hm, that would be bad. Let me check.

thanks for finding - its a pretty bad one since it doesn't only apply to the target_latency.

@xgu3 please verify and let me know if we can close this issue.