Repository: mlcommons/inference
Issue: The default config file should set the minimum QSL size for each benchmark
Root cause: Architectural Constraint

What's the syntax for that? I tried:
```
*.Offline.performance_sample_count = 1024
```
but this didn't seem to work.

In fact, we probably need something scenario-agnostic e.g.:
```
mobilenet.*.performance_sample_count = 1024
resnet50.*.performance_sample_count = 1024
ssd-mobilenet.*.performance_sample_count = 256
ssd-resnet34.*.performance_sample_count = 64
gnmt.*.performance_sample_count = 3003
```
but would the current parser understand this?

When we did do the config parser we have been debating to allow that 2nd wildcard and did not have a use for it and we did not add those 2 lines - I should have known better :( 

That's a shame. I suppose everybody can set it without using a config file. We can check how it was set from the logs:
```
$ grep "performance_sample_count" mlperf_log_*
mlperf_log_detail.txt:"pid": 6986, "tid": 6986, "ts": 25521ns : *TestSettings (performance_sample_count_override) can override
mlperf_log_detail.txt:"pid": 6986, "tid": 6986, "ts": 38542ns : performance_sample_count : 0
mlperf_log_detail.txt:"pid": 6986, "tid": 6986, "ts": 39584ns : performance_sample_count_override : 0
mlperf_log_summary.txt:performance_sample_count : 0
```
(It wasn't in this case.)

It was in this case:
```
$ grep "performance_sample_count" mlperf_log_*
mlperf_log_detail.txt:"pid": 92777, "tid": 92777, "ts": 7068ns : *TestSettings (performance_sample_count_override) can override
mlperf_log_detail.txt:"pid": 92777, "tid": 92777, "ts": 28407ns : performance_sample_count : 500
mlperf_log_detail.txt:"pid": 92777, "tid": 92777, "ts": 28813ns : performance_sample_count_override : 0
mlperf_log_summary.txt:performance_sample_count : 500
```

As I understand, this is set in the official app [here](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/python/main.py#L510):
```
qsl = lg.ConstructQSL(count, min(count, 500), ds.load_query_samples, ds.unload_query_samples)
```

However, if I run with:
```
$ export EXTRA_OPS="--scenario Offline --count=1024" && time ./run_local.sh tf mobilenet cpu
```
I still get 500:
```
$ grep performance_sample_count output/tf-cpu/mobilenet/*
output/tf-cpu/mobilenet/mlperf_log_detail.txt:"pid": 94268, "tid": 94268, "ts": 14219ns : *TestSettings (performance_sample_count_override) can override
output/tf-cpu/mobilenet/mlperf_log_detail.txt:"pid": 94268, "tid": 94268, "ts": 20003ns : performance_sample_count : 500
output/tf-cpu/mobilenet/mlperf_log_detail.txt:"pid": 94268, "tid": 94268, "ts": 20507ns : performance_sample_count_override : 0
output/tf-cpu/mobilenet/mlperf_log_summary.txt:performance_sample_count : 500
```

With the following patch, however:
```
diff --git a/v0.5/mlperf.conf b/v0.5/mlperf.conf
index cc094e3..9abb54b 100644
--- a/v0.5/mlperf.conf
+++ b/v0.5/mlperf.conf
@@ -7,6 +7,7 @@
 *.SingleStream.target_latency_percentile = 90
 *.SingleStream.min_duration = 60000
 *.SingleStream.min_query_count = 1024
+mobilenet.SingleStream.performance_sample_count_override = 1024
 
 *.MultiStream.target_qps = 20
 *.MultiStream.target_latency_percentile = 99
```
I can see the change:
```
$ export EXTRA_OPS="--scenario SingleStream --count=512" && time ./run_local.sh tf mobilenet cpu
$ grep sample_count output/tf-cpu/mobilenet/*
output/tf-cpu/mobilenet/mlperf_log_detail.txt:"pid": 97001, "tid": 97001, "ts": 9908ns : *TestSettings (performance_sample_count_override) can override
output/tf-cpu/mobilenet/mlperf_log_detail.txt:"pid": 97001, "tid": 97001, "ts": 15782ns : min_sample_count : 512
output/tf-cpu/mobilenet/mlperf_log_detail.txt:"pid": 97001, "tid": 97001, "ts": 15782ns : performance_sample_count : 1024
output/tf-cpu/mobilenet/mlperf_log_detail.txt:"pid": 97001, "tid": 97001, "ts": 16286ns : performance_sample_count_override : 1024
output/tf-cpu/mobilenet/mlperf_log_summary.txt:performance_sample_count : 1024
```

The question is whether we really need two different variables for the config file, `performance_sample_count` (which currently doesn't seem to work) and `performance_sample_count_override` (which works)?

@guschmue So is the following patch the best we can currently do?
```
diff --git a/v0.5/mlperf.conf b/v0.5/mlperf.conf
index cc094e3..640e402 100644
--- a/v0.5/mlperf.conf
+++ b/v0.5/mlperf.conf
@@ -7,6 +7,11 @@
 *.SingleStream.target_latency_percentile = 90
 *.SingleStream.min_duration = 60000
 *.SingleStream.min_query_count = 1024
+mobilenet.SingleStream.performance_sample_count_override = 1024
+resnet50.SingleStream.performance_sample_count_override = 1024
+ssd-mobilenet.SingleStream.performance_sample_count_override = 256
+ssd-resnet34.SingleStream.performance_sample_count_override = 64
+gnmt.SingleStream.performance_sample_count_override = 3903900
 
 *.MultiStream.target_qps = 20
 *.MultiStream.target_latency_percentile = 99
@@ -21,6 +26,11 @@ gnmt.MultiStream.min_query_count = 90112
 gnmt.MultiStream.target_latency = 100
 gnmt.MultiStream.target_qps = 10
 gnmt.MultiStream.target_latency_percentile = 97
+mobilenet.MultiStream.performance_sample_count_override = 1024
+resnet50.MultiStream.performance_sample_count_override = 1024
+ssd-mobilenet.MultiStream.performance_sample_count_override = 256
+ssd-resnet34.MultiStream.performance_sample_count_override = 64
+gnmt.MultiStream.performance_sample_count_override = 3903900
 
 *.Server.target_qps = 1.0
 *.Server.target_latency = 10
@@ -33,8 +43,18 @@ ssd-resnet34.Server.target_latency = 100
 gnmt.Server.min_query_count = 90112
 gnmt.Server.target_latency = 250
 gnmt.Server.target_latency_percentile = 97
+mobilenet.Server.performance_sample_count_override = 1024
+resnet50.Server.performance_sample_count_override = 1024
+ssd-mobilenet.Server.performance_sample_count_override = 256
+ssd-resnet34.Server.performance_sample_count_override = 64
+gnmt.Server.performance_sample_count_override = 3903900
 
 *.Offline.target_qps = 1.0
 *.Offline.target_latency_percentile = 90
 *.Offline.min_duration = 60000
 *.Offline.min_query_count = 1
+mobilenet.Offline.performance_sample_count_override = 1024
+resnet50.Offline.performance_sample_count_override = 1024
+ssd-mobilenet.Offline.performance_sample_count_override = 256
+ssd-resnet34.Offline.performance_sample_count_override = 64
+gnmt.Offline.performance_sample_count_override = 3903900
```

By the way, if I make the above change, should I rename the config file as `user.conf` and run e.g. as follows:
```
$ export EXTRA_OPS="--scenario SingleStream --count=499 --config=user.conf" && time ./run_local.sh tf mobilenet cpu
$ grep config output/tf-cpu/mobilenet/*
output/tf-cpu/mobilenet/mlperf_log_detail.txt:d9c4b77ed961c20bdff2b2a4c7d0904a91408882 Merge pull request #467 from guschmue/gs/config-fix
output/tf-cpu/mobilenet/mlperf_log_detail.txt:62eddd5fa123c0ed7f230868cf7e6d08f49f697c there was a dot missing when parsing items from config file
output/tf-cpu/mobilenet/results.json:    "cmdline": "Namespace(accuracy=False, backend='tensorflow', cache=0, config='user.conf', count=499, data_format=None, dataset='imagenet_mobilenet', dataset_list=None, dataset_path='/datasets/dataset-imagenet-ilsvrc2012-val', find_peak_performance=False, inputs=['input:0'], max_batchsize=32, max_latency=None, model='/home/anton/CK_TOOLS/model-tf-mlperf-mobilenet-downloaded/mobilenet_v1_1.0_224_frozen.pb', model_name='mobilenet', output='/data/anton/mlperf/dividiti-inference/v0.5/classification_and_detection/output/tf-cpu/mobilenet', outputs=['MobilenetV1/Predictions/Reshape_1:0'], profile='mobilenet-tf', qps=None, samples_per_query=None, scenario='SingleStream', threads=20, time=None)",
$ grep sample_count output/tf-cpu/mobilenet/*
output/tf-cpu/mobilenet/mlperf_log_detail.txt:"pid": 102258, "tid": 102258, "ts": 9145ns : *TestSettings (performance_sample_count_override) can override
output/tf-cpu/mobilenet/mlperf_log_detail.txt:"pid": 102258, "tid": 102258, "ts": 14730ns : min_sample_count : 499
output/tf-cpu/mobilenet/mlperf_log_detail.txt:"pid": 102258, "tid": 102258, "ts": 14730ns : performance_sample_count : 1024
output/tf-cpu/mobilenet/mlperf_log_detail.txt:"pid": 102258, "tid": 102258, "ts": 15239ns : performance_sample_count_override : 1024
output/tf-cpu/mobilenet/mlperf_log_summary.txt:performance_sample_count : 1024
```

yes. I can make a change to look model.*.key in the parser but its maybe a little late.
Maybe send a PR for mlperf.conf ?
For user.conf ... we could just make change to first load mlperf.conf and than user.conf where mlperf.conf should never be changed by the user.


I can make a PR now so not to forget. When you implement the parser, we can reduce the number of lines in `mlperf.conf` to the minimum. 

My next question is how to set the QSL size for the Open division, where the models may be custom.

I like the idea of never modifying the standard `mlperf.conf` (`default.conf` would be a more appropriate name), only overriding and appending via `user.conf`.

on my list ...