Repository: mlcommons/inference
Issue: Count in results.json is different in the Offline scenario
Root cause: Algorithmic Error

In the Offline scenario, the values of count as given via the `--count` flag in the command line and the `"count"` and `"total_items"` keys reported in `results.json` differ.

Let's start with the accuracy mode e.g. use `export EXTRA_OPS="--count 1000 --scenario Offline --accuracy"`

| `--count` | `"count"` | `"total_items"` |
|-|-|-|
| 1 | 1 | 1 |
| 2 | 1 | 2 |
| 8 | 1 | 8 |
| 64 | 2 | 64 |
| 128 | 4 | 128 |
| 256 | 8 | 256 |
| 512 | 17 | 512 |
| 1024 | 33 | 1024 |

I could see some logic until the last two lines: `--count == total_items`; `count == min(1, total_items / 32)`, where 32 is the default `max_batchsize` (contrary to the [README](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection#usage) claiming it is 128). Therefore, `count` probably means `batchcount`.

But why did it suddenly become 17, not 16 for 512?

It's even more puzzling in the performance mode e.g. with `export EXTRA_OPS="--count 1000 --scenario Offline"`:

| --count | "count" | "total_items" |
|-|-|-|
| 1 | 66 | 66 |
| 2 | 132 | 132 |
| 8 | 132 | 132 |
| 64 | 132 | 132 |
| 128 | 256 | 256 |
| 256 | 512 | 512 |
| 512 | 1024 | 1024 |

Here always `count == total_items`, but `count == --count * 2` for `--count >= 128`, and I can't pretend I understand 66 or 132.

I've run the above experiments with the official vision app and LoadGen from `5a4803e3`:
```
$ git log -1
commit 5a4803e37b990c0f294c8a7caa6029fb6f201573
Author: Leo Gordon <ens-lg4@users.noreply.github.com>
Date:   Tue Sep 24 16:29:25 2019 +0100

    On MacOS/LLVM thread_ids are not auto-convertible to uint64_t and have to be converted manually (#439)
```
configuring each run via CK with parameters:
```
$ ck install package --tags=mlperf,image-classification,model,tf,mobilenet,non-quantized
$ export MODEL_DIR=`ck locate env --tags=model,tf,mobilenet,non-quantized`
$ export DATA_DIR=`ck locate env --tags=dataset,imagenet,val`
$ export EXTRA_OPS="--count <N> --scenario Offline [--accuracy]"
$ ./run_and_time.sh tf mobilenet cpu
```
(See [here](https://github.com/ctuning/ck-mlperf/#mobilenet) for more details.)

@psyhtest I think the `count == --count * 2` is caused by a bug introduced by #414. Could you try if #461 fixes this issue?

For the accuracy mode, I wondered if https://github.com/mlperf/inference/pull/406 would fix it:
```
$ git log -1
commit ace74c303888e575826d2a8be8460608d156efae
Merge: 5269902 7f7cd27
Author: christ1ne <41205150+christ1ne@users.noreply.github.com>
Date:   Wed Sep 25 19:45:04 2019 -0700

    Merge pull request #406 from guschmue/gs/fix-no-results
    
    handle queries with empty results
```
Alas, `export EXTRA_OPS="--max-batchsize 32 --count 512 --scenario Offline --accuracy"` I still get `count == 17`. With `export EXTRA_OPS="--max-batchsize 1 --count 512 --scenario Offline --accuracy"`, however, `count == total_items == 512` as expected.


Thanks @nvpohanh. Good news. If I build LoadGen from your branch:
```
$ ck install package --tags=mlperf,inference,source,pr461
$ ck install package --tags=lib,python-package,mlperf,loadgen \
--dep_add_tags.mlperf-inference=pr461 --extra_tags=pr461
```
and run with:
```
$ export EXTRA_OPS="--count 1 --scenario Offline" \
&& ./run_local.sh tf mobilenet cpu > /dev/null 2>&1 \
&& grep \"count\" output/tf-cpu/mobilenet/results.json \
&& grep \"total_items\" output/tf-cpu/mobilenet/results.json
        "count": 66,
        "total_items": 66
```
I now get:

| --count | "count" | "total_items" |
|-|-|-|
| 1 | 66 | 66 |
| 2 | 66 | 66 |
| 8 | 66 | 66 |
| 64 | 66 | 66 |
| 65 | 66 | 66 |
| 66 | 66 | 66 |
| 67 | 67 | 67 |
| 128 | 128 | 128 |
| 256 | 256 | 256 | 
| 512 | 512 | 512 |

So the question is why do I get 66 for `--count == 65` and below? I have even tried with `--time 0`, hoping it would unset the minimum duration, and `--max-batchsize 8`, hoping it would increase the performance and change `count`, but to no avail:
```
$ export EXTRA_OPS="--max-batchsize 8 --time 0 --count 65 --scenario Offline" \
&& ./run_local.sh tf mobilenet cpu > /dev/null 2>&1 \
&& grep \"count\" output/tf-cpu/mobilenet/results.json \
&& grep \"total_items\" output/tf-cpu/mobilenet/results.json
        "count": 66,
        "total_items": 66
```

What does `count` do here? Shouldn't offline scenario only have 1 query for performance-only mode?

the results.json is a thing that pre-dates loadgen. In theory we should not be looking it and use mlperf_log_summary.txt instead but I'll take a look.

Thanks @guschmue. I'm just worried that 2 weeks until the submissions deadline, I still encounter behaviour that I can't explain, which may indicate bugs in the official app (only critical for those who are going to use it) or LoadGen (may be critical for all the submitters).

Are you ok with removing results.json ? That would make it clear that one need to look at the loadgen logs.
The loadgen logs are not very handy to to keep some history of the runs so I use  [this](https://github.com/mlperf/inference/blob/master/v0.5/classification_and_detection/tools/lglog2csv.py) script on top of loadgen.

I don't mind having `results.json`. The problem appears to be in LoadGen:
```
$ export EXTRA_OPS=" --count 65 --scenario Offline" && ./run_local.sh tf mobilenet cpu
$ grep sample output/tf-cpu/mobilenet/mlperf_log_summary.txt
samples_per_query : 66
sample_index_rng_seed : 0
performance_sample_count : 65
```
I've just seen the same `samples_per_query == 66` from our own C++ code with a low `count` (`performance_sample_count`). And it was on a different machine too...

We have noticed a correlation between the value of `*.Offline.min_duration` in `v0.5/mlperf.conf` and the value of `queries` in the output. With `--count == 1`:

| `*.Offline.min_duration` | `samples_per_query` |
|-|-|
| 60000 | 66 |
| 30000 | 33 |
| 20000 | 22 |
| 10000 | 11 |
|  5000 | 5 |
|  1900 | 2 |
|  1850 | 2 |
|  1820 | 2 |
|  1819 | 2 |
|  1818 | 1 |
|  1800 | 1 |
|  1000 | 1 |

Anything special and MLPerf-related happened in [1819](https://en.wikipedia.org/wiki/1819) and [1818](https://en.wikipedia.org/wiki/1818)?

@psyhtest What is your target_qps? 

 @nvpohanh We use the Offline default of 1.

so that makes sense. samples_per_query is calculated by `1.1 * min_duration * target_qps`. The idea is to ensure that samples_per_query is large enough so that the query will take just over min_duration to complete.

@nvpohanh Thanks for pointing that out. I was at my wits end with this puzzling behaviour.

Not sure I agree with the logic of it though. As I said, I didn't set `target_qps` myself. Hence, it's not not indicative of my SUT. I'd rather prefer LoadGen issue a single query and to wait until either `min_duration` is passed or `count` samples are processed. 

But the idea of offline scenario is that SUT receives a huge query with many samples so that it becomes a throughput measurement. If LoadGen issues a single query at a time, we can no longer rely on a lot of assumptions of throughput measurement (like data contiguity, etc.).

Indeed, I'm thinking of a single huge query with contiguous data. At the moment, for `count < 66` I get 66 queries, and then [as many queries as there are samples](https://github.com/mlperf/inference/issues/457#issuecomment-535467916).

I get as many queries as there are samples for `count >= 1` if I set `*.Offline.target_qps = 0.0` in `mlperf.conf`, but again it's not what we want.

Setting `*.Offline.min_query_count = 24576` didn't help either.

Neither did not setting `--count` explicitly:
```
export EXTRA_OPS="--scenario Offline" && ./run_local.sh tf mobilenet cpu
```
with:
```
*.Offline.target_qps = 0.0
*.Offline.target_latency_percentile = 90
*.Offline.min_duration = 60000
*.Offline.min_query_count = 1
*.Offline.samples_per_query = 24576
```
giving me one query with one sample:
```
TestScenario.Offline qps=28.85, mean=0.0116, time=0.035, queries=1, tiles=50.0:0.0116,80.0:0.0116,90.0:0.0116,95.0:0.0116,99.0:0.0116,99.9:0.0116
```

A submitter is supposed to set their own value for *.Offline.target_qps - why are you insisting on sticking with the default?

Also, does samples_per_query function correctly as a maximum cap for the value, or does it do nothing at all?

>  why are you insisting on sticking with the default?

@riking I'm not insisting :). I'm really struggling to make the Offline work. My QPS is roughly 240. If I try to set it explicitly (`mlperf.conf` is not modified):
```
$ export EXTRA_OPS="--scenario Offline --qps=240" && ./run_local.sh tf mobilenet cpu
...
TestScenario.Offline qps=240.98, mean=35.0239, time=65.730, queries=15840, tiles=50.0:35.5553,80.0:54.6040,90.0:60.9465,95.0:64.1066,99.0:65.6538,99.9:65.7052
```
I get 15840 queries - but no indication that 24576 has been reached.

> Also, does samples_per_query function correctly as a maximum cap for the value, or does it do nothing at all?

Doesn't appear to have any effect.

I agree that the actual samples_per_query should be `max(1.1 * min_duration * target_qps, min_query_count)` probably to follow the rules

But wait, looking at the rules, does Offline scenario has requirements on min_query_count? I thought that was the requirements for SingleStream, MultiStream, and Server only

You need at least one query, that's for sure :). So `*.Offline.min_query_count = 1` looks good to me.

One huge query would be ideal, but I'm not getting it no matter what I do :(.

I've tried `--qps 300` which increased the number of queries according to the output:
```
$ export EXTRA_OPS="--scenario Offline --qps=300" && ./run_local.sh tf mobilenet cpu
...
TestScenario.Offline qps=217.33, mean=47.7511, time=91.105, queries=19800, tiles=50.0:48.3166,80.0:74.6993,90.0:83.6093,95.0:88.3084,99.0:91.0210,99.9:91.0745
```
but `samples_per_query` according to `mlperf_log_summary.txt`:
```
================================================
Test Parameters Used
================================================
================================================
Test Parameters Used
================================================
samples_per_query : 19800
target_qps : 300
target_latency (ns): 0
max_async_queries : 1
min_duration (ms): 60000
max_duration (ms): 0
min_query_count : 1
max_query_count : 0
qsl_rng_seed : 0
sample_index_rng_seed : 0
schedule_rng_seed : 0
accuracy_log_rng_seed : 0
accuracy_log_probability : 0
print_timestamps : false
performance_issue_unique : false
performance_issue_same : false
performance_issue_same_index : 0
performance_sample_count : 500
```

So I get a valid number of `samples_per_query` (i.e. over 24,576) if I **lie** about the actual QPS of my SUT:
```
$ export EXTRA_OPS="--scenario Offline --qps=370" && ./run_local.sh tf mobilenet cpu
...
TestScenario.Offline qps=239.42, mean=53.7981, time=103.376, queries=24750, tiles=50.0:53.5663,80.0:84.7040,90.0:94.9856,95.0:99.8842,99.0:103.1668,99.9:103.3477
```
but lying doesn't seem right to me.