Repository: mlcommons/inference
Issue: Help with submission checker
Root cause: Documentation Deficiency

When running `submission-checker.py` I get several errors. Any assistance to any of them will be much appreciated:
```
1. ERROR:main:open/deci/results/openvino-cpu/resnet/Offline/accuracy not hash value for mlperf_log_accuracy.json
2. ERROR:main:open/deci/results/openvino-cpu/resnet/Offline/accuracy/mlperf_log_accuracy.json is not truncated
3. WARNING:main:open/deci/results/openvino-cpu/resnet/Offline/accuracy, accuracy not valid but taken for open
4. INFO:main:open/deci/results/openvino-cpu/resnet/Offline/accuracy, accuracy is 75.896
5. ERROR:main:open/deci/results/openvino-cpu/resnet/Offline/performance/run_1/mlperf_log_summary.txt performance_sample_count, found 1024, needs to be > 500
6. ERROR:main:open/deci/results/openvino-cpu/resnet/Offline/performance/run_1 has issues
7. WARNING:main:open/deci/results/openvino-cpu/resnet/SingleStream ignoring scenario SingleStream (neither required nor optional)
8. WARNING:main:open/deci/results/openvino-cpu/resnet ignorning missing scenarios in open division ({'Offline', 'Server'})
9. ERROR:main:NoResults open/deci/results/openvino-cpu/resnet/Offline
10. ERROR:main:NoResults open/deci/results/openvino-cpu/resnet/SingleStream
11. INFO:main:Results=0, NoResults=2
12. ERROR:main:SUMMARY: submission has errors
```
1+2. any workaround? these are files created in the official main.py
3+4. I don't understand these messages since the accuracy is sufficient for Resnet
5. is the message wrong? o.w. - found 1024, needs to be > 500. AFAIK this is defined in the main.py therefore I should not change it. Any idea?
7-10. Why does it skip SingleStream although results DO exist?


The submission checker looks for logs that have already been truncated with the script:
`https://github.com/mlperf/inference/blob/master/tools/submission/truncate_accuracy_log.py`

Regarding line 5: you need to change `main.py` to have 1024 instead of 500.

It is okay to change contents of `main.py` as long as you're following the rules. In fact it is required to modify `main.py` in some cases (e.g., if your expected latency is less than 1ms, then `SingleStream.target_latency` has to be set manually in `main.py` since the `user.conf` file can only accept integer multiples of 1 ms).

Also, are you using r0.7 branch? Master branch is a little out-dated at this point

> 1. ERROR:main:open/deci/results/openvino-cpu/resnet/Offline/accuracy not hash value for mlperf_log_accuracy.json
> 2. ERROR:main:open/deci/results/openvino-cpu/resnet/Offline/accuracy/mlperf_log_accuracy.json is not truncated

Please use the truncation script to truncate the accuracy results: https://github.com/mlperf/inference/blob/master/tools/submission/truncate_accuracy_log.py

> 3. WARNING:main:open/deci/results/openvino-cpu/resnet/Offline/accuracy, accuracy not valid but taken for open
> 4. INFO:main:open/deci/results/openvino-cpu/resnet/Offline/accuracy, accuracy is 75.896

This is fine for Open division.

> 5. ERROR:main:open/deci/results/openvino-cpu/resnet/Offline/performance/run_1/mlperf_log_summary.txt performance_sample_count, found 1024, needs to be > 500
> 6. ERROR:main:open/deci/results/openvino-cpu/resnet/Offline/performance/run_1 has issues

You need to set the performance sample count (or a.k.a. QSL size) to be at least 1024.

> 7. WARNING:main:open/deci/results/openvino-cpu/resnet/SingleStream ignoring scenario SingleStream (neither required nor optional)
> 8. WARNING:main:open/deci/results/openvino-cpu/resnet ignorning missing scenarios in open division ({'Offline', 'Server'})

Does the directory `open/deci/results/openvino-cpu/resnet/SingleStream` exist?

> 9. ERROR:main:NoResults open/deci/results/openvino-cpu/resnet/Offline
> 10. ERROR:main:NoResults open/deci/results/openvino-cpu/resnet/SingleStream
> 11. INFO:main:Results=0, NoResults=2
> 12. ERROR:main:SUMMARY: submission has errors

Finally, please use the checker on r0.7 branch.

**Thank you for your help! this saved a lot of time.**
I am down to these messages:
```
WARNING:main:open/deci/results/openvino-cpu/resnet/SingleStream ignoring scenario SingleStream (neither required nor optional)
ERROR:main:NoResults open/deci/results/openvino-cpu/resnet/SingleStream
INFO:main:Results=1, NoResults=1
ERROR:main:SUMMARY: submission has errors
```
The directory `open/deci/results/openvino-cpu/resnet/SingleStream` exists and I am running on the most recent r0.7 branch.
Any other ideas?
thanks again

It seems to ignore SingleStream... could you share the full log? Or maybe just anything that has "open/deci/results/openvino-cpu/resnet/SingleStream" in the line.

Yes, there is a PR for this here https://github.com/mlperf/inference/pull/732.
@christ1ne, we'd badly need to merge this one.


Try the r0.7 branch ... it should be merged now.

can I close this @sefibk