Repository: mlcommons/inference
Issue: Submission checker issue: Results are counted by the submission checker but rejected by the Web UI
Root cause: Validation Gap

When we run submission checker on our submission folder we see this. 
```
[2023-03-07 11:40:19,054 submission_checker.py:2651 INFO] ---
[2023-03-07 11:40:19,054 submission_checker.py:2652 INFO] Results=1957, NoResults=0
[2023-03-07 11:40:19,054 submission_checker.py:2658 INFO] SUMMARY: submission looks OK
/Users/arjun/CM/repos/local/cache/ac01ab3b22344c23/inference/tools/submission/generate_final_report.py:227: FutureWarning: save is not part of the public API, usage can give unexpected results and will be removed in a future version
  writer.save()
```

The submission checker also reports the below errors (which actually gets lost among other outputs)
```
[2023-03-07 11:45:30,622 submission_checker.py:2441 ERROR] open/cTuning/measurements/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/mobilenet-v2-precision_uint8-0.5-160/multistream is missing mlperf.conf
[2023-03-07 11:45:30,622 submission_checker.py:2441 ERROR] open/cTuning/measurements/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/mobilenet-v2-precision_uint8-0.5-160/multistream is missing user.conf
[2023-03-07 11:45:30,622 submission_checker.py:2475 ERROR] open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/mobilenet-v2-precision_uint8-0.5-160/multistream is missing s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config*.json
[2023-03-07 11:45:30,622 submission_checker.py:2244 ERROR] open/cTuning/measurements/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/mobilenet-v2-precision_uint8-0.5-160/multistream measurement_dir has issues
```

Despite these errors, all such results are counted as valid results by the submission checker while the submission UI rejects them. 

```
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite0/multistream 109.170664
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite0/offline 75.7832
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite0/singlestream 13.407167
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite1/multistream 166.040336
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite1/offline 49.5811
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite1/singlestream 20.497
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite2/multistream 235.829336
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite2/offline 34.8473
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite2/singlestream 29.161
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite3/multistream 374.929
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite3/offline 21.9922
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite3/singlestream 46.193333
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite4/multistream 753.836336
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite4/offline 11.9256
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-fp32-lite4/singlestream 85.460375
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-int8-lite0/multistream 35.852
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-int8-lite0/offline 232.321
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-int8-lite0/singlestream 4.374917
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-int8-lite1/multistream 56.387328
[2023-03-07 11:45:34,054 submission_checker.py:2643 INFO] Results open/cTuning/results/s4ch1n-tflite_cpp-cpu-tflite-vdefault-default_config/efficientnet-int8-lite1/offline 149.503
```