Repository: mlcommons/inference
Issue: Previously validated MLPerf submission results are now invalidated 
Root cause: External Dependency Breakage

After [this merge](https://github.com/mlcommons/inference/pull/1330) was done, our previously validated results are no longer passing the submission UI.

These are from the email yesterday
```
[2023-02-11 22:24:04,528 submission_checker.py:2643 INFO] Results open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-fp32-lite0/singlestream 8.369863 with power_metric = 198.179304
[2023-02-11 22:24:04,528 submission_checker.py:2643 INFO] Results open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-fp32-lite2/singlestream 14.706365 with power_metric = 346.365386
[2023-02-11 22:24:04,528 submission_checker.py:2643 INFO] Results open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-fp32-lite3/singlestream 21.842603 with power_metric = 519.070605
[2023-02-11 22:24:04,528 submission_checker.py:2643 INFO] Results open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-fp32-lite4/singlestream 32.920100 with power_metric = 856.106070
[2023-02-11 22:24:04,528 submission_checker.py:2643 INFO] Results open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-int8-lite2/singlestream 8.355014 with power_metric = 184.147006
[2023-02-11 22:24:04,528 submission_checker.py:2643 INFO] Results open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-int8-lite3/singlestream 11.522261 with power_metric = 249.986543
[2023-02-11 22:24:04,528 submission_checker.py:2643 INFO] Results open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-int8-lite4/singlestream 18.133046 with power_metric = 390.354031
[2023-02-11 22:24:04,528 submission_checker.py:2645 INFO] ---
[2023-02-11 22:24:04,528 submission_checker.py:2651 INFO] ---
[2023-02-11 22:24:04,528 submission_checker.py:2652 INFO] Results=7, NoResults=0
[2023-02-11 22:24:04,528 submission_checker.py:2658 INFO] SUMMARY: submission looks OK
```

And this is happening today
```
2023-02-12 11:53:01,708 submission_checker.py:1895 ERROR] Power WG check.py did not pass for: open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-int8-lite4/singlestream/performance
[2023-02-12 11:53:01,708 submission_checker.py:2336 ERROR] open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-int8-lite4/singlestream/performance/run_1 has issues
[2023-02-12 11:53:01,708 submission_checker.py:2639 INFO] ---
[2023-02-12 11:53:01,708 submission_checker.py:2645 INFO] ---
[2023-02-12 11:53:01,708 submission_checker.py:2648 ERROR] NoResults open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-fp32-lite0/singlestream
[2023-02-12 11:53:01,708 submission_checker.py:2648 ERROR] NoResults open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-fp32-lite2/singlestream
[2023-02-12 11:53:01,708 submission_checker.py:2648 ERROR] NoResults open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-fp32-lite3/singlestream
[2023-02-12 11:53:01,709 submission_checker.py:2648 ERROR] NoResults open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-fp32-lite4/singlestream
[2023-02-12 11:53:01,709 submission_checker.py:2648 ERROR] NoResults open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-int8-lite2/singlestream
[2023-02-12 11:53:01,709 submission_checker.py:2648 ERROR] NoResults open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-int8-lite3/singlestream
[2023-02-12 11:53:01,709 submission_checker.py:2648 ERROR] NoResults open/MLCommons/results/arjun_orin-tflite_cpp-cpu-armnn_tflite-v22.11-default_config/efficientnet-int8-lite4/singlestream
[2023-02-12 11:53:01,709 submission_checker.py:2651 INFO] ---
[2023-02-12 11:53:01,709 submission_checker.py:2652 INFO] Results=0, NoResults=7
[2023-02-12 11:53:01,709 submission_checker.py:2655 ERROR] SUMMARY: submission has errors
```

We know that PTDaemon version was updated in this PR. But such changes which causes result invalidation should not be done when submission window is open. 

@arjunsuresh I agree, it is rather unpleasant, but I'm afraid rather unavoidable under the circumstances. As a consolation, you only need about 10-11 (minutes per run) * 2 (runs per experiment) * 7 (experiments/models) =  ~2.5 hours to rerun the power experiments.

It is true that we only had these 7 plus 1 more 8 experiments taken before this change (I have no idea about other submitters). But it clearly took almost a day and not 2.5 hours (could be due to first time run which includes the accuracy runs) --  are you proposing that we'll be given this much time after the submission deadline to retake them?

Just for the official purpose I'm adding here from the inference notes.

**Dec 30, 2022 | Code freeze (functionality freeze)Includes automated submission checker ; Includes power repo**

And now it is changed to February 12. This I'm mentioning to ensure that we have a robust procedure "at least" for coming rounds.

@psyhtest In our case "most likely" we'll be able to redo the experiments thanks to our automation. But are you sure no other submitters (who have already taken results) will object to this change? - we are still in a weekend. I'm asking this again because if we revert back whatever results we are taking now will also be invalidated. I guess the only way forward will be to support multiple PTDaemons in the submission checker.  

Can you help me understand why other submitters (and broadly all submitters) were not consulted prior to making a change like this? @psyhtest @arjunsuresh 

@TheKanter The Power WG had asked submitters to test and validate the new PTDaemon well on time which was indeed done by those in the WG. But there was no one to do the required update in the power-dev repository on the related submission checker file. I came to know of this checker issue only 5 days back when our test submissions failed - [reported here](https://github.com/mlcommons/power-dev/issues/285) and since then @pgmpablo157321 made the required fix but this fix was not updated in the inference repository (unfortunately power-dev repo is a git sub module of inference repo) 

I guess since the power WG wanted this PTDaemon version to be used, @psyhtest reopened and merged this PR to do this git submodule update. I'm also using this version for collecting our results from the time I saw the email confirmation from the Power WG chair. 

The change was indeed discussed in both the WG meetings and in many email chains. But the implication of the change (need to redo the submissions) may not have been clear to most submitters.