Repository: mlcommons/inference
Issue: Error with power checker
Root cause: Algorithmic Error

We are seeing the below errors in 17 of our power results while running the submission_checker. I see [this commit ](https://github.com/mlcommons/power-dev/commit/362382f8154399a3067345acf7f686af379997b2)which changed this tolerance level from 200ms to 500ms for 2.1 round. May be we increase this to 1s?

ERROR: Not all checks passed
[x] Check client sources checksum
[x] Check server sources checksum
[x] Check PTD commands and replies
[x] Check UUID
[x] Check session name
[ ] Check time difference
    The time difference for 3 phase of ranging mode is more than 500ms. The difference is 535.0179672241211

All checks passed. Warnings encountered, check for audit!
[x] Check client sources checksum
[x] Check server sources checksum
[x] Check PTD commands and replies
[x] Check UUID
[x] Check session name
[ ] Check time difference
    The time difference for 3 phase of run_1 mode is more than 500ms. The difference is 890.5191421508789

8 of the 17 results we had problem with were inferred scenarios and the actual problem happened only with 9 results. We managed to do a rerun for them and all of them passed the submission checker. But this issue needs to be resolved in a proper way.

My guess is that this problem happens because the client process logs a timestamp, sends a command to the server process and the server process then logs its timestamp (as can be seen in the code [here](https://github.com/mlcommons/power-dev/blob/r3.0/ptd_client_server/lib/client.py#L290)). Here, the response time of the server process is ignored and this should be the reason for the high variance (>500ms) for the timestamps between the server and the client.

adding question here, since it will be easier to track
_Provably naive question, but since both machines are synced to same NTP, why do we think such thing would happen? Latencies in python?_

Hi @dmiskovic-NV , if we see the code [here](https://github.com/mlcommons/power-dev/blob/r3.0/ptd_client_server/lib/client.py#L290) the sequence of operations (considering just one phase) are as follows:

1. client logs time
2. client sends STOP signal to server
3. server logs time

But there could be delay for the server to listen to the command and this can be arbitrarily high on small systems. Probably doing [this sync_check](https://github.com/mlcommons/power-dev/blob/r3.0/ptd_client_server/lib/client.py#L288) after each phase is a better option. 

We are seeing issues here too - 

[ ] Check time difference
        The time difference for 3 phase of ranging mode is more than 500ms: 6.901392459869385
[ ] Check time difference
        The time difference for 3 phase of ranging mode is more than 500ms: 2.761815071105957
[ ] Check time difference
        The time difference for 3 phase of ranging mode is more than 500ms: 7.119086980819702

It looks like sometimes it syncs within a few milliseconds and sometimes there is a big skew. We think this is because our host is under powered and we're trying to get a new machine for our driver and move the NTP Server to a new host too.





I'm seeing the same issue as well during the ranging phase.

My errors are as follow:

```
[rakshith@node083 Dell]$ python3 power-dev/compliance/check.py results/XR5610_L4x1_TRT_MaxQ/3d-unet-99/Offline/performance/
[x] Check client sources checksum
[x] Check server sources checksum
[x] Check PTD commands and replies
[x] Check UUID
[x] Check session name
[ ] Check time difference
        The time difference for 3 phase of ranging mode is more than 500ms: 854.4766902923584

[x] Check client server messages
[x] Check results checksum
[x] Check errors and warnings from PTD logs
[x] Check PTD configuration
[x] Check debug is disabled on server-side

ERROR: Not all checks passed
[rakshith@node083 Dell]$ python3 power-dev/compliance/check.py results/XR5610_L4x1_TRT_MaxQ/3d-unet-99.9/Offline/performance/
[x] Check client sources checksum
[x] Check server sources checksum
[x] Check PTD commands and replies
[x] Check UUID
[x] Check session name
[ ] Check time difference
        The time difference for 3 phase of ranging mode is more than 500ms: 854.4766902923584

[x] Check client server messages
[x] Check results checksum
[x] Check errors and warnings from PTD logs
[x] Check PTD configuration
[x] Check debug is disabled on server-side

[rakshith@node083 Dell]$ python3 power-dev/compliance/check.py results/XR5610_L4x1_TRT_MaxQ/retinanet/MultiStream/performance/
[x] Check client sources checksum
[x] Check server sources checksum
[x] Check PTD commands and replies
[x] Check UUID
[x] Check session name
[ ] Check time difference
        The time difference for 3 phase of ranging mode is more than 500ms: 577.0492553710938

[x] Check client server messages
[x] Check results checksum
[x] Check errors and warnings from PTD logs
[x] Check PTD configuration
[x] Check debug is disabled on server-side
```

There are 3 logs that are not passing the 500ms test.  However, this has a snowball effect on other benchmark <-> scenario results as one failing scenario result can cause the non failing result to be discarded for that benchmark based on the inference suite rules. In my case, totally 6 results are not passing effectively because of the snowball effect. 3 from the 500ms issue and 2 others being knocked off. 

Here is the checker log :

```
[2023-03-02 02:44:06,929 submission_checker.py:2385 ERROR] closed/Dell/results/XR5610_L4x1_TRT_MaxQ/3d-unet-99 does not have all required scenarios, missing {'Offline'}
[2023-03-02 02:44:09,850 submission_checker.py:2385 ERROR] closed/Dell/results/XR5610_L4x1_TRT_MaxQ/3d-unet-99.9 does not have all required scenarios, missing {'Offline'}
[2023-03-02 02:44:04,383 submission_checker.py:2385 ERROR] closed/Dell/results/XR5610_L4x1_TRT_MaxQ/retinanet does not have all required scenarios, missing {'MultiStream'}
```

I made a change to our setup which had some improvement; I used a separate NTP server which was more powerful (I used to be running the NTP server from the controller). This seemed to help reduce the errors so far. So now my setup has 3 hosts on the same network: the SUT, the controller, the NTP server. 

Hi @bitfort Did all the errors go away? Also why are you running the NTP server yourself and not use an available one like Google NTP server? 

I have made [this PR] (https://github.com/mlcommons/power-dev/pull/290) if it can help avoid this issue until we have a proper solution available. 

I think they're intermittent enough that they can be avoided; not sure they're completely gone. 

I spoke too soon.

[ ] Check time difference
        The time difference for 3 phase of run_1 mode is more than 500ms.
        
 I think it's intermittent. I'll try to run again, but I'm running out of time to try again.

Its intermittent at best even with a local NTP server (from my previous submission experiences, couldn't run multiple times this time) . 

Thank you @bitfort and @rakshithvasudev for your replies. I belive doing my suggested code change can resolve this issue permanently. Reruns for no fault of submitters is not good for a benchmark suite. But I'm not sure who is responsible to do these fixes. 

@bitfort  Can you please give more information on how you reduced this difference as originally it was upto 10s IIRC? Did it require multiple runs with one of them passing or eventually with proper settings all the runs were passing? I don't think changing the code is that easy as the first time when client interacts with the server it is doing a sync -- and that's why the difference in time is in lower milliseconds and this cannot be done after individual phases. 

I finally managed to add a [fix for this issue](https://github.com/mlcommons/power-dev/pull/300) which does a proper sync of the times. Hopefully no more debate over this issue from next round of submissions.