Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect sometimes hangs on extremely large runs
Root cause: Resource Mishandling

[Originally discussed on Slack here.](https://inspectcommunity.slack.com/archives/C0805HJTK8U/p1731516474488299)

During multi-hour jobs, Inspect will sometimes hang. The user-visible manifestation of this bug is that the timers will stop. Anecdotally, memory and CPU usage do not appear to be high after the hang begins.

We currently do not have a consistent repro case of this behavior that doesn't involve running a real multi-hour job. In addition, when the issue happened, we were not able to diagnose the root cause with basic profiling steps.

Therefore, we'll need to: 
* Create a consistent repro of the behavior
  * Ideally we can do this without wasting compute on this (though that isn't guaranteed if the root cause is complex). We could try and make fairly artificial tasks using MockLLM or something similar and just run them for a very long time to see if the bug repros.
  * More sophisticated mock tasks may be required to trigger the bug, such as ones that use Docker sandboxes.
  * If we try the above steps and still can't repro the bug, it might be worth doing an actual run to see if it can be triggered as a last resort.
* Diagnose the issue once we can reproduce it at least occasionally
  * [@JJ Allaire](https://inspectcommunity.slack.com/team/U07V6G0G0GN) recommends [file logging for this](https://inspect.ai-safety-institute.org.uk/log-viewer.html#sec-external-file) so we can observe what it was doing right before it crashed.
  * We likely will need to attach debugging tools to the stuck process to Inspect what went wrong.
  * We can also look at things like what resource usage was like before the crash, what task it was on, etc. which might help make repros easier or give us some notion about root causes.

We have made several changes (some on PyPy and some still on `main`) that may address the underlying cause of this. In the stack traces we've seen there is a threading deadlock. There were AFAIK only two possible sources of threads being created in the system (both related to reading/writing logs):

1. We had a parallel log reader that used threads to get some improved concurrency, however we later discovered that s3fs is not fully threadsafe so it's possible that this use of threads was the cause of the hang. We no longer use threads here.

2. s3fs itself is implemented in an entirely async fashion, however provides sync wrappers using a fspec utility method (https://github.com/fsspec/filesystem_spec/blob/212ffb1867c6fa5ac9e9c7d61c3522fb7092bf90/fsspec/asyn.py#L63). This too could cause some deadlocking mischief. We now use exclusively async APIs for reading/writing to S3 during task execution.

The fact that people observed "freezing" and the stack traces we saw involved thread locks make me quite suspicious that the culprit is a log write or read that isn't returning (which when using sync APIs would cause the UI to become unresponsive). Hopefully with these recent changes this will no longer occur!


I am going to close this as I believe that the elimination of threading and the move to async writes to S3 has likely mitigated this problem. If we see this again let's try to capture debug logs (https://inspect.ai-safety-institute.org.uk/log-viewer.html#sec-external-file) and get a stack trace by attaching a debugger and then re-open another issue to concretely investigate these failures.