Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect always hangs at last question
Root cause: Resource Mishandling

Maybe if no timeouts are set it defaults to infinite?

<img width="1808" height="360" alt="Image" src="https://github.com/user-attachments/assets/a383ebd8-27bf-4edd-816b-e9a933fa0157" />

I have observed something similar.  A few notes that may help narrow it down:
- A month or two ago, I was able to do many experiments without seeing this happen
- In the past couple weeks, I've seen it happen quite a bit, even though I didn't change the InspectAI version (I've been on 0.3.114 for a while)
    - I'm primarily on a `e2-highmem-16` GCP instance, running inside a `python:3.11.3` docker image.
- Seems to happen sporadically with various models (including at least sonnet-4 from anthropic, gpt-4.1 from openai, llama-scout from togetherai, and gemini-flash from google), but experiments using `google/gemini-2.5-flash-preview-05-20` have run into it particularly persistently, sometimes hanging even before the final sample.  Also saw it semi-frequently with llama-scout from togetherAI (but that one was always within the last 1-3 samples, not before).
    - A bit tricky to pin down; I can see when it's stuck it is in `epoll_wait` within asyncio, which AFAICT is waiting for external input (usually waiting for llm api connection(s), but I've also seen a case where this waiting was happening even when all api connections were closed).  But this waiting also happens during normal execution, and wakes up when the llm response comes back; I don't know what causes it to become infinite at some point.
    - From `inspect eval --help`, it is true that `--timeout` defaults to infinite, but I still got a (seemingly) infinite hang when I tried **`--timeout=1200`** recently with gemini (over an hour with no output before I intervened).  Oddly, attaching a debugger while hanging showed that the epoll_wait timeout was still infinite (-1) despite specifying `--timeout` on the cli; I'm not very familiar with asyncio so I don't know if that's expected.
- In many (but not all) cases, interrupting the run and trying again with `inspect eval-retry` fixes it and gets past the last-sample hang.
- It's been difficult to get a MWE since it seemed fine with a toy task containing just a few simple problems.  It's possible that the issue is related to scale in some way; my guess is that it's something semi-random, but some heavy loads increase the probability, maybe to near 100% in the worst case.
- There is a related but separate issue where an experiment might _appear_ to be hanging infinitely on the last sample, but actually is just taking several hours to write/read the `.eval` file (happens if the traces are large, e.g. when the `.eval` file is a few hundred MB; for some reason it seems like at the end there is both a write and a read, and the read takes a really long time for `model_validate` on the samples).
    - You can tell when this is happening because CPU usage will be near 100% doing (de)compression in this case, whereas the aforementioned infinite case has 0% CPU usage.

Some more details: I always use Gemini flash as a judge.  When I've opened the log files after this happens I can see all my questions are answered, but that it hasn't computed the averages or summary statistics, maybe these are what takes the time?  Also when this is happening I can always see my last 1 or 2 judge connections hanging in there, stagnant

> When I've opened the log files after this happens I can see all my questions are answered, but that it hasn't computed the averages or summary statistics, maybe these are what takes the time?

This would be unusual as computing these statistics are typically really fast and minimally resource intensive. 

> Also when this is happening I can always see my last 1 or 2 judge connections hanging in there, stagnant

This seems more likely to be related to the issue somehow. Have you tried using `trace` to see if it gives you any insight into anomalies:

https://inspect.aisi.org.uk/tracing.html#anomalies

> > When I've opened the log files after this happens I can see all my questions are answered, but that it hasn't computed the averages or summary statistics, maybe these are what takes the time?
> 
> This would be unusual as computing these statistics are typically really fast and minimally resource intensive.
> 
> > Also when this is happening I can always see my last 1 or 2 judge connections hanging in there, stagnant
> 
> This seems more likely to be related to the issue somehow. Have you tried using `trace` to see if it gives you any insight into anomalies:
> 
> https://inspect.aisi.org.uk/tracing.html#anomalies

I haven't tried this. I am running in a Colab notebook and I don't think a shell command would work for me. I can try next time I run into the issue but as  @mdarcy220 said it only happens on very large evals

You should be able to run shell commands using `!`

For future reference and in case others are still hitting this, I (mostly) got to the bottom of my case, which turned out to be two different issues:
- gemini seemingly may infinitely hang on certain requests; I don't know why but suspect some kind of internal output-parsing failure (some requests preceding the hang returned with `MALFORMED_FUNCTION_CALL` or `RECITATION` as the `finishReason`).  The `--timeout` option doesn't work for gemini in the Inspect version I was using; that was fixed in `0.3.121`.
- An [SSE parsing bug](https://github.com/florimondmanca/httpx-sse/issues/34) in MCP

These issues could occur anywhere throughout an eval, but due to parallel execution the other workers carry on even when one sample gets frozen, which made it appear as though the hang happened on just the last sample or two.