Repository: UKGovernmentBEIS/inspect_ai
Issue: More bottlenecks in cheap evals
Root cause: Resource Mishandling

The previous fix, https://github.com/UKGovernmentBEIS/inspect_ai/issues/2275 gave a significant speedup, however, I'm still finding that the cheaper evals are severely bottlenecked by something other than the model sampling. 
My sampling code logs how many concurrent samples we're processing, and it hovers around 25 on evals like MMLU and DROP, even though I've set max_connections to 512.
My experiments are on a non-public codebase, I'd guess that this can be reproduced running vllm with a small model, and running the MMLU eval.

I've attached screenshots of py-spy, sorted by OwnTime and TotalTime. My guess is that there's some io blocking the asyncio event loop, which would benefit from asyncio.to_thread.

<img width="672" height="458" alt="Image" src="https://github.com/user-attachments/assets/8cc5c82d-8667-47bb-aecb-ee6386124b30" />
<img width="789" height="456" alt="Image" src="https://github.com/user-attachments/assets/49412b37-4cc4-4687-a57d-7758681e2ecd" />

It does indeed look like there may be some blocking io bottlenecks -- I am see zipfile/__init__.py taking a bunch of time (that is indeed a blocking call to opening a zip file) and then again in log events (which is blocking call to a write to a sqllite database). Generally these things are diminimus in relative time but in the case of a large number of very cheap to evaluate samples they can dominate. We will try to reproduce this locally, but a couple of things to try on your end to explore this theory:

1) What happens if you set `--log-buffer` very high. This will control how often we flush samples to the log (zip) file. Note that the default is 10 -- it seems like you could easily go to 1000 in this scenario.

2) What happens if you specify `--no-log-realtime`? By default we log samples and events to a sqlite database so we can provide a realtime view in the log viewer (especially useful for long running agent trajectories). In your case though you don't really benefit so should be able to disable entirely.

Sadly in both of these cases we've found that `to_thread()` has not helped (the zip file writer holds the GIL, and the aiosqlite package we found had stability issues. There still might be another path there, but at least initial explorations weren't fruitful.



Did some local experimentation. To approximate a very cheap generation I ran MMLU with inference cached (so the cost of inference is just unpickling the cached output) then ran again with `--log-buffer=1000` and `--no-log-realtime`:

|                                         | Total Time | Per Sample |
|-----------------------------------------|:----------:|:------------:|
| Baseline (cached generate)              |    5:24    |     23ms     |
| `--log-buffer=1000`                     |    3:00    |     13ms     |
| `--log-buffer=1000` `--no-log-realtime` |    0:51    |     3.6ms    |

So it looks like logging (mostly realtime event logging) is adding ~ 20ms per generation of overhead. Hopefully you will find the same and we end up closer to maximum inference throughput.

Thanks, I can confirm that it's about 6x faster with both of these settings changed, and --log-buffer seems to make an especially big difference. I ran the DROP eval (10k samples) with and without the settings, and it goes from about 15 minutes to 2.5 minutes, which would correspond to about 100ms overhead. With the settings changed, I'm seeing in the ballpark of 512 (=max_connections) concurrent requests as expected.

It's possible that IO is slower on my machine. For one thing, I have my home directory on an NFS, though switching the inspect log directory off of the NFS doesn't immediately seem to help.

These settings are sufficient for my purposes, as I can make my library plug them in when calling `eval_async`.

Okay, very glad to hear!

I'm on a 2023 MB Pro w/ SSD so fast IO but not likely that much faster than a recent Linux machine w/ an SSD?

The database writes are done to the user data dir (XDG_DATA_HOME) which defaults to `~/.local/share` so those are still hitting NFS. Flushing the log buffer also involves the use of temp file so if the tempdir is somehow also pointing at NFS that could explain the extra time there.