Repository: UKGovernmentBEIS/inspect_ai
Issue: For big eval datasets, parsing the eval log can take up to hours
Root cause: Algorithmic Error

Hi,

## Situation

I have a rather large dataset (200k samples) which results in eval log files (in .eval format) of up to 1GB. The dataset is multimodal but I guess it should not matter much since I pass media content by URL and disable logging images (so no media content is effectively added into the file) with `INSPECT_EVAL_LOG_IMAGES=false`. Each samples then consists of a few messages (system prompt, prompt, media content, model generation). I'm running evaluations against a local vLLM instance (with OpenAI connector, not with vLLM one, so there is no startup time due to spawning a vLLM instance).

Loading the eval log file in the VSCode extension is rather fast (a matter of minutes, but no more than 5 minutes), but any operations on the log file through the CLI (e.g. running eval-retry) or the Python API (e.g. to convert to a Pandas df) is taking hours.

Typically, if running eval-retry on this dataset, it takes up to 4 hours before the CLI updates and first samples get run. Similarly, if running conversion to Pandas dataframe, it takes hours as well to retrieve all the samples.


## Steps to reproduce my dataset

* My dataset is 200k videos passed by URL (see #2787 for detail on my actual setup) to an externally managed vLLM.
* My pipeline is system prompt + generate. Each sample has a ChatMessageUser consisting of a ContentVideo (passed by local file:/// URL) and a ContentText (actual prompt used on this file).
* Apart from `INSPECT_EVAL_LOG_IMAGES=false` and relevant env variables to point to the OPENAI API server and model to use, I'm using the default values and the default CLI `inspect eval file.py@task`. 
* Then, stop the eval in the middle (partial eval log file would be 500-800MB) and resume it with `inspect eval-retry <log_file>`. It takes hours before passing the first sample through OPENAI server and updating the CLI display accordingly.
 

## Things I have tested

* Should not be due to startup time of a model serving instance, as I'm using an externally managed vLLM instance (with OpenAI API connector). So should not be a variant of #2758.
* It is very similar to https://github.com/UKGovernmentBEIS/inspect_ai/issues/2418 although also slightly different, and with different order of magnitudes.
* I'm not exactly sure whether the eval log file should be this big for my dataset. I converted it into a pretty-printed JSON representation to debug. Each samples (there are 200k of them) is about 500 lines of pretty-printed JSON. Nothing striking in there, but most of the lines (400 out of 500) are under the "events" key (not sure if there should be so much events emitted/logged and whether this might be an issue).


## Open questions

I'm happy to help debug this or look on my own, but I must confess I'm slightly puzzled by this issue and where to look for first. Random thoughts on my side are:
* Not sure if my eval log file is regular and reasonable in size or whether there is an issue with the file being too large (e.g. too much events emitted compared to regular baseline)?
* I'm using `INSPECT_EVAL_LOG_IMAGES=false` which should prevent emitting any multimodal content in the log file. I also needed it in order to pass the true files through URL and not base64 encode them before passing to OpenAI server (see #2787). Not sure if this base64 conversion (or any access to files) might be done in other part of the code and resulting in poor performances (because it would iterate on 200k video files, and not simply load a single eval log file for whatever reason)?
* I'll try to log in JSON and see whether the same issues arise.


EDIT: Clarification on inspect_ai version, as I tried to run it again against the latest version to date (0.3.152). It now takes way less time to get to the base UI of `inspect eval-retry` (about 10 minutes to drop to the tabbed view of Tasks/Running samples/Console). At this point, the "Tasks" view display the parameters in use (e.g. `max_retries: 1, fail_on_error: False, log_images: False, dataset: dataset`). From there, it takes a few hours before sending the first sample to the external OpenAI API server.