Repository: UKGovernmentBEIS/inspect_ai
Issue: Bug: JSON log file fails schema validation during `read_eval_log`
Root cause: Documentation Deficiency

# Problem Description
Calling `read_eval_log` on a freshly-written JSON log file raises this validation error:
```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
pydantic_core._pydantic_core.ValidationError: 1 validation error for EvalLog
eval
  Field required [type=missing, input_value={'2025-03-11T23-05-20+00-..., 'sample_id': 195}]}]}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.10/v/missing
```
The files do contain the `"eval"` field. There's no more in the traceback, but debugging has it failing in the pydantic `BaseModel` constructor.

# Context
I run evals with the default log format, which produces both a .eval file and a small .json file with some small subset of the full data (e.g., no `samples` field). I was initially reading in the 20 MB .eval file, but it seemed to be too large and was causing VS Code to crash. So I tried reading in the json file, which contains all the data I need at the moment.

## Problematic json log files
- https://github.com/aaron-sandoval/inspect-control-settings/blob/d3ce7a92c337caaf8205d433a3d1c1b1bb9c5f29/experiments/logs/calibration/2025-03-11_230519/logs.json
- https://github.com/aaron-sandoval/inspect-control-settings/blob/d3ce7a92c337caaf8205d433a3d1c1b1bb9c5f29/experiments/logs/apps_baseline/2025-03-11_223646/logs.json

## Stack
WSL Ubuntu
`python=3.13.1`
`inspect-ai=0.3.72`

I also had this kind of error with `read_eval_log`, but, looking at your first linked file, I think my issue was different. For you, if you just remove the first layer of the JSON, `read_eval_log` no longer raises an error. (That is, delete `"2025-03-11T23-05-20+00-00_t-u-decomp_kGCvVVvrH3fqZpm7HW99oP.eval": {` along with the final closing `}`.)

Where did this JSON come from?

That file is actually a manifest produced by bundling (it summarizes the contents of a log directory, which allows the viewer to display all the logs in a directory statically). That definitely can't be opened as an eval_log. 

Open the eval file itself in VSCode shouldn't be crashing - maybe we should be digging into how that is happening (as in general the JSON representations of log files are actually much more difficult to deal with efficiently). 

Note that `read_eval_log` also includes `header_only`, which will load only the summary information (not the sample data), which is useful when working at a summary level (and may be what you are looking for).

Ah, these JSONs came out of calls to `eval_set` rather than `eval`. I've only run `eval_set` with one task at a time so far, but I just realized that it adds an extra layer to the json at the root. In the docs I don't see an arg or any way to specify to `read_eval_log` which log to select at that first level. Does `read_eval_log` not support JSON logs run with `eval_set`?

I'm not totally following what you're trying to do (`read_eval_log` is just a function to read log contents). Maybe could you share a little more about what you're trying to do (e.g. pseudo code or just describe the workflow you're going for in vscode)?

Sorry, let me clarify. In my workflow. I'm running a set of evals with `eval_set` to generate log files. When the evals are done, I'll have a a few Jupyter notebooks which read different subsets of those eval logs to do some data analysis. Right now I don't need the sample data, so I tried `read_eval_log(my_log.eval, header_only=True)` and everything works fine. Thanks for that recommendation.
**This solves all of my immediate problems** since there's no reason for me to touch the manifest, and all the data I need is in the header, so the high memory usage (or whatever the issue is) is gone.
There do still seem to be 2 problems here which could be important for other users:
1. Medium-size log files degrading performance and causing crashes.
2. `read_eval_log` lack of support for JSON logs run with `eval_set`. The workaround you described is pretty easy, but seems like there should be native support. Or alternatively, as you said "That definitely can't be opened as an eval_log", maybe it could be clearer for users that these manifests that are almost identical to JSON logs aren't meant to be used with `read_eval_log`.

> Medium-size log files degrading performance and causing crashes.

I would be happy to work on debugging this - I'm not sure why that is happening. I'd need a bit of repro case to dig into, lmk if you can share.

> read_eval_log lack of support for JSON logs run with eval_set. The workaround you described is pretty easy, but seems like there should be native support. Or alternatively, as you said "That definitely can't be opened as an eval_log", maybe it could be clearer for users that these manifests that are almost identical to JSON logs aren't meant to be used with read_eval_log.

Yes, JSON log files can be opened with `read_eval_log`, but the manifest files are not log files. You can use the `--log-format` arg to emit actual JSON log files, though I don't really recommend this unless you have a good reason. We can add a note to the eval-set docs describing that a manifest is produced next to the log files that are produced when running eval-set...

Sure, let's see if this will reproduce it. I'm working in Ubuntu 24.04 through WSL.

1. Install Cursor, latest version is 0.46.11, corresponding to VSCode v1.96.2.
2. Install the `uv` package manager
3. `git clone https://github.com/aaron-sandoval/inspect-control-settings.git`
4. `git checkout 9db841982a30b4dc40de1dd8b6a63467d0373da1`
5. Install Python 3.13.1 and the environment in uv.lock with uv
6. In VS Code, install the basic Jupyter extensions shown here:
![Image](https://github.com/user-attachments/assets/003f120c-2fcf-49b3-9eb7-9f6067314a54)
7. Open compute_calibration_thresholds.ipynb in VS Code
8. Partially run the notebook up to the final header "# Causes VS Code Performance Degradation and Crash". This causes no issues.
9. Run the remainder of the notebook. The final cell takes > 10 seconds to run and then causes VS Code to stop responding.et me know if any you need any clarification on any of these steps, further details about my environment, etc.

It looks like the cell that is crashing the notebook is printing the contents of the log file object. The contents of that complete log file uncompressed are something in the neighborhood of 148MB. I'm not completely shocked that the notebook crashes in this scenario, as I know that when dealing with JSON object in excess of 100MB I have had issues in the Inspect extension (the viewer and extension don't load or print complete logs like this anymore in order to avoid being required to load the complete the log into memory).

This doesn't look to me to be an Inspect issue (related more to the notebook extension afaict) so I'm going to go ahead and close this issue.