Repository: UKGovernmentBEIS/inspect_ai
Issue: Dataclass Objects in Score Metadata Not Preserved in Binary Log Format
Root cause: Architectural Constraint

When using the new binary log format (`log_format="eval"`), dataclass objects stored in `Score.metadata` are converted to dictionaries, losing their original class structure. The same objects are correctly preserved when using the JSON log format (`log_format="json"`).

This causes a regression in our tests when we try to upgrade to inspect 0.3.46

### Minimal Repro

```python
from dataclasses import dataclass

from inspect_ai import Task, eval, task
from inspect_ai.dataset import MemoryDataset, Sample
from inspect_ai.scorer import Score, Target, mean, scorer
from inspect_ai.solver import TaskState


@dataclass
class ResultState:
    foo: str = ""


@scorer(metrics=[mean()])
def check_state():
    async def score(state: TaskState, target: Target) -> Score:
        tool_state = ResultState(foo="test reasoning")
        return Score(value=1.0, metadata={"tool_state": tool_state})

    return score


@task
def repro_task() -> Task:
    return Task(dataset=MemoryDataset([Sample(input="test")]), plan=[], scorer=check_state())


if __name__ == "__main__":
    # Should work with JSON format
    json_result = eval(repro_task(), log_format="json")[0]
    assert json_result.samples is not None and json_result.samples[0].score is not None
    json_score = json_result.samples[0].score
    assert json_score.metadata is not None
    assert hasattr(json_score.metadata["tool_state"], "foo")
    print("JSON serialisation works")
    print(f"{json_score.metadata['tool_state']=}")
    print(f"Type of tool_state: {type(json_score.metadata['tool_state'])}")

    # Should fail with binary format
    eval_result = eval(repro_task(), log_format="eval")[0]
    assert eval_result.samples is not None and eval_result.samples[0].score is not None
    eval_score = eval_result.samples[0].score
    assert eval_score.metadata is not None
    assert not hasattr(eval_score.metadata["tool_state"], "foo")
    print("Eval serialisation fails")
    print(f"{eval_score.metadata['tool_state']=}")
    print(f"Type of tool_state: {type(eval_score.metadata['tool_state'])}")
```

The issue here is that with JSON logs we kept the entire log in memory and then returned it to you at the end. This means that everything (including non-serializable types) are preserved for the _return value_ from `eval()` (since they are still in memory as originally created) however they would not survive round-tripping to disk (reading the same JSON file from disk would also fail to preserve the dataclass).

With eval files we do not hold the entire log in memory as we write, rather we just spool out samples as they are completed. At the end, we then read the entire log from disk and return it (so you don't have the dataclass anymore). We adopted this behavior because keeping all of those samples in memory can be quite taxing (whereas just reading them in one shot at the end per-log will spike a task at a time rather than have all tasks keeping all of their samples resident). Honestly I'd prefer if we didn't return samples at all, but changing this would break too much code at this point.


Closing as the new behaviour is intended (note that now the `log` returned from `eval()` will be the same as the log you'd get later by reading from disk (the previous behaviour was an unintended side effect of the fact that we kept the entire `EvalLog` in memory for JSON).

Thanks for the info