Repository: UKGovernmentBEIS/inspect_ai
Issue: inspect_ai.log._transcript.eval_events_with_content fails to lookup content in EvalSample.transcript
Root cause: Algorithmic Error

`EvalSample.transcript` stores `EvalEvents`, which can be populated with their content via `inspect_ai.log._transcript.eval_events_with_content`.

Currently this function is implemented as:

```python
def eval_events_with_content(events: EvalEvents) -> list[Event]:
    def content_fn(text: str) -> str:
        if text.startswith(CONTENT_PROTOCOL):
            return events.content.get(text, text)
        else:
            return text

    return walk_events(events.events, content_fn)
```

but `events.content` looks like:

```
{'5424e76eb25ff940165e396a2564527d': 'Simplify the expression: 1 + (2 + 3). Your final output must ONLY contain the final answer.'}
```

so seems like the prefix (`tc://`) needs to be removed before resolving the content, ex:

```python
def eval_events_with_content(events: transcript.EvalEvents) -> list[transcript.Event]:
    def content_fn(text: str) -> str:
        if text.startswith(transcript.CONTENT_PROTOCOL):
            # remove the prefix when lookup up content
            return events.content.get(text[len(transcript.CONTENT_PROTOCOL) :], text)
        else:
            return text

    return transcript.walk_events(events.events, content_fn)
```

For more context as to how we're using `eval_events_with_content` (mentioning in case there's easier way to do this that avoids using non public functions from `_transcript`:

```python
def get_model_outputs_from_eval_sample(eval_sample: inspect_ai.log.EvalSample) -> list[inspect_ai.model.ModelOutput]:
    """
    Get the model outputs from an `EvalSample`, useful for creating tests which replay model outputs.

    Example:

        # Assuming we have `eval_logs` is a list of `EvalLog`, from either:
        # - `inspect_ai.eval(...)`
        # - `inspect_ai.log.read_eval_log(path)`

        eval_log: inspect_ai.log.EvalLog = eval_logs[0]
        eval_sample: inspect_ai.log.EvalSample = eval_log.samples[0]

        model_outputs: list[inspect_ai.model.ModelOutput] = get_model_outputs_from_eval_sample(eval_sample)

        # we could either store these directly in the codebase (which is probably fine)
        # - logs/2024-10-10T23-09-09+01-00_example_task_gmK2mp6CqyBoSvZrw8S5DE.json

        # or can store them actually in the code

        # they can then be used like this:
        model = get_model("mockllm/model", custom_outputs=model_outputs)

        # here we are rerunning them on the exact same task that generated them, only
        # substituting the original model with our mock model
        eval_logs = inspect_ai.eval(
            example_task(),
            model=model,
        )

    """
    # get the transcript
    eval_events: inspect_ai.log._transcript.EvalEvents = eval_sample.transcript

    # use functions in transcript module to load the events with content (as the transcript itself otherwise
    # often uses placeholders for content)
    eval_events_with_content_resolved: list[transcript.Event] = _eval_events_with_content(eval_events)

    # events can be one of many different types, but we're only looking for model events
    model_outputs: list[inspect_ai.model.ModelOutput] = [
        event.output for event in eval_events_with_content_resolved if isinstance(event, transcript.ModelEvent)
    ]

    return model_outputs


def get_mock_model_from_eval_sample(eval_sample: inspect_ai.log.EvalSample) -> inspect_ai.model.Model:
    """
    Create a mock model which will replay the outputs from an `EvalSample`.

    Example:

        # assuming `path` is path from a previous run, ex:
        #   logs/2024-10-10T23-46-14+01-00_example_task_STkCa2nNUosqHANMQwQUDz.json
        #
        eval_log: inspect_ai.log.EvalLog = inspect_ai.log.read_eval_log(path)
        eval_sample: inspect_ai.log.EvalSample = eval_logs.samples[0]

        # we can then create the mock model, and pass it to `eval` just like a normal model
        mock_model: inspect_ai.model.Model = get_mock_model_from_eval_sample(eval_sample)

        # this will give the exact same results as the original task which produced the `eval_sample`
        eval_logs = inspect_ai.eval(
            example_task(),
            model=mock_model,
        )

    """

    model_outputs: list[inspect_ai.model.ModelOutput] = get_model_outputs_from_eval_sample(eval_sample)

    return get_model("mockllm/model", custom_outputs=model_outputs)
```

Thanks for reporting this! (we actually aren't using that function so that is how this slipped our attention). Will resolve this ASAP. 

The transcript events are currently non-public (as we wanted to be absolutely certain we had them right before exposing) so calling that function is today the only way to reverse out the content aliases (once we make the events public we'll do this automatically)

@bronson-apollo A heads up that when the `feature/eval-log-format` branch is merged (probably late today or tomorrow) there will be public API for this (and the private API you are using will be gone). The new API is `resolve_sample_attachments()`: https://github.com/UKGovernmentBEIS/inspect_ai/commit/0ae23b4a04cd0e3cbb881bc4c993179801060031. 

Hi JJ, I'm just trying to upgrade now and not sure exactly I should update the code? I gather I should call `resolve_sample_attachments()` to update the sample object, but then what how do I get the ModelOutputs after that? 

You should be able to get them the exact same way, `resolve_sample_attachments()` just resolves all of the attachment references, it doesn't change the topology of the API (i.e. where previously an attachment would be "attachment://de67a678b" it will now have the actual content resolved.

Thanks! It worked. For posterity, I replaced
```python
    # get the transcript
    eval_events: inspect_ai.log._transcript.EvalEvents = eval_sample.transcript

    # use functions in transcript module to load the events with content (as the transcript itself otherwise
    # often uses placeholders for content)
    eval_events_with_content_resolved: list[transcript.Event] = _eval_events_with_content(eval_events)
```

with
```python
    eval_events_with_content_resolved = inspect_ai.log.resolve_sample_attachments(eval_sample).events
```

and left the rest of the implementation unchanged