Repository: UKGovernmentBEIS/inspect_ai
Issue: [Bug report] Agent caching is broken due to cached messages having source: 'cache'
Root cause: Algorithmic Error

Caching is currently broken for solvers (e.g. agents) that generate a transcript of messages. This is because when you fetch a model response from cache, the `source` field is marked as `cache` (instead of it being marked as `generate` in the original run of the agent):

https://github.com/UKGovernmentBEIS/inspect_ai/blob/ff1d013ace94b0f61890fb8f17e71f7f77dd9633/src/inspect_ai/model/_cache.py#L219-L221

Current behavior: Let's say an agent generates n messages. Then if you run the script n times, then it will get completely cached. Each run caches one more message properly.

Proposed fix: We have a different way of marking model responses as coming from cache that doesn't break caching itself.

Ouch! This PR should mitigate the issue you are seeing: https://github.com/UKGovernmentBEIS/inspect_ai/pull/432

@dragonstyle I think we need to a feature that adds cache fetch/store events to the transcript.

@tw-aisi - sorry, I'm really struggling to understand the issue you're describing, can you ELI5 please 😅 ? As @jjallaire-aisi mentioned in the associated PR this field was originally set simply so that it was clear from the logs that responses were pulled from cache instead from a generate call

There may be something specific relating to `agents` (which I'm not familiar with) at play here perhaps? Or does the `model_dump` just need to exclude the source here (I'm not at all sure of this though because I'm definitely missing something from the issue) https://github.com/UKGovernmentBEIS/inspect_ai/blob/ff1d013ace94b0f61890fb8f17e71f7f77dd9633/src/inspect_ai/model/_cache.py#L143

Ah sorry for not being more clear. I think it is actually a bit of a tricky bug.

The issue is not specific to agents, it pops up whenever you are running a solver that will append cached responses to the message history and then resend that message history to a model `generate` call.

The issue is indeed related to the line you linked. The `model_dump` is used to generate the cache key, and thus the cache key will change if the source of a message changes from`generate` to `cache`.

Here's a walkthrough of what I was running into in practice:

### First run (nothing cached)
1. I start an agent run. Agent makes a call to `generate` with the initial user prompt: `[user_message1]`
2. `generate` returns `assistant_message2(source="generate")`.
3. Agent appends this message to the message history makes a call to `generate` with the prompt `[user_message1, assistant_message2(source="generate")]`.
4. `generate` returns `assistant_message3(source="generate")`.

### Second run (everything in principle should be cached, but is actually not)
1. I start an agent run. Agent makes a call to `generate` with the initial user prompt: `[user_message1]`
2. `generate` returns `assistant_message2(source="cache")`.
3. Agent appends this message to the message history makes a call to `generate` with the prompt `[user_message1, assistant_message2(source="cache")]`.
4. `generate` does not have a cache hit because the prompt we submitted has a different source field for `assistant_message2`!



Aha, I see what you're saying @tw-aisi, thanks for the clarification! Given that, I suspect excluding the `source` field from the cache key is probably an alternative solution here, but #432 should sort this more cleanly anyway now we're tracking whether it's from cache in the transcript instead of the log 👍 

Resolved by https://github.com/UKGovernmentBEIS/inspect_ai/pull/432