zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] set default cache policy
Root cause: Unimplemented Feature Gap

I love being able to cache model responses in inspect and the fine grained control I have over the cache policy. However, I often find myself writting a new solver and forgetting to add a cache policy when I call generate or wanting to change the cache policy in a bunch of places all at once. Is there some way to set a defeault cache policy for an eval?

If not, here are some potential options for an interface:
1. In the Task
```python
Task(
    dataset=dataset,
    solver=solver,
    scorer=[...],
    config=GenerateConfig(cache=CachePolicy("1M")), # in the generate config
    cache=CachePolicy("1M") # or a seperate parameter
)
```
2. As an environment variable
```bash
export INSPECT_DEFAULT_CACHE_POLICY="expiry=1M, per_epoch=True"
```
3. As a argument to `inspect eval`
```bash
inspect eval task.py --model openai/gpt-4o --cache "expiry=1M, per_epoch=True"
```

If we added it to `GenerateConfig` then all of the above would actually work! (syntax for the CLI and environment variables would be JSON encoded rather than what you show here). We'll plan on doing that.

Any updates on this? Happy to help out if needed!

`cache` is now part of generate config: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2749/
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] progress of forked solvers / subtasks are not visible in terminal viewer
Root cause: Unimplemented Feature Gap

If running an agent that forks a solver the terminal viewer will just show something like:
```
--------- subtask: chain ---------
chain()
```

Ideally, we would see the live transcript of subtasks like we do for top level solvers



+1

Yes, we know about this one. Sadly it's pretty involved to be able to dig into the subtask for the events (as its running on another co-routine). Not impossible, just on the expensive side engineering wise so we haven't prioritised.

Resovled with https://github.com/UKGovernmentBEIS/inspect_ai/commit/b91f5e58c2d547485149133ce3ec25780ed15c29
Repository: UKGovernmentBEIS/inspect_ai
Issue: No support for Ollama/offline models?
Root cause: Unimplemented Feature Gap

It would be useful if you could incorporate testing of offline LLMs as this community of models is growing rapidly and with smaller models performing not as well, plus some doing really well,  this toolset could be very useful to help set standards and highlight risks (and benefits in stronger model responses) to users.

Main site: www.ollama.com

Great to see this btw, awesome work!!!

Thanks! Currently support for offline models exists but only via HF transformers (https://ukgovernmentbeis.github.io/inspect_ai/models.html#sec-hugging-face-transformers). We'd definitely like to see more offline providers though! 

We need to make one improvement and then it will be possible to implement support for other providers in external packages. So, for example, the [ollama-python](https://github.com/ollama/ollama-python) package could include an Inspect model provider, which would e.g. make `model="ollama/llama3"` "just work" when its installed alongside inspect.

Will follow up on this thread once we have the plumbing in place to make this work.


Discovered that Ollama actually supports the OpenAI API directly, so it was quite easy to add to Inspect. This is now available in the v0.3.9 release. Assuming you have Ollma running locally and have the `llama3` model pulled, you can use it as follows:

```bash
inspect eval example.py --model ollama/llama3
```

More details in the full docs on models: https://ukgovernmentbeis.github.io/inspect_ai/models.html

Fantastic work aisi-inspect team! 

Thank you @jimccadm for the issue, and thank you @aisi-inspect for being so fast
Repository: UKGovernmentBEIS/inspect_ai
Issue: Installation raises: "The dependency name for inspect-ai does not match the actual package's name: inspect-evals"
Root cause: Environment Incompatibility

The error is mysterious to me since the string `inspect-evals` does not even appear in your `pyproject.toml`. 

It was introduced sometime between 62dfb859a5be132f28c35b7db7c416cab6f4bbf6 (working) and ec5c6f073c28c5b6ef845382a892d772c48c065e (broken). 

# Steps to reproduce
```shell
# Show that 62dfb859a5 works fine
❯ cat pyproject.toml | grep "inspect-ai "
inspect-ai = {git = "https://github.com/UKGovernmentBEIS/inspect_ai/", rev = "62dfb859a5be132f28c35b7db7c416cab6f4bbf6"}

❯ poetry i
Running 'poetry lock --no-update && poetry install --sync'
Resolving dependencies... (3.8s)
Installing dependencies from lock file

Package operations: 0 installs, 1 update, 0 removals

  - Downgrading inspect-ai (0.3.60.dev18+g62dfb859 62dfb85 -> 0.3.59.dev73+gac268d19 62dfb85)

Installing the current project: benchmarks (0.1.0)



# Now, upgrade to ec5c6f073c28 and get the error
❯ cat pyproject.toml | grep "inspect-ai "
inspect-ai = {git = "https://github.com/UKGovernmentBEIS/inspect_ai/", rev = "ec5c6f073c28c5b6ef845382a892d772c48c065e"}

❯ poetry i
Running 'poetry lock --no-update && poetry install --sync'
Resolving dependencies... (0.0s)

The dependency name for inspect-ai does not match the actual package's name: inspect-evals
```



Still fails in the same way with `poetry lock --no-update --no-cache`, so not a caching issue.

I am not sure what's going on here. I will say that people at AISI have had some wicked hard to track down poetry versioning issues across many different packages (sometimes requiring full purging of the poetry cache). Inspect itself doesn't attempt to do anything with Poetry nor does it reference inspect-evals so messages from Poetry about inspect-evals are something local to your environment.

It's of course certainly possible that some change in Inspect is causing Poetry to go down a different path (e.g. we added the `griffe` and `markdown` packages to our dev dependencies recently). You might want to git bisect to better pin things down.


Yes, it's super weird since I find nothing blameworthy in your `pyproject.toml`. Will keep you posted. 

After:

1. Deleting lockfile
2. `poetry env remove <name of env>`
3. `poetry lock --no-update`
4. `poetry install`

I can no longer reproduce the problem 👻 

Probably fair to blame either Poetry or my particular setup, not Inspect. Closing.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Incorrect typing in anthropic provider (edit: not actually incorrect, closing)

~~I believe in line 610, content is a list of Anthropic `ContentBlock` objects, not a list of inspect `Content` objects.~~
https://github.com/UKGovernmentBEIS/inspect_ai/blob/25e3bdc2cc7c299d7c72c87579d469c472b4b6b9/src/inspect_ai/model/_providers/anthropic.py#L608-L617

On a closer read actually `content` is correctly typed. Disregard the above.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Openrouter integration not working
Root cause: Environment Incompatibility

Openrouter does not seem to be available as a model serving option, despite docs claiming that it is [here](https://inspect.ai-safety-institute.org.uk/providers.html#openrouter) A minimal error replication is to run 

$ inspect eval helloworldeval.py --model openrouter/gryphe/mythomax-12-13b

(where helloworldeval.py is the hello world evaluation as at [here](https://inspect.ai-safety-institute.org.uk/tutorial.html))

This produces the error "ValueError: Model name gryphe/mythomax-12-13b from openrouter not recognized"

(changing the model name to other models available at openrouter produces the same result).

I have tried to find the underlying cause of this bug and not been successful, but I believe it may be a missing file in the model._providers submodule, where there is no openrouter.py file.

 




It's there as of v0.3.63 (published to PyPI around a week ago)

I updated and now it's there. Thanks.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Reasoning Traces for OpenRouter
Root cause: Validation Gap

Thanks as always for the nice library!

I'm using OpenRouter to access `deepseek/deepseek-r1`. It works well, except that the reasoning traces are not displayed in the log, and that the `message.reasoning` trait of the response message seems not to be populated. Am I doing something wrong, or is this a bug?

I was trying to verify that this is indeed a problem with deepseek, and found that `o3-mini` for example also doesn't give reasoning traces. This makes sense, because the API just doesn't expose them, right?

While looking for the potential problem in the source (none found yet), I stumbled on a likely typo in `openrouter.py`. I assume `params[EXTRA_BODY]["tranforms"] = self.transforms` should be `params[EXTRA_BODY]["transforms"] = self.transforms`.

Right now we capture reasoning traces either via a `reasoning` or `reasoning_content` field sent alongside the response `content` OR via a `<thinking>` tag inside the `content`. If neither of those appear than OpenRouter has effectively stripped this from the response from deepseek -- I would file an issue there in hopes its an easy to remedy oversight.

Fixed the transforms issue you mentioned.

I don't think there is anything further we can do here (OpenRouter needs to forward along the traces either in the message body or in `<thinking>` tags as is done by Ollma and Together in their OpenAI compatible back ends.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Include Gemini Reasoning Traces
Root cause: Unimplemented Feature Gap

#1229 states that it is a to-do to 

>Implement support for Gemini 2.0 Flash Thinking

and importantly, that

> Note that the Gemini item requiring migrating to Google's new universal [GenAI SDK](https://ai.google.dev/gemini-api/docs/sdks#python-quickstart), which they are currently saying is not yet ready for production.

Since I would really like this feature, I hope it's OK to restate this as an issue here, as it was not yet resolved by #1229.

Note that one of the maintainers said in another thread yesterday:

> Finally, Google is in the course of introducing a brand new Python client for Gemini. They have said it is "not ready for production". We nevertheless have an implementation of it internally that we are hoping to promote soon (but we've observed numerous issues with it).

https://github.com/UKGovernmentBEIS/inspect_ai/issues/1299#issuecomment-2653805278

Yeah, we've seen lots of instances where their model definitions don't actually correspond to what's sent over the wire (which just blows everything up). As soon as things settle down we'll implement the new client.

Resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1349.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Log viewer produces `An unexpected error occurred.` when user stores certain data types in a `TaskState`.
Root cause: Validation Gap

Both the `inspect view` command and the VSCode extension started showing the error below today. The viewer shows the Sample, Info, and JSON tabs just fine, but when I click on a sample to see details, this error shows:
```
Error: Cannot read properties of null (reading 'name')
at TypeError: Cannot read properties of null (reading 'name')
    at http://localhost:7575/assets/index.js:16097:17
    at Array.map (<anonymous>)
    at x$1.MetaDataView [as constructor] (http://localhost:7575/assets/index.js:16090:47)
    at x$1.B$1 [as render] (http://localhost:7575/assets/index.js:266:19)
    at j$1 (http://localhost:7575/assets/index.js:188:49)
    at $ (http://localhost:7575/assets/index.js:99:153)
    at j$1 (http://localhost:7575/assets/index.js:190:204)
    at $ (http://localhost:7575/assets/index.js:99:153)
    at N (http://localhost:7575/assets/index.js:237:45)
    at j$1 (http://localhost:7575/assets/index.js:199:97)
```
This is only for logs that I produced today, old logs from yesterday and earlier are fine. I'm not sure if it's an issue with inspect or with my evals

# Stack:
Python=3.13.1
inspect-ai=0.3.56, 0.3.64, broken with both
Repo, broken at this commit: https://github.com/aaron-sandoval/inspect-control-settings/tree/7b21bcbc31b212c341a4dc32fd6eb642df66f883
Working at this commit: https://github.com/aaron-sandoval/inspect-control-settings/tree/e02a4da8e4b04a6b0b0398893c8dadc49f4942f7

I did some experimenting and found that the error comes from my storing a function in the `TaskState` instance. Seems like a serialization error. It would be nice to have a more graceful failure if the user stores non-serializable data in the `TaskState` rather than the sample details being totally inaccessible.

@dragonstyle There should be a fallback handler for serialisation which ignores unserialisable objects. Not sure why that isn't kicking in here.

Could you share a log that causes this issue (you can DM it to me if you'd rather not share it here)? That would help immensely in tracking the issue down.

@dragonstyle Here's a log that's broken: https://github.com/aaron-sandoval/inspect-control-settings/tree/7b21bcbc31b212c341a4dc32fd6eb642df66f883/experiments/logs/apps_baseline/2025-02-14_131938
Repository: UKGovernmentBEIS/inspect_ai
Issue: Side by side comparison of two models
Root cause: Unimplemented Feature Gap

Is there a way to get a side by side view comparing two models on the same tasks?
For example seeing which instances both models got wrong or when one got something wrong while the other didn't, etc.

Thanks for the great library.

Right now there isn't an automated way to do this (though it is on our todo list). You can sort of set this up using the Inspect VSCode extension by just opening two logs side by side and visually comparing them (tedious but perhaps helpful).

We would also find this a very interesting feature for the log viewer. I had kind of assumed this would be too much work to be worth it, as it seems to invalidate a fundamental assumption of the log viewer. But great to hear it's on the todo list! Maybe it's easier than I thought.

It isn't easy, unfortunately, but I think we'd still like to do it!

If you wanted to approach this problem how would you do it?
Can you break down this feature into 4-5 high-level tasks so that an
outsider can contribute?

On Tue, Feb 18, 2025 at 5:37 AM Charles Teague ***@***.***>
wrote:

> It isn't easy, unfortunately, but I think we'd still like to do it!
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/UKGovernmentBEIS/inspect_ai/issues/1327#issuecomment-2665743807>
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACGSB7ONVEY2LIMPIPGVCD2QMZSRBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOJIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVE3TCOBWGY2TAMJZQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSHA2TKOBZHE4DOMVHORZGSZ3HMVZKMY3SMVQXIZI>
> .
> You are receiving this email because you authored the thread.
>
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>
> .
>
>


Thanks, this is a great suggestion. I'm right in the middle of quite a bit of refactoring of the main application logic which is necessary to tackle a feature like this (and will cause a lot of merge issues as I work through it). Let me get through this work and then I'll revert here with some suggestions as I see a good path!

Just here to bring up that same need.

But also the need to compare two evals with different parameters (in a broad sense: different vllm version / sampling parameters...), which could be equivalently implemented through the same feature.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect viewer doesn't show transcript for forked processes
Root cause: Algorithmic Error

I have a solver that uses `fork` to run a sub-agent:
```python
    async def solve(state: TaskState, generate: Generate) -> TaskState:

        ...

        try:
            state = await fork(
                state, sub_agent_initializer(max_attempts, state.token_limit)
            )
        except SampleLimitExceededError as e:
            if e.state is None:
                raise e
            state = e.state

        ...
```

Previously I had been able to see the forked agents transcript nested in the parent transcript. Now I just see:

![Image](https://github.com/user-attachments/assets/bc8444f6-2b66-4c21-88b7-f74461abb6cf)

with no option to expand. 

---

Inspect version: 0.3.62

Thanks for reporting this - is there any chance you have a log file which reproduces this that you could share? I just did a little testing with forking locally and things appeared to be working properly for my toy test cases....
Repository: UKGovernmentBEIS/inspect_ai
Issue: [docs] wrong paper link to ReAct in Agent Basics

https://inspect.ai-safety-institute.org.uk/agents.html

In the first paragraph


> Agents combine planning, memory, and tool usage to pursue more complex, longer horizon tasks (e.g. a [Capture the Flag](https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurity)) challenge). Agents are an area of active research, and many schemes for implementing them have been developed, including [AutoGPT](https://arxiv.org/abs/2306.02224), [ReAct](https://arxiv.org/pdf/2303.11366.pdf), and [Reflexion](https://arxiv.org/pdf/2303.11366.pdf).


Both ReAct and Reflexion link to the same Reflexion paper.

Thx for reporting! Fixed here https://github.com/UKGovernmentBEIS/inspect_ai/commit/a31fcc66350fa7bb263b47df84a51583fefd283c
Repository: UKGovernmentBEIS/inspect_ai
Issue: Representing refusals

Is there already a canonical (and provider-agnostic) way to represent refusals in Inspect? Do we need to add one?

From the discussion in https://github.com/UKGovernmentBEIS/inspect_ai/issues/1299, this seemed to be what you are suggesting, @jjallaire? 

```python
ChatCompletionChoice(
    message=ChatMessageAssistant(
        content="",
        source="generate",
    ),
    stop_reason="content_filter",
)
```

What are the pros and cons of this vs returning something other than a `ChatCompletionChoice`?

No the content in that case has a refusal message (e.g. "I'm sorry I can't help with that" or whatever various model API provider return explicitly). The main pro is that it maps to the documented APIs of nearly all model providers (which return standard assistant messages with "content_filter" or something similar as the stop reason). So we are basically matching exactly the form and structure of the core model APIs (generally a design goal). As it so happens some providers also return content filter error out of band (e.g. in a proxy server that returns a 400) but we normalise these in Inspect.

We can decide that Inspect should deviate from the standard structure of model APIs but I'm not sure why we would do this. As it stands the stop_reason enables agents to program explicitly around refusals and scorers to also take them into account (which safeguard scorers do all the time).

> The main pro is that it maps to the documented APIs of nearly all model providers (which return standard assistant messages with "content_filter" or something similar as the stop reason). 

Perfect, I didn't know that, and it's very persuasive! Seems like we've already got basically the right approach. :) 

The only thing I would perhaps add (low confidence suggestion), is an `is_refusal` field on a `ChatCompletionChoice`, to abstract over the different `stop_reason`s that providers may return (if there are indeed different ones). Wdyt? 

We normalise stop reasons to this:

```python
StopReason = Literal[
    "stop",
    "max_tokens",
    "model_length",
    "tool_calls",
    "content_filter",
    "unknown",
]
```

So `stop_reason == "content_filter"` is essentially `is_refusal`. I'd rather have people be comfortable with and accustomed to programming against stop reason than introduce an abstraction for this equality check.

> nearly all model providers

Is Google the odd one out then?

> As it so happens some providers also return content filter error out of band (e.g. in a proxy server that returns a 400) but we normalise these in Inspect.

How do you currently do this?

> We normalise stop reasons to this

Fantastic, again I didn't know we already did this.

> I'd rather have people be comfortable with and accustomed to programming against stop reason than introduce an abstraction for this equality check.

Agreed!
Repository: UKGovernmentBEIS/inspect_ai
Issue: `python` and `bash` tools don't include the timeout in the tool description sent to LLM
Root cause: Documentation Deficiency

You can see this e.g. here in the docstring of the `execute` function:

https://github.com/UKGovernmentBEIS/inspect_ai/blob/30ef955e10ee6895ba8d99457d80c7969d24836b/src/inspect_ai/tool/_tools/_execute.py#L61-L107

I believe this could be fixed by instantiating a [ToolDef](https://inspect.ai-safety-institute.org.uk/reference/inspect_ai.tool.html#tooldef) and having `python` return that, instead of defining the nested function `execute`. 

Are there downsides to that? We'll probably make this change in our own repo. If you think it's a good idea I can contribute it to Inspect. 

So essentially tell the model exactly what the timeout is? I haven't seen this done so my only question would be do we think this would make any difference in the model's performance?  It's possible it even makes the model perform worse because it holds back on doing useful things b/c it's "worried" about the timeout.  I definitely like to be evidence based when making changes like this as there are a lot of agents/evals sitting on top of this code that I don't want to negatively affect.

> So essentially tell the model exactly what the timeout is?

Yes

> It's possible it even makes the model perform worse because it holds back on doing useful things b/c it's "worried" about the timeout. I definitely like to be evidence based when making changes like this as there are a lot of agents/evals sitting on top of this code that I don't want to negatively affect.

I think prima facie it makes sense to tell the model more about the setting. But of course, it's hard to predict anything with high confidence. We don't know how it will affect performance, and the effect probably varies by model and benchmark! I don't think I have the capacity to personally run a comprehensive empirical experiment, so closing for now.

Yes, I agree that the intuition is that the model would make better choices, but we've learned over time that elicitation changes that "make sense" often don't do anything or make things worse! One of the problems is our intuitions often project our "human intelligence" onto the problem space whereas we are dealing with a different type of intelligence here :-)
Repository: UKGovernmentBEIS/inspect_ai
Issue: Search message contents in log viewer
Root cause: Unimplemented Feature Gap

## Current behaviour
The log viewer offers some filtering, including on the contents of the `input` and `target` fields:

<img width="439" alt="Image" src="https://github.com/user-attachments/assets/53e81963-9a0b-4f93-ac31-a62ff882b5c7" />

## Feature suggestion
Enable filtering on message contents, e.g. via an option named `message_contains`.

Perhaps one would want to filter on the role or other attributes of the message as well, e.g. find samples where a particular string appears in Assistant messages.


## Challenges
I realise this may be non-trivial, since the text to search through may be large. In addition, when using the `.eval` format, the text to search through will be in the sample-level files, not in the header or summary files.

This is out of scope for the current architecture (since we don't have access to the sample data and this data in total is often multiple GB for agentic evals). At some point we may implement an external database which could have improved searching semantics and we'll naturally revisit this if and when we do that.