zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: Gemini web_search 400 errors: `Function calling config is set without function_declarations`
Root cause: External Dependency Breakage

We have been performing some Q&A benchmarking on a few different commercially available models (gemini, grok, openai, etc.) and we are attempting to enable the web_search feature for each and have had success with all except for Google Gemini. 

With Gemini we always receive the following 400 error:

```
ClientError: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'Function 
calling config is set without function_declarations.', 'status':                
'INVALID_ARGUMENT'}}
``` 

Here is a very silly but simple example script to replicate the issue:

``` python
from inspect_ai import Task, task, eval_set
from inspect_ai.dataset import Sample
from inspect_ai.scorer import match
from inspect_ai.solver import generate, use_tools
from inspect_ai.tool import web_search


@task
def websearch_addition_problem():
    return Task(
        dataset=[Sample(
            input="What is 1 + 1?",
            target=["2", "2.0"]
        )],
        solver=[use_tools(web_search("gemini")), generate()],
        scorer=match(numeric=True),
    )


eval_set(
    tasks=[websearch_addition_problem],
    retry_attempts=3,
    model=["google/gemini-2.5-pro"],
    log_dir="logs-websearch",
    log_level="http",
    log_format="json"
)
```

Here are the details (`pyproject.toml`) for the `uv` environment with which I am running this script:

``` toml
[project]
name = "llm-benchmarking-inspect-ai"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = [
    "google-genai>=1.36.0",
    "inspect-ai>=0.3.131",
    "openai>=1.107.1",
    "pandas>=2.3.2",
    "requests>=2.32.5",
]
``` 

Am I doing something with the configuration here or is there something else going on? 

Thanks! 

Thanks for reporting! Fixed here https://github.com/UKGovernmentBEIS/inspect_ai/pull/2463
Repository: UKGovernmentBEIS/inspect_ai
Issue: Linter Errors for Metric Types: Use Sequence instead of list and Mapping instead of dict

## Setup

We define a new metric to be passed to a scorer.

```python
# ideally we want this to work
def get_metrics1() -> list[Metric]:
    return [accuracy(), stderr()]

@scorer(metrics=get_metrics1()) # linter throws at get_metrics1
def custom_scorer1():
    return match()

# this is the current approach but messy
def get_metrics2() -> list[Metric | dict[str, list[Metric]]] | dict[str, list[Metric]]:
    return [accuracy(), stderr()]

@scorer(metrics=get_metrics2())
def custom_scorer2():
    return match()
```

## Expected behavior

`custom_scorer1` should not receive any linter errors.


## Actual behavior

```
Argument of type "list[Metric]" cannot be assigned to parameter "metrics" of type "list[Metric | dict[str, list[Metric]]] | dict[str, list[Metric]]" in function "scorer"
  Type "list[Metric]" is not assignable to type "list[Metric | dict[str, list[Metric]]] | dict[str, list[Metric]]"
    "list[Metric]" is not assignable to "list[Metric | dict[str, list[Metric]]]"
      Type parameter "_T@list" is invariant, but "Metric" is not the same as "Metric | dict[str, list[Metric]]"
      Consider switching from "list" to "Sequence" which is covariant
    "list[Metric]" is not assignable to "dict[str, list[Metric]]"basedpyright[reportArgumentType](https://docs.basedpyright.com/v1.31.3/configuration/config-files/#reportArgumentType)
```

## Possible fix

Change all types using `list` to `typing.Sequence` and `dict` to `typing.Mapping`.

## Minimal reproducible repo

See [here](https://github.com/jeqcho/inspect-bug-reducer/tree/metric-list).


Agreed this would be a good change. Can you give us a PR for it?

Sure, I will try to get one up by the end of the week.
Repository: UKGovernmentBEIS/inspect_ai
Issue: `grouped`: Identical group display strings in viewer when multiple metrics
Root cause: Unimplemented Feature Gap

## Setup

Define a task using `grouped` with multiple metrics, e.g.

```python
@task
def some_task():
    return Task(
        ...
        metrics=[
            grouped(accuracy(), group_key="category", all_label="all (accuracy)"),
            grouped(stderr(), group_key="category", all_label="all (stderr)"),
        ],
    )
```

## Ideal behavior

It will be nice if the metric name is provided alongside the groups. e.g.

```bash
match
red (accuracy)     0.000
green (accuracy)   0.000
all (accuracy)     0.000
red (stderr)       0.000
green2 (stderr)    0.000
all (stderr)       0.000
```


## Actual behavior

Currently, it just shows up as the display string without the metric which row is accuracy and which is stderr without checking what `all_label` is.

In the shell

```bash
match                                                                                                                                                           
red             0.000                                                                                                                                           
green           0.000                                                                                                                                           
all (accuracy)  0.000                                                                                                                                           
red2            0.000                                                                                                                                           
green2          0.000                                                                                                                                           
all (stderr)    0.000 
```

In the viewer


<img width="359" height="38" alt="Image" src="https://github.com/user-attachments/assets/de292c28-b93f-4020-82bf-e0846b49cec6" />

## Possible fix

There are two ideas:
1. Introduce a new parameter to `grouped` that takes in a function that will take in the unique values of that group and output a string. e.g.

```python
@metric
def grouped(
    metric: Metric,
    group_key: str,
    *,
    all: Literal["samples", "groups"] | Literal[False] = "samples",
    all_label: str = "all",
    value_to_float: ValueToFloat = value_to_float(),
    group_key_pretty_printer: Callable[[str],str] # add this new parameter
) -> Metric:
    ...
    # when computing the group_key string to be displayed
    group_key_display_strings = [group_key_pretty_printer(title) for title in group_values]
```

This approach gives users more customizability.

2. Directly output for each group value in the form `value (metric)` as in `Ideal behavior` above. In fact, we can use approach #1 and set the default pretty printer to be this form.

## Minimal repo to see the actual behavior

See [here](https://github.com/jeqcho/inspect-bug-reducer/tree/grouped)
Repository: UKGovernmentBEIS/inspect_ai
Issue: Not possible to turn off thinking for gemini models
Root cause: Configuration Error

There seems to be a bug when passing down the configuration of reasoning tokens from the generated config to the actual vertex call. If setting the reasoning_tokens to 0, the inspect view and json outputs do show that the setting was set to 0, but the model doesn't seem to be receiving that setting since it still uses reasoning tokens (the usage is displayed and does seem like it is being the default).


Looking at [this part of the code](https://github.com/UKGovernmentBEIS/inspect_ai/blob/a988548f1fc49eaf0b918c2ad4013b492dd7e77a/src/inspect_ai/model/_providers/google.py#L391) my suspicion is that now the default for the gemini 2.5+ models is to use dynamic reasoning (set to -1), so not passing the setting when explicitly set to 0 is not turning off the reasoning anymore. Currently there is no alternative, and no way to run these models with thinking turned off.

It looks like 0 will disable thinking for 2.5 Flash and 2.5 Flash Lite but there is currently no way to disable thinking for 2.5 Pro: https://ai.google.dev/gemini-api/docs/thinking.

So I think the solution is to pass the 0 through for Flash models and to warn for Pro models. This is done here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2486
Repository: UKGovernmentBEIS/inspect_ai
Issue: Type error when Toolcalling
Root cause: Interface Contract Mismatch

I am implementing a feature using CustomModelAPI for ToolCalling, following the instructions in your documentation for modelapi. When overriding the generate method, the code is as follows:
```
return ModelOutput(
    model=model,
    choices=[
        ChatCompletionChoice(
            message=ChatMessageAssistant(
                content=response_text,
                model=model,
                source="generate",
                tool_calls=[
                    ToolCall(
                        id=str(id),
                        function=tool.name,
                        arguments=tool.parameters.properties
                    )
                    for id, tool in enumerate(tools)
                ]
            ),
            stop_reason="tool_calls",
        )
    ],
)
```
While defining the ToolCall object, the arguments parameter expects a dict[str, Any], so I passed tool.parameters.properties which has the type dict[str, Schema]. However, during execution, I encountered the following error:
```
./python3.12/site-packages/inspect_ai/model/_call_tools.py:259 in execute_tools
input was not a valid JSON value [type=invalid-json-value, input_value=JSONSchema(type='integer'...yOf=None, required=None), input_type=JSONSchema]
```
Could you please advise on how to resolve this issue?

Additionally, I have another question:

As a researcher from another country, I do not have access to the original vendor's LLM API, so I plan to use my country's proxy API to access the model. How can I implement ToolCalling to be compatible with other functionalities in the inspect_ai framework?

I would appreciate your guidance and solutions. Thank you!


I think you just need to do `arguments=tool.parameters.properties.model_dump()`. `ToolCall` is expecting something that can be directly serialized to JSON rather than a pydantic model
Repository: UKGovernmentBEIS/inspect_ai
Issue: toy example for multi-turn dialogues using inspect

I created a toy example using Inspect for multi-turn dialogue. https://github.com/Lovkush-A/inspect_multiturn_dialogue

This might be a lot to ask, but can somebody in AISI skim through and let me know if there is cleaner way of doing things? There is only one notebook to read through, in which i create a dataset, solver, scorer, task and run the evals. I plan to share this around AI safety community so I want to make sure I am using framework properly.

Thank you!

The `messages` will always a list of messages initialized from the input. I would definitely code against the `messages` not the `input`, which will allow for other solvers to be able to tweak the messages in some fashion (not likely, but its nice to not shut the door on this entirely). So something more like this:

```python
async def solve(state: TaskState, generate: Generate) -> TaskState:
    
    input = state.messages.copy()
    state.messages = []

    for turn in input:
        state.messages.append(turn)
        state = await generate(state)

    return state
```

In terms of `always_false_scorer()`, you should just be able to have no `scorer` at all in your Task (then eval will just skip the scoring step).



@aisi-inspect Better late than never, but thanks for the feedback! Very helpful.

FYI we did actually reply to this immediately when originally posted (maybe you just saw the reply now b/c we closed it?). Just didn't want you to think we were that unresponsive :-) 

Oh no! I totally mis communicated there. I was referring to my thanks being late, not your help. I saw you helped quickly but I have been delayed to come back to this.

Again, apologies. Re reading my message, I really come across as passive aggressive... 

Thx for the follow up, no worries at all! :-)
Repository: UKGovernmentBEIS/inspect_ai
Issue: React agent crashes if context window length exceeded
Root cause: Interface Contract Mismatch

I think there may be a bug in [Inspect's react agent](https://github.com/UKGovernmentBEIS/inspect_ai/blob/c3e669cbea97000d0bbf6c8a1ece371fdcf0fa78/src/inspect_ai/agent/_react.py) that causes it to crash if truncation is set to `auto` and the model context length is reached:

- On line 186, the code `if state.output.stop_reason == "model_length":` is followed by `await _handle_overflow(state, overflow)`
- On line 417 inside `_handle_overflow()` is the statement `state.messages = await overflow(previous_messages)`
- The value of the overflow argument comes from the statement `overflow = _resolve_overflow(truncation)` on line 174...
- ...which comes from the statement `overflow = cast(MessageFilter | None, trim_messages)` on line 402 inside `_resolve_overflow()`
- ...but `trim_messages`, imported from `inspect_ai.model._trim`, is [not an async function](https://github.com/UKGovernmentBEIS/inspect_ai/blob/c3e669cbea97000d0bbf6c8a1ece371fdcf0fa78/src/inspect_ai/model/_trim.py#L6)

Can reproduce using this test code:

```python
task = Task(
    dataset=[Sample(input="What is the capital of Denmark?", target="Copenhagen")],
    solver=react(truncation=truncation),
    scorer=includes(),
)

model = get_model(
    "mockllm/model",
    custom_outputs=[
        *(
            [
                ModelOutput.from_content(
                    model="mockllm/model",
                    content="This is a repeated message",
                )
            ] * 99
        ),
        ModelOutput.from_content(
            model="mockllm/model",
            content="Final message with stop_reason=model_length",
            stop_reason="model_length",
        ),
        ModelOutput.for_tool_call(
            model="mockllm/model",
            tool_name="submit",
            tool_arguments={"answer": "Copenhagen"},
        ),
    ],
)

log = eval(task, model=model)[0]
assert log.results
```

Closed with https://github.com/UKGovernmentBEIS/inspect_ai/pull/2506
Repository: UKGovernmentBEIS/inspect_ai
Issue: Better names for scorer variations
Root cause: Unimplemented Feature Gap

Currently when you want to create several instances of the same scorer, inspect creates a unique metric name for each of them by appending an index to each one (in [metrics_unique_key](https://github.com/UKGovernmentBEIS/inspect_ai/blob/fab509736be20d1b489f7eefea450bf40df277b8/src/inspect_ai/_eval/task/results.py#L538)).

In the example below, the scores will show up as something like `model_graded_qa` and `model_graded_qa2` from which it is rather difficult to back out which model they refer to.
```python
Task(
    dataset=dataset,
    solver=[
        system_message(SYSTEM_MESSAGE),
        generate()
    ],
    scorer=[
        model_graded_qa(model="openai/gpt-4"), 
        model_graded_qa(model="google/gemini-2.5-pro")
    ],
)
```

Ideally there would be an easy way to dynamically set the scorer's name.

Two proposals for how to do this:

1. Some way to override the name of a scorer (and maybe other attributes like metrics): `scorer_with(model_graded_qa(model="openai/gpt-4"), 'model_graded_qa:gpt-4')`
2. Some way to set the name dynamically within the scorer
```python
@scorer()
def model_graded_qa(model = None):
     async def score...

     return override(score, name=f"model_graded_qa:{model}" if model else "model_graded_qa")
```
Repository: UKGovernmentBEIS/inspect_ai
Issue: Does it work only with API key
Root cause: Documentation Deficiency

Hi Inspect-AI,

This is an amazing framework. Thanks for developing one for the community.
We wanted to explore the tool and incorporate in one of our projects, my question is does it work only with API keys?. For example i have a dataset can we able to get the metrics using python library itself.

Regards,
Roshini

There are various local model providers that don't require an API key: [Hugging Face](https://inspect.aisi.org.uk/providers.html#hugging-face), [vLLM](https://inspect.aisi.org.uk/providers.html#vllm), [Ollama](https://inspect.aisi.org.uk/providers.html#ollama), [Lllama-cpp-python](https://inspect.aisi.org.uk/providers.html#llama-cpp-python), [SGLang](https://inspect.aisi.org.uk/providers.html#sglang), [TransformerLens](https://inspect.aisi.org.uk/providers.html#transformer-lens)
Repository: UKGovernmentBEIS/inspect_ai
Issue: [bug] Devcontainer build fails due to `bitnami/kubectl` deprecation
Root cause: External Dependency Breakage

### Issue 

As of 28th August 2025 Bitnami have deprecated and moved all images from their public catalog (https://hub.docker.com/u/bitnami) to a legacy catalog (https://hub.docker.com/u/bitnamilegacy).

The devcontainer Dockerfile uses `bitnami/kubectl` ([docker repo](https://hub.docker.com/r/bitnami/kubectl), [line](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/.devcontainer/Dockerfile#L7)) so fails to build if you do not have a cached version of the image.

### Workaround

Replace `bitnami/kubectl` with `bitnamilegacy/kubectl`. [Up to kubectl 1.33.4](https://hub.docker.com/r/bitnamilegacy/kubectl/tags?name=1.3) is available, but [per the deprecation notice](https://hub.docker.com/r/bitnami/kubectl#%EF%B8%8F-important-notice-upcoming-changes-to-the-bitnami-catalog) new versions will not be made available there and the `bitnamilegacy/kubectl` repository may be removed at anytime.

### Potential fix

Switch to `bitnamisecure/kubectl:latest` - This is a rolling version of the same image. Specific version tags are not published (publicly) though, so not a viable option if reproducability is a concern.

Other options:
- Get access to Bitnami Secure to keep using their images
- Find an alternative source for kubectl



@sjawhar could you take a look at this?

https://github.com/UKGovernmentBEIS/inspect_ai/pull/2574
Repository: UKGovernmentBEIS/inspect_ai
Issue: vLLM with a local model path
Root cause: Configuration Error

I would like to use local models (for example, loaded from finetuned checkpoint folders) as Solvers and Scorers. I was successful in setting this up with HuggingFace as the provider API, but am running into errors with vLLM.

The following call:
`model = get_model("vllm/local", model_path=local_model_path)`
throws an error:
`Server process exited unexpectedly with code 2. Try rerunning with '--log-level debug' to see the full traceback. Alternatively, you can run the following launch command manually to see the full traceback: ... `

When I run the launch command separately:
`vllm serve local --host 0.0.0.0 --api-key inspectai --model-path local_model_path --port 37716`
the error is as follows:
`usage: vllm [-h] [-v] {chat,complete,serve,bench,collect-env,run-batch} ...
vllm: error: unrecognized arguments: --model-path local_model_path`

which is in line with the vLLM documentation, where the command to start a server with a local model path is simply:
`vllm serve local_model_path`

Any suggestions for a fix?
Thanks!




cc @kaifronsdal 

the command to run local models with vllm is I think something like `vllm serve /path/to/your/local/model` but you should check the docs for vllm to be sure. Can you try `model = get_model("vllm/path/to/your/local/model")`. If that doesn't work, we may need to implement a workaround. Alternatively, you can run the vllm server yourself and connect to the api.

Thanks for quick response!

I was able to connect to a running local server by specifying the model argument as `vllm/<path-to-local-model>`. (works with Inspect starting its own server as well).
Repository: UKGovernmentBEIS/inspect_ai
Issue: Support relative paths for text_editor tool
Root cause: Unimplemented Feature Gap

Right now, the text_editor tool requires an absolute path for the path parameter.  This is confusing for the models, and they often just give up on the text editor entirely.  From a recent run:

`text_editor(command: "view", path: "problem.md")`
The path problem.md is not an absolute path, it should start with `/`. Maybe you meant problem.md?

`text_editor(command: "view", path: "/problem.md")`
The path /problem.md does not exist. Please provide a valid path.

`sed -n '1,200p' problem.md`

Could relative paths be supported?  If not, could the error message be updated to give the correct absolute path?

It should 100% support relative paths and I'm surprised it doesn't! 

What model are you using w/ the text editor tool?

@epatey could you take a look at this?


@JonathanGabor, thanks for the report. I'll have a PR ready to go as soon as I get to the office. It's a straightforward fix.

I am curious about the scenario in which you're encountering this. Of course I can make the `text_editor` tool support relative paths based on `CWD`. Did `problem.md` get into the sandbox via a `files=` on your sandbox?

Thank you!
> I am curious about the scenario in which you're encountering this.

I'm using a SandboxEnvironmentSpec, similar to [Agent Bench](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/agent_bench).
Repository: UKGovernmentBEIS/inspect_ai
Issue: CLI: Support requests per minute (RPM)
Root cause: Unimplemented Feature Gap

It would be nice if the `inspect eval` command supports specifying the max rpm, since model providers usually specify rpm limits. e.g. OpenAI on GPT-5

<img width="973" height="397" alt="Image" src="https://github.com/user-attachments/assets/45aa71b9-3d34-46a1-a04a-235796e2e4da" />

We should be approximating this by just honoring their retry headers (which we do). i.e. they will automatically back us off appropriately to fall within the RPM.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] promt caching to reduce API costs
Root cause: Unimplemented Feature Gap

Anthropic and Google now both support versions of prompt caching which reduces API costs a lot for multi-turn agents/tasks with long documents. It would be fantastic if inspect had native support for these options.

https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
https://ai.google.dev/gemini-api/docs/caching?lang=python (Note: requires at least 32,768 tokens to cache)


Yes, this is next on our list!

PR for Anthropic caching up here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/262

We have implemented this for Anthropic. For Google the benefits are somewhat less compelling as cache requests must be at least 32K tokens (for Anthropic its 1K tokens). Their API also seems to be a bit more "static" (i.e. harder to make work automagically based on heuristics). Will close the issue and we can bring it back to life as a request for Google token caching if/when that becomes more compelling.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Evaluation mismatch in theory_of_mind() results using ollama/gemma3:1b
Root cause: Documentation Deficiency

Hi,

I tested the basic theory_of_mind() method using ollama/gemma3:1b, and as you can see below, some of the answers are actually correct but have been marked as incorrect — specifically the ones highlighted in red.

```
def theory_of_mind():
    return Task(
        dataset=example_dataset("theory_of_mind"),
        solver=[
          chain_of_thought(),
          generate(),
          self_critique()
        ],
        scorer=model_graded_fact()
    )
```

<img width="1339" height="562" alt="Image" src="https://github.com/user-attachments/assets/2b91d7f1-8bc2-4e54-9052-7f4077587cb3" />

Could it be that the model is too small to recognize that the short answer is actually contained within the longer one?

Thanks in advance :)


Correct, the model is indeed too small to correctly score t hese.

Thanks. I wasn’t sure whether the model used to answer the questions was the same one used for evaluation. It’s true that only one model is passed as a parameter, but I haven’t gone through all the documentation to check whether it’s possible to decouple the two: using a more powerful model to assess the answers generated by the smaller one.